Amazon EC2 cloud profile error after server restart: Unable to access AWS

Problem

After a TeamCity server restart, an Amazon EC2 cloud profile shows a profile error and starts no agents. Other profiles keep working, including profiles with the same settings. The error stays until someone resets the profile. The UI shows this error:

Cloud profile error
Failed to initialize cloud client 'amazon'.Unable to access AWS. Check your network settings and try again.

jetbrains.buildServer.util.amazon.AWSException: Unable to access AWS. Check your network settings and try again.
	at jetbrains.buildServer.clouds.amazon.client.AmazonClientFactory.createNewClient(AmazonClientFactory.java:89)
	...
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Connect to sts.eu-west-3.amazonaws.com:443 [sts.eu-west-3.amazonaws.com/13.39.225.156] failed: Read timed out (SDK Attempt Count: 3)
	...
	at software.amazon.awssdk.services.sts.DefaultStsClient.getSessionToken(DefaultStsClient.java:1478)
	at jetbrains.buildServer.clouds.amazon.connector.impl.staticType.StaticSessionCredentialsHolder.requestSession(StaticSessionCredentialsHolder.java:71)
	...
Caused by: java.net.SocketTimeoutException: Read timed out
	...
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)

teamcity-server.log, teamcity-clouds.log and teamcity-connections.log have no line about the failure.

Cause

TeamCity creates the cloud client of the profile when it first needs it after startup. For an AWS connection that uses access keys with session credentials, the client creation requests a session token from AWS STS. When that request times out, TeamCity marks the profile as broken and does not try again. It also does not write the error to a log. This is a bug: EC2 cloud profile never recovers from a transient error during client creation.

The first timeout happens on the network path between the TeamCity server and the regional STS endpoint, sts.<region>.amazonaws.com:443. In the error above, the TLS handshake timed out on all three attempts.

Resolution

1. Click Reset on the cloud profile, or disable and enable it. TeamCity creates a new cloud client, and the profile starts agents again.
2. Check the network path from the TeamCity server to sts.<region>.amazonaws.com:443. If a proxy or a firewall is on that path, check its logs for the time of the server startup.
3. Follow EC2 cloud profile never recovers from a transient error during client creation for the fix. Until it ships, repeat step 1 after each restart in which the profile fails.