Symptoms
TeamCity build agents repeatedly disconnect from the server every few minutes and reconnect. The agent log shows the same cycle:
WARN ... PollingProtocol - Error while asking server for the next command by GET http://<server>/app/agents/v1/commands/next.
Will continue polling: java.net.SocketTimeoutException: Read timed out
INFO ... PollingProtocol - Agent has been unregistered from server.
INFO ... registration - Registered on server with id <id> and authorization token '<token>'
A common variant has java.net.SocketException: Connection reset instead of SocketTimeoutException, indicating the connection was actively closed by an intermediate device rather than silently dropped.
The agent re-registers with the same agent id and the same authorization token across cycles, so authentication is intact.
Cause
The TeamCity agent uses HTTP long-polling: it opens a connection to the server's /app/agents/v1/commands/next endpoint and waits for the server to respond when there is work to do. The connection can stay quiet for several minutes by design.
When a network device between the agent and the server has an idle timeout shorter than the long-poll window, that device cuts the connection mid-poll. The agent treats the dropped poll as a server-side error, marks itself unregistered, and restarts the poll. The cycle repeats indefinitely.
Common culprits:
- A reverse proxy (nginx, HAProxy, IIS ARR, Apache, AWS Application Load Balancer, Cloudflare, etc.) in front of TeamCity with a read timeout shorter than the long-poll window. Default timeouts of 60 seconds (nginx, AWS ALB) are too short.
- A stateful firewall or NAT device between the agent's network and the server's network dropping idle TCP connections.
Resolution
If TeamCity is behind a reverse proxy, raise the relevant idle/read timeout to at least 5 minutes (1200 seconds). The directive name varies by product. The TeamCity documentation on configuring a proxy server covers nginx. Settings for other common proxies are below.
nginx
In the location or upstream block for TeamCity:
proxy_read_timeout 1200s;
proxy_send_timeout 1200s;
Apache httpd
In the TeamCity VirtualHost or Location block:
ProxyTimeout 1200
Timeout 1200
HAProxy
In the TeamCity backend, or in defaults:
timeout server 1200s
timeout client 1200s
timeout server governs how long HAProxy waits for the TeamCity backend to respond, which is what bounds a long-poll request. timeout tunnel does not apply to TeamCity agent traffic, since agents use plain HTTP long-poll, not WebSocket or HTTP CONNECT tunnels.
IIS with Application Request Routing (ARR)
In IIS Manager, select the server node, double-click Application Request Routing Cache, click Server Proxy Settings on the right, and raise Time-out (seconds) to 1200. Also check the TeamCity site's connection timeout under Sites > TeamCity > Advanced Settings > Limits > connectionTimeout.
AWS Application Load Balancer
The load balancer has an idle-timeout attribute that defaults to 60 seconds. Raise it to 1200 (the maximum is 4000). Console: EC2 > Load Balancers > select the ALB > Attributes > Edit > Idle timeout.
Cloudflare
Cloudflare is not suitable in front of TeamCity for agent traffic on Free or Pro plans. Idle timeout is fixed at 100 seconds on Free and Pro, and 300 seconds on Business. Both are shorter than the long-poll window. If Cloudflare is in the path, route agent traffic to bypass Cloudflare or use an Enterprise plan with raised limits.
No reverse proxy
If TeamCity is exposed directly without a reverse proxy, the dropped connections are coming from a firewall or NAT device on the path between the agent and the server. Most enterprise firewalls drop idle TCP connections after 60 to 300 seconds. Raise the idle TCP timeout, or add an exception for the agent-to-server connection on the TeamCity port.
Comments
0 comments
Please sign in to leave a comment.