Mitigating TeamCity Outage After SQL AlwaysOn Failover

Background/Situation

A TeamCity outage can occur following the successful failover of a SQL AlwaysOn availability group hosting the TeamCity database. TeamCity failed to trigger and run jobs until the services on both TeamCity server nodes were restarted. Logs showed repeated errors regarding the connection being closed, pointing to a stale database connection issue.

Errors in logs:

2026-02-14 15:31:24,831] WARN - etbrains.buildServer.HEARTBEAT - Heartbeat error: jetbrains.buildServer.serverSide.db.UnexpectedDBException: Unexpected exception: SQL error when doing 'Rollback': com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.

[2026-02-14 15:31:29,530] WARN - etbrains.buildServer.HEARTBEAT - Heartbeat error: jetbrains.buildServer.serverSide.db.UnexpectedDBException: Unexpected exception: SQL error when doing 'Rollback': com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.

A remediation plan was requested to prevent such manual intervention in future SQL failovers.

Question

How can TeamCity outages caused by SQL AlwaysOn failovers be prevented, detected, and resolved automatically in the future?

Answer

The issue is likely due to stale database connections left in the pool following the AlwaysOn failover. To ensure TeamCity connects to the new primary database following the SQL AlwaysOn failover, follow these steps:

1. Adjust JDBC Connection String

Update your JDBC connection string in database.properties with the following parameters:

ApplicationIntent=ReadWrite;MultiSubnetFailover=True

For example:

connectionUrl=jdbc:sqlserver://<your-database-server>:1433;databaseName=<your teamcity database>;integratedSecurity=true;ApplicationIntent=ReadWrite;MultiSubnetFailover=True

This ensures the driver quickly redirects TeamCity to the new primary upon failover.

Learn more: https://learn.microsoft.com/en-us/sql/connect/jdbc/jdbc-driver-support-for-high-availability-disaster-recovery?view=sql-server-ver17

2. Enable Connection Pool Health Checks

In your database.properties file, add the following properties for connection pool validation:

testOnBorrow=true
testOnReturn=false
testWhileIdle=true
timeBetweenEvictionRunsMillis=60000

3. Implement a Validation Query

Add a validationQuery to ensure the database connection points to a writable node. Use the following query, which will cause a divide-by-zero error and force the JDBC driver to drop the current database connection and establish a new database connection, if the query returns anything other than READ_WRITE:

validationQuery=SELECT CASE WHEN DATABASEPROPERTYEX(DB_NAME(), 'Updateability') = 'READ_WRITE' THEN 1 ELSE (SELECT 1/0) END

4. Restart TeamCity Servers

Restart your TeamCity servers to apply the new settings. Once implemented, the following behavior is expected:

1. SQL Failover begins.
2. TeamCity attempts to execute a job and "borrows" a connection from the pool.
3. With testOnBorrow=true, the validationQuery checks if the database is writable.
4. If the database is not writable, the connection is killed, and TeamCity requests a new connection.
5. With MultiSubnetFailover=True and ApplicationIntent=ReadWrite, the JDBC driver connects to the new primary.
6. The job executes successfully with no need for manual intervention.

Implementing these changes should reduce the likelihood of outages during future SQL failovers.

Notes

• Test these changes in a non-production environment before deploying to production.
• Initially, a restart of the TeamCity servers is required to apply the updated configuration.
• Please share your test results for further feedback or improvement suggestions.

Following this setup ensures smoother failovers and continuous job execution in TeamCity.