Backing up and restoring a multinode TeamCity environment: minimizing downtime during a major upgrade

Question

A multinode (high-availability) TeamCity On-Premises setup shares a single Data Directory and one external database across all server nodes. The Data Directory usually lives on a network file share, so infrastructure snapshots of the node machines do not capture it. How the backup and restore are structured determines how much downtime a major upgrade, and a rollback if it fails, will cost.

How can a multinode TeamCity environment be backed up and restored for a major upgrade with the least downtime, and how is the shared Data Directory returned to its exact pre-upgrade state?

Answer

Keep the Data Directory and database consistent

TeamCity stores its data in two places that must stay in sync: the Data Directory on disk and the external database. Capturing one without the other, or capturing them at different moments, can leave the server inconsistent after a restore. Back them up together and restore them together, from the same point in time. See Manual Backup and Restore and TeamCity Data Backup.

Expected downtime: minor versus major upgrades

Bugfix (minor) updates do not change the data format, so nodes can be updated one at a time with effectively no outage: update the main node while a secondary keeps serving, then update the secondary.

A major upgrade changes the data format of both the database and the Data Directory. The conversion is one-time and cannot be reversed, and all nodes share the same data, so a major upgrade needs a coordinated conversion step during which the server is not fully available. When the upgraded main node starts, secondary nodes detect the different data format and switch to read-only mode until the conversion completes. Schedule major upgrades in a maintenance window. See Upgrading TeamCity Server and Agents and Multinode Setup.

Fastest rollback: decide before confirming the data conversion

On the first start of a newer major version, TeamCity keeps the data in the old format and waits for confirmation of the conversion on the Maintenance page. Until that confirmation, the existing data is untouched. If the upgraded main node shows a problem at that stage, the quickest recovery is to abort before confirming: revert that node to its pre-upgrade state and start the previous version. No data restore is needed, because no conversion has happened, and only the upgraded node is affected. Reverting a VM snapshot or checkpoint of the node is the fastest way back to the previous binaries. See TeamCity Maintenance Mode.

Rollback after the conversion: snapshots restore faster than backup files

Once the conversion is confirmed, the data format has changed and cannot be downgraded, so a rollback means restoring the Data Directory and database to their pre-upgrade state. Restoring a large dataset with maintainDB or a full SQL restore is slow. Where the infrastructure supports it, reverting snapshots is faster: a storage or VM snapshot of the Data Directory share together with a database snapshot, both taken while the services are stopped so they form a consistent pair. The documentation notes that a backup tailored to the specific infrastructure is often the most performant to create and restore. See Manual Backup and Restore.

What the built-in TeamCity backup includes

The backup created from the TeamCity web UI or with the maintainDB tool includes server settings, projects and build configurations, plugins, and (in the Basic scope) the database. It does not include build artifacts, because of their size, and build logs are stored as part of artifacts, so those are excluded too. To preserve artifacts, copy <Data Directory>/system/artifacts separately or take a full copy of the Data Directory. To restore from a backup file, reinstall the matching TeamCity version, create an empty Data Directory, run maintainDB restore against the backup, then copy the artifacts back. See Restoring TeamCity Data from Backup.

Build agents update automatically

Build agents update themselves to match the server version, for both upgrades and downgrades, so a server rollback returns the agents to the previous version with no manual step. The one exception is the Windows build agent service wrapper, which is not updated automatically and needs a manual step when it changes. See Upgrading TeamCity Server and Agents.