The main node of the multinode TeamCity installation was stopped, but is still reported as "stopping"

Symptoms

1. The TeamCity multinode setup consists of the main node and a secondary one.
2. The main node is stopped by stopping the TeamCity Server Windows service for a disaster recovery test.
3. After some time, the /app/rest/server/nodes endpoint is still returning two nodes. The main node's state is stopping.

As per the documentation, the MAIN_NODE responsibility can be reassigned only if the current main node is offline. Otherwise, the request will fail. How can the MAIN_NODE responsibility be reassigned in this case, when the main node is clearly offline, but its state remains stopping?

Cause

This is expected behavior in some cases. What happens is:

1. The TeamCity server Windows service is stopped.
2. This initiated the shutdown sequence.
3. The main node changed its startupStage to SHUTTING_DOWN.
4. The node was unable to complete the shutdown sequence before Windows killed the process.
5. The startupStage is persistent. Meaning, it will continue to be SHUTTING_DOWN until the node is back online and changes it.

Resolution

It doesn't affect the reassignment of the MAIN_NODE responsibility.

Instead of using the /app/rest/server/nodes endpoint, the recommended in the documentation /app/rest/server/nodes?locator=state:online endpoint should be used to check whether the main node is offline:

1. The main node writes SHUTTING_DOWN and dies.
2. After it stops heartbeating for a few minutes (3 by default), other nodes will mark it offline and show the “main node is inactive” health report.
3. At this point, the /app/rest/server/nodes?locator=state:online will not show the main node.
4. The MAIN_NODE responsibility is considered vacant and can be given to the secondary node.