TeamCity REST API returns state deleted or unknown for a queued build: build queue merge

Background/Situation

An integration queues builds with POST /app/rest/buildQueue. Sometimes several identical requests for the same build configuration arrive at the same moment, with the same branch, revision and parameters. The response to one of them then has state: deleted or state: unknown, and only one of the identical builds runs. In that response, status, number, queuedDate, startDate, finishDate, href and webUrl are null, while changes, artifacts and buildType are filled in. The audit has no entry for that build ID.

The REST API reference for Build describes state as "the actual stage of the build's lifecycle: queued, running, or finished". The OpenAPI definition also lists deleted and unknown.

Question

1. When does the REST API return a build with state: deleted or state: unknown?
2. What removes a queued build right after it is queued, and where is this logged?

Answer

What the states mean

TeamCity computes state when it builds the response:

• queued: the build is in the queue.
• running or finished: the build started.
• deleted: the build is neither queued nor started, and TeamCity marked it as deleted while the request was processed. Rest api: build has state='unknown' added this value.
• unknown: the build is neither queued nor started, and is not marked as deleted yet. API issue: Build can be in "unknown" state on canceling from the build queue: not in the queue, not yet in the history and not deleted describes this window.

What removes the build

The build queue optimization handles equivalent builds in two ways:

1. A request that is equivalent to a build already in the queue adds no build. Its response returns the queued build, so several requests receive the same build ID with state: queued.
2. When two equivalent builds are both in the queue, the queue merge keeps the newer build and removes the older one. This happens a few milliseconds after they are queued. TeamCity takes the build out of the queue first and marks it as deleted afterwards. A response built between the two steps has state: unknown, and a response built after them has state: deleted.

GET /app/rest/builds/id:<id> with the ID of a merged build returns the build that replaced it. TeamCity keeps the record of the replacement in memory for 30 minutes, and a server restart loses it (Reliable finding of a build optimized in the queue: after server restart and not very limited by the total count). After that, the request returns 404 Not Found.

For builds that are part of a build chain, the chain optimization can also replace a queued build with an equivalent build that already started.

The behavior was reproduced on TeamCity 2026.2.1 (239120), and the code is the same in 2026.1.3. In 15 rounds of 6 identical parallel requests, 7 of the 90 responses had state: deleted and 17 had state: unknown.

Log lines and audit records

The merge writes these lines. Both are in teamcity-server.log, and the second one is also in teamcity-activities.log:

Build queue optimizer substituted 1 builds in the queue with builds containing more changes
Build removed from queue: <build ID>, comment=substituted by build with more changes

A build that the chain optimization replaces has comment=optimized to already started build or comment=optimized to already started or another queued build.

At the default log level, teamcity-rest.log can also have Promotion with id <build ID> was replaced by promotion with id <build ID> during request processing. This line appears for only part of the affected responses, so teamcity-activities.log is the reliable source. By default, both files roll over at 10 MB and keep one older file. On a busy server they cover a short period, so collect them soon after an occurrence.

The audit has no removal record for a merged build, because no user removed it. The "added to queue" record is written. The build cleanup then deletes the audit records of builds that no longer exist, and a merged build is one of them. This follows from the source code, and the reproduction did not run a cleanup.

How to handle it in a client

If one build per set of identical requests is enough, treat deleted and unknown in the POST response as "merged into an equivalent build". Find that build with GET /app/rest/builds/id:<id> within 30 minutes.

If every request must run its own build, add a parameter with a unique value to properties in each request. Builds with different parameter values are not equivalent, so the queue does not merge them:

"properties": { "property": [ { "name": "request.id", "value": "<unique value>" } ] }

In the reproduction, all 90 builds stayed in the queue, and every response had state: queued. Builds started with rapid succession get merged in the queue even with teamcity.buildQueue.allowMerging set to false lists the same workaround.

For parallel requests, teamcity.buildQueue.allowMerging=false is not enough, both as a configuration parameter and as a request property. In the reproduction it stopped the deleted and unknown responses. But of 90 requests, only 71 builds stayed in the queue with the configuration parameter, and 28 with the request property. The other requests received the ID of a build that another request queued. This is the open bug Builds started with rapid succession get merged in the queue even with teamcity.buildQueue.allowMerging set to false.

Data for an investigation

• The build ID from the response, and the time of the request in UTC.
• teamcity-activities.log and teamcity-server.log that cover that time.
• The request body, and whether other requests for the same build configuration ran at the same moment.