NuGet feed fails with "Unable to put new task into the queue because maximum capacity is exceeded."

Environment:

TeamCity 2023.11.x and older

Cause

Corrupted indexes in nuget's database

Fix

Fixed in 2024.03 https://youtrack.jetbrains.com/issue/TW-86597/Upgrade-HSQLDB-to-2.7.2

Workaround

Upgrade the HSQLDB version on your TeamCity installation without waiting for the new TeamCity release as per these steps:

1. Stop TeamCity server.
2. Download the new HSQL jar: https://download.jetbrains.com/teamcity-repository/org/jetbrains/teamcity/hsqldb/2.7.2/hsqldb-2.7.2.jar
3. Save the original jar, <TeamCity home directory>/webapps/ROOT/WEB-INF/lib/org.hsqldb-hsqldb.jar, elsewhere as a backup.
4. Place the download jar instead of the original one: <TeamCity home directory>/webapps/ROOT/WEB-INF/lib/org.hsqldb-hsqldb.jar
5. Start TeamCity server.

HSQL DB's CHECKPOINT DEFRAG operation can be a problem. To verify that, you may try disabling it by setting the teamcity.hsqlMetadataStorage.checkpoint.enableDefrag=false internal property , and restarting the server. If the issue still reproduces after you do this, please remove the property.

If none of the above helps, and the issue still reproduces with the new HSQLDB version in place, please collect the following data and send it for further analysis

<TeamCity data directory>/system/caches/buildsMetadata
<TeamCity home directory>/logs/teamcity-server.log*
<TeamCity home directory>/logs/teamcity-nuget.log*
<TeamCity home directory>/logs/threadDumps-<date_when_the_issue_reproduced>. If they don't exist, generate them manually through Administration - Diagnostics