Abnormally large artifacts produced by the build left no free disk space for Data Directory

Symptoms

1. The TeamCity Data Directory and TeamCity Server Home Directory are located on the same disk partition.
2. Build artifacts are stored in the default <TeamCity Data Directory>/system/artifacts directory.
3. The build produced large artifacts, causing a spike in the size of the <TeamCity Data Directory>/system/artifacts directory and completely filling the disk.
4. TeamCity Server becomes unresponsive or completely unavailable. Even the scheduled cleanup, which would've helped with freeing some space, can't start.

Cause

When the partition with the TeamCity Data Directory and TeamCity Server Home Directory is completely full, TeamCity Server won't be able to start any new processes, including cleanup.

Resolution

Freeing even a little space manually will allow TeamCity to start the cleanup process, which will delete old data in accordance with the configured cleanup rules.

Prevention

While increasing the total space of the partition should help, it may eventually fill up, leading to the same issue again. As such, it is recommended to implement additional prevention measures. There are multiple ways to prevent this issue from repeating:

• Introducing external artifact storage. TeamCity may store its artifacts in external cloud storage, which will prevent such scenarios.
• Limiting the total size of the <TeamCity Data Directory>/system/artifacts directory. The exact steps needed to configure this will differ depending on the host machine's OS or available tools. In this case, if the build produces large artifacts, leading to a spike in artifact storage size and leaving no space, it'll affect only artifact creation. The TeamCity Data Directory and TeamCity Server Home Directory will still have free space to use and will not be affected by it. Meaning that TeamCity will continue working.
• Using a separate disk partition for artifact storage. This partition should have enough free space for everyday normal use. TeamCity can then be configured to use this partition for artifact storage. In this case, if the build produces large artifacts, leading to a spike in artifact storage size and leaving no space, it'll affect only artifact creation. The TeamCity Data Directory and TeamCity Server Home Directory are located in a separate partition and will not be affected by it. Meaning that TeamCity will continue working.