Problem
Git operations run by the TeamCity server against a public HTTPS repository fail while collecting changes:
fatal: unable to access 'https://github.com/...':
SSL certificate problem: unable to get local issuer certificate
The affected VCS roots stop collecting changes, so builds either fail or no longer start on new commits. Repositories behind an enterprise root certificate that was uploaded to TeamCity keep working.
The same repository works when Git runs manually from a shell on the server, under the OS user that runs TeamCity. That test has no GIT_SSL_CAINFO in the environment and no http.sslCAInfo in the Git configuration. curl validates the same host against the OS trust store.
In the mirror repository, git config http.sslCAInfo returns <TeamCity Data Directory>/system/caches/ssl/git_custom_certificates.crt. That file holds only the certificates uploaded to TeamCity, so it is a few kilobytes rather than the size of a full CA bundle.
Two conditions are required: server-side operations with native Git are enabled, and at least one certificate is uploaded under Administration | Projects | <Root project> | SSL/HTTPS Certificates.
Cause
TeamCity does not point native Git at git_custom_certificates.crt from the start. A server-side Git command runs against the OS trust store first. Only when that attempt fails with an SSL error does TeamCity retry the command once, and the retry does three things:
- Read the PEM content of every certificate uploaded to TeamCity.
- Write that content to
<TeamCity Data Directory>/system/caches/ssl/git_custom_certificates.crt. - Set
http.sslCAInfoin the mirror repository to that path.
http.sslCAInfo replaces Git's default CA resolution instead of extending it. After step 3, Git no longer reads the OS trust store for that mirror. Public issuer chains stop validating, while the uploaded enterprise roots keep working. The option is written into the mirror repository's Git configuration, so it stays in effect for later operations on that mirror.
TeamCity does not re-evaluate that option. Once http.sslCAInfo points at the generated file and the uploaded certificate set has not changed, the next SSL error rethrows immediately instead of retrying. TeamCity never tests the OS trust store again for that mirror, so the failure outlives the condition that caused it. One transient SSL fault is enough to pin a mirror to the generated bundle permanently.
Trust is additive on the other two code paths, which is why the same certificate set works there. JGit builds an aggregate trust manager over the JVM default certificates plus the uploaded ones. Agent-side native Git sets http.sslCAInfo only after it verifies that the uploaded certificates alone can validate the host. When they cannot, the agent leaves Git on its default trust. The server-side path runs no such check.
One uploaded file contributes one certificate
Each file in the trusted certificate storage becomes a single entry. TeamCity writes the first certificate it can parse from that entry into the generated bundle. A concatenated bundle uploaded as one file therefore adds one certificate, not all of the certificates inside it. The generated file holds as many certificates as there are uploaded files.
The generated file is not rewritten on a schedule
TeamCity rewrites git_custom_certificates.crt only inside the SSL-error retry described above, and only when the content on disk differs from the uploaded certificate list. An external process that keeps replacing the file with a wider bundle creates a rewrite loop, because the next SSL error restores the narrow version. Frequent rewrites indicate an external writer rather than a TeamCity timer.
Resolution
Clear the pin on the affected mirrors
When the server's OS trust store already validates the failing host, the generated bundle is the only thing rejecting it. Unsetting the option returns those mirrors to the default trust.
Confirm the OS trust store first, on the server, as the OS user that runs TeamCity:
curl -sSI --cacert /etc/pki/tls/certs/ca-bundle.crt https://<host>/ -o /dev/null; echo "OS bundle exit: $?"
curl -sSI --cacert <TeamCity Data Directory>/system/caches/ssl/git_custom_certificates.crt https://<host>/ -o /dev/null; echo "TeamCity bundle exit: $?"
Exit 0 means the certificate validated and exit 60 means it did not. OS bundle 0 with TeamCity bundle 60 is the case this section covers. If the OS bundle also returns 60, the host genuinely does not validate against the system store, and the certificate upload below is the fix instead.
Then clear the option across every mirror:
for d in <TeamCity Data Directory>/system/caches/git/*.git; do
git --git-dir="$d" config --unset http.sslCAInfo 2>/dev/null
done
grep -l sslCAInfo <TeamCity Data Directory>/system/caches/git/*.git/config
The grep prints nothing when the option is gone everywhere. Cover the whole directory rather than the mirrors named in the logs, because those are only the ones polled while logging was on.
This is a mitigation and not a fix. A later SSL error against the same host pins the mirrors again. Upload the missing root certificate as well if you want the recovery to survive that.
Do not delete the uploaded certificates to clear the pin
Removing every uploaded certificate deletes the generated bundle but leaves http.sslCAInfo pointing at it in each mirror. Git then fails on a missing CA file rather than an untrusted issuer:
error setting certificate verify locations
Every later operation on that mirror fails the same way, because TeamCity recognizes this message as an SSL error too. Clear the option per mirror as described above, then decide which certificates to upload.
Upload the missing public root certificates
Add the root certificate that anchors the failing host's chain to the same storage that holds the enterprise roots:
- Go to Administration | Projects and click <Root project>.
- Open the SSL/HTTPS Certificates tab.
- Click Upload certificate and add one root certificate per file, in PEM, DER, or PKCS#7 format.
Upload one file per root certificate, because one file contributes one certificate. A concatenated OS CA bundle uploaded as a single file adds only its first certificate. Upload the roots that the failing hosts need rather than the whole OS bundle.
The generated bundle then validates the public chains and the enterprise chains. A mirror whose http.sslCAInfo already points at the narrow file recovers without manual cleanup. The next SSL error regenerates the file from the updated certificate list and retries against it.
See Adding trusted certificates to TeamCity server for the supported formats and the scope of the certificate storage.
Route the affected hosts through JGit
Set an internal property so that Git operations for one host use JGit, where trust is additive:
teamcity.git.nativeOperationsEnabled.github.com=false
The text after the property name is matched as a substring against the repository URL, so only VCS roots whose URL contains it change behavior. The property covers the remote operations: fetch, ls-remote, push, and tag. Local Git commands on the mirror always use native Git and are unaffected.
JGit fetch is slower than native Git fetch.
Capture the trigger if the failure recurs
The retry starts only after a first SSL error against the OS trust store. That first error identifies the trigger and is not visible in the default logs. Once a mirror is pinned, that attempt no longer happens, so you cannot capture the trigger until you clear the pin. Collect it:
- Unset
http.sslCAInfoon the affected mirror, as described in "Clear the pin on the affected mirrors". - Set the
teamcity.git.debugNativeGit=trueinternal property. - Enable the
debug-vcslogging preset on the Administration | Diagnostics | Troubleshooting page. - Reproduce the failure.
- Collect all
teamcity-vcs.log*files. - Remove the property and set the logging preset back to <Default>.
The entries that precede the first write of git_custom_certificates.crt name the SSL error that started the fallback.
No fix as of 2026.1.3
The behavior is present in 2026.1.3 (build 222742). Neither the 2026.1.2 (build 222647) nor the 2026.1.3 release notes contain a certificate fix.
One SSL error permanently pins server-side native Git to the uploaded CA bundle tracks the recovery behavior described here: the option is never re-evaluated, and deleting the uploaded certificates does not clear it. Vote there to be notified of a fix.
Incorrrect git_custom_certificates.crt is not recreated after error "Problem with the SSL CA cert (path? access rights?)" , fixed in 2025.07, describes a different failure: the generated file was not recreated after an SSL error whose message TeamCity did not recognize. A build that contains that fix still shows the behavior described here.
Custom certificate support for server-side native Git was added in 2022.10.1 by Custom ssl certificate support for native git on server, so this is not a regression in a recent release.
Comments
0 comments
Please sign in to leave a comment.