Icon

You are viewing the documentation of TeamCity 2018.x, which is not the most recently released version of TeamCity.
View this page in the latest documentation or refer to the listing to choose the documentation corresponding to your TeamCity version.

 

Versions Compared

Old Version 5

changes.mady.by.user Yegor Yarko

Saved on

compared with

New Version Current

changes.mady.by.user Yegor Yarko

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Info
  • if you have issue with your build logic, make sure it is a TeamCity-related issue (i.e. it does not reproduce in the same environment without TeamCity) and include details of your investigation
  • note the TeamCity version in use, including the build number (can be found in the footer and the teamcity-server.log). Consider checking if the issue is still relevant in the most recent TeamCity version;
  • update previous postings of yours on the topic instead of creating new ones, if you still create a duplicate make sure to note all previous postings on the same topic you have made or found;
  • post one issue per submission, post the most important issue first, or post several issues noting others if they are related;
  • note the pattern of issue occurrence (first time, recurring), how it was mitigated before, whether there were any recent environment changes;
  • be specific: note exact times, error messages, etc.;
  • describe the expected and actual behavior;
  • when reporting build procedure issues, try to reproduce them without TeamCity on the agent machine in the same environment and let us know the results;
  • detail the related settings configured in TeamCity (include screenshots, settings files, actual values, REST API entity representations);
  • include related screenshots of the TeamCity UI (always include the entire page and the browser URL in the capture);
  • include related text messages as text (not as image), include the messages with all the details;
  • when reporting build procedure issues, try to reproduce them without TeamCity on the agent machine in the same environment and let us know the results;
  • do not include large portions (above 10Kb) of the textual data into the email text, rather attach it in a file;
  • use common multi-platform file formats like .png, .txt, .zip, do not use combining formats (.pdf) until really necessary, do not use proprietary formats like .doc, .eml, rar, etc.;
  • when sending files greater than 500Kb in size or more than three files, package them into a single archive;
  • attach/upload archive with TeamCity server logs (see details, ideally, the entire <server home>\logs directory with all the directories inside. If impractical, all the files updated around or later than the issue time); if related to the build-time or agent behavior, attach the entire <agent home>\logs directory and
  • if a specific build is affected, include the build logs for the related build builds (downloaded via dedicated link from the build results' Build Log tab);
  • for performance/slowness/delays issues, take a set of (10+ spread across the issue time) server or agent thread dumps during the issue occurrence and make sure to send us the dumps with the related data;
  • when sending files greater than 500Kb in size or more than three files, package them into a single archive;when replacing/masking data in logs, note the replacements patterns used;
  • note if there is an anti-virus installed and if there is a network proxy or reverse proxy in front of the TeamCity server;
  • when relevant, note the OS, versions of any manually installed components like Java, the TeamCity distribution used (.exe, .tar.gz)
  • note any customizations/not standard environment settings;
  • list any non-bundled plugins installed;
  • if any, note the previous manual modifications of the TeamCity Data Directory or the database;
  • when suggesting an improvement or feature or asking for settings advice, detail why you need the feature and what the original goal you want to achieve is. Suggestions as to how you would like to see the feature are welcome too;
  • check the sections below for common cases and specific information to collect and send to us.

...

If you experience a slow TeamCity web UI response, checking for changes process, server-side sources checkout, long cleanup times or other slow server activity, your target should be the machine where the TeamCity server is installed.
If the issue is related only to a single build, you will also need need to investigate the TeamCity agent machine which is running the build and also the server.

Investigate the system resources (CPU, memory, IO) load. If there is a high load, determine the process causing it. If it is not a TeamCity-related process, that might need addressing outside of the TeamCity scope. Also check for generic slow-down reasons like anti-virus software, maintenance times, etc.

If it is the TeamCity server that is loading the CPU/IO or there is no substantial CPU/IO load and everything runs just fine except for TeamCity, then this is to be investigated further.
Please check Check if you have any Conflicting Software like anti-virus running on the machine and disable/uninstall it.
Check that the database used by TeamCity and the file storage of the TeamCity Data Directory do not have performance issues.

If you have a substantial TeamCity installation, please check you have appropriate your memory settings as the first step.

...

During the slow operation, take several thread dumps of the slow process (see below for thread dump taking approaches) with 5-10 seconds interval. If the slowness continues, please take several more thread dumps (e.g. 3-5 within several minutes) and then repeat after some time (e.g. 10 minutes) while the process is still being slow.

Then send us a detailed description of the issue accompanied with the thread dumps and full server (or agent) logs covering the issue. Unless it is undesirable for some reason, the preferred way is to file an issue into our issue tracker and let us know via feedback support email. Please include all the relevant details of investigation, including the CPU/IO load information, what specifically is slow and what is not, note affected URLs, visible effects, etc.
For large amounts of data, please use our FTPfile upload service to share the archives with us.

Server Thread Dump

When an operation on the server is slow, please take a set of the server thread dumps (10+) spread over the time of the slowness. TeamCity automatically saves thread dumps on super slow operations. , so there might already be some saved in logs/threadDumps-<date> directories.
It is recommended to send us an archive of the entire content of server's <TeamCity Home>/logs/threadDumps-<date> directories for all the recent dates.

It is recommended that you take a thread dump of the TeamCity server from the Web UI ( if the hanging is local and you can still open the TeamCity Administration pages): go to the Administration | Server Administration | Diagnostics page and click the Save Thread Dump button to save a dump under the <TeamCity Home>/logs/threadDumps-<date> directory (where you can later download the files from "Server Logs").
If the server is fully started but the web UI is not responsive, try the direct URL using the actual URL of your TeamCity server.

If the UI is not accessible (or the server is not yet fully started), you can take a server thread dump manually using the approaches described below.

You can also adjust the teamcity.diagnostics.requestTime.threshold.ms=30000 internal property to change the timeout after which a thread dump is automatically created in the threadDumps-<date> directory under TeamCity logs whenever there is a user-originated web request taking longer than timeout.

...

If the UI is not accessible, you can take the dump thread manually using the approaches described below. Note that the TeamCity agent consists of two java processes: the launcher and agent itself. The agent is triggered by the launcher. You will usually be interested in the agent (nested) process and not the launcher one.

...

  • run jstack <pid_of_java_process> (using jstack from the Java installation as used by the process) or kill -3 <pid_of_java_process>. In the latter case output will appear in <TeamCity Home>/logs/catalina.out or <TeamCity agent home>/logs/error.log.

See also Server Performance section above.

Database-related Slowdowns

...

The log can also be sent to us for analysis.

Back to top

Anchor
oom
oom

OutOfMemory Problems

...

  • Determine what process encounters the error (the actual building process, the TeamCity server, or the TeamCity agent). You can track memory and CPU usage by TeamCity with the charts on the Administration | Server Administration | Diagnostics page of your TeamCity web UI.
  • If the server is to blame, please check you have increased memory settings from the default ones for using the server in production (see the section).
  • If the build process is to blame, set "JVM Command Line Parameters" settings in the build runner.  Increase value for '-Xmx' JVM option, like -Xmx1200m, e.g. Java Inspections builds may specifically need to increase -Xmx value.
  • Anchor
    serverMemoryDump
    serverMemoryDump
    If the TeamCity server is to blame and increasing the memory size does not help, please report the case for us to investigate. For this, while the server is high on memory consumption, take several server thread dumps as described above, get the memory dump (see below) and all the server logs including threadDumps-* sub-directories, archive the results and send them to us for further analysis. Make sure that Xmx setting is less than 8Gb before getting the dump:
    • if a memory dump (hprof file) is created automatically the java_xxx.hprof file is be created in the process startup directory (<TeamCity Home>/bin or <TeamCity Agent home>/bin);
    • for the server, you can also take memory dump manually when the memory usage is at its peak. Go to the Administration | Server Administration | Diagnostics page of your TeamCity web UI and click Dump Memory Snapshot.
    • another approach to take a memory dump manually is to use the jmap standard JVM command line utility of the full JVM installation of the same version as the Java used by the process. Example command line is:
      jmap -dump:file=<file_on_disk_to_save_dump_into>.hprof <pid_of_the_process>

See how to change JVM options for the server and for agents.

Back to top

"Too many open files" Error

...

If the number of files is large and looks suspicions suspicious and the locking process is a TeamCity one (the TeamCity agent or server with no other web applications running), then, while the issue is still occurring, grab the list of open handles several times with several minutes interval and send the result to us for investigation together with the relevant details.

...

TeamCity Server Logs
Viewing Build Agent Logs

Back to top

Version Control debug logging
Anchor
vcs
vcs

...

First, please enable the generic VCS debug logging, as described above.

Uncomment the SVN-related parts (the SVN.LOG appender and javasvn.output category) of the Log4j configuration file on the server and on the agent (if the agent-side checkout is used). The log will be saved to the logs/teamcity-svn.log file. Generic VCS log should be also taken from logs/teamcity-vcs.log

...

In case the server-side checkout is used, the "patch" that is passed from the server to the agent can be retrieved by:

  • add property the property system.agent.save.patch=true to the build configuration.
  • trigger the build.

the build log and the agent log will contain the line "Patch is saved to file ${file.name}"
Get the file and provide supply it with the problem description.

Back to top

Logging for .NET Runners

To investigate process launch issues for .Net-related runners, enable debugging as described below. The detailed information will then be printed into the build log. It is recommended not to have the debug logging for a long time and revert the settings after investigation.

...

Expand
titleAlternative way to enable the logging
  1. Open the <agent home>/plugins/dotnetPlugin/bin folder.
  2. Make a backup copy of teamcity-log4net.xml
  3. Replace teamcity-log4net.xml with the content of teamcity-log4net-debug.xml
Note

After a debug log is created, it is recommended to roll back the change.
The change in the teamcity-log4net.xml will be removed on the build agent autoupgrade.

Back to top

Anchor
remote
remote

Remote Run Problems

The changes that are sent from the IDE to the server on a remote run can be retrieved from the server .BuildServer/system/changes directory. Locate the <change_number>.changes file that corresponds to your change (you can pick the latest number available or deduce the URL of the change from the web UI).
The file contains the patch in the binary form. Please provide send it with the problem description.

Back to top

Logging in IntelliJ IDEA/Platform-based IDEs

...

If the settings are the same and you do not use the manual checkout mode but the problem is there, do the following:

  • Provide us with your IDEA VCS settings and TeamCity VCS settings (for the build configurations you expect to be suitable with your IDEA project)
  • Enable debug logs for the TeamCity IntelliJ plugin (see above)
  • Enable the TeamCity server debug logs (see above)
  • In the TeamCity IntelliJ plugin, try to start a remote run build
  • Provide us with the debug logs from the TeamCity IntelliJ plugin and from the TeamCity server.

Back to top

Logging in TeamCity Eclipse plugin

...

Read more about Eclipse Debug mode Gathering Information About Your Plug-in and built-in Eclipse help.

Back to top

TeamCity Visual Studio Addin issues

...

To capture logs from the TeamCity Visual Studio Addin, please run Microsoft Visual Studio executable (:

1. Locate the Visual Studio installation directory (in the example below c:\Program Files (x86)\Microsoft Visual Studio\2017\Common7\IDE)

2. Run Microsoft Visual Studio executable (INSTALLATION_DIRECTORY\Common7\IDE\devenv.exe) with additional from the command line with the ReSharper-related command line arguments:

  • For TeamCity VS Add-in as a part of ReSharper Ultimate use /ReSharper.LogFile <PATH_TO_FILE> and /ReSharper.LogLevel <Normal|Verbose|Trace> switches

    Code Block
    c:\Program Files (x86)\Microsoft Visual Studio\2017\Common7\IDE>devenv.exe /ReSharper.LogFile C:\Users\jetbrains\Desktop\vs.log /ReSharper.LogLevel Verbose
  • For Legacy version of TeamCity VS Add-in, use /TeamCity.LogFile <PATH_TO_FILE> and  /TeamCity.LogLevel <Normal|Verbose|Trace> switches

Visual Studio logging

To troubleshoot common Visual Studio problems please run , run Microsoft Visual Studio executable file with (INSTALLATION_DIRECTORY\Common7\IDE\devenv.exe) with the /Log command Line switch and send us resulting log file.

Back to top

dotCover Issues

To collect additional logs generated by JetBrains dotCover, add the teamcity.agent.dotCover.log configuration parameter to the build configuration with a path to an empty directory on the agent.
All dotCover log files will be placed there and TeamCity will publish zipped logs as hidden build artifact .teamcity/.NETCoverage/dotCoverLogs.zip.

Wiki Markup
{hidden-data}from http://youtrack.jetbrains.com/issue/TW-17058#comment=27-240592{hidden-data}

...

On a rare occasion of the TeamCity server or agent process terminating unexpectedly with no apparent reason, it can happen that this is caused by a Java runtime crash.
If this happens, the Oracle JVM regularly creates a file named hs_err_pid*.log in the working directory of the process. The working directory is usually <TeamCity server or agent home>/bin Under WIndows when running as a service it can be other like C:\Windows\SysWOW64. You can also search the disk for the recent files with "hs_err_pid" in the name.
See also Oracle documentation, related the Fatal Error Log section in the document.

Please send this file to us for investigation and consider updating the JVM for the server (or for agents) to the latest version available.

...

In case of access issues, time-out errors, etc. please try using passive FTP mode.

Wiki Markup
{hidden-data}
curl -v -u anonymous:xxx ftp://ftp.intellij.net/.uploads/ -T "companyName_date_archive.zip"
{hidden-data}

HTTP

You can upload a file via https://uploads.services.jetbrains.com/ form and let us know the exact file name.

If you cannot upload a large file in one go, try splitting the file into parts and upload them separately.

Back to top