Icon

You are viewing the documentation of TeamCity 8.x, which is not the most recently released version of TeamCity.
Go to the latest TeamCity documentation or refer to the listing to choose the documentation corresponding to your TeamCity version.

 

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: New uploads link

Anchor
top
top

If you experience problems running TeamCity and believe it's they are related to the software, please contact us with a detailed description of the issue.

To fix a problem, we may need a wide range of information about your system as well as various logs. The section below explains how to collect such information for different issues.

In this section:

Table of Contents
maxLevel3
minLevel2

Anchor
hang
hang
Anchor
Hangs and Thread Dumps
Hangs and Thread Dumps
Anchor
serverperformance
serverperformance

Slowness, Hangings and Low Performance

If TeamCity is running slower then than you would expect, please use the notes below to locate the slow process and send us all the relevant details if the process is a TeamCity one.

Determine

...

Which Process Is Slow

If you experience a slow TeamCity web UI response, checking for changes process, server-side sources checkout, long cleanup times or other slow server activity, your target should be the machine where the TeamCity server is installed.
If the issue is related only to a single build, you will need to also investigate the TeamCity agent machine which is running the build.
Please investigate Investigate the system resources (CPU, memory, IO) load. If there is a high load, determine the process to blamecausing it. If it is not a TeamCity-related process, that might need addressing outside of the TeamCity scope. Also check for generic slow-down reasons like anti-virus software, etc.

If it is the TeamCity server which that is loading the CPU/IO or there is no substantial CPU/IO load and all everything runs just fine except for TeamCity, then this is to be investigated further.

Please check if you have any Conflicting Software like anti-virus running on the machine and disable/uninstall it.

If you have a substantial TeamCity installation, please check you have appropriate memory settings as a the first step.

Collect Data

During the slow operation, take several thread dumps of the slow process (see below for thread dump taking approaches) with 5-10 seconds interval. If the slowness continues, please take several more thread dumps (e.g. 3-5 within several minutes) and then repeat after some time (e.g. 10 minutes) while the process is still being slow.

Then send the us a detailed description of the issue to us accompanied with the thread dumps and full server (or agent) logs covering the issue. Unless not possible it is undesirable for some reasonsreason, the preferred way is to file an issue into our issue tracker and let us know via feedback email. Please include all the relevant details of investigation, including the CPU/IO load information, what specifically is slow and what is not, note URLs, visible effects, etc.

Server thread dump

You can For large amounts of data, please use our FTP.

Server Thread Dump

It is recommended that you take a thread dump of the TeamCity server right from the web Web UI (if the hanging is local and you can still open administration the TeamCity Administration pages): go to the Administration | Server Administration | Diagnostics page and click the View server thread dump link to open the thread dump in a new browser window or Save Thread Dump button to save it to the <TeamCity home>/logs directory (where you can later download the files from "Server Logs"). A small hint: if
If the web UI is not responsive, try the direct URL substituting using the actual URL of your TeamCity server URL.

If the server UI is not usable please use accessible, you can take a server thread dump manually using the approaches described below. Please note that if you can take a thread dump without using TeamCity UI, that can be preferable since the thread dump taken from UI can lack some diagnostics information.

Agent thread dump

You can take a thread dump of the TeamCity agent process right from the web UI: Use Dump threads on agent action on the Agent page.

Please note that if you can take a thread dump by other means, that can be preferable since the thread dump taken from UI can lack some diagnostics information
If you will take agent thread dump manually, please note that You can also add the teamcity.diagnostics.requestTime.threshold.ms=30000 internal property to automatically create a thread dump in the threadDumps-<date> directory under TeamCity logs whenever there is a user-originated web request taking longer than 30 seconds.

Anchor
serverperformance
serverperformance

Collecting CPU Profiling Data on Server

If you experience degraded server performance and the TeamCity server process is producing a large CPU load, take a CPU profiling snapshot and send it to us accompanied with the detailed description of what you were doing and what your system setup is.
You can take CPU profiling and memory snapshots by installing the server profiling plugin and following the instructions on the plugin page.

Here are some hints to get the best results from CPU profiling:

  • after starting the server, wait for some time to allow it to "warm up". This can take from 5 to 20 minutes depending on the data volume that TeamCity stores.
  • when a CPU usage increase is found on the server, please try to indicate what actions cause the load.
  • start CPU profiling and repeat the action several times (5 - 10).
  • capture a snapshot.
  • archive the snapshot and send it to us including the description of the actions that cause the CPU load.

Agent Thread Dump

It is recommended that you take an agent thread dump from the Web UI: go to the Agent page, Agent Summary tab, and use the Dump threads on agent action.

If the UI is not accessible, you can take the dump thread manually using the approaches described below. Note that the TeamCity agent consists of two java processes: the launcher and agent itself. Agent The agent is run triggered by the launcher process. You will usually be interested in the agent (more nested one) process and not the launcher processone.

Taking Thread Dump

Anchor
Thread dump taking approaches
Thread dump taking approaches

These can help if the approach you are unable to take the a thread dump for the server from Administration / Diagnostics page is not applicablefrom the TeamCity web UI.

To take a thread dump:

Under Windows

You have several options:

...

  • To take a server thread dump

...

  • if the server is run from the console, press Ctrl+Break in the console window (this will not work for an agent, since

...

  • its console belongs to the launcher process).

...

  • Alternatively, run jstack <pid_of_java_process>

...

  • in the

...

  • bin

...

  • directory of

...

  • the JVM used to by the application.
  • To take an agent thread dump or if your server is running as a service, use the TeamCity-bundled thread dump tool (can be found in the agent's plugins). Run the command:

    Code Block

...

  • <TeamCity agent>\plugins\stacktracesPlugin\bin\x86\JetBrains.TeamCity.Injector.exe <pid_of_java_process>
    

...

  • Note that if the hanging process is run as a service,

...

Alternatively, you can try to run a thread dumping tool using Microsoft PsExec.
Use the following command line to get the dump:

Code Block

psexec -u <user> -p <password> <full path to the util> <process pid> stacktrace <output file>

...

  • the server or the agent process must be run from a console with elevated permissions (using Run as Administrator). If the service is run under System account, you might also need to launch the thread dumping tools via "PsExec.exe -s <path to the tool>\<tool> <options>"

Anchor
linux_thread_dump
linux_thread_dump

Under Linux
  • run jstack <pid_of_java_process> (using jstack from the same JVM as used by the application) or kill -3 <pid_of_java_process>. In the latter case output will appear in <TeamCity server home>/logs/catalina.out or <TeamCity agent home>/logs/error.log.

See also Server Performance section below.

Database-related

...

Slowdowns

When the server is slow it makes sense to , check if the problem is caused by database operations.
It is recommended to use database-specific tools.

You can also use the debug-sql server logging preset. Upon enabling, all the queries which take longer 1 second will be logged into the teamcity-sql.log file. The time can be changed by setting "the teamcity.sqlLog.slowQuery.threshold" internal property. The value should be set in milliseconds and is 1000 by default.

MySQL

Along with the server thread dump, please attach the output of the "show processlist;" SQL command executed in MySQL console. Much like the Like with thread dumps, it makes sense to execute the command , several times if slowness occurred several times and send us the output.
Also, MySQL can be set up to keep a log of long queries executed with the changes in my.ini:

...

The log can also be sent to us for analysis.

Back to top

Anchor
oom
oom

OutOfMemory Problems

If you experience problems with TeamCity "eating" too much memory or OutOfMemoryError/"Java heap space" errors in the server log, please do the following:

  • Determine what process encounters the error (the actual building process, the TeamCity server, or the TeamCity agent). Since TeamCity 8.1.2, you can track memory and CPU usage by TeamCity with the charts on the Administration | Server Administration | Diagnostics page of your TeamCity web UI.
  • If the server is to blame, please check you have increased memory settings from the default ones for using the server in production (see the section).
  • If you use x64 JVM, please consider using 32 bit JVM, as it will require less memory (instructions for server).
  • Try to increase the memory for the process via '-Xmx' JVM option, like -Xmx1200m. If the error message is "java.lang.OutOfMemoryError: PermGen space", increase the value in -XX:MaxPermSize=270m JVM option.
    The option needs to be passed to the process with problems:
    • if it is the building process itself, use "JVM Command Line Parameters" settings in the build runner. e.g. Inspections builds may specifically need to increase the parameter;
    • refer to the corresponding documentation for TeamCity server or agent. See also Installing and Configuring the TeamCity serverServer;
      Anchor
      serverMemoryDump
      serverMemoryDump
  • If the TeamCity server is to blame and increasing the memory size does not help, please report the case for us to investigate. For this, while the server is high on memory consumption, take several server thread dumps as described above, get the memory dump (see below), archive the results and send them to us for further analysis. If you have increased the Xmx setting, please reduce it to the usual one before getting the dump (smaller snapshots are easier to analyze and easier to upload):
    • to get a memory dump (hprof file) automatically when an OutOfMemory error occurs, add the following server JVM option (works for JDK 1.5.0_07+): -XX:+HeapDumpOnOutOfMemoryError. When OOM error occurs next time, the java_xxx.hprof file will be created in the process startup directory (<TeamCity home>/bin or <TeamCity Agent home>/bin);
    • you can also take memory dump manually when the memory usage is at its peak. Go to the Administration | Server Administration | Diagnostics page of your TeamCity web UI and click Dump Memory Snapshot.
    • another approach to take the a memory dump manually is to run the TeamCity server with JDK 1.6+ and use the jmap standard JVM util. e.g. jmap -dump:file=<file_on_disk_to_save_dump_into>.hprof <pid_of_your_TeamCity_server_process>

See how to change JVM options for the server and for agents.

Back to top

"Too many open files" Error

  1. Determine what computer it occurs on
  2. Determine the process locking the files (on Linux use lsof, on Windows you can use handle or TCPView for listing sockets) and the files list
  3. If the number is not large, check the OS and the process limits on the file handles (on Linux use ulimit -n) and increase them if necessary. Please note that default Linux 1024 handles per process is way too small for a server application like TeamCity. Please increase the number to at least 16000. Please check the actual process limits after the change as there are different settings in the OS for settings global and per-session limits (e.g. see the post)

If the number of files is large and is looking looks suspicions and the locking process is a TeamCity one (the TeamCity agent or server with no other web applications running), then, while the issue is still occurring, grab the list of open handles several times with several minutes interval and send the result to us for investigation together with the relevant details.

Please note that you will most probably need to reboot the machine with the error after the investigation to restore normal functioning of the applications.

...

Agent does not connect to the server

Please refer to Common Problems

Logging events
Anchor
logging
logging

The TeamCity server and agent create logs that can be used to investigate issues.

Info
titleHow to enable DEBUG logging on server?

Before reproducing the problem it makes sense to enable 'DEBUG' log level for TeamCity classes.
On the server side, go to the Administration | Server Administration | Diagnostics page and select debug logging level ('debug-all', 'debug-vcs', etc).

After that, DEBUG messages will go to teamcity-*.log files (read more).

...

TeamCity Server Logs
Viewing Build Agent Logs

Back to top

Version Control

...

debug logging
Anchor
vcs
vcs

In general, to debug VCS problems we need information for jetbrains.buildServer.VCS Log4j category.

TeamCity server performs checking for changes and if server-side checkout is used also the checkout.
TeamCity agent performs the checkout if agent-side checkout is used.

...

Info

To enable VCS logging on the server side, switch logging preset to "debug-vcs" in administration web UI and then retrieve logs/teamcity-vcs.log log file.

Most VCS operations occur on the TeamCity server, but if you're using the agent-side checkout, VCS checkout occurs on the build agents.

For the agent and the server, you can change the Log4j configuration manually in <TeamCity home>\/conf\/teamcity-server-log4j.xml or <BuildAgent home>\/conf\/teamcity-agent-log4j.xml files to have include the following fragment:

Code Block
xml
xml

<category name="jetbrains.buildServer.VCS" additivity="false">
    <appender-ref ref="ROLL.VCS"/>
    <appender-ref ref="CONSOLE-ERROR"/>
    <priority value="DEBUG"/>
</category>

<category name="jetbrains.buildServer.buildTriggers.vcs" additivity="false">
    <appender-ref ref="ROLL.VCS"/>
    <appender-ref ref="CONSOLE-ERROR"/>
    <priority value="DEBUG"/>
</category>

Please also update the <appender name="ROLL.VCS" node to increase the number of the files to store:

Code Block
    <param name="maxBackupIndex" value="30"/>

If there are separate logging options for specific version controls, they are described below.

Subversion

...

debug logging

Info

To enable SVN logging on the server side, switch the logging preset to "debug-

...

SVN"

...

in the administration web UI and then retrieve the logs/teamcity-vcs.log and logs/teamcity-svn.log

...

files.

Alternative An alternative manual approach that is also necessary for agent-side logging:.

First, please enable the generic VCS debug logging, as described above.

Uncomment the SVN-related parts (the SVN.LOG appender and javasvn.output category) of the Log4j configuration file on the server and on the agent (if the agent-side checkout is used). The log will be saved to the logs/teamcity-svn.log file. Generic VCS log should be also taken from logs/teamcity-vcs.log

ClearCase

Uncomment the Clearcase-related lines in the <TeamCity home>\/conf\/teamcity-server-log4j.xml file.
The log will be saved to logs\/teamcity-clearcase.log directory.

...

Patch Application Problems

In case the server-side checkout is used, the "patch" that is passed from the server to the agent can be retrieved by:

  • add property system.agent.save.patch=true to the build configuration.
  • trigger the build.

Build the build log and the agent log will contain the line "Patch is saved to file ${file.name}"
Get the file and provide it with the problem description.

Back to top

Logging for .NET Runners

To investigate process launch issues for .Net-related runners, enable debugging as described below. The detailed information will then be printed into the build log.

Do the following:

  1. Open the <agent home>/plugins/dotnetPlugin/bin folder.
  2. Make a backup copy of teamcity-log4net.xml
  3. Replace teamcity-log4net.xml with the content of teamcity-log4net-debug.xml
Note

After a debug log is created, it is recommended to roll back the change.
The change in the teamcity-log4net.xml will be removed on the build agent autoupgrade.

Alternatively, you can set the teamcity.agent.dotnet.debug=true configuration parameter in a build configuration or on an agent.

Back to top

Anchor
remote
remote

Remote Run Problems

The changes that are sent form from the IDE to the server on a remote run can be retrieved from the server 's .BuildServer\/system\/changes directory. Locate the <change_number>.changes file that corresponds to your change (you can pick the latest number available or deduce the from the URL of the change form from the web UI).
The file contains the patch in the binary form. Please provide it with the problem description.

Back to top Anchorserverperformanceserverperformance

Server Performance

If you experience degraded server performance and TeamCity server process is producing large CPU load, please take the CPU profiling snapshot and send it to us accompanied with the detailed description of what you were doing and what is your system setup.
You can take the CPU profiling and memory snapshots by installing the server profiling plugin and following the instructions provided on the plugin page.

Here are some hints to get the best results from CPU profiling:

  • after server startup wait for some time to allow it to "warm up". This can take from 5 to 20 minutes depending on the data volume that TeamCity stores.
  • when CPU usage increase is found on the server, please try to indicate what actions cause the load
  • start CPU profiling and repeat the action several times (5 - 10)
  • capture the snapshot
  • archive the snapshot and send it to us including description of the actions that cause CPU load

Back to top

Logging in IntelliJ IDEA/Platform-based IDEs

To enable debug logging for the IntelliJ Platform-based IDE plugin you can manually change the , include the following fragment into the Log4j configuration in of the <IDE home>\/bin\/log.xml file to have fragment:

Code Block
xml
xml

<appender name="TC-FILE" class="org.apache.log4j.RollingFileAppender">
  <param name="MaxFileSize" value="10Mb"/>
  <param name="MaxBackupIndex" value="10"/>
  <param name="file" value="$LOG_DIR$/idea-teamcity.log"/>
  <layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" value="%d [%7r] %6p - %30.30c - %m \n"/>
  </layout>
</appender>

<appender name="TC-XMLRPC-FILE" class="org.apache.log4j.RollingFileAppender">
  <param name="MaxFileSize" value="10Mb"/>
  <param name="MaxBackupIndex" value="10"/>
  <param name="file" value="$LOG_DIR$/idea-teamcity-xmlrpc.log"/>
  <layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" value="%d [%7r] %6p - %30.30c - %m \n"/>
  </layout>
</appender>

<category name="jetbrains.buildServer.XMLRPC" additivity="false">
  <priority value="DEBUG"/>
  <appender-ref ref="TC-XMLRPC-FILE"/>
</category>

<category name="jetbrains.buildServer" additivity="false">
  <priority value="DEBUG"/>
  <appender-ref ref="TC-FILE"/>
</category>

After changing this file please , restart the IDE. The TeamCity plugin debug logs are saved into idea-teamcity* files and will appear in the logs directory of the IDE settings ([<IDE settings/data directory>/system/log directory (idea-teamcity* files).

Open in IDE

...

Functionality Logging

(Applicable to IntelliJ IDEA and Eclipse)
Add the following JVM option before starting IDE:
-Dteamcity.activation.debug=true
Logging the logging related to the open in IDE functionality will appear in the IDE console.

No

...

Suitable Build Configurations Found for Remote Run

First of all, check that your VCS settings in IDEA corresponds correspond to the VCS settings in TeamCity.
If they do not, then it is the cause of the problem.If so, then change them and it should fix the problem.

Secondly, check that the build configurations you expect to be suitable with your IDEA project has either server-side VCS checkout mode or agent-side checkout and NOT manual VCS checkout mode (it is not possible to apply a personal patch for a build with the manual checkout mode because TeamCity must apply that patch after the VCS checkout is done, but it does not know or manage the time when it is performed).

If the settings are the same and you do not use the manual checkout mode but the problem is there, do the following:

  • Provide us your IDEA VCS settings and TeamCity VCS settings (for the build configurations you expect to be suitable with your IDEA project)
  • Enable debug logs for the TeamCity IntelliJ plugin (see above)
  • Enable the TeamCity server debug logs (see above)
  • In the TeamCity IntelliJ plugin, try to start a remote run build
  • Provide us debug logs from the TeamCity IntelliJ plugin and from the TeamCity server.

Back to top

Logging in TeamCity Eclipse plugin

To enable tracing for the plugin, run Eclipse IDE with the -debug <filename> command line parameter. The <filename> portion of the argument should be a properties file containing key-value pairs. Name The name of each property corresponds to the plugin module and the value is either 'true' (to enable debug) or 'false'. Here is an example of enabling most common tracing options:

...

Read more about Eclipse Debug mode Gathering Information About Your Plug-in and built-in Eclipse help.

Back to top

TeamCity Visual Studio Addin issues

TeamCity Addin logging

To capture logs from the TeamCity Visual Studio Addin please do the following:

...

, please run Microsoft Visual Studio executable (devenv.exe) with the /TeamCity.LogFile <PATH_TO_FILE> switch specified.

...

Visual Studio

...

Visual Studio logging

To troubleshoot common Visual Studio problems please run Visual Studio executable file with /Log command Line switch and send us resulting log file.

Back to top Anchorserverperformanceserverperformance

dotCover Issues

To collect additional logs generated by JetBrains dotCover, add the teamcity.agent.dotCover.log configuration parameter to the build configuration with a path to an empty directory on the agent.
All dotCover log files will be placed there and TeamCity will publish zipped logs as hidden build artifact .teamcity/.NETCoverage/dotCoverLogs.zip.

Wiki Markup
{hidden-data}from http://youtrack.jetbrains.com/issue/TW-17058#comment=27-240592{hidden-data}

JVM Crashes

On a rare occasion of the TeamCity server or agent process terminating unexpectedly with no apparent reason, it can happen that this is caused by a Java runtime crash.
If this happens, the Oracle JVM creates a file named hs_err_pid*.log in the working directory of the process. Working The working directory can be <TeamCity server or agent home>\/bin or other like C:\Windows\SysWOW64. You can also search the disk for the recent files with "hs_err_pid" in the name.
See also related descriptionOracle documentation, the Fatal Error Log section.

Please send this file to us for investigation and consider updating the JVM for the server (or for agents) to the latest version available.

If you get the "There is insufficient memory for the Java Runtime Environment to continue. Native memory allocation (malloc) failed to allocate ..." message with the crash or in the crash report file, make sure to switch to 64 bits JVM or reduce -Xmx setting not to increase 1200m, see details in the memory configuration section.

Anchor
rawBuildLog
rawBuildLog

...

While investigating issues related to a build log, we might need the raw binary build log as stored by TeamCity.
It can be downloaded via the Web UI from the Build Log build's tab: select "Verbose" log detail and use the "raw messages file" link at the top-right.

...

Sending Information to the Developers

Files under 5Mb 10 MB in size can be attached right into the tracker issue (if you do not want the attachments to be publicly accessible, limit viewing the attachment visibility to "teamcity-developers" user group only).
You can also send small files (up to 2 MbMB) via email: teamcity-feedback@jetbrainssupport@jetbrains.com or via online form (up to 20 MB). Please do not forget to mention your TeamCity version and environment and archive the files before attaching.

If the file is over 5 Mb, you You can upload the archived files to ftplarger files via https://ftp.intellij.net/.uploads and let us know the exact file name. If you receive permission denied error on upload attempt, please rename the file. It's OK that you do not see the file listing on the FTP.
The FTP accepts standard anonymous credentials: username: "anonymous", password: "<your e-mail>"..jetbrains.com.

Back to top