Page History

...

To fix a problem, we may need a wide range of information about your system as well as various logs. The section below explains how to collect such information for different issues.

In this section:

...

Then send us a detailed description of the issue accompanied with the thread dumps and full server (or agent) logs covering the issue. Unless it is undesirable for some reason, the preferred way is to file an issue into our issue tracker and let us know via feedback email. Please include all the relevant details of investigation, including the CPU/IO load information, what specifically is slow and what is not, note URLs, visible effects, etc.
For large amounts of data, please use our FTP.

Server Thread Dump

It is recommended that you take a thread dump of the TeamCity server from the Web UI (if the hanging is local and you can still open the TeamCity Administration pages): go to the Administration | Server Administration | Diagnostics page and click the View server thread dump link to open the thread dump in a new browser window or Save Thread Dump button to save it to the <TeamCity home>/logs directory (where you can later download the files from "Server Logs").
If the web UI is not responsive, try the direct URL using the actual URL of your TeamCity server.

If the UI is not accessible, you can take a server thread dump manually using the approaches described below.

You can also add the teamcity.diagnostics.requestTime.threshold.ms=30000 internal property to automatically create a thread dump in the threadDumps-<date> directory under TeamCity logs whenever there is a user-originated web request taking longer than 30 seconds.

...

If the UI is not accessible, you can take the dump thread manually using the approaches described below. Note that the TeamCity agent consists of two java processes: the launcher and agent itself. The agent is triggered by the launcher. You will usually be interested in the agent (nested) process and not the launcher one.

...

• To take a server thread dump if the server is run from the console, press Ctrl+Break in the console window (this will not work for an agent, since its console belongs to the launcher process).
• Alternatively, run jstack <pid_of_java_process> in the bin directory of the JVM used to by the application.

• To take an agent thread dump or if your server is running as a service, use the TeamCity-bundled thread dump tool (can be found in the agent's plugins). Run the command:

  Code Block
  <TeamCity agent>\plugins\stacktracesPlugin\bin\x86\JetBrains.TeamCity.Injector.exe <pid_of_java_process>

  Note that if the hanging process is run as a service, the server or the agent process must be run from a console with elevated permissions (using Run as Administrator). If the service is run under System account, you might also need to launch the thread dumping tools via "PsExec.exe -s <path to the tool>\<tool> <options>"

...

• run jstack <pid_of_java_process> (using jstack from the same JVM as used by the application) or kill -3 <pid_of_java_process>. In the latter case output will appear in <TeamCity server home>/logs/catalina.out or <TeamCity agent home>/logs/error.log.

See also Server Performance section below.

Database-related Slowdowns

...

The log can also be sent to us for analysis.

Back to top

OutOfMemory Problems

...

• Determine what process encounters the error (the actual building process, the TeamCity server, or the TeamCity agent). Since TeamCity 8.1.2, you can track memory and CPU usage by TeamCity with the charts on the Administration | Server Administration | Diagnostics page of your TeamCity web UI.
• If the server is to blame, please check you have increased memory settings from the default ones for using the server in production (see the section).
• If you use x64 JVM, please consider using 32 bit JVM, as it will require less memory (instructions for server).
• Try to increase the memory for the process via '-Xmx' JVM option, like -Xmx1200m. If the error message is "java.lang.OutOfMemoryError: PermGen space", increase the value in -XX:MaxPermSize=270m JVM option.
  The option needs to be passed to the process with problems:
  • if it is the building process itself, use "JVM Command Line Parameters" settings in the build runner. e.g. Inspections builds may specifically need to increase the parameter;
  • refer to the corresponding documentation for TeamCity server or agent. See also Installing and Configuring the TeamCity serverServer;
    

    Anchor
    serverMemoryDump serverMemoryDump

• If the TeamCity server is to blame and increasing the memory size does not help, please report the case for us to investigate. For this, while the server is high on memory consumption, take several server thread dumps as described above, get the memory dump (see below), archive the results and send them to us for further analysis. If you have increased the Xmx setting, please reduce it to the usual one before getting the dump (smaller snapshots are easier to analyze and easier to upload):
  • to get a memory dump (hprof file) automatically when an OutOfMemory error occurs, add the following server JVM option (works for JDK 1.5.0_07+): -XX:+HeapDumpOnOutOfMemoryError. When OOM error occurs next time, the java_xxx.hprof file will be created in the process startup directory (<TeamCity home>/bin or <TeamCity Agent home>/bin);
  • you can also take memory dump manually when the memory usage is at its peak. Go to the Administration | Server Administration | Diagnostics page of your TeamCity web UI and click Dump Memory Snapshot.
  • another approach to take a memory dump manually is to run the TeamCity server with JDK 1.6+ and use the jmap standard JVM util. e.g. jmap -dump:file=<file_on_disk_to_save_dump_into>.hprof <pid_of_your_TeamCity_server_process>

See how to change JVM options for the server and for agents.

Back to top

"Too many open files" Error

...

TeamCity Server Logs
Viewing Build Agent Logs

Back to top

Version Control debug logging

Anchor
vcs vcs

...

For the agent and the server, you can change the Log4j configuration manually in <TeamCity home>/conf/teamcity-server-log4j.xml or <BuildAgent home>/conf/teamcity-agent-log4j.xml files to include the following fragment:

<category name="jetbrains.buildServer.VCS" additivity="false">
    <appender-ref ref="ROLL.VCS"/>
    <appender-ref ref="CONSOLE-ERROR"/>
    <priority value="DEBUG"/>
</category>

<category name="jetbrains.buildServer.buildTriggers.vcs" additivity="false">
    <appender-ref ref="ROLL.VCS"/>
    <appender-ref ref="CONSOLE-ERROR"/>
    <priority value="DEBUG"/>
</category>

Please also update the <appender name="ROLL.VCS" node to increase the number of the files to store:

     <param name="maxBackupIndex" value="30"/>

...

First, please enable the generic VCS debug logging, as described above.

Uncomment the SVN-related parts (the SVN.LOG appender and javasvn.output category) of the Log4j configuration file on the server and on the agent (if the agent-side checkout is used). The log will be saved to the logs/teamcity-svn.log file. Generic VCS log should be also taken from logs/teamcity-vcs.log

...

the build log and the agent log will contain the line "Patch is saved to file ${file.name}"
Get the file and provide it with the problem description.

Back to top

Logging for .NET Runners

To investigate process launch issues for .Net-related runners, enable debugging as described below. The detailed information will then be printed into the build log.

...

Alternatively, you can set the teamcity.agent.dotnet.debug=true configuration parameter in a build configuration or on an agent.

Back to top

Remote Run Problems

The changes that are sent from the IDE to the server on a remote run can be retrieved from the server .BuildServer/system/changes directory. Locate the <change_number>.changes file that corresponds to your change (you can pick the latest number available or deduce the URL of the change from the web UI).
The file contains the patch in the binary form. Please provide it with the problem description.

Back to top

Logging in IntelliJ IDEA/Platform-based IDEs

To enable debug logging for the IntelliJ Platform-based IDE plugin, include the following fragment into the Log4j configuration of the <IDE home>/bin/log.xml file:

<appender name="TC-FILE" class="org.apache.log4j.RollingFileAppender">
  <param name="MaxFileSize" value="10Mb"/>
  <param name="MaxBackupIndex" value="10"/>
  <param name="file" value="$LOG_DIR$/idea-teamcity.log"/>
  <layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" value="%d [%7r] %6p - %30.30c - %m \n"/>
  </layout>
</appender>

<appender name="TC-XMLRPC-FILE" class="org.apache.log4j.RollingFileAppender">
  <param name="MaxFileSize" value="10Mb"/>
  <param name="MaxBackupIndex" value="10"/>
  <param name="file" value="$LOG_DIR$/idea-teamcity-xmlrpc.log"/>
  <layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" value="%d [%7r] %6p - %30.30c - %m \n"/>
  </layout>
</appender>

<category name="jetbrains.buildServer.XMLRPC" additivity="false">
  <priority value="DEBUG"/>
  <appender-ref ref="TC-XMLRPC-FILE"/>
</category>

<category name="jetbrains.buildServer" additivity="false">
  <priority value="DEBUG"/>
  <appender-ref ref="TC-FILE"/>
</category>

...

• Provide us your IDEA VCS settings and TeamCity VCS settings (for the build configurations you expect to be suitable with your IDEA project)
• Enable debug logs for the TeamCity IntelliJ plugin (see above)
• Enable the TeamCity server debug logs (see above)
• In the TeamCity IntelliJ plugin, try to start a remote run build
• Provide us debug logs from the TeamCity IntelliJ plugin and from the TeamCity server.

Back to top

Logging in TeamCity Eclipse plugin

...

Read more about Eclipse Debug mode Gathering Information About Your Plug-in and built-in Eclipse help.

Back to top

TeamCity Visual Studio Addin issues

...

To troubleshoot common Visual Studio problems please run Visual Studio executable file with /Log command Line switch and send us resulting log file.

Back to top

dotCover Issues

To collect additional logs generated by JetBrains dotCover, add the teamcity.agent.dotCover.log configuration parameter to the build configuration with a path to an empty directory on the agent.
All dotCover log files will be placed there and TeamCity will publish zipped logs as hidden build artifact .teamcity/.NETCoverage/dotCoverLogs.zip.

{hidden-data}from http://youtrack.jetbrains.com/issue/TW-17058#comment=27-240592{hidden-data}

...

Files under 10 MB in size can be attached right into the tracker issue (if you do not want the attachments to be publicly accessible, limit the attachment visibility to "teamcity-developers" user group only).
You can also send small files (up to 2 MB) via email: teamcity-support@jetbrains.com or via online form (up to 20 MB). Please do not forget to mention your TeamCity version and environment and archive the files before attaching.

FTP
If the file is over 10 MB, you You can upload the archived files to ftp://ftp.intellij.net/.uploads and let us know the exact file name. If you receive the permission denied error on an upload attempt, please rename the file. It's OK that you do not see the file listing on the FTP.
The FTP accepts standard anonymous credentials: username: "anonymous", password: "<your e-mail>".
In addition to usual, unencrypted connections, TLS ones are also supported.

In case of access issues, time-out errors, etc. please try using passive FTP mode.

HTTP
You can upload a file larger files via https://uploads.jetbrains.com/ form and let us know the exact file name.

Back to top