Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: New uploads link

...

To fix a problem, we may need a wide range of information about your system as well as various logs. The section below explains how to collect such information for different issues.

In this section:

Table of Contents
maxLevel3
minLevel2

Anchor
hang
hang
Anchor
Hangs and Thread Dumps
Hangs and Thread Dumps

...

Then send the detailed description of the issue to us accompanied with the thread dumps and full server (or agent) logs covering the issue. Unless not possible for some reasons, the preferred way is to file an issue into our issue tracker . Please include all the relevant details of investigation, including CPU/IO load information, what specifically is slow and what is not, etc.

...

If the server UI is not usable please use approaches described below.
Please note that if you can take a thread dump without using TeamCity UI, that can be preferable since the thread dump taken from UI can lack some diagnostics information.

...

  • if you need server thread dump and the server is run from console, press Ctrl+Break in the console window (this will not work for agent, since it's console belongs to launcher process).
    Attention: do not use this approach in TeamCity 7.1.2 when the server is run with teamcity-server.bat. See TW-24531 for details.
  • if JDK 1.6 is used, use jstack <pid_of_java_process> command (jstack is located in the "bin" directory of JDK installation). If it does not work, ensure you use the tools from the same version of JDK as is used to run the process you take the dump from (same x86/x64 and same version).
  • you can also use TeamCity-bundled thread dump tool (can be found in agent's plugins). Run the command:

    Code Block
    
    <TeamCity agent>\plugins\stacktracesPlugin\bin\x86\JetBrains.TeamCity.Injector.exe <pid_of_java_process>
    

...

Alternatively, you can try to run a thread dumping tool using Microsoft PsExec.
Use the following command line to get the dump:

Code Block

psexec -u <user> -p <password> <full path to the util> <process pid> stacktrace <output file>

...

  • run jstack <pid_of_java_process> or kill -3 <pid_of_java_process>. In the latter case output will appear in <TeamCity server home>/logs/catalina.out or <TeamCity agent home>/logs/error.log.

See also Server Performance section below.

Database-related slowdowns

...

The log can also be sent to us for analysis.

Back to top

Anchor
oom
oom

OutOfMemory Problems

...

  • Determine what process encounters the error (the actual building process, the TeamCity server, or the TeamCity agent)
  • If server is to blame, please check you have increased memory settings from default ones for using the server in production (see the section).
  • If you use x64 JVM, please consider using 32 bit JVM, as it will require less memory (instructions for server).
  • Try to increase the memory for the process via '-Xmx' JVM option, like -Xmx1200m. If the error message is "java.lang.OutOfMemoryError: PermGen space", increase the value in -XX:MaxPermSize=270m JVM option.
    The option needs to be passed to the process with problems:
    • if it is the building process itself, use "JVM Command Line Parameters" settings in the build runner. e.g. Inspections builds may specifically need to increase the parameter;
    • refer to the corresponding documentation for TeamCity server or agent. See also Installing and Configuring the TeamCity serverServer;
      Anchor
      serverMemoryDump
      serverMemoryDump
  • If increasing memory size does not help, please take several server thread dumps as described above, get the memory dump, archive it and send it to us for further analysis. Please reduce the Xmx setting to normal before getting the dump (smaller snapshots are easier to analyze and easier to upload):
    • to get a memory dump (hprof file) automatically when an OutOfMemory error occurs, add the following JVM option (works for JDK 1.5.0_07+): -XX:+HeapDumpOnOutOfMemoryError. When OOM error occurs next time, java_xxx.hprof file will be created in the process startup directory (<TeamCity home>/bin or <TeamCity Agent home>/bin);
    • you can also take memory dump manually when the memory usage is at its peak. Go to the Administration | Server Administration | Diagnostics page of your TeamCity web UI and click Dump Memory Snapshot.
    • another approach to take the memory dump manually is to run TeamCity server with JDK 1.6+ and use jmap standard JVM util. e.g. jmap -dump:file=<file_on_disk_to_save_dump_into>.hprof <pid_of_your_TeamCity_server_process>

See how to change JVM options for the server and for agents.

Back to top

"Too many open files" Error

...

TeamCity Server Logs
Viewing Build Agent Logs

Back to top

Version Control Debug Logging
Anchor
vcs
vcs

...

For agent and the server you can change the Log4j configuration manually in <TeamCity home>\conf\teamcity-server-log4j.xml or <BuildAgent home>\conf\teamcity-agent-log4j.xml files to have fragment:

Code Block
xml
xml

<category name="jetbrains.buildServer.VCS" additivity="false">
    <appender-ref ref="ROLL.VCS"/>
    <appender-ref ref="CONSOLE-ERROR"/>
    <priority value="DEBUG"/>
</category>

<category name="jetbrains.buildServer.buildTriggers.vcs" additivity="false">
    <appender-ref ref="ROLL.VCS"/>
    <appender-ref ref="CONSOLE-ERROR"/>
    <priority value="DEBUG"/>
</category>

...

First, please enable generic VCS debug logging, as described above.

Uncomment SVN-related parts (SVN.LOG appender and javasvn.output category) of Log4j configuration file on server and on agent (if agent-side checkout is used). The log will be saved to the logs/teamcity-svn.log file. Generic VCS log should be also taken from logs/teamcity-vcs.log

...

Build log and agent log will contain the line "Patch is saved to file ${file.name}"
Get the file and provide it with the problem description.

Back to top

Anchor
remote
remote

Remote Run Problems

The changes that are sent form the IDE to the server on a remote run can be retrieved from server's .BuildServer\system\changes directory. Locate the <change_number>.changes file that corresponds to your change (you can pick the latest number available or deduce the from the URL of the change form the web UI).
The file contains the patch in the binary form. Please provide it with the problem description.

Back to top

Anchor
serverperformance
serverperformance

...

  • after server startup wait for some time to allow it to "warm up". This can take from 5 to 20 minutes depending on the data volume that TeamCity stores.
  • when CPU usage increase is found on the server, please try to indicate what actions cause the load
  • start CPU profiling and repeat the action several times (5 - 10)
  • capture the snapshot
  • archive the snapshot and send it to us including description of the actions that cause CPU load

Back to top

Logging in IntelliJ IDEA/Platform-based IDEs

To enable debug logging for IntelliJ Platform based IDE plugin you can manually change the Log4j configuration in <IDE home>\bin\log.xml file to have fragment:

Code Block
xml
xml

<appender name="TC-FILE" class="org.apache.log4j.RollingFileAppender">
  <param name="MaxFileSize" value="10Mb"/>
  <param name="MaxBackupIndex" value="10"/>
  <param name="file" value="$LOG_DIR$/idea-teamcity.log"/>
  <layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" value="%d [%7r] %6p - %30.30c - %m \n"/>
  </layout>
</appender>

<appender name="TC-XMLRPC-FILE" class="org.apache.log4j.RollingFileAppender">
  <param name="MaxFileSize" value="10Mb"/>
  <param name="MaxBackupIndex" value="10"/>
  <param name="file" value="$LOG_DIR$/idea-teamcity-xmlrpc.log"/>
  <layout class="org.apache.log4j.PatternLayout">
    <param name="ConversionPattern" value="%d [%7r] %6p - %30.30c - %m \n"/>
  </layout>
</appender>

<category name="jetbrains.buildServer.XMLRPC" additivity="false">
  <priority value="DEBUG"/>
  <appender-ref ref="TC-XMLRPC-FILE"/>
</category>

<category name="jetbrains.buildServer" additivity="false">
  <priority value="DEBUG"/>
  <appender-ref ref="TC-FILE"/>
</category>

...

  • Provide us your IDEA VCS settings and TeamCity VCS settings (for the build configurations you expect to be suitable with your IDEA project)
  • Enable debug logs for TeamCity IntelliJ plugin (see above)
  • Enable TeamCity server debug logs (see above)
  • In TeamCity IntelliJ plugin try to start a remote run build
  • Provide us debug logs from TeamCity IntelliJ plugin and from TeamCity server

Back to top

Logging in TeamCity Eclipse plugin

...

Read more about Eclipse Debug mode Gathering Information About Your Plug-in and built-in Eclipse help.

Back to top

TeamCity Visual Studio Addin issues

...

To troubleshoot common Visual Studio problems please run Visual Studio executable file with /Log command Line switch and send us resulting log file.

Back to top

Anchor
serverperformance
serverperformance

...

If the file is over 5 Mb, you can upload the archived files to ftpvia https://ftp.intellij.net/.uploads and let us know the exact file name. If you receive permission denied error on upload attempt, please rename the file. It's OK that you do not see the file listing on the FTP.
The FTP accepts standard anonymous credentials: username: "anonymous", password: "<your e-mail>"..jetbrains.com.

Back to top