If you experience problems running TeamCity and believe it's related to the software, please contact us with detailed description of the issue.
If you experience problems running TeamCity and believe it's related to the software, please contact us with detailed description of the issue.
To fix a problem, we may need a wide range of information about your system as well as various logs. The section below explains how to collect such information for different issues.
In this section:
If TeamCity is running slower then you would expect, please use the notes below to locate the slow process and send us all the relevant details if the process is a TeamCity one.
If you experience slow TeamCity web UI response, checking for changes process, server-side sources checkout or other slow server activity, your target should be machine where TeamCity server is installed.
If the issue is related only to a single build, you will need to also investigate TeamCity agent machine which is running the build.
Please investigate the system resources (CPU, memory, IO) load. If there is a high load, determine the process to blame. If it is not TeamCity-related process, that might need addressing outside of the TeamCity scope.
If it is TeamCity server which is loading CPU/IO or there is no substantial CPU/IO load and all runs just fine except for TeamCity, then this is to be investigated further.
If you have a substantial TeamCity installation, please check you have appropriate memory settings as a first step.
During the slow operation, take several thread dumps of the slow process (see below for thread dump taking approaches) with 5-10 seconds interval. If the slowness continues, please take several more thread dumps (e.g. 3-5 within several minutes) and then repeat after some time (e.g. 10 minutes) while the process is still being slow.
Then send the detailed description of the issue to us accompanied with the thread dumps and full server (or agent) logs covering the issue. Unless not possible for some reasons, the preferred way is to file an issue into our issue tracker . Please include all the relevant details of investigation, including CPU/IO load information, what specifically is slow and what is not, etc.
You can take a thread dump of the TeamCity server right from the web UI (if the hanging is local and you can still open administration pages): go to the Administration | Server Administration | Diagnostics page and click the View server thread dump link to open the thread dump in a new browser window or Save Thread Dump button to save it to <TeamCity home>/logs directory (where you can later download the files from "Server Logs"). A small hint: if web UI is not responsive try direct URL substituting your server URL.
If the server UI is not usable please use approaches described below.
Please note that if you can take a thread dump without using TeamCity UI, that can be preferable since the thread dump taken from UI can lack some diagnostics information.
You can take a thread dump of the TeamCity agent process right from the web UI: Use Dump threads on agent action on the Agent page.
Please note that if you can take a thread dump by other means, that can be preferable since the thread dump taken from UI can lack some diagnostics information
If you will take agent thread dump manually, please note that TeamCity agent consists of two java processes: launcher and agent itself. Agent is run by the launcher process. You will usually be interested in the agent (more nested one) and not the launcher process.
These can help if the approach to take the thread dump for the server from Administration / Diagnostics page is not applicable.
To take a thread dump:
You have several options:
jstack <pid_of_java_process> command (jstack is located in the "bin" directory of JDK installation). If it does not work, ensure you use the tools from the same version of JDK as is used to run the process you take the dump from (same x86/x64 and same version).<TeamCity agent>\plugins\stacktracesPlugin\bin\x86\JetBrains.TeamCity.Injector.exe <pid_of_java_process> |
If the hanging process is run as a service, taking the thread dump is more complicated and under some environments does not have a reliable approach. Recommended approach is to run the agent or server process from a console and use Ctrl+Break approach above.
Alternatively, you can try to run a thread dumping tool using Microsoft PsExec.
Use the following command line to get the dump:
psexec -u <user> -p <password> <full path to the util> <process pid> stacktrace <output file> |
where:
<user> and <password> – correspond to the same account that was used to run the hanging process as a service. If service uses "System", use "-s" switch instead of -u and -p.
<full path to the util> – full path to the thread dumping util
<process pid> – the process id of the hanging process
<output file> – a full path to a file to save output to (should be accessible for the specified user).
jstack <pid_of_java_process> or kill -3 <pid_of_java_process>. In the latter case output will appear in <TeamCity server home>/logs/catalina.out or <TeamCity agent home>/logs/error.log.You can also use third-party GUI tool: AdaptJ StackTrace Utility.
It supports Windows, Linux or Mac OS X. Choose "Launch" on the page (if you have Java installed). When the application is started, select the process id via "Process > Select", then dump its thread dump with "Process > Thread Dump".
See also Server Performance section below.
When the server is slow it makes sense to check if the problem is caused by database operations.
It is recommended to use database-specific tools.
You can also use debug-sql server logging preset. Upon enabling, all the queries which take longer 1 second will be logged into teamcity-sql.log file. The time can be changed by setting "teamcity.sqlLog.slowQuery.threshold" internal property. The value should be set in milliseconds and is 1000 by default.
Along with server thread dump, please attach output of "show processlist;" SQL command executed in MySQL console. Much like the thread dumps it makes sense to execute the command, if slowness occurred several times and send us the output.
Also, MySQL can be set up to keep a log of long queries executed with the changes in my.ini:
[mysqld] ... log-slow-queries long_query_time=15 |
The log can also be sent to us for analysis.
If you experience problems with TeamCity "eating" too much memory or OutOfMemoryError in the server log, please do the following:
'-Xmx' JVM option, like -Xmx1200m. If the error message is "java.lang.OutOfMemoryError: PermGen space", increase the value in -XX:MaxPermSize=270m JVM option.-XX:+HeapDumpOnOutOfMemoryError. When OOM error occurs next time, java_xxx.hprof file will be created in the process startup directory (<TeamCity home>/bin or <TeamCity Agent home>/bin);jmap standard JVM util. e.g. jmap -dump:file=<file_on_disk_to_save_dump_into>.hprof <pid_of_your_TeamCity_server_process>See how to change JVM options for the server and for agents.
lsof, on Windows you can use handle or TCPView for listing sockets) and the files listulimit -n) and increase them if necessary. Please note that default Linux 1024 handles per process is way too small for a server application like TeamCity. Please increase the number to at least 16000.If the number of files is large and is looking suspicions and the locking process is a TeamCity one (TeamCity agent or server with no other web applications running), grab the list of open handles several times with several minutes interval and send the result to us for investigation together with relevant details.
Please note that you wil most probably need to reboot the machine with the error after the investigation to restore normal functioning of the applications.
TeamCity server and agent create logs that can be used to investigate issues.
Before reproducing the problem it makes sense to enable 'DEBUG' log level for TeamCity classes. |
For detailed information, please refer to the corresponding sections:
TeamCity Server Logs
Viewing Build Agent Logs
In general, to debug VCS problems we need information for jetbrains.buildServer.VCS Log4j category.
TeamCity server performs checking for changes and if server-side checkout is used also the checkout.
TeamCity agent performs the checkout if agent-side checkout is used.
For the server, you can switch logging preset to "debug-vcs" in administration web UI and then retrieve logs/teamcity-vcs.log log file.
For agent and the server you can change the Log4j configuration manually in <TeamCity home>\conf\teamcity-server-log4j.xml or <BuildAgent home>\conf\teamcity-agent-log4j.xml files to have fragment:
<category name="jetbrains.buildServer.VCS" additivity="false">
<appender-ref ref="ROLL.VCS"/>
<appender-ref ref="CONSOLE-ERROR"/>
<priority value="DEBUG"/>
</category>
<category name="jetbrains.buildServer.buildTriggers.vcs" additivity="false">
<appender-ref ref="ROLL.VCS"/>
<appender-ref ref="CONSOLE-ERROR"/>
<priority value="DEBUG"/>
</category>
|
If there are separate logging options for specific version controls, they are described below.
For server-side logging use "debug-vcs" logging preset in administration web UI and then retrieve logs/teamcity-svn.log log file.
Alternative manual approach that is also necessary for agent-side logging:
First, please enable generic VCS debug logging, as described above.
Uncomment SVN-related parts (SVN.LOG appender and javasvn.output category) of Log4j configuration file on server and on agent (if agent-side checkout is used). The log will be saved to the logs/teamcity-svn.log file. Generic VCS log should be also taken from logs/teamcity-vcs.log
Uncomment Clearcase-related lines in the <TeamCity home>\conf\teamcity-server-log4j.xml file.
The log will be saved to logs\teamcity-clearcase.log directory.
In case server-side checkout is used, the "patch" that is passed from server to the agent can be retrieved by:
system.agent.save.patch=true to the build configuration.Build log and agent log will contain the line "Patch is saved to file ${file.name}"
Get the file and provide it with the problem description.
The changes that are sent form the IDE to the server on a remote run can be retrieved from server's .BuildServer\system\changes directory. Locate the <change_number>.changes file that corresponds to your change (you can pick the latest number available or deduce the from the URL of the change form the web UI).
The file contains the patch in the binary form. Please provide it with the problem description.
If you experience degraded server performance and TeamCity server process is producing large CPU load, please take the CPU profiling snapshot and send it to us accompanied with the detailed description of what you were doing and what is your system setup.
You can take the CPU profiling and memory snapshots by installing the server profiling plugin and following the instructions provided on the plugin page.
Here are some hints to get the best results from CPU profiling:
To enable debug logging for IntelliJ Platform based IDE plugin you can manually change the Log4j configuration in <IDE home>\bin\log.xml file to have fragment:
<appender name="TC-FILE" class="org.apache.log4j.RollingFileAppender">
<param name="MaxFileSize" value="10Mb"/>
<param name="MaxBackupIndex" value="10"/>
<param name="file" value="$LOG_DIR$/idea-teamcity.log"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d [%7r] %6p - %30.30c - %m \n"/>
</layout>
</appender>
<appender name="TC-XMLRPC-FILE" class="org.apache.log4j.RollingFileAppender">
<param name="MaxFileSize" value="10Mb"/>
<param name="MaxBackupIndex" value="10"/>
<param name="file" value="$LOG_DIR$/idea-teamcity-xmlrpc.log"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d [%7r] %6p - %30.30c - %m \n"/>
</layout>
</appender>
<category name="jetbrains.buildServer.XMLRPC" additivity="false">
<priority value="DEBUG"/>
<appender-ref ref="TC-XMLRPC-FILE"/>
</category>
<category name="jetbrains.buildServer" additivity="false">
<priority value="DEBUG"/>
<appender-ref ref="TC-FILE"/>
</category>
|
After changing this file please restart the IDE. TeamCity plugin debug logs will appear in <IDE data directory>/system/log directory (idea-teamcity* files).
(Applicable to IntelliJ IDEA and Eclipse)
Add the following JVM option before starting IDE:
-Dteamcity.activation.debug=true
Logging related to open in IDE functionality will appear in the IDE console.
To enable tracing for the plugin, run Eclipse IDE with the -debug <filename> command line parameter. The <filename> portion of the argument should be a properties file containing key-value pairs. Name of each property corresponds to the plugin module and value is either 'true' (to enable debug) or 'false'. Here is an example of enabling most common tracing options:
jetbrains.teamcity.core/debug = true jetbrains.teamcity.core/debug/communications = false jetbrains.teamcity.core/debug/ui = true jetbrains.teamcity.core/debug/vcs = true jetbrains.teamcity.core/debug/vcs/detail = true jetbrains.teamcity.core/debug/parser = true jetbrains.teamcity.core/debug/platform = true jetbrains.teamcity.core/debug/teamcity = true jetbrains.teamcity.core/perfomance/vcs = true jetbrains.teamcity.core/perfomance/teamcity = true |
Read more about Eclipse Debug mode Gathering Information About Your Plug-in and built-in Eclipse help.
To capture logs from TeamCity Visual Studio Addin please do the following:
<username-profile-folder>\Local Settings\Temp\JetLogs folder.<username-profile-folder>\Local Settings\Temp\JetLogs folder. All of the files that were created are logs.To troubleshoot common Visual Studio problems please run Visual Studio executable file with /Log command Line switch and send us resulting log file.
On a rare occasion of TeamCity server or agent process terminating unexpectedly with no reason, it can happen that this is caused by Java runtime crash.
If this happens, Oracle JVM creates a file named hs_err_pid*.log in the working directory of the process. Working directory can be <TeamCity server or agent home>\bin or other like C:\Windows\SysWOW64. You can search the disk for the recent files with "hs_err_pid" in the name.
See also related description.
Please send this file to us for investigation and consider updating JVM for the server (or for agents) to the latest version available.
While investigating issues related to build log we might need raw binary build log as stored by TeamCity.
It can be downloaded via Web UI from the Build Log build's tab: select "Verbose" log detail and use "raw messages file" link at the top-right.
Files under 5Mb in size can be attached right into the tracker issue (if you do not want the attachments to be publicly accessible, limit viewing the attachment to "teamcity-developers" user group only).
You can also send small files (up to 2 Mb) via email: teamcity-feedback@jetbrains.com Please do not forget to mention your TeamCity version and environment.
If the file is over 5 Mb, you can upload the archived files to ftp://ftp.intellij.net/.uploads and let us know the exact file name. If you receive permission denied error on upload attempt, please rename the file. It's OK that you do not see the file listing on the FTP.
The FTP accepts standard anonymous credentials: username: "anonymous", password: "<your e-mail>".