...
If you have a substantial TeamCity installation, please check you have appropriate memory settings as a the first step.
Collect Data
During the slow operation, take several thread dumps of the slow process (see below for thread dump taking approaches) with 5-10 seconds interval. If the slowness continues, please take several more thread dumps (e.g. 3-5 within several minutes) and then repeat after some time (e.g. 10 minutes) while the process is still being slow.
...
It is recommended that you take a thread dump of the TeamCity server from the Web UI (if the hanging is local and you can still open the TeamCity Administration pages): go to the Administration | Server Administration | Diagnostics page and click the View server thread dump link to open the thread dump in a new browser window or Save Thread Dump button to save it to the <TeamCity home>/logs directory (where you can later download the files from "Server Logs").
If the web UI is not responsive, try the direct URL using your actualthe actual URL of your TeamCity server.
If the UI is not accessible, you can take a server thread dump manually using one of the approaches described below.
...
If you experience degraded server performance and the TeamCity server process is producing a large CPU load, take the a CPU profiling snapshot and send it to us accompanied with the detailed description of what you were doing and what your system setup is.
You can take the CPU profiling and memory snapshots by installing the server profiling plugin and following the instructions on the plugin page.
Here are some hints to get the best results from CPU profiling:
- after starting the server start-up, wait for some time to allow it to "warm up". This can take from 5 to 20 minutes depending on the data volume that TeamCity stores.
- when a CPU usage increase is found on the server, please try to indicate what actions cause the load.
- start CPU profiling and repeat the action several times (5 - 10).
- capture the a snapshot.
- archive the snapshot and send it to us including the description of the actions that cause the CPU load.
...
It is recommended that you take an agent thread dump from the Web UI: go to the Agent page, Agent Summary tab, and use the Dump threads on agent action.
If the UI is not accessible, you can take the dump thread manually using one of the approaches described below. Note that the TeamCity agent consists of two java processes: the launcher and agent itself. Agent The agent is run triggered by the launcher process. You will usually be interested in the agent (the nested one) process and not the launcher processone.
Taking Thread Dump
| Anchor | ||||
|---|---|---|---|---|
|
These can help if you are unable to take a thread dump from the TeamCity web UI.
To take a thread dump:
...
- To take a server thread dump if the server is run from the console, press Ctrl+Break in the console window (this will not work for agent, since its console belongs to the launcher process).
- To take the an agent thread dumps dump or if your server is running as a service, use the TeamCity-bundled thread dump tool (can be found in the agent's plugins). Run the command:
Note that if the hanging process is run as a service, the server or the agent process must be run from a console with elevated permissions (using Run as Administrator).Code Block <TeamCity agent>\plugins\stacktracesPlugin\bin\x86\JetBrains.TeamCity.Injector.exe <pid_of_java_process>
| Anchor | ||||
|---|---|---|---|---|
|
...
Database-related Slowdowns
When the server is slow it makes sense to , check if the problem is caused by database operations.
It is recommended to use database-specific tools.
You can also use the debug-sql server logging preset. Upon enabling, all the queries which take longer 1 second will be logged into the teamcity-sql.log file. The time can be changed by setting "the teamcity.sqlLog.slowQuery.threshold" internal property. The value should be set in milliseconds and is 1000 by default.
...