Blog

Abandoned connection on Tomcat: how to find the root cause

Tomcat gives us a set of useful configuration properties that allow us to discover the root cause of an abandoned connection in two ways:

  • logging the stack trace showing you the line of code that opened the connection
  • receiving jmx notifications of abandoned connections. Each notification includes the time and the stack trace as in the log

The properties we need are presented below using the Tomcat documentation about the Jdbc connection pool:

property name property description
logAbandoned (boolean) Flag to log stack traces for application code which abandoned a Connection. Logging of abandoned Connections adds overhead for every Connection borrow because a stack trace has to be generated. The default value is false.
removeAbandonedTimeout (int) Timeout in seconds before an abandoned(in use) connection can be removed. The default value is 60 (60 seconds). The value should be set to the longest running query your applications might have.
removeAbandoned (boolean) Flag to remove abandoned connections if they exceed the removeAbandonedTimeout. If set to true a connection is considered abandoned and eligible for removal if it has been in use longer than the removeAbandonedTimeout Setting this to true can recover db connections from applications that fail to close a connection. See also logAbandoned The default value is false.
jmxEnabled (boolean) Register the pool with JMX or not. The default value is true.

In other words:

  • logAbandoned requests tomcat to print the stacktrace of the code that is responsible of the abandoned connection.
  • removeAbandonedTimeout tells how many seconds to mark a connection as abandoned. Be careful if your system uses long running processes that hang up the connection for too many seconds. Take a look also at the ResetAbandonedTimer
  • removeAbandoned is in charge of
    1. enabling abandoned connections check
    2. if logAbandoned is true, logging the stacktrace
    3. really closing the connection
  • jmxEnabled register a new MBean on jmx called tomcat.jdbc

In practice, I assume that you have configured one or more resources in your Tomcat Context. You could add the above properties in your tag like below:


<Resource name="jdbc/backoffice"
...
    removeAbandoned="true"
    removeAbandonedTimeout="10" 
    logAbandoned="true"
    jmxEnabled="true"
...
/>

You are achiving these benefits:

  1. when a abandoned connection happens you should see a log like the follow telling you the line of code where the connection was opened. In the default tomcat installation the log is shown in the catalina log file and in the console output. I suggest you to check your conf/logging.properties configuration.
  2. 
    [Wed Aug 31 17:24:33 CEST 2016] WARNING: [org.apache.tomcat.jdbc.pool.ConnectionPool] Connection has been abandoned PooledConnection[com.mysql.jdbc.JDBC4Connection@173432e9]:java.lang.Exception
        at org.apache.tomcat.jdbc.pool.ConnectionPool.getThreadDump(ConnectionPool.java:1072)
        at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:715)
        at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:644)
        at org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:187)
        at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
        at com.fgiovannetti.FireConnector.execute(FireConnector.java:29)
    
  3. the abandoned connection is closed avoiding to exhaust the connections in the pool
  4. you could be notified about the abandoned connection on JMX Console activating the notifications of the MBean tomcat.jdbc like in figure
  5. Abandoned connections logged on jmx

For a full understanding of this topic I suggest you to read the complete tutorial.

Share it!
Fabrizio Giovannetti