JDBC connection pool running out of connections in JBoss EAP

Issue

• Application Server runs out of available connections in the connection pool
• Message appears in the server.log file

·         No ManagedConnections available within configured blocking timeout

• Server hangs

Environment

• JBoss Enterprise Application Platform (EAP)
  • 4.x
  • 5.x

Resolution

• Increase your connection pool size.  Add the following

<max-pool-size>whatever</max-pool-size> 

to the *-ds.xml file to increase the number of connections available. The default is 20.

• The following attribute of the ManagedConnectionPool MBean in the jmx-concole, for the datasource in question, can help understand how many connections are really needed.

MaxConnectionsInUse

• Tell your threads to wait longer for a connection to open up.  Add the following

<blocking-timeout-millis>whatever</blocking-timeout-millis> 

to the *-ds.xml file to increase the length of time to wait. The default is 30000 milli-seconds.

• Fix the offending code so that connections are closed. Especially if you see - 

Closing a connection for you. Please close them yourself

Diagnostic Steps

• Ensure the CachedConnectionValve is installed and CachedConnectionManager debugging is enabled in order to show where the connection leak is coming from. The CachedConnectionValve is installed by default in EAP.  CachedConnectionManager debugging depends on the server profile (enabled  in default and all, disabled in production).
• Configuration details is as follows:

• EAP 4.x
  • CachedConnectionValve: JBOSS_HOME/server/$PROFILE/deploy/deploy/jboss-web.deployer/server.xml
  • CachedConnectionManager is configured in  JBOSS_HOME/server/$PROFILE/deploy/jbossjca-service.xml .  Debugging is enabled as follows:

  <mbean code="org.jboss.resource.connectionmanager.CachedConnectionManager" 

         name="jboss.jca:service=CachedConnectionManager">

    <depends optional-attribute-name="TransactionManagerServiceName">jboss:service=TransactionManager</depends>

    <!-- Enable connection close debug monitoring -->

    <attribute name="Debug">true</attribute>

• EAP 5.x
  • CachedConnectionValve:  JBOSS_HOME/server/$PROFILE/deploy/deploy/jbossweb.sar/server.xml
  • CachedConnectionManager is configured in JBOSS_HOME/server/$PROFILE/deploy/jca-jboss-beans.xml. Debugging is enabled as follows:

  <bean name="CachedConnectionManager" class="org.jboss.resource.connectionmanager.CachedConnectionManager">

      <!-- Whether to track unclosed connections and close them -->

     <property name="debug">true</property>

• Search server.log for the following:

INFO [CachedConnectionManager] Closing a connection for you. Please close them yourself

• Review associated stack trace to find the code that's not releasing connections.
• Identify the code which is leaking the connection.  By using the CachedConnectionManager from the JMX Console:

1. Log into the JMX Console (e.g. http://jboss-host:8080/jmx-console).
2. Scroll down to the "jboss.jca" section.
3. Click the "service=CachedConnectionManager" link.
4. Scroll down and invoke the "listInUseConnections" operation.
5. A stack trace for each in-use connection will be printed. (Only if CachedConnectionManager is in debug mode)
• Take subsequent readings from listInUseConnections() say after the interval of 10 seconds and figure out from the trace which application code is holding the connections for longer time.  This will help in detecting the application code which probably would be leaking the connections.
• Look at the database connections from the database side to see what SQL is running on the connections.  Make sure there is not any long running SQL or recursion bugs in the SQL that could be causing the connections to never be released.

Regards

Kuldeep Sharma