Mantis - Resin
Viewing Issue Advanced Details
ID: Category: Severity: Reproducibility: Date Submitted: Last Update:
4734 crash always 08-30-11 11:04 06-22-12 10:24
Reporter: drsskelkar Platform:  
Assigned To: ferg OS:  
Priority: normal OS Version:  
Status: closed Product Version: 3.1.10  
Product Build: Resolution: no change required  
Projection: none      
ETA: none Fixed in Version:  
Summary: 0004734: Resin hangs on concurrent database connection load
Description: I have noticed strange behavior of resin under concurrent database access load
Environment:
(1) OS : Ubuntu Linux 10.04.3 32 bit
(2) JDK Version 6 and 7 both
(3) MySQL Both 5.1 and 5.5
(4) JDBC driver connector J 5.1.17
(5) Resin 3.1.10

On intranet if about 50 user login naturally ( may be one after another ) all are able to login ( which requires database access )and application works properly.

But if all user login instantly ( special case to test the load ) resin hangs
and becomes totally non-responsive ( Other web applications also get stalled )

database configuration as follows

<database>
   <jndi-name>jdbc/schoolquiz</jndi-name>
     <driver>
      <type>com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource</type>
      <url>jdbc:mysql://localhost:3306/schoolquiz?characterEncoding=UTF-8</url> [^]
            <user>schoolquiz</user>
      <password>quizmagic</password>
     </driver>
     <prepared-statement-cache-size>8</prepared-statement-cache-size>
   <max-connections>100</max-connections>
   <max-idle-time>30s</max-idle-time>
 </database>

the jvm-dafault.log file mostly shows errors of type

Closing dangling connections. All connections must have a close() in a finally block.

I have repeatedly checked the code but there are no dangling connections !
Every connection is closed in finally block

Thanks and regards
Sharad Kelkar






 
Steps To Reproduce:
Additional Information:
Relationships
Attached Files:

Notes
(0005523)
kain   
09-22-11 13:12   
I have a similar occurrence with Resin 4.0.22. We have a production machine running and under extremely odd (and rare) circumstances, Resin will stop responding.

We have audited our code quite a bit now and similarly can see no issues with connections being left open and while we do not have the same database connectivity declaration in the configuration, resin will stop responding for long periods of time and usually requires a restart (or two) to resolve.
(0005524)
ferg   
09-22-11 16:27   
For 4.0.22, please get a thread dump to track down the problem.

This description reads like it's guessing to the cause of a server freeze, which is almost always wrong. (And not a particularly useful bug report.)

Instead of guessing, it's important to get an actual thread dump to see what the threads are doing.

In 4.0.22, you can get a thread dump with the command-line

  bin/resin.sh thread-dump

Or with a "kill -QUIT" of the Resin process on Unix. Or with jps and jstack from the Java tools.

For the "dangling connection" message, Resin will give a stack trace to show where the connection was allocated. So you can look at that specific allocation instance to see how the finally block is constructed.
(0005525)
drsskelkar   
09-22-11 19:25   
( Note from original bug reporter )
After trying various options one trick significantly mitigated the problem.

In web.xml file I made changes in database tag
I changed driver type from 'com.mysql.jdbc.jdbc2.optional.MysqlConnectionPool DataSource' to 'com.mysql.jdbc.Driver' . This has practically solved my problem, all though I do not have hard benchmarks to prove my point.

This change probably replaced connection pool manager.
Earlier it was using Connector-J supplied connection pool manager and now it uses Resin's internal connection pool manager.
It seems that Connector-J's connection pool manager causes connection leaks on concurrent load. On the other hand Resin's internal connection pool manager looks more reliable
(0005526)
mate   
09-25-11 23:36   
We're using Connector/J 5.1.12 with com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource under Resin 4.0.18 without any such issues.

It does sound like an application error.
(I don't remember whether Connector/J treats unclosed ResultSets and Statements differently depending on com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource vs com.mysql.jdbc.Driver; might be worth checking)
(0005542)
ferg   
10-05-11 16:39   
It might be a mysql driver issue. We just got a support request with a thread dump that looks very similar.

For some reason, the getPooledConnection() call is being synchronized, when it really shouldn't be. I'll need to look into the driver to see why it's acting that way.
(0005543)
ferg   
10-05-11 16:59   
Also reported in Mysql 5.0.8.
(0005550)
ferg   
10-07-11 10:16   
The MysqlConnectionPoolDataSource has a synchronized on its getConnection (from looking at the source), which means it's not possible for Resin to open multiple connections at a time. (This is probably a driver bug.)

Because of that lock, it's probably best to use the Driver configuration, because the Driver will allow multiple simultaneous new connections.
(0005554)
drsskelkar   
10-07-11 23:33   
( Note from issue reporter )
Dear Scott,
I really appreciate the rapt attention you are giving to all bug reports !
Kudos to Resin...

I have just small request, could you please mentions this fact in your product information / manual,so that Resin is not blamed for somebody elses mistake.

It will also avoid frustrating sleepless nights of many developers, believe me, I just got trapped in totally clueless loop...
Thanks and regards
(0005935)
ferg   
06-22-12 10:24   
See http://wiki4.caucho.com/Application_Server:_MySql_JDBC_Pooling. [^]