Mantis Bugtracker
Main | My View | View Issues | Change Log | Docs
  

Viewing Issue Advanced Details [ Jump to Notes ] [ View Simple ] [ Issue History ] [ Print ]
ID Category Severity Reproducibility Date Submitted Last Update
0004734 [Resin] crash always 08-30-11 11:04 06-22-12 10:24
Reporter drsskelkar View Status public  
Assigned To ferg
Priority normal Resolution no change required Platform
Status closed   OS
Projection none   OS Version
ETA none Fixed in Version Product Version 3.1.10
  Product Build
Summary 0004734: Resin hangs on concurrent database connection load
Description I have noticed strange behavior of resin under concurrent database access load
Environment:
(1) OS : Ubuntu Linux 10.04.3 32 bit
(2) JDK Version 6 and 7 both
(3) MySQL Both 5.1 and 5.5
(4) JDBC driver connector J 5.1.17
(5) Resin 3.1.10

On intranet if about 50 user login naturally ( may be one after another ) all are able to login ( which requires database access )and application works properly.

But if all user login instantly ( special case to test the load ) resin hangs
and becomes totally non-responsive ( Other web applications also get stalled )

database configuration as follows

<database>
   <jndi-name>jdbc/schoolquiz</jndi-name>
     <driver>
      <type>com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource</type>
      <url>jdbc:mysql://localhost:3306/schoolquiz?characterEncoding=UTF-8</url> [^]
            <user>schoolquiz</user>
      <password>quizmagic</password>
     </driver>
     <prepared-statement-cache-size>8</prepared-statement-cache-size>
   <max-connections>100</max-connections>
   <max-idle-time>30s</max-idle-time>
 </database>

the jvm-dafault.log file mostly shows errors of type

Closing dangling connections. All connections must have a close() in a finally block.

I have repeatedly checked the code but there are no dangling connections !
Every connection is closed in finally block

Thanks and regards
Sharad Kelkar






 
Steps To Reproduce
Additional Information
Attached Files

- Relationships

- Notes
(0005523)
kain
09-22-11 13:12

 I have a similar occurrence with Resin 4.0.22. We have a production machine running and under extremely odd (and rare) circumstances, Resin will stop responding.

We have audited our code quite a bit now and similarly can see no issues with connections being left open and while we do not have the same database connectivity declaration in the configuration, resin will stop responding for long periods of time and usually requires a restart (or two) to resolve.
 
(0005524)
ferg
09-22-11 16:27

 For 4.0.22, please get a thread dump to track down the problem.

This description reads like it's guessing to the cause of a server freeze, which is almost always wrong. (And not a particularly useful bug report.)

Instead of guessing, it's important to get an actual thread dump to see what the threads are doing.

In 4.0.22, you can get a thread dump with the command-line

  bin/resin.sh thread-dump

Or with a "kill -QUIT" of the Resin process on Unix. Or with jps and jstack from the Java tools.

For the "dangling connection" message, Resin will give a stack trace to show where the connection was allocated. So you can look at that specific allocation instance to see how the finally block is constructed.
 
(0005525)
drsskelkar
09-22-11 19:25

 ( Note from original bug reporter )
After trying various options one trick significantly mitigated the problem.

In web.xml file I made changes in database tag
I changed driver type from 'com.mysql.jdbc.jdbc2.optional.MysqlConnectionPool DataSource' to 'com.mysql.jdbc.Driver' . This has practically solved my problem, all though I do not have hard benchmarks to prove my point.

This change probably replaced connection pool manager.
Earlier it was using Connector-J supplied connection pool manager and now it uses Resin's internal connection pool manager.
It seems that Connector-J's connection pool manager causes connection leaks on concurrent load. On the other hand Resin's internal connection pool manager looks more reliable
 
(0005526)
mate
09-25-11 23:36

 We're using Connector/J 5.1.12 with com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource under Resin 4.0.18 without any such issues.

It does sound like an application error.
(I don't remember whether Connector/J treats unclosed ResultSets and Statements differently depending on com.mysql.jdbc.jdbc2.optional.MysqlConnectionPoolDataSource vs com.mysql.jdbc.Driver; might be worth checking)
 
(0005542)
ferg
10-05-11 16:39

 It might be a mysql driver issue. We just got a support request with a thread dump that looks very similar.

For some reason, the getPooledConnection() call is being synchronized, when it really shouldn't be. I'll need to look into the driver to see why it's acting that way.
 
(0005543)
ferg
10-05-11 16:59

 Also reported in Mysql 5.0.8.
 
(0005550)
ferg
10-07-11 10:16

 The MysqlConnectionPoolDataSource has a synchronized on its getConnection (from looking at the source), which means it's not possible for Resin to open multiple connections at a time. (This is probably a driver bug.)

Because of that lock, it's probably best to use the Driver configuration, because the Driver will allow multiple simultaneous new connections.
 
(0005554)
drsskelkar
10-07-11 23:33

 ( Note from issue reporter )
Dear Scott,
I really appreciate the rapt attention you are giving to all bug reports !
Kudos to Resin...

I have just small request, could you please mentions this fact in your product information / manual,so that Resin is not blamed for somebody elses mistake.

It will also avoid frustrating sleepless nights of many developers, believe me, I just got trapped in totally clueless loop...
Thanks and regards
 
(0005935)
ferg
06-22-12 10:24

 See http://wiki4.caucho.com/Application_Server:_MySql_JDBC_Pooling. [^]
 

- Issue History
Date Modified Username Field Change
08-30-11 11:04 drsskelkar New Issue
09-22-11 13:12 kain Note Added: 0005523
09-22-11 16:27 ferg Note Added: 0005524
09-22-11 19:25 drsskelkar Note Added: 0005525
09-25-11 23:36 mate Note Added: 0005526
10-05-11 16:39 ferg Note Added: 0005542
10-05-11 16:59 ferg Note Added: 0005543
10-07-11 10:16 ferg Note Added: 0005550
10-07-11 23:33 drsskelkar Note Added: 0005554
06-22-12 10:24 ferg Note Added: 0005935
06-22-12 10:24 ferg Assigned To  => ferg
06-22-12 10:24 ferg Status new => closed
06-22-12 10:24 ferg Resolution open => no change required

Mantis 1.0.0rc3[^]
Copyright © 2000 - 2005 Mantis Group
47 total queries executed.
36 unique queries executed.
Powered by Mantis Bugtracker