Mantis - Resin
Viewing Issue Advanced Details
944 minor always 02-16-06 10:50 03-27-06 11:41
ferg  
 
urgent  
closed  
fixed  
none    
none 3.0.19  
0000944: thread pile-up on log
(rep by Bill Au)

This is resin pro 3.0.17 with SUN JDK 1.5.0_06 running on
32-bit RedHat Enterprise Linux 4.

We have a problem when all of a sudden the threads count in
the JVM goes through the roof. Enclosed are two JVM threads
dump taken 10 seconds apart. In each one there are close to
400 threads waiting on a lock in AccessLog.log().

The same thread is holding the lock in the both dumps was it
had been running for at least 10 seconds when the dumps were
taken. Is there any way to get around this? The high threads
count is causing OutOfMemoryError as we are running out of stack
space for all the threads:

java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:574)
        at
com.caucho.util.ThreadPool$ThreadLauncher.startConnection(ThreadPool.java:568)
        at
com.caucho.util.ThreadPool$ThreadLauncher.run(ThreadPool.java:607)
        at java.lang.Thread.run(Thread.java:595)

We are using the default stack size (-Xss1m). Not sure if we
can reduce it. Doing so may just delay the problem. We need to
fix the bottleneck for the threads pile-up.


Notes
(0000945)
ferg   
03-27-06 11:41   
That lock has been removed and the access log write synchronization has been modified to avoid the deadlock.