Mantis - Resin
Viewing Issue Advanced Details
4765 block always 09-16-11 02:52 10-04-11 16:07
apleshi  
ferg  
normal  
closed 4.0.22  
fixed  
none    
none 4.0.24  
0004765: ThreadPool full, same as 0004683
[11-09-15 06:27:00.838] {resin-port-null:8080} WarningService: java.lang.IllegalStateException: Thread pool full
[11-09-15 06:27:00.838] {resin-port-null:8080} java.lang.IllegalStateException: Thread pool full
                                                at com.caucho.env.thread.ThreadPool.scheduleImpl(ThreadPool.java:651)
                                                at com.caucho.env.thread.ThreadPool.start(ThreadPool.java:566)
                                                at com.caucho.network.listen.TcpSocketLink.requestAccept(TcpSocketLink.java:644)
                                                at com.caucho.network.listen.SocketLinkThreadLauncher.launchChildThread(SocketLinkThre
adLauncher.java:77)
                                                at com.caucho.env.thread.AbstractThreadLauncher.startConnection(AbstractThreadLauncher
.java:471)
                                                at com.caucho.env.thread.AbstractThreadLauncher.runTask(AbstractThreadLauncher.java:55
0)
                                                at com.caucho.env.thread.AbstractTaskWorker.run(AbstractTaskWorker.java:172)
                                                at com.caucho.env.thread.ResinThread.runTasks(ResinThread.java:164)
                                                at com.caucho.env.thread.ResinThread.run(ResinThread.java:130)

Our current thread pool settings are:
            <thread-max>1024</thread-max>
 jvm-default.log [^] (106,075 bytes) 09-16-11 02:52

Notes
(0005508)
apleshi   
09-16-11 03:06   
we have three servers running resin as web service, I send request from http_load and then all three servers run out of thread the same time.
(0005510)
ferg   
09-16-11 14:20   
Take a look at your adentry.java line 62.

From the thread dump, there are a large number of threads writing data to something like System.out or System.err.
(0005513)
apleshi   
09-18-11 18:57   
yes, there is System.out writing something to file, and at that time we run out of harddisk, is this the reason?
(0005538)
ferg   
10-04-11 15:56   
Yes, that's very possible.

In a production system, System.out should be very limited. Normally, you would want to use java.util.logging for that kind of information and use the "fine" level, so it's not normally getting logged.

I'll check the logging routines to see if they should handle out-of-disk issues better, but this doesn't directly look like a Resin bug.
(0005539)
ferg   
10-04-11 16:07   
I'm changing the flushing to reduce contention, which should help this case a bit, but the main change should be to reduce the System.out logging and use a finer java.util.logging level instead.