Mantis - Resin
Viewing Issue Advanced Details
ID: Category: Severity: Reproducibility: Date Submitted: Last Update:
4765 block always 09-16-11 02:52 10-04-11 16:07
Reporter: apleshi Platform:  
Assigned To: ferg OS:  
Priority: normal OS Version:  
Status: closed Product Version: 4.0.22  
Product Build: Resolution: fixed  
Projection: none      
ETA: none Fixed in Version: 4.0.24  
Summary: 0004765: ThreadPool full, same as 0004683
Description: [11-09-15 06:27:00.838] {resin-port-null:8080} WarningService: java.lang.IllegalStateException: Thread pool full
[11-09-15 06:27:00.838] {resin-port-null:8080} java.lang.IllegalStateException: Thread pool full
                                                at com.caucho.env.thread.ThreadPool.scheduleImpl(ThreadPool.java:651)
                                                at com.caucho.env.thread.ThreadPool.start(ThreadPool.java:566)
                                                at com.caucho.network.listen.TcpSocketLink.requestAccept(TcpSocketLink.java:644)
                                                at com.caucho.network.listen.SocketLinkThreadLauncher.launchChildThread(SocketLinkThre
adLauncher.java:77)
                                                at com.caucho.env.thread.AbstractThreadLauncher.startConnection(AbstractThreadLauncher
.java:471)
                                                at com.caucho.env.thread.AbstractThreadLauncher.runTask(AbstractThreadLauncher.java:55
0)
                                                at com.caucho.env.thread.AbstractTaskWorker.run(AbstractTaskWorker.java:172)
                                                at com.caucho.env.thread.ResinThread.runTasks(ResinThread.java:164)
                                                at com.caucho.env.thread.ResinThread.run(ResinThread.java:130)

Our current thread pool settings are:
            <thread-max>1024</thread-max>
Steps To Reproduce:
Additional Information:
Relationships
Attached Files:  jvm-default.log [^] (106,075 bytes) 09-16-11 02:52

Notes
(0005508)
apleshi   
09-16-11 03:06   
we have three servers running resin as web service, I send request from http_load and then all three servers run out of thread the same time.
(0005510)
ferg   
09-16-11 14:20   
Take a look at your adentry.java line 62.

From the thread dump, there are a large number of threads writing data to something like System.out or System.err.
(0005513)
apleshi   
09-18-11 18:57   
yes, there is System.out writing something to file, and at that time we run out of harddisk, is this the reason?
(0005538)
ferg   
10-04-11 15:56   
Yes, that's very possible.

In a production system, System.out should be very limited. Normally, you would want to use java.util.logging for that kind of information and use the "fine" level, so it's not normally getting logged.

I'll check the logging routines to see if they should handle out-of-disk issues better, but this doesn't directly look like a Resin bug.
(0005539)
ferg   
10-04-11 16:07   
I'm changing the flushing to reduce contention, which should help this case a bit, but the main change should be to reduce the System.out logging and use a finer java.util.logging level instead.