Mantis - Resin
Viewing Issue Advanced Details
ID: Category: Severity: Reproducibility: Date Submitted: Last Update:
2728 major always 06-09-08 11:59 06-11-08 21:35
Reporter: paulberto Platform:  
Assigned To: ferg OS:  
Priority: normal OS Version:  
Status: closed Product Version: 3.1.6  
Product Build: Resolution: fixed  
Projection: none      
ETA: none Fixed in Version: 3.1.7  
Summary: 0002728: Running latest snapshot -- cache stops working after InterruptedIOException
Description: After which, the load on the machine skyrockets since cache is disabled.

[21:50:17.369] Caused by: java.io.InterruptedIOException: timeout fd=1395 errno=11
[21:50:17.369]
[21:50:17.369] at com.caucho.vfs.JniSocketImpl.writeNative(Native Method)
[21:50:17.369] at com.caucho.vfs.JniSocketImpl.write(JniSocketImpl.java:288)
[21:50:17.369] at com.caucho.vfs.JniStream.write(JniStream.java:124)
[21:50:17.369] at com.caucho.vfs.WriteStream.nextBuffer(WriteStream.java:327)
[21:50:17.369] at com.caucho.server.connection.ResponseStream.writeNext(ResponseStream.java:480)
[21:50:17.369] at com.caucho.server.connection.ToByteResponseStream.write(ToByteResponseStream.java:336)
[21:50:17.369] at com.caucho.vfs.VfsStream.write(VfsStream.java:201)
[21:50:17.369] at com.caucho.vfs.WriteStream.print(WriteStream.java:553)
[21:50:17.369] at com.caucho.vfs.WriteStream.print(WriteStream.java:631)
[21:50:17.369] at com.caucho.quercus.program.TextStatement.execute(TextStatement.java:63)
[21:50:17.369] ... 26 more
Steps To Reproduce:
Additional Information: I'm being forced to monitor the machines in question and restart them when this is occuring. Please issue fix!!!! I am running resin-pro-3.1.s080503 SNAPSHOT in an attempt to get the fix from a previous bug ID 0002704.

Please advise
Relationships
Attached Files:

Notes
(0003154)
paulberto   
06-09-08 12:19   
sorry here is the real log error:

[2008/06/09 22:12:08.800] java.io.InterruptedIOException: timeout fd=134 errno=11
[2008/06/09 22:12:08.800]
[2008/06/09 22:12:08.800] at com.caucho.vfs.JniSocketImpl.writeNative(Native Method)
[2008/06/09 22:12:08.800] at com.caucho.vfs.JniSocketImpl.write(JniSocketImpl.java:288)
[2008/06/09 22:12:08.800] at com.caucho.vfs.JniStream.write(JniStream.java:124)
[2008/06/09 22:12:08.800] at com.caucho.vfs.WriteStream.nextBuffer(WriteStream.java:327)
[2008/06/09 22:12:08.800] at com.caucho.server.connection.ResponseStream.nextBuffer(ResponseStream.java:324)
[2008/06/09 22:12:08.800] at com.caucho.server.cache.CacheInode.writeToStream(CacheInode.java:102)
[2008/06/09 22:12:08.800] at com.caucho.server.cache.CacheInode.writeToStream(CacheInode.java:79)
[2008/06/09 22:12:08.800] at com.caucho.server.cache.CacheFilterChain.fillFromCache(CacheFilterChain.java:530)
[2008/06/09 22:12:08.800] at com.caucho.server.cache.CacheFilterChain.useCache(CacheFilterChain.java:301)
[2008/06/09 22:12:08.800] at com.caucho.server.cache.CacheFilterChain.doFilter(CacheFilterChain.java:174)
[2008/06/09 22:12:08.800] at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:187)
[2008/06/09 22:12:08.800] at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:266)
[2008/06/09 22:12:08.800] at com.caucho.server.http.HttpRequest.handleRequest(HttpRequest.java:270)
[2008/06/09 22:12:08.800] at com.caucho.server.port.TcpConnection.run(TcpConnection.java:678)
[2008/06/09 22:12:08.800] at com.caucho.util.ThreadPool$Item.runTasks(ThreadPool.java:721)
[2008/06/09 22:12:08.800] at com.caucho.util.ThreadPool$Item.run(ThreadPool.java:643)
[2008/06/09 22:12:08.800] at java.lang.Thread.run(Thread.java:619)
[2008/06/09 22:12:08.801] cache disabled due to java.io.InterruptedIOException: timeout fd=134 errno=11
(0003155)
paulberto   
06-09-08 15:26   
This is a show stopper for my site. We rely on the cache heavily and once it stops working the site dies. It almost spirals out of control because of the amount of work it performs when the caching gets automatically disabled. So much so that even the new feature which attempts to kill resin on high loads doesn't occur.

Would it be possible to get the latest snapshot with the bug fix committed (reported in 0002704) or any suggestions as to how we can workaround it in the meantime ?
 
License #: 1007371
(0003156)
ferg   
06-09-08 16:27   
snapshots are not supposed to be used for production code. They're essentially early alpha code. It's possible to release a new snapshot, but it will still be an alpha snapshot.
(0003158)
paulberto   
06-09-08 17:19   
I cant revert to 3.1.5 because of a quercus issue and 3.1.6 seems to have this issue. I don't mind giving the snapshot a whirl. Unless of course, you can suggest a work around for this issue in 3.1.6 ?
(0003159)
ferg   
06-09-08 18:51   
You can try remove libresin.so. That would change the interruption to something the cache can handle.
(0003160)
paulberto   
06-10-08 09:09   
Thanks! Seems to have had a positive effect.
(0003161)
paulberto   
06-10-08 14:33   
Under low traffic removing libresin.so helped tremendously but under high load (right now) we're having the same issue and it's bringing us down every few minutes.

The current snapshot release i'm running seems to be much more stable than 3.1.5 --- we have a big issue with rewrites (rewrite-dispatch) not working properly in 3.1.6 so i have to stick with the snapshot.

In any case, would it be possible to get the alpha snapshot discussed yesterday -- i really need it at this point. Understanding it is alpha i can revert back to any version from a click of a button if i deem it to unstable.

Thanks Ferg!
(0003162)
paulberto   
06-10-08 14:35   
Sorry disregard -- i see that you have published new snapshot.. gona give it a whirl !!!! thanks!!!!!!
(0003164)
paulberto   
06-10-08 16:42   
Still happening using latest snapshot build (s080609):

[2008/06/11 02:28:03.054] java.io.InterruptedIOException: timeout fd=2494 errno=11
[2008/06/11 02:28:03.054]
[2008/06/11 02:28:03.054] at com.caucho.vfs.JniSocketImpl.writeNative(Native Method)
[2008/06/11 02:28:03.054] at com.caucho.vfs.JniSocketImpl.write(JniSocketImpl.java:288)
[2008/06/11 02:28:03.054] at com.caucho.vfs.JniStream.write(JniStream.java:124)
[2008/06/11 02:28:03.054] at com.caucho.vfs.WriteStream.nextBuffer(WriteStream.java:327)
[2008/06/11 02:28:03.054] at com.caucho.server.connection.ResponseStream.nextBuffer(ResponseStream.java:324)
[2008/06/11 02:28:03.054] at com.caucho.server.cache.CacheInode.writeToStream(CacheInode.java:102)
[2008/06/11 02:28:03.054] at com.caucho.server.cache.CacheInode.writeToStream(CacheInode.java:79)
[2008/06/11 02:28:03.054] at com.caucho.server.cache.CacheFilterChain.fillFromCache(CacheFilterChain.java:530)
[2008/06/11 02:28:03.054] at com.caucho.server.cache.CacheFilterChain.useCache(CacheFilterChain.java:301)
[2008/06/11 02:28:03.054] at com.caucho.server.cache.CacheFilterChain.doFilter(CacheFilterChain.java:174)
[2008/06/11 02:28:03.054] at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:187)
[2008/06/11 02:28:03.054] at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:266)
[2008/06/11 02:28:03.054] at com.caucho.server.http.HttpRequest.handleRequest(HttpRequest.java:270)
[2008/06/11 02:28:03.054] at com.caucho.server.port.TcpConnection.run(TcpConnection.java:678)
[2008/06/11 02:28:03.054] at com.caucho.util.ThreadPool$Item.runTasks(ThreadPool.java:721)
[2008/06/11 02:28:03.054] at com.caucho.util.ThreadPool$Item.run(ThreadPool.java:643)
[2008/06/11 02:28:03.054] at java.lang.Thread.run(Thread.java:619)

Any suggestions ? Should i remove the JNI library again -- will that help ?
(0003165)
paulberto   
06-10-08 16:43   
Sorry was cut off:

[2008/06/11 02:28:03.054] java.io.InterruptedIOException: timeout fd=2494 errno=11
[2008/06/11 02:28:03.054]
[2008/06/11 02:28:03.054] at com.caucho.vfs.JniSocketImpl.writeNative(Native Method)
[2008/06/11 02:28:03.054] at com.caucho.vfs.JniSocketImpl.write(JniSocketImpl.java:288)
[2008/06/11 02:28:03.054] at com.caucho.vfs.JniStream.write(JniStream.java:124)
[2008/06/11 02:28:03.054] at com.caucho.vfs.WriteStream.nextBuffer(WriteStream.java:327)
[2008/06/11 02:28:03.054] at com.caucho.server.connection.ResponseStream.nextBuffer(ResponseStream.java:324)
[2008/06/11 02:28:03.054] at com.caucho.server.cache.CacheInode.writeToStream(CacheInode.java:102)
[2008/06/11 02:28:03.054] at com.caucho.server.cache.CacheInode.writeToStream(CacheInode.java:79)
[2008/06/11 02:28:03.054] at com.caucho.server.cache.CacheFilterChain.fillFromCache(CacheFilterChain.java:530)
[2008/06/11 02:28:03.054] at com.caucho.server.cache.CacheFilterChain.useCache(CacheFilterChain.java:301)
[2008/06/11 02:28:03.054] at com.caucho.server.cache.CacheFilterChain.doFilter(CacheFilterChain.java:174)
[2008/06/11 02:28:03.054] at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:187)
[2008/06/11 02:28:03.054] at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:266)
[2008/06/11 02:28:03.054] at com.caucho.server.http.HttpRequest.handleRequest(HttpRequest.java:270)
[2008/06/11 02:28:03.054] at com.caucho.server.port.TcpConnection.run(TcpConnection.java:678)
[2008/06/11 02:28:03.054] at com.caucho.util.ThreadPool$Item.runTasks(ThreadPool.java:721)
[2008/06/11 02:28:03.054] at com.caucho.util.ThreadPool$Item.run(ThreadPool.java:643)
[2008/06/11 02:28:03.054] at java.lang.Thread.run(Thread.java:619)
[2008/06/11 02:28:03.055] not-modified: /camcom.css
[2008/06/11 02:28:03.056] cache disabled due to java.io.InterruptedIOException: timeout fd=2494 errno=11
(0003166)
ferg   
06-11-08 08:06   
There's a new snapshot. The previous one did not include the InterruptedIOException fixes.