Mantis Bugtracker
  

Viewing Issue Simple Details Jump to Notes ] View Advanced ] Issue History ] Print ]
ID Category Severity Reproducibility Date Submitted Last Update
0003645 [Resin] minor always 08-21-09 13:44 08-24-09 10:53
Reporter ferg View Status public  
Assigned To ferg
Priority immediate Resolution fixed  
Status closed   Product Version 3.1.9
Summary 0003645: syn_recv between mod_caucho and Resin
Description (rep by Daniel WIgenfors)


After a couple of hours, around noon, suddently all threads in apache where eaten up, and when checking the caucho-status page, many of the resin-instanses where marked red and unavailable. When using telnet, to try to access port 6801 of the resin-servers, alot of times, there was no response, just timeout, just as mod_caucho would have seen it probably.

We have a http port, 8081, configured on the resin-instances, and it was possible to reach that port, and access the /resin-status page, there we could see that the instance was more or less idling, with a minimal number of active threads. When doing a thread-dump of the instance, it looks like no threads are running, they are just waiting for new connections.

a netstat on the resin-machine revelead alot of (up to around 1000) connections in SYN_RECV, all from the webserver (web64). We first re-compiled mod caucho and lowered the connect timout back to the default 2s, and restarted everything, after a while, some of the resin servers startet to be unresponsive again, and the number of SYN_RECV connections increased. We then rolled back to the 3.1.8 mod_caucho with the default connect timeout.
After a while we suffered from the same problems again, and we decieded to roll back to resin 3.1.8 on the resin-servers as well.
This seems to have fixed the problems, as we havent seen any more of these since the application-servers where restarted with 3.1.8 again.

The closest conclution would be to suspect that there is something wrong with the communication between resin and mod_cacucho in 3.1.9. Have you had any other reports with this kind of problem?
Additional Information
Attached Files

- Relationships

- Notes
(0004137)
ferg
08-24-09 10:53

Issue is related to a low thread-max (200) with a heavy load using all threads.
 

- Issue History
Date Modified Username Field Change
08-21-09 13:44 ferg New Issue
08-24-09 10:52 ferg Assigned To  => ferg
08-24-09 10:52 ferg Status new => closed
08-24-09 10:52 ferg Resolution open => fixed
08-24-09 10:52 ferg Fixed in Version  => 4.0.2
08-24-09 10:53 ferg Note Added: 0004137


Mantis 1.0.0rc3[^]
Copyright © 2000 - 2005 Mantis Group
28 total queries executed.
25 unique queries executed.
Powered by Mantis Bugtracker