0003645: syn_recv between mod_caucho and Resin

Viewing Issue Advanced Details [ Jump to Notes ]

[ View Simple ] [ Issue History ] [ Print ]

Category

Severity

Reproducibility

Date Submitted

Last Update

0003645

[Resin]

minor

always

08-21-09 13:44

08-24-09 10:53

Reporter

ferg

View Status

public

Assigned To

ferg

Priority

immediate

Resolution

fixed

Platform

Status

closed

Projection

none

OS Version

ETA

none

Fixed in Version

4.0.2

Product Version

3.1.9

Product Build

Summary

0003645: syn_recv between mod_caucho and Resin

Description

(rep by Daniel WIgenfors)

After a couple of hours, around noon, suddently all threads in apache where eaten up, and when checking the caucho-status page, many of the resin-instanses where marked red and unavailable. When using telnet, to try to access port 6801 of the resin-servers, alot of times, there was no response, just timeout, just as mod_caucho would have seen it probably.

We have a http port, 8081, configured on the resin-instances, and it was possible to reach that port, and access the /resin-status page, there we could see that the instance was more or less idling, with a minimal number of active threads. When doing a thread-dump of the instance, it looks like no threads are running, they are just waiting for new connections.

a netstat on the resin-machine revelead alot of (up to around 1000) connections in SYN_RECV, all from the webserver (web64). We first re-compiled mod caucho and lowered the connect timout back to the default 2s, and restarted everything, after a while, some of the resin servers startet to be unresponsive again, and the number of SYN_RECV connections increased. We then rolled back to the 3.1.8 mod_caucho with the default connect timeout.
After a while we suffered from the same problems again, and we decieded to roll back to resin 3.1.8 on the resin-servers as well.
This seems to have fixed the problems, as we havent seen any more of these since the application-servers where restarted with 3.1.8 again.

The closest conclution would be to suspect that there is something wrong with the communication between resin and mod_cacucho in 3.1.9. Have you had any other reports with this kind of problem?

Steps To Reproduce

Additional Information

Attached Files

Relationships

Notes
(0004137) ferg 08-24-09 10:53	Issue is related to a low thread-max (200) with a heavy load using all threads.

Issue History
Date Modified	Username	Field	Change
08-21-09 13:44	ferg	New Issue
08-24-09 10:52	ferg	Assigned To	=> ferg
08-24-09 10:52	ferg	Status	new => closed
08-24-09 10:52	ferg	Resolution	open => fixed
08-24-09 10:52	ferg	Fixed in Version	=> 4.0.2
08-24-09 10:53	ferg	Note Added: 0004137

Mantis 1.0.0rc3[^]

28 total queries executed.
25 unique queries executed.