Mantis - Resin
Viewing Issue Advanced Details
6166 major random 06-14-18 02:49 07-26-18 16:01
closed 4.0.56  
none 4.0.58  
0006166: All threads associated with a port disappear
We are seeing resin servers randomly stop accepting requests on one port (but continue to process requests on other ports).

From jstacks and heap dumps it looks like the problem is that the resin-port-####-launcher thread dies, and the thread pool for the port then depletes until there are no threads left to handle requests.

We can't see any errors indicating why the launcher thread would have died.

From a heap dump taken after one occurrence we saw:

All the :9090 threads, including the -launcher thread have disappeared.

The :9090 TcpPort and its SocketLinkThreadLauncher objects are present in the heap dump, but the SocketLinkThreadLauncher is in what we think is a wierd state.

 - It has no thread assigned, but is in ACTIVE_WAKE state.

 - If it was in IDLE or PARK state then the TcpPort$SuspendReaper Alarm job (which looks to be scheduled) would reschedule or wake up the launcher thread, but in ACTIVE_WAIT state it does nothing.

 - The :9090 SocketLinkThreadLauncher's threadIdleExpireTime was around the time when we started seeing problems.

There are no notes attached to this issue.