Mantis Bugtracker
  

Viewing Issue Advanced Details Jump to Notes ] View Simple ] Issue History ] Print ]
ID Category Severity Reproducibility Date Submitted Last Update
0001194 [Quercus] major sometimes 06-12-06 15:15 06-28-06 11:47
Reporter slamb View Status public  
Assigned To ferg
Priority normal Resolution fixed Platform
Status closed   OS
Projection none   OS Version
ETA none Fixed in Version 3.0.20 Product Version 3.0.19
  Product Build
Summary 0001194: Two running wrapper.pls after restart
Description Encountered this on Resin 2.1.17; looks like wrapper.pl in 3.0.19 is essentially the same.

Resin doesn't seem to completely let go of the logfiles unless I completely restart it. Thus, I do so in logrotate nightly. (Perhaps that's a separate bug, and I haven't checked if it applies to 3.0.)

In any case, sometimes after restart there are two wrapper.pls running. One of them starts up Java, which realizes that it can't bind to the port and dies. It then starts it again, and again, and again. The result is horrible VM and CPU spikes bringing up a JVM all the time for nothing.

I added some syslogs to diagnose and found this happens about once a week:

1. wrapper.pl B starts, SIGTERMs wrapper.pl A, creates a pidfile, and then wrapper.pl A unlinks it.
2. wrapper.pl C starts and has no way of knowing about wrapper.pl B. Both continue to run, with nasty consequences.

I found this comment in wrapper.pl:

    # unlink needs to happen relatively soon so restart's pid won't
    # get unlinked

That's inadequate. There's still a race, and it's still sometimes being lost.

My solution is to not make the new pidfile until it knows the old process is dead and therefore won't be unlinking any files. Something like this:

        my $signal = 15;
        while (kill($signal, $pid)) {
            syslog('info', "pid $$ delivered signal $signal to $pid (old wrapper)");
            sleep(1);
            $signal = 9;
        }
        syslog('info', "pid $$: old wrapper $pid should be dead now");
Steps To Reproduce
Additional Information
Attached Files

- Relationships

There are no notes attached to this issue.

- Issue History
Date Modified Username Field Change
06-12-06 15:15 slamb New Issue
06-12-06 16:31 slamb Issue Monitored: slamb
06-28-06 11:47 ferg Assigned To  => ferg
06-28-06 11:47 ferg Status new => closed
06-28-06 11:47 ferg Resolution open => fixed
06-28-06 11:47 ferg Fixed in Version  => 3.0.20


Mantis 1.0.0rc3[^]
Copyright © 2000 - 2005 Mantis Group
28 total queries executed.
25 unique queries executed.
Powered by Mantis Bugtracker