Our traffic continues to increase placing more load on our Apaches and Resin servers. We have noticed a marked increase in the number of 503 errors reported by users and started tailing the access log. We were generating something on the order of 50-100 503 errors per minute. We tried tuning a bunch of things on the backend like thread pools, etc. thinking there may have been some chance of being the problem, but it made no difference. We could achieve 503 error within about 1-2 minutes of restarting apache. The only thing that solved the problem was disabling the native socket libraries used by resin. As soon as we moved the .so files in libexec out of the way and restarted the application the problem completely stopped.
Our resin servers all run on quad CPU Ubuntu 7.04 x86-64
Our front end web servers run mod_caucho w/ Apache 2.0 and Apache 2.2 and both versions were experiencing the error.