Mantis - Resin
Viewing Issue Advanced Details
ID: Category: Severity: Reproducibility: Date Submitted: Last Update:
3569 minor always 06-18-09 19:26 08-24-09 10:10
Reporter: ferg Platform:  
Assigned To: ferg OS:  
Priority: normal OS Version:  
Status: closed Product Version:  
Product Build: Resolution: fixed  
Projection: none      
ETA: none Fixed in Version: 4.0.1  
Summary: 0003569: deadlock display
Description: (rep by Rob Lockstone)

I've never seen this happen with Resin. But ever since upgrading to Resin Pro 3.1.9 (from Pro 3.0.21), it's happening several times per week with different servers during restarts.

Environment: Windows 2003 64-bit Server (SP2), Resin Pro 3.1.9 (100 Server License), JDK 1.6.0_13 (64-bit)

I looked in the 1.6.0_14 release notes and didn't see anything specific to deadlock situations that were fixed in the .14 release. I don't know if this is a problem with our code, or Resin, or the JDK, or some combination thereof.

Other than the obvious, "Why is this happening and how can I fix it?" question, I'm also curious about the 15 minutes that it always takes resin to notice the deadlock and restart. Is this 15 minutes configurable? If so, where can I adjust it and what are the implications of making it < 15 minutes?

Rob

Here's an example of the arguments we pass to Java. Most of our servers are configured similarly with variations in the heap sizes.

      <jvm-arg>-server</jvm-arg>
      <jvm-arg>-verbose:gc</jvm-arg>
      <jvm-arg>-Xmn500m</jvm-arg>
      <jvm-arg>-Xms5000m</jvm-arg>
      <jvm-arg>-Xmx5000m</jvm-arg>
      <jvm-arg>-Xss128k</jvm-arg>
      <jvm-arg>-Xrs</jvm-arg>
      <jvm-arg>-Xloggc:log/gc.log</jvm-arg>
      <jvm-arg>-Xdebug</jvm-arg>
      <jvm-arg>-XX:+UseConcMarkSweepGC</jvm-arg>
      <jvm-arg>-XX:+PrintGCTimeStamps</jvm-arg>
      <jvm-arg>-XX:+PrintGCDetails</jvm-arg>
      <jvm-arg>-Dhttp.maxConnections=400</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote.port=9337</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote.ssl=false</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote.authenticate=false</jvm-arg>
      <jvm-arg>-agentlib:resin</jvm-arg>

And here's the portion of the log file from when a deadlock occurs.

RESIN [02:58:38.369] {resin-destroy} Server[id=,cluster=app-tier] stopping [This is when the restart request is sent. --Rob]
RESIN [02:58:38.416] {http--80-2843$1916052035} WebApp[] active
RESIN [02:59:06.136] {main} Proxy Cache disk-size=1024M memory-size=64M [Resin starts to restart here. --Rob]
RESIN [02:59:06.168] {main} PingThread[] starting, checking []
RESIN [02:59:06.340] {main}
RESIN [02:59:06.340] {main} Windows 2003 5.2 amd64
RESIN [02:59:06.340] {main} Java(TM) SE Runtime Environment 1.6.0_13-b03, Cp1252, en
RESIN [02:59:06.340] {main} Java HotSpot(TM) 64-Bit Server VM 11.3-b02, 64, mixed mode, Sun Microsystems Inc.
RESIN [02:59:06.340] {main} user.name: SYSTEM
RESIN [02:59:06.340] {main} resin.home = d:\resin
RESIN [02:59:06.340] {main} resin.root = d:\resin
RESIN [02:59:06.340] {main} resin.conf = /d:/resin/conf/resin.conf
RESIN [02:59:06.340] {main}
RESIN [03:14:06.169] {resin-7} JDK detected deadlock. Restarting Resin. [15 minutes later is when Resin has detected the deadlock and restarts. --Rob]
RESIN [03:14:06.169] {resin-7} javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.ThreadInfo,items=((itemName=blockedCount,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=blockedTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=inNative,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=lockInfo,itemType=javax.management.openmbean.CompositeType(name=java.lang.management.LockInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer))))),(itemName=lockName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lockOwnerId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=lockOwnerName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lockedMonitors,itemType=javax.management.openmbean.ArrayType(name=[Ljavax.management.openmbean.CompositeData;,dimension=1,elementType=javax.management.openmbean.CompositeType(name=java.lang.management.MonitorInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=lockedStackDepth,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=lockedStackFrame,itemType=javax.management.openmbean.CompositeType(name=java.lang.StackTraceElement,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=fileName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lineNumber,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=methodName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=nativeMethod,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean))))))),primitiveArray=false)),(itemName=lockedSynchronizers,itemType=javax.management.openmbean.ArrayType(name=[Ljavax.management.openmbean.CompositeData;,dimension=1,elementType=javax.management.openmbean.CompositeType(name=java.lang.management.LockInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)))),primitiveArray=false)),(itemName=stackTrace,itemType=javax.management.openmbean.ArrayType(name=[Ljavax.management.openmbean.CompositeData;,dimension=1,elementType=javax.management.openmbean.CompositeType(name=java.lang.StackTraceElement,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=fileName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lineNumber,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=methodName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=nativeMethod,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)))),primitiveArray=false)),(itemName=suspended,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=threadId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=threadName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=threadState,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=waitedCount,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=waitedTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)))),contents={blockedCount=12, blockedTime=-1, inNative=false, lockInfo=javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.LockInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)))),contents={className=com.caucho.loader.EnvironmentClassLoader, identityHashCode=324813035}), lockName=com.caucho.loader.EnvironmentClassLoader@135c40eb, lockOwnerId=25, lockOwnerName=resin-8, lockedMonitors=[Ljavax.management.openmbean.CompositeData;@62135133, lockedSynchronizers=[Ljavax.management.openmbean.CompositeData;@62381d92, stackTrace=[Ljavax.management.openmbean.CompositeData;@521d8e9e, suspended=false, threadId=27, threadName=resin-10, threadState=BLOCKED, waitedCount=7, waitedTime=-1})
RESIN [03:14:06.184] {resin-7} javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.ThreadInfo,items=((itemName=blockedCount,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=blockedTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=inNative,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=lockInfo,itemType=javax.management.openmbean.CompositeType(name=java.lang.management.LockInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer))))),(itemName=lockName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lockOwnerId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=lockOwnerName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lockedMonitors,itemType=javax.management.openmbean.ArrayType(name=[Ljavax.management.openmbean.CompositeData;,dimension=1,elementType=javax.management.openmbean.CompositeType(name=java.lang.management.MonitorInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=lockedStackDepth,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=lockedStackFrame,itemType=javax.management.openmbean.CompositeType(name=java.lang.StackTraceElement,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=fileName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lineNumber,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=methodName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=nativeMethod,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean))))))),primitiveArray=false)),(itemName=lockedSynchronizers,itemType=javax.management.openmbean.ArrayType(name=[Ljavax.management.openmbean.CompositeData;,dimension=1,elementType=javax.management.openmbean.CompositeType(name=java.lang.management.LockInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)))),primitiveArray=false)),(itemName=stackTrace,itemType=javax.management.openmbean.ArrayType(name=[Ljavax.management.openmbean.CompositeData;,dimension=1,elementType=javax.management.openmbean.CompositeType(name=java.lang.StackTraceElement,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=fileName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lineNumber,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=methodName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=nativeMethod,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)))),primitiveArray=false)),(itemName=suspended,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=threadId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=threadName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=threadState,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=waitedCount,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=waitedTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)))),contents={blockedCount=7, blockedTime=-1, inNative=false, lockInfo=javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.LockInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)))),contents={className=com.caucho.loader.ClassEntry, identityHashCode=587476940}), lockName=com.caucho.loader.ClassEntry@23042fcc, lockOwnerId=27, lockOwnerName=resin-10, lockedMonitors=[Ljavax.management.openmbean.CompositeData;@67de0c09, lockedSynchronizers=[Ljavax.management.openmbean.CompositeData;@22e85825, stackTrace=[Ljavax.management.openmbean.CompositeData;@43f2f70a, suspended=false, threadId=25, threadName=resin-8, threadState=BLOCKED, waitedCount=1, waitedTime=-1})
RESIN [03:14:11.091] {main} Proxy Cache disk-size=1024M memory-size=64M
RESIN [03:14:11.122] {main} PingThread[] starting, checking []
RESIN [03:14:11.263] {main}
RESIN [03:14:11.263] {main} Windows 2003 5.2 amd64
RESIN [03:14:11.278] {main} Java(TM) SE Runtime Environment 1.6.0_13-b03, Cp1252, en
RESIN [03:14:11.278] {main} Java HotSpot(TM) 64-Bit Server VM 11.3-b02, 64, mixed mode, Sun Microsystems Inc.
RESIN [03:14:11.278] {main} user.name: SYSTEM
RESIN [03:14:11.278] {main} resin.home = d:\resin
RESIN [03:14:11.278] {main} resin.root = d:\resin
RESIN [03:14:11.278] {main} resin.conf = /d:/resin/conf/resin.conf
RESIN [03:14:11.278] {main}

Steps To Reproduce:
Additional Information:
Relationships
Attached Files:

There are no notes attached to this issue.