Mantis Bugtracker
  

Viewing Issue Simple Details Jump to Notes ] View Advanced ] Issue History ] Print ]
ID Category Severity Reproducibility Date Submitted Last Update
0003569 [Resin] minor always 06-18-09 19:26 08-24-09 10:10
Reporter ferg View Status public  
Assigned To ferg
Priority normal Resolution fixed  
Status closed   Product Version
Summary 0003569: deadlock display
Description (rep by Rob Lockstone)

I've never seen this happen with Resin. But ever since upgrading to Resin Pro 3.1.9 (from Pro 3.0.21), it's happening several times per week with different servers during restarts.

Environment: Windows 2003 64-bit Server (SP2), Resin Pro 3.1.9 (100 Server License), JDK 1.6.0_13 (64-bit)

I looked in the 1.6.0_14 release notes and didn't see anything specific to deadlock situations that were fixed in the .14 release. I don't know if this is a problem with our code, or Resin, or the JDK, or some combination thereof.

Other than the obvious, "Why is this happening and how can I fix it?" question, I'm also curious about the 15 minutes that it always takes resin to notice the deadlock and restart. Is this 15 minutes configurable? If so, where can I adjust it and what are the implications of making it < 15 minutes?

Rob

Here's an example of the arguments we pass to Java. Most of our servers are configured similarly with variations in the heap sizes.

      <jvm-arg>-server</jvm-arg>
      <jvm-arg>-verbose:gc</jvm-arg>
      <jvm-arg>-Xmn500m</jvm-arg>
      <jvm-arg>-Xms5000m</jvm-arg>
      <jvm-arg>-Xmx5000m</jvm-arg>
      <jvm-arg>-Xss128k</jvm-arg>
      <jvm-arg>-Xrs</jvm-arg>
      <jvm-arg>-Xloggc:log/gc.log</jvm-arg>
      <jvm-arg>-Xdebug</jvm-arg>
      <jvm-arg>-XX:+UseConcMarkSweepGC</jvm-arg>
      <jvm-arg>-XX:+PrintGCTimeStamps</jvm-arg>
      <jvm-arg>-XX:+PrintGCDetails</jvm-arg>
      <jvm-arg>-Dhttp.maxConnections=400</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote.port=9337</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote.ssl=false</jvm-arg>
      <jvm-arg>-Dcom.sun.management.jmxremote.authenticate=false</jvm-arg>
      <jvm-arg>-agentlib:resin</jvm-arg>

And here's the portion of the log file from when a deadlock occurs.

RESIN [02:58:38.369] {resin-destroy} Server[id=,cluster=app-tier] stopping [This is when the restart request is sent. --Rob]
RESIN [02:58:38.416] {http--80-2843$1916052035} WebApp[] active
RESIN [02:59:06.136] {main} Proxy Cache disk-size=1024M memory-size=64M [Resin starts to restart here. --Rob]
RESIN [02:59:06.168] {main} PingThread[] starting, checking []
RESIN [02:59:06.340] {main}
RESIN [02:59:06.340] {main} Windows 2003 5.2 amd64
RESIN [02:59:06.340] {main} Java(TM) SE Runtime Environment 1.6.0_13-b03, Cp1252, en
RESIN [02:59:06.340] {main} Java HotSpot(TM) 64-Bit Server VM 11.3-b02, 64, mixed mode, Sun Microsystems Inc.
RESIN [02:59:06.340] {main} user.name: SYSTEM
RESIN [02:59:06.340] {main} resin.home = d:\resin
RESIN [02:59:06.340] {main} resin.root = d:\resin
RESIN [02:59:06.340] {main} resin.conf = /d:/resin/conf/resin.conf
RESIN [02:59:06.340] {main}
RESIN [03:14:06.169] {resin-7} JDK detected deadlock. Restarting Resin. [15 minutes later is when Resin has detected the deadlock and restarts. --Rob]
RESIN [03:14:06.169] {resin-7} javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.ThreadInfo,items=((itemName=blockedCount,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=blockedTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=inNative,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=lockInfo,itemType=javax.management.openmbean.CompositeType(name=java.lang.management.LockInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer))))),(itemName=lockName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lockOwnerId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=lockOwnerName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lockedMonitors,itemType=javax.management.openmbean.ArrayType(name=[Ljavax.management.openmbean.CompositeData;,dimension=1,elementType=javax.management.openmbean.CompositeType(name=java.lang.management.MonitorInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=lockedStackDepth,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=lockedStackFrame,itemType=javax.management.openmbean.CompositeType(name=java.lang.StackTraceElement,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=fileName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lineNumber,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=methodName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=nativeMethod,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean))))))),primitiveArray=false)),(itemName=lockedSynchronizers,itemType=javax.management.openmbean.ArrayType(name=[Ljavax.management.openmbean.CompositeData;,dimension=1,elementType=javax.management.openmbean.CompositeType(name=java.lang.management.LockInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)))),primitiveArray=false)),(itemName=stackTrace,itemType=javax.management.openmbean.ArrayType(name=[Ljavax.management.openmbean.CompositeData;,dimension=1,elementType=javax.management.openmbean.CompositeType(name=java.lang.StackTraceElement,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=fileName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lineNumber,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=methodName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=nativeMethod,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)))),primitiveArray=false)),(itemName=suspended,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=threadId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=threadName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=threadState,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=waitedCount,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=waitedTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)))),contents={blockedCount=12, blockedTime=-1, inNative=false, lockInfo=javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.LockInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)))),contents={className=com.caucho.loader.EnvironmentClassLoader, identityHashCode=324813035}), lockName=com.caucho.loader.EnvironmentClassLoader@135c40eb, lockOwnerId=25, lockOwnerName=resin-8, lockedMonitors=[Ljavax.management.openmbean.CompositeData;@62135133, lockedSynchronizers=[Ljavax.management.openmbean.CompositeData;@62381d92, stackTrace=[Ljavax.management.openmbean.CompositeData;@521d8e9e, suspended=false, threadId=27, threadName=resin-10, threadState=BLOCKED, waitedCount=7, waitedTime=-1})
RESIN [03:14:06.184] {resin-7} javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.ThreadInfo,items=((itemName=blockedCount,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=blockedTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=inNative,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=lockInfo,itemType=javax.management.openmbean.CompositeType(name=java.lang.management.LockInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer))))),(itemName=lockName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lockOwnerId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=lockOwnerName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lockedMonitors,itemType=javax.management.openmbean.ArrayType(name=[Ljavax.management.openmbean.CompositeData;,dimension=1,elementType=javax.management.openmbean.CompositeType(name=java.lang.management.MonitorInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=lockedStackDepth,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=lockedStackFrame,itemType=javax.management.openmbean.CompositeType(name=java.lang.StackTraceElement,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=fileName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lineNumber,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=methodName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=nativeMethod,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean))))))),primitiveArray=false)),(itemName=lockedSynchronizers,itemType=javax.management.openmbean.ArrayType(name=[Ljavax.management.openmbean.CompositeData;,dimension=1,elementType=javax.management.openmbean.CompositeType(name=java.lang.management.LockInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)))),primitiveArray=false)),(itemName=stackTrace,itemType=javax.management.openmbean.ArrayType(name=[Ljavax.management.openmbean.CompositeData;,dimension=1,elementType=javax.management.openmbean.CompositeType(name=java.lang.StackTraceElement,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=fileName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=lineNumber,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)),(itemName=methodName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=nativeMethod,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)))),primitiveArray=false)),(itemName=suspended,itemType=javax.management.openmbean.SimpleType(name=java.lang.Boolean)),(itemName=threadId,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=threadName,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=threadState,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=waitedCount,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=waitedTime,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)))),contents={blockedCount=7, blockedTime=-1, inNative=false, lockInfo=javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.LockInfo,items=((itemName=className,itemType=javax.management.openmbean.SimpleType(name=java.lang.String)),(itemName=identityHashCode,itemType=javax.management.openmbean.SimpleType(name=java.lang.Integer)))),contents={className=com.caucho.loader.ClassEntry, identityHashCode=587476940}), lockName=com.caucho.loader.ClassEntry@23042fcc, lockOwnerId=27, lockOwnerName=resin-10, lockedMonitors=[Ljavax.management.openmbean.CompositeData;@67de0c09, lockedSynchronizers=[Ljavax.management.openmbean.CompositeData;@22e85825, stackTrace=[Ljavax.management.openmbean.CompositeData;@43f2f70a, suspended=false, threadId=25, threadName=resin-8, threadState=BLOCKED, waitedCount=1, waitedTime=-1})
RESIN [03:14:11.091] {main} Proxy Cache disk-size=1024M memory-size=64M
RESIN [03:14:11.122] {main} PingThread[] starting, checking []
RESIN [03:14:11.263] {main}
RESIN [03:14:11.263] {main} Windows 2003 5.2 amd64
RESIN [03:14:11.278] {main} Java(TM) SE Runtime Environment 1.6.0_13-b03, Cp1252, en
RESIN [03:14:11.278] {main} Java HotSpot(TM) 64-Bit Server VM 11.3-b02, 64, mixed mode, Sun Microsystems Inc.
RESIN [03:14:11.278] {main} user.name: SYSTEM
RESIN [03:14:11.278] {main} resin.home = d:\resin
RESIN [03:14:11.278] {main} resin.root = d:\resin
RESIN [03:14:11.278] {main} resin.conf = /d:/resin/conf/resin.conf
RESIN [03:14:11.278] {main}

Additional Information
Attached Files

- Relationships

There are no notes attached to this issue.

- Issue History
Date Modified Username Field Change
06-18-09 19:26 ferg New Issue
08-24-09 10:10 ferg Assigned To  => ferg
08-24-09 10:10 ferg Status new => closed
08-24-09 10:10 ferg Resolution open => fixed
08-24-09 10:10 ferg Fixed in Version  => 4.0.1


Mantis 1.0.0rc3[^]
Copyright © 2000 - 2005 Mantis Group
26 total queries executed.
24 unique queries executed.
Powered by Mantis Bugtracker