0006183: Resin daily performance issue and Resin process high CPU utlization issue

Viewing Issue Simple Details [ Jump to Notes ]

[ View Advanced ] [ Issue History ] [ Print ]

Category

Severity

Reproducibility

Date Submitted

Last Update

0006183

[Resin]

major

always

08-31-18 06:17

09-11-18 12:52

Reporter

venkatesan

View Status

public

Assigned To

ferg

Priority

normal

Resolution

no change required

Status

closed

Product Version

4.0.53

Summary

0006183: Resin daily performance issue and Resin process high CPU utlization issue

Description

Java Version : Java(TM) SE Runtime Environment 1.7.0_171-b11
OS flavor : Unix SunOS 5.11 sparcv9
Resin Version : Resin-4.0.53

Hi,

The resin process taking more CPU memory and performance issue after 3 hours of up time, even in idle time also .No errors in the JVM logs and Watchdog manager logs.

Screen shots of Resin process,CPU utilization ,prstat Attachment

We tried jvm args with the following combinations and find below the results of the same.

1.
jvm_args : -Xmx512m -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled -XX:-UseGCOverheadLimit

And its result to MEMORY issue

[2018/08/31 11:09:02.004] {resin-27} Watchdog received warning from Resin[app-b,pid=0]:
                                       Shutdown: TcpSocketLink OutOfMemory
[2018/08/31 11:10:04.134] {watchdog-app-b} Watchdog detected close of Resin[app-b,pid=0]
                                             exit reason: MEMORY (exit code=6)
[2018/08/31 11:10:04.134] {watchdog-app-b} WatchdogChild[app-b] starting

2. jvm_args : -Xmx1024m -XX:MaxPermSize=1024m -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled -XX:-UseGCOverheadLimit

Result to Memory issue and performance, high CPU utilization issue.

3. jvm_args : -Xmx2048m -XX:MaxPermSize=2048m -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:+CMSParallelRemarkEnabled -XX:+CMSClassUnloadingEnabled -XX:-UseGCOverheadLimit

Result to performance / slowness issue.

Please find below the jps and jstack output:
=============================================================================
$jps

31742 Jps
31573 WatchdogManager
31574 Resin

$ jstack 31574
2018-08-31 11:34:34
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.171-b11 mixed mode):

"Attach Listener" daemon prio=3 tid=0x0000000101985800 nid=0x2ec waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"resin-713" daemon prio=3 tid=0x0000000102683800 nid=0x2df waiting on condition [0xffffffff4baff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-712" daemon prio=3 tid=0x000000010144a800 nid=0x2de waiting on condition [0xffffffff4c2ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-711" daemon prio=3 tid=0x0000000104707000 nid=0x2dd waiting on condition [0xffffffff4dcff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-710" daemon prio=3 tid=0x00000001025f4000 nid=0x2dc waiting on condition [0xffffffff5d6ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-709" daemon prio=3 tid=0x0000000100e20000 nid=0x2db waiting on condition [0xffffffff5f2ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-708" daemon prio=3 tid=0x000000010144a000 nid=0x2da waiting on condition [0xffffffff4f2ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"FlushWorker[]-17" daemon prio=3 tid=0x0000000102dab800 nid=0x2d9 runnable [0xffffffff523ff000]
   java.lang.Thread.State: RUNNABLE
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:345)
        at com.caucho.vfs.VfsStream.write(VfsStream.java:201)
        at com.caucho.vfs.WriteStream.flush(WriteStream.java:400)
        at com.caucho.log.AbstractRolloverLog.flushStream(AbstractRolloverLog.java:474)
        - locked <0x000000071774dc78> (a java.lang.Object)
        at com.caucho.log.AbstractRolloverLog$FlushWorker.runTask(AbstractRolloverLog.java:988)
        at com.caucho.env.thread2.AbstractTaskWorker2.run(AbstractTaskWorker2.java:247)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"DependencyCheckWorker[]-20" daemon prio=3 tid=0x0000000102764800 nid=0x2d7 runnable [0xffffffff3e3fe000]
   java.lang.Thread.State: RUNNABLE
        at java.io.UnixFileSystem.getLength(Native Method)
        at java.io.File.length(File.java:968)
        at com.caucho.vfs.FilePath.getLength(FilePath.java:433)
        at com.caucho.vfs.Depend.isModified(Depend.java:171)
        at com.caucho.make.DependencyContainer.checkImpl(DependencyContainer.java:298)
        at com.caucho.make.DependencyContainer.isModified(DependencyContainer.java:259)
        at com.caucho.make.DependencyContainer.isModified(DependencyContainer.java:234)
        at com.caucho.loader.JarListLoader.isModified(JarListLoader.java:93)
        at com.caucho.loader.TreeLoader.isModified(TreeLoader.java:166)
        at com.caucho.make.DependencyContainer.checkImpl(DependencyContainer.java:298)
        at com.caucho.make.DependencyContainer.access$100(DependencyContainer.java:47)
        at com.caucho.make.DependencyContainer$DependencyCheckWorker.runTask(DependencyContainer.java:365)
        at com.caucho.env.thread2.AbstractTaskWorker2.run(AbstractTaskWorker2.java:247)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-704" daemon prio=3 tid=0x00000001007a5000 nid=0x2d6 waiting on condition [0xffffffff4eaff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-703" daemon prio=3 tid=0x00000001014a6800 nid=0x2d5 waiting on condition [0xffffffff521ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-702" daemon prio=3 tid=0x0000000101eab800 nid=0x2d4 waiting on condition [0xffffffff4d6ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-701" daemon prio=3 tid=0x00000001034e2800 nid=0x2d3 waiting on condition [0xffffffff4c4ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-677" daemon prio=3 tid=0x0000000103495000 nid=0x2bb waiting on condition [0xffffffff4f0ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-port-9000-675" daemon prio=3 tid=0x0000000103163000 nid=0x2b9 runnable [0xffffffff5fefe000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:153)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at com.caucho.vfs.SocketStream.read(SocketStream.java:187)
        at com.caucho.vfs.SocketStream.readTimeout(SocketStream.java:239)
        at com.caucho.vfs.ReadStream.fillWithTimeout(ReadStream.java:1146)
        at com.caucho.network.listen.TcpSocketLink.threadKeepalive(TcpSocketLink.java:1468)
        at com.caucho.network.listen.TcpSocketLink.processKeepalive(TcpSocketLink.java:1446)
        at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1285)
        at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1201)
        at com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:997)
        at com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
        at com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
        at com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
        at com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-674" daemon prio=3 tid=0x0000000100aec000 nid=0x2b8 waiting on condition [0xffffffff61eff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-672" daemon prio=3 tid=0x0000000102683000 nid=0x2b6 waiting on condition [0xffffffff620ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-port-9000-670" daemon prio=3 tid=0x0000000101c09000 nid=0x2b4 runnable [0xffffffff62ffe000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:153)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at com.caucho.vfs.SocketStream.read(SocketStream.java:187)
        at com.caucho.vfs.SocketStream.readTimeout(SocketStream.java:239)
        at com.caucho.vfs.ReadStream.fillWithTimeout(ReadStream.java:1146)
        at com.caucho.network.listen.TcpSocketLink.threadKeepalive(TcpSocketLink.java:1468)
        at com.caucho.network.listen.TcpSocketLink.processKeepalive(TcpSocketLink.java:1446)
        at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1285)
        at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1201)
        at com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:997)
        at com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
        at com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
        at com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
        at com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-669" daemon prio=3 tid=0x00000001035f1800 nid=0x2b3 waiting on condition [0xffffffff4feff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-668" daemon prio=3 tid=0x00000001033dc800 nid=0x2b2 waiting on condition [0xffffffff52eff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-667" daemon prio=3 tid=0x0000000102d73000 nid=0x2b1 waiting on condition [0xffffffff528ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-port-9000-664" daemon prio=3 tid=0x0000000103204000 nid=0x2ae runnable [0xffffffff4d0fe000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:153)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at com.caucho.vfs.SocketStream.read(SocketStream.java:187)
        at com.caucho.vfs.SocketStream.readTimeout(SocketStream.java:239)
        at com.caucho.vfs.ReadStream.fillWithTimeout(ReadStream.java:1146)
        at com.caucho.network.listen.TcpSocketLink.threadKeepalive(TcpSocketLink.java:1468)
        at com.caucho.network.listen.TcpSocketLink.processKeepalive(TcpSocketLink.java:1446)
        at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1285)
        at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1201)
        at com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:997)
        at com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
        at com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
        at com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
        at com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-662" daemon prio=3 tid=0x000000010320c800 nid=0x2ac waiting on condition [0xffffffff4bcff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-port-9000-646" daemon prio=3 tid=0x0000000102573800 nid=0x29c runnable [0xffffffff4a6fe000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:153)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at com.caucho.vfs.SocketStream.read(SocketStream.java:187)
        at com.caucho.vfs.SocketStream.readTimeout(SocketStream.java:239)
        at com.caucho.vfs.ReadStream.fillWithTimeout(ReadStream.java:1146)
        at com.caucho.network.listen.TcpSocketLink.threadKeepalive(TcpSocketLink.java:1468)
        at com.caucho.network.listen.TcpSocketLink.processKeepalive(TcpSocketLink.java:1446)
        at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1285)
        at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1201)
        at com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:997)
        at com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
        at com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
        at com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
        at com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-643" daemon prio=3 tid=0x0000000101935000 nid=0x299 waiting on condition [0xffffffff490ff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-port-9000-642" daemon prio=3 tid=0x00000001009f1800 nid=0x298 runnable [0xffffffff500fe000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:153)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at com.caucho.vfs.SocketStream.read(SocketStream.java:187)
        at com.caucho.vfs.SocketStream.readTimeout(SocketStream.java:239)
        at com.caucho.vfs.ReadStream.fillWithTimeout(ReadStream.java:1146)
        at com.caucho.network.listen.TcpSocketLink.threadKeepalive(TcpSocketLink.java:1468)
        at com.caucho.network.listen.TcpSocketLink.processKeepalive(TcpSocketLink.java:1446)
        at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1285)
        at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1201)
        at com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:997)
        at com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
        at com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
        at com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
        at com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-641" daemon prio=3 tid=0x0000000101ff9000 nid=0x297 waiting on condition [0xffffffff51bff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-port-9000-640" daemon prio=3 tid=0x0000000100cf9000 nid=0x296 runnable [0xffffffff614fc000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:153)
        at java.net.SocketInputStream.read(SocketInputStream.java:122)
        at oracle.net.ns.Packet.receive(Unknown Source)
        at oracle.net.ns.DataPacket.receive(Unknown Source)
        at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.net.ns.NetInputStream.read(Unknown Source)
        at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:930)
        at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:894)
        at oracle.jdbc.ttc7.Oclose.receive(Oclose.java:103)
        at oracle.jdbc.ttc7.TTC7Protocol.close(TTC7Protocol.java:693)
        - locked <0x000000076a33d6b8> (a oracle.jdbc.ttc7.TTC7Protocol)
        at oracle.jdbc.driver.OracleStatement.close(OracleStatement.java:731)
        - locked <0x000000076ca8c770> (a oracle.jdbc.driver.OracleStatement)
        - locked <0x000000076a33ce38> (a oracle.jdbc.driver.OracleConnection)
        at com.caucho.sql.UserStatement.close(UserStatement.java:177)
        at com.gbm.common.GBMDBPool.getConnection(GBMDBPool.java:107)
        at com.gbm.common.GBMDBCommonComponent.getResultSet(GBMDBCommonComponent.java:88)
        at com.gbm.common.GBMGeneralComponent.getResultSet(GBMGeneralComponent.java:205)
        at com.gbm.common.GBMGeneralComponent.getSingletonResult(GBMGeneralComponent.java:5299)
        at com.gbm.pension.GBMPensionLevelMasterComponent.GetTrsyDesc(GBMPensionLevelMasterComponent.java:1151)
        at _jsp._gbm._jsp._pension._gbm_0pension_0menu__jsp._jspService(_gbm_0pension_0menu__jsp.java:766)
        at _jsp._gbm._jsp._pension._gbm_0pension_0menu__jsp._jspService(_gbm_0pension_0menu__jsp.java:60)
        at com.caucho.jsp.JavaPage.service(JavaPage.java:64)
        at com.caucho.jsp.Page.pageservice(Page.java:548)
        at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
        at com.caucho.server.webapp.DispatchFilterChain.doFilter(DispatchFilterChain.java:131)
        at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:290)
        at com.caucho.server.webapp.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:298)
        at com.caucho.server.webapp.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:116)
        at com.caucho.jsp.PageContextImpl.forward(PageContextImpl.java:1158)
        at _jsp._gbm._jsp._pension._gbm_0pension_0index__jsp._jspService(_gbm_0pension_0index__jsp.java:155)
        at _jsp._gbm._jsp._pension._gbm_0pension_0index__jsp._jspService(_gbm_0pension_0index__jsp.java:31)
        at com.caucho.jsp.JavaPage.service(JavaPage.java:64)
        at com.caucho.jsp.Page.pageservice(Page.java:548)
        at com.caucho.server.dispatch.PageFilterChain.doFilter(PageFilterChain.java:194)
        at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:156)
        at com.caucho.server.webapp.AccessLogFilterChain.doFilter(AccessLogFilterChain.java:95)
        at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:290)
        at com.caucho.server.http.HttpRequest.handleRequest(HttpRequest.java:838)
        at com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:1353)
        at com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:1309)
        at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1293)
        at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1201)
        at com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:997)
        at com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
        at com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
        at com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
        at com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

"resin-573" daemon prio=3 tid=0x000000010124e000 nid=0x253 waiting on condition [0xffffffff48cff000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:315)
        at com.caucho.env.thread2.ResinThread2.park(ResinThread2.java:196)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:147)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

Additional Information

GC logs :
=================================================================
Java HotSpot(TM) 64-Bit Server VM (24.171-b11) for solaris-sparc JRE (1.7.0_171-b11), built on Dec 20 2017 00:03:46 by "java_re" with Sun Studio 12u1
Memory: 8k page, physical 56623104k(42444928k free)
CommandLine flags: -XX:AllocateInstancePrefetchLines=2 -XX:AllocatePrefetchInstr=1 -XX:AllocatePrefetchLines=6 -XX:AllocatePrefetchStyle=3 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=905969664 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=327155712 -XX:MaxPermSize=2147483648 -XX:MaxTenuringThreshold=6 -XX:OldPLABSize=16 -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit -XX:+UseParNewGC
1.938: [GC 236032K->29378K(855296K), 0.0318638 secs]
2.199: [Full GC 64654K->22939K(855296K), 0.1323400 secs]
4.576: [GC 259035K->118594K(855360K), 0.1175130 secs]
7.338: [GC 248095K(855360K), 0.0844811 secs]
12.513: [GC 287230K(855360K), 0.1856703 secs]
38.139: [GC 310644K->142949K(855360K), 0.0676450 secs]
104.312: [GC 379045K->142894K(855360K), 0.0129912 secs]
165.232: [GC 378990K->155745K(855360K), 0.0274070 secs]
243.172: [GC 391841K->163603K(855360K), 0.0214620 secs]
268.554: [Full GC 286748K->105187K(855360K), 0.4493113 secs]
330.724: [GC 341283K->126752K(855360K), 0.0068089 secs]
389.481: [GC 362848K->120000K(855360K), 0.0081963 secs]
393.013: [GC 356096K->124805K(855360K), 0.0075988 secs]
394.326: [Full GC 280864K->124355K(855360K), 0.5280932 secs]
394.860: [Full GC 127241K->115156K(855360K), 0.4205903 secs]
447.250: [GC 351252K->131068K(855360K), 0.0088413 secs]
448.625: [GC 367164K->197753K(855360K), 0.0864380 secs]
450.014: [GC 433849K->266529K(855360K), 0.1046741 secs]
450.798: [GC 474784K->365265K(855360K), 0.0797336 secs]
450.879: [GC 406036K(855360K), 0.0334903 secs]
451.309: [GC 601361K->390652K(855360K), 0.0408717 secs]
452.185: [GC 526541K(855360K), 0.0753252 secs]
452.841: [GC 610039K->353395K(855360K), 0.0027686 secs]
454.153: [GC 589491K->355007K(855360K), 0.0037258 secs]
455.461: [GC 591103K->359002K(855360K), 0.0044464 secs]
456.781: [GC 595098K->363973K(855360K), 0.0051950 secs]
458.084: [GC 600069K->369908K(855360K), 0.0059426 secs]
459.363: [GC 606004K->372749K(855360K), 0.0127552 secs]
460.638: [GC 608845K->378202K(855360K), 0.0102421 secs]
461.864: [GC 614298K->382822K(855360K), 0.0096062 secs]
463.174: [GC 618918K->382417K(855360K), 0.0092969 secs]
464.485: [GC 618513K->387513K(855360K), 0.0086131 secs]
465.759: [GC 623609K->387164K(855360K), 0.0086527 secs]
467.037: [GC 623260K->392313K(855360K), 0.0087535 secs]
467.253: [GC 628409K->395090K(855360K), 0.0083423 secs]
467.312: [GC 631127K->390224K(855360K), 0.0049476 secs]
467.355: [GC 625817K->394573K(855360K), 0.0065044 secs]
467.400: [GC 630581K->388925K(855360K), 0.0062645 secs]
467.447: [GC 624695K->390687K(855360K), 0.0058321 secs]
467.483: [GC 626758K->389267K(855360K), 0.0098007 secs]
467.493: [GC 389367K(855360K), 0.0027341 secs]
467.520: [GC 625363K->386571K(855360K), 0.0017833 secs]
467.558: [GC 622633K->385930K(855360K), 0.0016979 secs]
467.587: [GC 621815K->385839K(855360K), 0.0017685 secs]
467.615: [GC 621696K->385857K(855360K), 0.0016638 secs]
467.654: [GC 621809K->386355K(855360K), 0.0017232 secs]
467.683: [GC 622269K->386211K(855360K), 0.0016806 secs]
467.710: [GC 621671K->385924K(855360K), 0.0016531 secs]
467.741: [GC 621991K->385960K(855360K), 0.0016953 secs]
467.770: [GC 621970K->385997K(855360K), 0.0017608 secs]
467.800: [GC 622039K->386030K(855360K), 0.0017456 secs]
467.832: [GC 621990K->386063K(855360K), 0.0016987 secs]
467.859: [GC 621973K->386100K(855360K), 0.0016810 secs]
467.887: [GC 621822K->387173K(855360K), 0.0018346 secs]
467.913: [GC 622613K->386154K(855360K), 0.0017754 secs]
467.941: [GC 622182K->386188K(855360K), 0.0017874 secs]
467.968: [GC 621903K->386710K(855360K), 0.0017127 secs]
467.996: [GC 622765K->386757K(855360K), 0.0016996 secs]
468.023: [GC 622131K->386271K(855360K), 0.0018931 secs]
468.051: [GC 622159K->386952K(855360K), 0.0017301 secs]
468.078: [GC 623030K->386988K(855360K), 0.0018656 secs]
468.106: [GC 622378K->386906K(855360K), 0.0016824 secs]
468.134: [GC 622940K->387639K(855360K), 0.0018215 secs]
468.161: [GC 623184K->387079K(855360K), 0.0017266 secs]
468.189: [GC 623002K->387001K(855360K), 0.0017952 secs]
468.217: [GC 622876K->387752K(855360K), 0.0017180 secs]
468.245: [GC 623441K->386436K(855360K), 0.0017616 secs]
468.273: [GC 621957K->387097K(855360K), 0.0018657 secs]
468.301: [GC 623013K->386478K(855360K), 0.0016875 secs]
468.329: [GC 622015K->387260K(855360K), 0.0018284 secs]
468.357: [GC 622844K->386520K(855360K), 0.0018052 secs]
468.384: [GC 622585K->387230K(855360K), 0.0017907 secs]
468.412: [GC 622309K->387356K(855360K), 0.0017880 secs]
468.440: [GC 623175K->386588K(855360K), 0.0017849 secs]
468.468: [GC 622098K->386604K(855360K), 0.0015817 secs]
468.496: [GC 622027K->386629K(855360K), 0.0017595 secs]
468.524: [GC 622715K->388206K(855360K), 0.0018113 secs]
468.552: [GC 623371K->386659K(855360K), 0.0016659 secs]
468.580: [GC 621425K->386680K(855360K), 0.0017557 secs]
468.609: [GC 622587K->386696K(855360K), 0.0016613 secs]
468.637: [GC 622032K->386716K(855360K), 0.0017908 secs]
468.665: [GC 621414K->387513K(855360K), 0.0018115 secs]
468.693: [GC 623020K->387625K(855360K), 0.0018158 secs]
468.723: [GC 623185K->386765K(855360K), 0.0017782 secs]
468.751: [GC 622309K->387595K(855360K), 0.0016754 secs]
468.779: [GC 622280K->386800K(855360K), 0.0017671 secs]
468.808: [GC 621931K->386817K(855360K), 0.0016609 secs]
468.837: [GC 622426K->387672K(855360K), 0.0018669 secs]
468.865: [GC 623004K->388624K(855360K), 0.0018384 secs]
468.894: [GC 623527K->386867K(855360K), 0.0018031 secs]
468.922: [GC 622212K->387825K(855360K), 0.0018144 secs]
468.950: [GC 623482K->386899K(855360K), 0.0018494 secs]
468.978: [GC 621955K->386917K(855360K), 0.0016863 secs]
469.007: [GC 622210K->387899K(855360K), 0.0018111 secs]
469.035: [GC 623269K->386949K(855360K), 0.0016985 secs]
469.065: [GC 622609K->387945K(855360K), 0.0017492 secs]
469.094: [GC 623339K->386980K(855360K), 0.0018138 secs]
469.123: [GC 622427K->387989K(855360K), 0.0018261 secs]
469.152: [GC 623332K->387007K(855360K), 0.0016872 secs]
469.181: [GC 622323K->387959K(855360K), 0.0017161 secs]
469.209: [GC 623151K->387037K(855360K), 0.0017355 secs]
469.238: [GC 623094K->387052K(855360K), 0.0016858 secs]
469.267: [GC 622836K->388097K(855360K), 0.0018118 secs]
469.296: [GC 622762K->387082K(855360K), 0.0017983 secs]
469.325: [GC 622482K->388140K(855360K), 0.0018414 secs]
469.354: [GC 622971K->388093K(855360K), 0.0017944 secs]
469.383: [GC 623662K->388115K(855360K), 0.0017776 secs]
469.411: [GC 623186K->388136K(855360K), 0.0017208 secs]
469.440: [GC 623925K->387150K(855360K), 0.0017824 secs]
469.470: [GC 623246K->388244K(855360K), 0.0017305 secs]
469.499: [GC 623822K->388199K(855360K), 0.0018188 secs]
469.528: [GC 622375K->387193K(855360K), 0.0016494 secs]
469.557: [GC 622802K->387207K(855360K), 0.0017500 secs]
469.586: [GC 623097K->387221K(855360K), 0.0017174 secs]
469.615: [GC 622586K->388281K(855360K), 0.0018771 secs]
469.645: [GC 624140K->389419K(855360K), 0.0019259 secs]
469.674: [GC 624522K->389446K(855360K), 0.0018410 secs]
469.703: [GC 624801K->387275K(855360K), 0.0018109 secs]
469.733: [GC 622042K->387288K(855360K), 0.0018589 secs]
469.762: [GC 623347K->388444K(855360K), 0.0017112 secs]
469.791: [GC 623609K->389552K(855360K), 0.0018769 secs]
469.821: [GC 624993K->388482K(855360K), 0.0018257 secs]
469.850: [GC 624304K->387340K(855360K), 0.0017552 secs]
469.880: [GC 622178K->388461K(855360K), 0.0018590 secs]
469.909: [GC 623520K->387366K(855360K), 0.0018005 secs]
469.938: [GC 622570K->388500K(855360K), 0.0019407 secs]
469.967: [GC 623970K->388519K(855360K), 0.0018603 secs]
470.009: [GC 624108K->389734K(855360K), 0.0018684 secs]
470.039: [GC 625419K->388560K(855360K), 0.0018301 secs]
470.068: [GC 623211K->388637K(855360K), 0.0019444 secs]
470.098: [GC 624473K->387444K(855360K), 0.0018418 secs]
470.128: [GC 622222K->387456K(855360K), 0.0017725 secs]
470.157: [GC 623396K->388635K(855360K), 0.0018615 secs]
470.187: [GC 624624K->387479K(855360K), 0.0019176 secs]
470.217: [GC 623479K->387491K(855360K), 0.0017436 secs]
470.246: [GC 623439K->387503K(855360K), 0.0018790 secs]
470.276: [GC 623440K->389953K(855360K), 0.0017568 secs]
470.306: [GC 625912K->388779K(855360K), 0.0019655 secs]
470.336: [GC 624497K->388742K(855360K), 0.0019003 secs]
470.366: [GC 624462K->390021K(855360K), 0.0019940 secs]
470.396: [GC 625482K->388830K(855360K), 0.0017453 secs]
470.426: [GC 622843K->388795K(855360K), 0.0019754 secs]
470.456: [GC 623974K->388812K(855360K), 0.0018602 secs]
470.487: [GC 622574K->387595K(855360K), 0.0018204 secs]
470.517: [GC 623568K->387606K(855360K), 0.0018292 secs]
470.548: [GC 623376K->388913K(855360K), 0.0018117 secs]
470.579: [GC 624423K->388879K(855360K), 0.0018301 secs]
470.609: [GC 624195K->387639K(855360K), 0.0018058 secs]
470.639: [GC 621318K->387650K(855360K), 0.0018146 secs]
470.670: [GC 623025K->389069K(855360K), 0.0021084 secs]
470.673: [GC 393092K(855360K), 0.0187855 secs]
470.722: [GC 623986K->387832K(855360K), 0.0019365 secs]
470.753: [GC 623139K->387821K(855360K), 0.0017303 secs]
470.784: [GC 622761K->387741K(855360K), 0.0016988 secs]
470.815: [GC 623573K->387597K(855360K), 0.0017100 secs]
470.848: [GC 623685K->387582K(855360K), 0.0017905 secs]
470.878: [GC 552070K->318028K(855360K), 0.0018245 secs]
470.908: [GC 550662K->316913K(855360K), 0.0018223 secs]
470.938: [GC 552782K->318200K(855360K), 0.0018796 secs]
470.969: [GC 553728K->318267K(855360K), 0.0018525 secs]
471.000: [GC 553372K->316912K(855360K), 0.0018284 secs]
471.029: [GC 551537K->318249K(855360K), 0.0020620 secs]
471.060: [GC 553814K->316933K(855360K), 0.0018022 secs]
471.090: [GC 550758K->316944K(855360K), 0.0016769 secs]
471.120: [GC 551684K->316955K(855360K), 0.0018040 secs]
471.151: [GC 551217K->318883K(855360K), 0.0197023 secs]
471.202: [GC 554084K->318592K(855360K), 0.0103168 secs]
471.242: [GC 553742K->317296K(855360K), 0.0108516 secs]
471.283: [GC 553346K->317429K(855360K), 0.0099138 secs]
471.340: [GC 552510K->318939K(855360K), 0.0157058 secs]
471.403: [GC 553399K->317512K(855360K), 0.0019390 secs]
471.434: [GC 552971K->317511K(855360K), 0.0019059 secs]
471.466: [GC 553212K->317497K(855360K), 0.0018737 secs]
471.497: [GC 553338K->318929K(855360K), 0.0021699 secs]
471.529: [GC 553539K->317509K(855360K), 0.0018451 secs]
471.560: [GC 552244K->317468K(855360K), 0.0017910 secs]
471.592: [GC 551599K->317483K(855360K), 0.0018039 secs]
471.622: [GC 551048K->317480K(855360K), 0.0018375 secs]
471.653: [GC 553290K->317490K(855360K), 0.0017577 secs]
471.684: [GC 552676K->318924K(855360K), 0.0019133 secs]
471.715: [GC 553515K->317508K(855360K), 0.0017823 secs]
471.745: [GC 552867K->317518K(855360K), 0.0017586 secs]
471.776: [GC 550727K->318963K(855360K), 0.0021490 secs]
471.808: [GC 554523K->317535K(855360K), 0.0018475 secs]
471.839: [GC 551003K->319031K(855360K), 0.0019676 secs]
471.870: [GC 554698K->319002K(855360K), 0.0019111 secs]
471.901: [GC 553988K->319062K(855360K), 0.0019077 secs]

Attached Files

RESIN_CPU_UTILIZATION_ISSUE.docx [^] (281,628 bytes) 08-31-18 06:17

gc.log [^] (850,113 bytes) 08-31-18 06:19

jps.txt [^] (211,647 bytes) 08-31-18 06:20

Relationships

Notes
(0006849) ferg 09-10-18 15:24	This is a memory leak issue, as you can see from the OutOfMemory and the MEMORY exit code. You need to get a heap dump, either from Resin's heap dump command line, or the PDF report or the JVM's own heap dump to see what's taking up the memory. The data provided here doesn't give any information about the memory use.

(0006850) venkatesan 09-10-18 23:21	Hi, We are not faced any " Memory " issue in default JVM args values and we observed "performance/slowness" issue only after sometime(3-4 hours of active time) . We had tried with JVM args parameters like attached information. at that time only we faced OutOfMemory and the MEMORY exit code. Kindly suggest us ,how to address the issue (performance/slowness issue ).

(0006851) ferg 09-11-18 12:50	This bug report specifically contains a complaint about OutOfMemory. If you have a performance problem, you need to get a profile. Resin Pro contains a profile option. If you don't have Resin Pro, you can use the JVM's profile tools. None of the information indicates any Resin problem, by the way. Memory and performance problems are generally application issues. Unless you have specific profile or heap data showing a Resin problem, the most likely cause is an application problem. You'd need to use profiling and heap information to track those down first.

(0006852) ferg 09-11-18 12:52	Closed, because nothing in this bug report shows a Resin bug.

Issue History
Date Modified	Username	Field	Change
08-31-18 06:17	venkatesan	New Issue
08-31-18 06:17	venkatesan	File Added: RESIN_CPU_UTILIZATION_ISSUE.docx
08-31-18 06:19	venkatesan	File Added: gc.log
08-31-18 06:20	venkatesan	File Added: jps.txt
08-31-18 06:28	venkatesan	Issue Monitored: venkatesan
09-10-18 15:24	ferg	Note Added: 0006849
09-10-18 23:21	venkatesan	Note Added: 0006850
09-11-18 12:50	ferg	Note Added: 0006851
09-11-18 12:52	ferg	Note Added: 0006852
09-11-18 12:52	ferg	Assigned To	=> ferg
09-11-18 12:52	ferg	Status	new => closed
09-11-18 12:52	ferg	Resolution	open => no change required
09-11-18 12:52	ferg	Description Updated
09-11-18 12:52	ferg	Additional Information Updated

Mantis 1.0.0rc3[^]

41 total queries executed.
29 unique queries executed.