0006125: Resin shutdown from unexpected watchdog exit - exit reason: MEMORY (exit code=6)

Viewing Issue Simple Details [ Jump to Notes ]

[ View Advanced ] [ Issue History ] [ Print ]

Category

Severity

Reproducibility

Date Submitted

Last Update

0006125

[Resin]

crash

random

01-03-18 05:07

03-12-18 17:11

Reporter

venkatesan

View Status

public

Assigned To

ferg

Priority

normal

Resolution

no change required

Status

closed

Product Version

4.0.53

Summary

0006125: Resin shutdown from unexpected watchdog exit - exit reason: MEMORY (exit code=6)

Description

Hi,

We are resin 4.0.53 in Solaris 5.11.
We are facing an issue due which resin is stopping automatically by the watch dog.After our analysis it is been observed that the resin getting stopped after afternoon one time (i.e. Around & After 2:00 PM ) and at mid night (i.e. around & After 10:00 PM) daily. At this time the resin will be mostly idle.

We are getting following logs in the watchdog-manager.log
--------------
[2017/12/27 19:22:59.197] {main} watchdog = 127.0.0.1:-1 (watchdog:default)
[2017/12/27 19:22:59.197] {main} stage = production
[2017/12/27 19:22:59.434] {resin-17} WebApp[production/webapp/admin.resin/ROOT] active
[2017/12/27 19:22:59.437] {main} Host[production/host/admin.resin] active
[2017/12/27 19:22:59.437] {main} ServletService[id=default,cluster=watchdog] active
[2017/12/27 19:22:59.437] {main}
[2017/12/27 19:22:59.445] {main} http listening to localhost:9001
[2017/12/27 19:22:59.450] {main}
[2017/12/27 19:22:59.518] {watchdog-app-b} WatchdogChild[app-b] starting
[2017/12/27 19:22:59.521] {watchdog-app-b} Watchdog starting Resin[app-b]
[2017/12/27 21:03:15.158] {watchdog-app-b} Watchdog detected close of Resin[app-b,pid=0]
exit reason: [2017/12/27 21:03:15.158] {watchdog-app-b} WatchdogChild[app-b] starting
[2017/12/27 21:03:15.159] {watchdog-app-b} Watchdog starting Resin[app-b]
[2017/12/27 22:06:26.957] {watchdog-app-b} Watchdog detected close of Resin[app-b,pid=0]
exit reason: MEMORY (exit code=6)
------------

And we are getting following log in the stdout.log
---------
[17-12-28 15:04:14.097] {resin-191} DataStore removing 3 orphans (remove=3,entry-count=474)
[17-12-28 15:04:20.436] {resin-main-link} WarningService: Shutdown: Resin shutdown from unexpected watchdog exit.
[17-12-28 15:04:20.448] {resin-shutdown} ServletService[id=app-b,cluster=app] stopping
[17-12-28 15:04:20.450] {resin-shutdown} Host[production/host/default] stopping
[17-12-28 15:04:20.450] {resin-shutdown} WebApp[production/webapp/default/myapp] stopping
[17-12-28 15:04:20.484] {resin-shutdown} WebApp[production/webapp/default/ROOT] stopping
[17-12-28 15:04:20.485] {resin-shutdown} WebApp[production/webapp/default/resin-admin] stopping
[17-12-28 15:04:20.606] {resin-shutdown} Shutdown Resin reason: WATCHDOG_EXIT
-------------------------------------------------------------

We have adequate RAM (24GB) in the server.

We have changed the following parameters to address this issue.

1. Reduced the stack size : 512kb to 256kb

2. Initial heap size defined as : 256 MB

3. Maximum extending of heap size defined as : 2GB

Even though we are getting the same error.Please let us know how to resolve this issue.

Additional Information

Attached Files

Relationships

Notes
(0006815) ferg 01-04-18 15:13	Those two logs don't match. You'd want the part of the ivm-default.log/stdout.log at 2017-12-27 22:06. Since the code is MEMORY, it's most likely a memory leak in the application. The *.pdf report for the restart should include the heap dump on exit.

(0006830) ferg 03-12-18 17:11	As noted, a memory issue is an application error. It is possible on Solaris that Resin was started with 32 bits. Resin requires a 64-bit JVM. Use -d64 on the command line if it starts with 32 bits.

Issue History
Date Modified	Username	Field	Change
01-03-18 05:07	venkatesan	New Issue
01-03-18 05:09	venkatesan	Issue Monitored: venkatesan
01-04-18 15:13	ferg	Note Added: 0006815
03-12-18 17:11	ferg	Note Added: 0006830
03-12-18 17:11	ferg	Assigned To	=> ferg
03-12-18 17:11	ferg	Status	new => closed
03-12-18 17:11	ferg	Resolution	open => no change required

Mantis 1.0.0rc3[^]

32 total queries executed.
27 unique queries executed.