Mantis - Resin
Viewing Issue Advanced Details
ID: Category: Severity: Reproducibility: Date Submitted: Last Update:
5894 major random 05-06-15 08:12 06-03-15 09:09
Reporter: Mathias Lagerwall Platform:  
Assigned To: ferg OS:  
Priority: normal OS Version:  
Status: closed Product Version: 4.0.44  
Product Build: Resolution: unable to reproduce  
Projection: none      
ETA: none Fixed in Version:  
Summary: 0005894: A lot of threads stuck in java.lang....
Description: We use the 4.0.s150407 Pro version (4.0.43 with some .44 fixes)

After upgrading to this version we have had some incidents were the application has stopped responding. A stack-trace reveals that there are a lot of threads stuck deep in java.lang classes.

We have also upgraded to Java 8 (from Java 7) in the same release so it might be a Java problem? (I have search the net but hasn't found anything that could explain this) We are running the Oracle SE 8u45.
Can this be due to changes in the ClassLoader?


Example of some of the traces from one occasion:

Thread 118: (state = BLOCKED)
 - java.lang.Class.getDeclaredConstructors0(boolean) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.Class.privateGetDeclaredConstructors(boolean) @bci=58, line=2671 (Compiled frame)
 - java.lang.Class.getConstructor0(java.lang.Class[], int) @bci=10, line=3075 (Compiled frame)
 - java.lang.Class.newInstance() @bci=46, line=412 (Compiled frame)
 - sun.reflect.MethodAccessorGenerator$1.run() @bci=24, line=403 (Compiled frame)
 - sun.reflect.MethodAccessorGenerator$1.run() @bci=1, line=394 (Compiled frame)
......
......

Thread 116: (state = BLOCKED)
 - java.lang.Class.getDeclaredConstructors0(boolean) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.Class.privateGetDeclaredConstructors(boolean) @bci=58, line=2671 (Compiled frame)
 - java.lang.Class.getConstructor0(java.lang.Class[], int) @bci=10, line=3075 (Compiled frame)
 - java.lang.Class.newInstance() @bci=46, line=412 (Compiled frame)
 - sun.reflect.MethodAccessorGenerator$1.run() @bci=24, line=403 (Compiled frame)
 - sun.reflect.MethodAccessorGenerator$1.run() @bci=1, line=394 (Compiled frame)
......
......

Thread 115: (state = BLOCKED)
 - java.lang.Class.getDeclaredConstructors0(boolean) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.Class.privateGetDeclaredConstructors(boolean) @bci=58, line=2671 (Compiled frame)
 - java.lang.Class.getConstructor0(java.lang.Class[], int) @bci=10, line=3075 (Compiled frame)
 - java.lang.Class.newInstance() @bci=46, line=412 (Compiled frame)
 - sun.reflect.MethodAccessorGenerator$1.run() @bci=24, line=403 (Compiled frame)
 - sun.reflect.MethodAccessorGenerator$1.run() @bci=1, line=394 (Compiled frame)
......
......

Thread 105: (state = BLOCKED)
 - java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int, java.security.ProtectionDomain, java.lang.String) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=27, line=760 (Compiled frame)
 - java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int) @bci=7, line=642 (Compiled frame)
 - sun.reflect.GeneratedMethodAccessor41.invoke(java.lang.Object, java.lang.Object[]) @bci=212 (Compiled frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)
......
......

Thread 102: (state = BLOCKED)
 - java.lang.reflect.Field.declaredAnnotations() @bci=0, line=1145 (Compiled frame)
 - java.lang.reflect.Field.declaredAnnotations() @bci=18, line=1148 (Compiled frame)
 - java.lang.reflect.Field.getDeclaredAnnotations() @bci=1, line=1139 (Compiled frame)
 - java.lang.reflect.AccessibleObject.getAnnotations() @bci=1, line=207 (Compiled frame)
......
......

Thread 95: (state = BLOCKED)
 - java.lang.reflect.Field.declaredAnnotations() @bci=0, line=1145 (Compiled frame)
 - java.lang.reflect.Field.declaredAnnotations() @bci=18, line=1148 (Compiled frame)
 - java.lang.reflect.Field.getDeclaredAnnotations() @bci=1, line=1139 (Compiled frame)
 - java.lang.reflect.AccessibleObject.getAnnotations() @bci=1, line=207 (Compiled frame)
......
......
Steps To Reproduce:
Additional Information:
Relationships
Attached Files:  stacktrace_20150513_151058_1100_2.trace [^] (173,130 bytes) 05-13-15 06:25

Notes
(0006627)
nam   
05-12-15 19:22   
Hi Mathias,

Can you post a complete stack trace? I'm wondering who's calling the java.lang classes...
(0006628)
Mathias Lagerwall   
05-13-15 06:27   
The uploaded stacktrace is not from the same occasion as the sample above. It is from a new hang we had a couple of minutes ago.
The CPU is really busy when this happens and the site becomes unresponsive.
(0006629)
nam   
05-13-15 13:03   
Those threads are in code initiated by your application. It can be a JVM bug, but it's definitely not a Resin bug. Are your classes on an NFS or network drive?

 - java.lang.Class.getDeclaredConstructors0(boolean) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.Class.privateGetDeclaredConstructors(boolean) @bci=58, line=2671 (Compiled frame)
 - java.lang.Class.getConstructor0(java.lang.Class[], int) @bci=10, line=3075 (Compiled frame)
 - java.lang.Class.newInstance() @bci=46, line=412 (Compiled frame)
 - sun.reflect.MethodAccessorGenerator$1.run() @bci=24, line=403 (Compiled frame)
 - sun.reflect.MethodAccessorGenerator$1.run() @bci=1, line=394 (Compiled frame)
 - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame)
 - sun.reflect.MethodAccessorGenerator.generate(java.lang.Class, java.lang.String, java.lang.Class[], java.lang.Class, java.lang.Class[], int, boolean, boolean, java.lang.Class) @bci=755, line=393 (Compiled frame)
 - sun.reflect.MethodAccessorGenerator.generateConstructor(java.lang.Class, java.lang.Class[], java.lang.Class[], int) @bci=14, line=92 (Compiled frame)
 - sun.reflect.NativeConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=65, line=55 (Compiled frame)
 - sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=5, line=45 (Compiled frame)
 - java.lang.reflect.Constructor.newInstance(java.lang.Object[]) @bci=79, line=422 (Compiled frame)
 - java.lang.Class.newInstance() @bci=138, line=442 (Compiled frame)
 - java.util.ResourceBundle$Control.newBundle(java.lang.String, java.util.Locale, java.lang.String, java.lang.ClassLoader, boolean) @bci=41, line=2645 (Compiled frame)
 - java.util.ResourceBundle.loadBundle(java.util.ResourceBundle$CacheKey, java.util.List, java.util.ResourceBundle$Control, boolean) @bci=54, line=1501 (Compiled frame)
 - java.util.ResourceBundle.findBundle(java.util.ResourceBundle$CacheKey, java.util.List, java.util.List, int, java.util.ResourceBundle$Control, java.util.ResourceBundle) @bci=213, line=1465 (Compiled frame)
 - java.util.ResourceBundle.findBundle(java.util.ResourceBundle$CacheKey, java.util.List, java.util.List, int, java.util.ResourceBundle$Control, java.util.ResourceBundle) @bci=37, line=1419 (Compiled frame)
 - java.util.ResourceBundle.findBundle(java.util.ResourceBundle$CacheKey, java.util.List, java.util.List, int, java.util.ResourceBundle$Control, java.util.ResourceBundle) @bci=37, line=1419 (Compiled frame)
 - java.util.ResourceBundle.getBundleImpl(java.lang.String, java.util.Locale, java.lang.ClassLoader, java.util.ResourceBundle$Control) @bci=187, line=1361 (Compiled frame)
 - java.util.ResourceBundle.getBundle(java.lang.String, java.util.Locale, java.util.ResourceBundle$Control) @bci=9, line=890 (Compiled frame)
 - sun.util.resources.LocaleData$1.run() @bci=11, line=164 (Compiled frame)
 - sun.util.resources.LocaleData$1.run() @bci=1, line=160 (Compiled frame)
 - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Compiled frame)
 - sun.util.resources.LocaleData.getBundle(java.lang.String, java.util.Locale) @bci=9, line=160 (Compiled frame)
 - sun.util.resources.LocaleData.getNumberFormatData(java.util.Locale) @bci=26, line=156 (Interpreted frame)
 - sun.util.locale.provider.LocaleResources.getDecimalFormatSymbolsData() @bci=46, line=167 (Interpreted frame)
 - java.text.DecimalFormatSymbols.initialize(java.util.Locale) @bci=28, line=616 (Compiled frame)
 - java.text.DecimalFormatSymbols.<init>(java.util.Locale) @bci=11, line=113 (Compiled frame)
 - sun.util.locale.provider.DecimalFormatSymbolsProviderImpl.getInstance(java.util.Locale) @bci=17, line=85 (Compiled frame)
 - java.text.DecimalFormatSymbols.getInstance(java.util.Locale) @bci=14, line=180 (Compiled frame)
 - java.util.Formatter.getZero(java.util.Locale) @bci=15, line=2283 (Compiled frame)
 - java.util.Formatter.<init>(java.util.Locale, java.lang.Appendable) @bci=16, line=1892 (Compiled frame)
 - java.util.Formatter.<init>() @bci=14, line=1914 (Compiled frame)
 - java.lang.String.format(java.lang.String, java.lang.Object[]) @bci=4, line=2928 (Compiled frame)
 - com.zaxxer.hikari.pool.HikariPool.getConnection() @bci=253, line=205 (Compiled frame)
 - com.zaxxer.hikari.HikariDataSource.getConnection() @bci=28, line=91 (Compiled frame)
 - se.netset.persistence3.util.ConnectionPoolAdaptorHikariCP.getConnection() @bci=6, line=47 (Compiled frame)
 - se.netset.persistence3.Database.openConnection() @bci=22, line=213 (Interpreted frame)
 - se.netset.persistence3.Transaction.open() @bci=5, line=397 (Compiled frame)
 - se.netset.m4.NodeHttp.lazyInitPersistence() @bci=21, line=310 (Compiled frame)
 - se.netset.m4.NodeHttp.transaction() @bci=8, line=353 (Compiled frame)
 - se.netset.m4.M4Wrapper.transaction() @bci=4, line=68 (Compiled frame)
 - se.netset.m4.M4Wrapper.transaction() @bci=4, line=68 (Compiled frame)
 - se.netset.persistence3.PersistenceProviderService.getTransaction() @bci=4, line=15 (Compiled frame)
 - se.netset.comp.service.generic.impl.AbstractPersistenceDAOService.getTransaction() @bci=4, line=15 (Compiled frame)
 - se.netset.comp.service.generic.impl.AbstractDAOService.get(se.netset.persistence3.OID) @bci=1, line=39 (Compiled frame)
 - se.netset.atea.comp.customer.service.CustomerService.getCurrentCustomer() @bci=40, line=165 (Compiled frame)
 - se.netset.atea.comp.filter.service.FilterPopulaterService.populateSessionWithFilters(se.netset.etailer.Customer, se.netset.etailer.User) @bci=10, line=47 (Compiled frame)
 - se.netset.atea.comp.filter.service.FilterPopulaterService.populateSessionWithFilters() @bci=3, line=37 (Compiled frame)
 - se.netset.etailer.EtailerComp$Rt.createSessionInformation() @bci=95, line=3327 (Compiled frame)
 - se.netset.etailer.EtailerComp$Rt.getSessionInformation() @bci=28, line=3297 (Compiled frame)
 - se.netset.atea.comp.etailer.services.SessionService.getSessionInformation() @bci=3, line=40 (Compiled frame)
 - se.netset.atea.comp.etailer.services.SessionService.getBool(se.netset.etailer.session.SIKEY) @bci=1, line=200 (Compiled frame)
 - se.netset.etailer.service.ViewSettingsService.isAteaDirect() @bci=6, line=77 (Compiled frame)
 - se.netset.etailer.skin0.SkinUtil.updateADFlag() @bci=36, line=76 (Compiled frame)
 - se.netset.etailer.web.P01.rootElement() @bci=16, line=22 (Compiled frame)
 - se.netset.ui.Controller1.run() @bci=48, line=72 (Compiled frame)
 - se.netset.m4.M4Wrapper.run() @bci=4, line=78 (Compiled frame)
 - se.netset.m4.NodeHttp.doRun(se.netset.m4.M4) @bci=42, line=404 (Compiled frame)
 - se.netset.m4.NodeHttp.run(se.netset.m4.M4, java.lang.Class, java.lang.String, java.lang.Object) @bci=395, line=466 (Compiled frame)
 - se.netset.m4.NodeHttp.doGet() @bci=69, line=147 (Compiled frame)
 - se.netset.m4.NodeHttp.execute() @bci=79, line=71 (Compiled frame)
 - se.netset.m4.NodeHttp.newInstance(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=11, line=57 (Compiled frame)
 - se.netset.m4.M4Comp.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=11, line=173 (Compiled frame)
 - se.netset.m4.ServletAPIAdapter.doGet(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=77, line=96 (Compiled frame)
 - javax.servlet.http.HttpServlet.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=62, line=120 (Compiled frame)
(0006630)
Mathias Lagerwall   
05-13-15 13:18   
Thanks for the feedback!
No, they are located on a normal local drive.
We will try with another version of the JVM to see if this problem goes away.
(0006638)
ferg   
06-03-15 09:09   
From looking at the stack trace, this does not appear to be a Resin issue. None of the threads are in Resin classes that hold a lock that might be causing the introspection block.

If more information becomes available showing a Resin-specific issue, please feel free to open a new bug report.