0006047: resin-data db corruption creates large log file, prevents restart

Viewing Issue Advanced Details [ Jump to Notes ]

[ View Simple ] [ Issue History ] [ Print ]

Category

Severity

Reproducibility

Date Submitted

Last Update

0006047

[Resin]

minor

always

03-23-17 11:51

03-30-17 15:24

Reporter

wileysaw

View Status

public

Assigned To

ferg

Priority

normal

Resolution

fixed

Platform

Status

closed

Projection

none

OS Version

ETA

none

Fixed in Version

4.0.52

Product Version

4.0.51

Product Build

Summary

0006047: resin-data db corruption creates large log file, prevents restart

Description

ava.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1329)
        at com.caucho.db.block.BlockStore.init(BlockStore.java:545)
        at com.caucho.db.table.Table.loadFromFile(Table.java:332)
        at com.caucho.db.Database.getTable(Database.java:217)
        at com.caucho.db.sql.Parser.parseFromItem(Parser.java:530)
        at com.caucho.db.sql.Parser.parseFromItems(Parser.java:373)
        at com.caucho.db.sql.Parser.parseSelect(Parser.java:249)
        at com.caucho.db.sql.Parser.parseSelect(Parser.java:213)
        at com.caucho.db.sql.Parser.parse(Parser.java:178)
        at com.caucho.db.sql.Parser.parse(Parser.java:151)
        at com.caucho.db.Database.parseQuery(Database.java:317)
        at com.caucho.db.jdbc.StatementImpl.executeQuery(StatementImpl.java:108)
        at com.caucho.server.distcache.DataStore.initDatabase(DataStore.java:205)
        at com.caucho.server.distcache.DataStore.init(DataStore.java:173)
        at com.caucho.server.distcache.CacheDataBackingImpl.start(CacheDataBackingImpl.java:502)
        at com.caucho.server.distcache.CacheStoreManager.start(CacheStoreManager.java:374)
        at com.caucho.server.distcache.DistCacheSystem.start(DistCacheSystem.java:162)
        at com.caucho.env.service.ResinSystem.startServices(ResinSystem.java:544)
        at com.caucho.env.service.ResinSystem.start(ResinSystem.java:511)
        at com.caucho.server.resin.Resin.start(Resin.java:875)
        at com.caucho.server.resin.Resin.initMain(Resin.java:1008)
        at com.caucho.server.resin.Resin.main(Resin.java:1467)
NODATA: 464d1 1490061183803
java.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1329)
        at com.caucho.server.distcache.LocalDataManager.readData(LocalDataManager.java:179)
        at com.caucho.server.distcache.DistCacheEntry.get(DistCacheEntry.java:933)
        at com.caucho.server.distcache.DistCacheEntry.get(DistCacheEntry.java:166)
        at com.caucho.server.distcache.CacheImpl.get(CacheImpl.java:318)
        at com.caucho.cloud.globalcache.GlobalCacheManager.notifyUpdate(GlobalCacheManager.java:301)
        at com.caucho.cloud.globalcache.GlobalCacheActor$NotifyUpdateWorker.runTask(GlobalCacheActor.java:98)
        at com.caucho.env.thread2.AbstractTaskWorker2.run(AbstractTaskWorker2.java:247)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
Suspicious change: 0 1
java.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1329)
        at com.caucho.db.block.BlockStore.setAllocation(BlockStore.java:991)
        at com.caucho.db.block.BlockStore.findFreeBlock(BlockStore.java:778)
        at com.caucho.db.block.BlockStore.allocateBlock(BlockStore.java:725)
        at com.caucho.db.block.BlockStore.allocateRow(BlockStore.java:636)
        at com.caucho.db.table.TableRowAllocator.allocateInsertRowBlock(TableRowAllocator.java:161)
        at com.caucho.db.table.Table.insert(Table.java:983)
        at com.caucho.db.sql.InsertQuery.execute(InsertQuery.java:118)
        at com.caucho.db.jdbc.PreparedStatementImpl.execute(PreparedStatementImpl.java:367)
        at com.caucho.db.jdbc.PreparedStatementImpl.executeUpdate(PreparedStatementImpl.java:337)
        at com.caucho.server.distcache.DataStore.insert(DataStore.java:486)
        at com.caucho.server.distcache.DataStore.save(DataStore.java:459)
        at com.caucho.server.distcache.CacheDataBackingImpl.saveData(CacheDataBackingImpl.java:389)
        at com.caucho.server.distcache.LocalDataManager.writeData(LocalDataManager.java:117)
        at com.caucho.server.distcache.DistCacheEntry.localUpdate(DistCacheEntry.java:1250)
        at com.caucho.server.distcache.DistCacheEntry.putLocalValue(DistCacheEntry.java:1276)
        at com.caucho.server.distcache.DistCacheEntry.putIfNew(DistCacheEntry.java:557)
        at com.caucho.server.distcache.CacheImpl.putIfNew(CacheImpl.java:545)
        at com.caucho.cloud.globalcache.GlobalCacheManager.put(GlobalCacheManager.java:249)
        at com.caucho.cloud.globalcache.GlobalCacheActor.pushData(GlobalCacheActor.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.caucho.bam.proxy.ProxySkeleton$QueryMethodInvoker.invoke(ProxySkeleton.java:488)
        at com.caucho.bam.proxy.ProxySkeleton.query(ProxySkeleton.java:248)
        at com.caucho.bam.proxy.ProxyActor.query(ProxyActor.java:117)
        at com.caucho.bam.query.QueryActorFilter.query(QueryActorFilter.java:95)
        at com.caucho.bam.packet.Query.dispatch(Query.java:86)
        at com.caucho.bam.mailbox.MultiworkerMailbox$PacketProcessor.process(MultiworkerMailbox.java:423)
        at com.caucho.bam.mailbox.MultiworkerMailbox$PacketProcessor.process(MultiworkerMailbox.java:409)
        at com.caucho.env.actor.ValueActorQueue$ValueItemProcessor.process(ValueActorQueue.java:183)
        at com.caucho.env.actor.ValueActorQueue$ValueItemProcessor.process(ValueActorQueue.java:161)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.doConsume(ActorQueue.java:454)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.consumeAll(ActorQueue.java:407)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.access$300(ActorQueue.java:341)
        at com.caucho.env.actor.ActorQueue$ActorWorker.runTask(ActorQueue.java:556)
        at com.caucho.env.thread2.AbstractTaskWorker2.run(AbstractTaskWorker2.java:247)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
Suspicious change: 0 1
java.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1329)
        at com.caucho.db.block.BlockStore.setAllocation(BlockStore.java:991)
        at com.caucho.db.block.BlockStore.findFreeBlock(BlockStore.java:778)
        at com.caucho.db.block.BlockStore.allocateBlock(BlockStore.java:725)
        at com.caucho.db.block.BlockStore.allocateRow(BlockStore.java:636)
        at com.caucho.db.table.TableRowAllocator.allocateInsertRowBlock(TableRowAllocator.java:161)
        at com.caucho.db.table.Table.insert(Table.java:983)
        at com.caucho.db.sql.InsertQuery.execute(InsertQuery.java:118)
        at com.caucho.db.jdbc.PreparedStatementImpl.execute(PreparedStatementImpl.java:367)
        at com.caucho.db.jdbc.PreparedStatementImpl.executeUpdate(PreparedStatementImpl.java:337)
        at com.caucho.server.distcache.DataStore.insert(DataStore.java:486)
        at com.caucho.server.distcache.DataStore.save(DataStore.java:459)
        at com.caucho.server.distcache.CacheDataBackingImpl.saveData(CacheDataBackingImpl.java:389)
        at com.caucho.server.distcache.LocalDataManager.writeData(LocalDataManager.java:117)
        at com.caucho.server.distcache.DistCacheEntry.localUpdate(DistCacheEntry.java:1250)
        at com.caucho.server.distcache.DistCacheEntry.putLocalValue(DistCacheEntry.java:1276)
        at com.caucho.server.distcache.DistCacheEntry.putIfNew(DistCacheEntry.java:557)
        at com.caucho.server.distcache.CacheImpl.putIfNew(CacheImpl.java:545)
        at com.caucho.cloud.globalcache.GlobalCacheManager.put(GlobalCacheManager.java:249)
        at com.caucho.cloud.globalcache.GlobalCacheActor.pushData(GlobalCacheActor.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.caucho.bam.proxy.ProxySkeleton$QueryMethodInvoker.invoke(ProxySkeleton.java:488)
        at com.caucho.bam.proxy.ProxySkeleton.query(ProxySkeleton.java:248)
        at com.caucho.bam.proxy.ProxyActor.query(ProxyActor.java:117)
        at com.caucho.bam.query.QueryActorFilter.query(QueryActorFilter.java:95)
        at com.caucho.bam.packet.Query.dispatch(Query.java:86)
        at com.caucho.bam.mailbox.MultiworkerMailbox$PacketProcessor.process(MultiworkerMailbox.java:423)
        at com.caucho.bam.mailbox.MultiworkerMailbox$PacketProcessor.process(MultiworkerMailbox.java:409)
        at com.caucho.env.actor.ValueActorQueue$ValueItemProcessor.process(ValueActorQueue.java:183)
        at com.caucho.env.actor.ValueActorQueue$ValueItemProcessor.process(ValueActorQueue.java:161)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.doConsume(ActorQueue.java:454)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.consumeAll(ActorQueue.java:407)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.access$300(ActorQueue.java:341)
        at com.caucho.env.actor.ActorQueue$ActorWorker.runTask(ActorQueue.java:556)
        at com.caucho.env.thread2.AbstractTaskWorker2.run(AbstractTaskWorker2.java:247)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
Suspicious change: 0 1
java.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1329)
        at com.caucho.db.block.BlockStore.setAllocation(BlockStore.java:991)
        at com.caucho.db.block.BlockStore.findFreeBlock(BlockStore.java:778)
        at com.caucho.db.block.BlockStore.allocateBlock(BlockStore.java:725)
        at com.caucho.db.block.BlockStore.allocateRow(BlockStore.java:636)
        at com.caucho.db.table.TableRowAllocator.allocateInsertRowBlock(TableRowAllocator.java:161)
        at com.caucho.db.table.Table.insert(Table.java:983)
        at com.caucho.db.sql.InsertQuery.execute(InsertQuery.java:118)
        at com.caucho.db.jdbc.PreparedStatementImpl.execute(PreparedStatementImpl.java:367)
        at com.caucho.db.jdbc.PreparedStatementImpl.executeUpdate(PreparedStatementImpl.java:337)
        at com.caucho.server.distcache.DataStore.insert(DataStore.java:486)
        at com.caucho.server.distcache.DataStore.save(DataStore.java:459)
        at com.caucho.server.distcache.CacheDataBackingImpl.saveData(CacheDataBackingImpl.java:389)
        at com.caucho.server.distcache.LocalDataManager.writeData(LocalDataManager.java:117)
        at com.caucho.server.distcache.DistCacheEntry.localUpdate(DistCacheEntry.java:1250)
        at com.caucho.server.distcache.DistCacheEntry.putLocalValue(DistCacheEntry.java:1276)
        at com.caucho.server.distcache.DistCacheEntry.putIfNew(DistCacheEntry.java:557)
        at com.caucho.server.distcache.CacheImpl.putIfNew(CacheImpl.java:545)
        at com.caucho.cloud.globalcache.GlobalCacheManager.put(GlobalCacheManager.java:249)
        at com.caucho.cloud.globalcache.GlobalCacheActor.pushData(GlobalCacheActor.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.caucho.bam.proxy.ProxySkeleton$QueryMethodInvoker.invoke(ProxySkeleton.java:488)
        at com.caucho.bam.proxy.ProxySkeleton.query(ProxySkeleton.java:248)
        at com.caucho.bam.proxy.ProxyActor.query(ProxyActor.java:117)
        at com.caucho.bam.query.QueryActorFilter.query(QueryActorFilter.java:95)
        at com.caucho.bam.packet.Query.dispatch(Query.java:86)
        at com.caucho.bam.mailbox.MultiworkerMailbox$PacketProcessor.process(MultiworkerMailbox.java:423)
        at com.caucho.bam.mailbox.MultiworkerMailbox$PacketProcessor.process(MultiworkerMailbox.java:409)
        at com.caucho.env.actor.ValueActorQueue$ValueItemProcessor.process(ValueActorQueue.java:183)
        at com.caucho.env.actor.ValueActorQueue$ValueItemProcessor.process(ValueActorQueue.java:161)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.doConsume(ActorQueue.java:454)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.consumeAll(ActorQueue.java:407)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.access$300(ActorQueue.java:341)
        at com.caucho.env.actor.ActorQueue$ActorWorker.runTask(ActorQueue.java:556)
        at com.caucho.env.thread2.AbstractTaskWorker2.run(AbstractTaskWorker2.java:247)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
Suspicious change: 0 1

Steps To Reproduce

Additional Information

[ &8203;log-handler log]
&8203;
[10:54:14.366] [main] CONFIG com.caucho.loader.ClassEntry - In-place class redefinition (HotSwap) is available.
[10:54:14.658] [main] INFO com.caucho.env.service.ResinSystem -
[10:54:14.658] [main] INFO com.caucho.env.service.ResinSystem - Resin Professional 4.0.51 (built Tue, 28 Feb 2017 02:27:46 PST)
[10:54:14.658] [main] INFO com.caucho.env.service.ResinSystem -
[10:54:14.658] [main] INFO com.caucho.env.service.ResinSystem - Linux 2.6.32-642.15.1.el6.x86_64 amd64
[10:54:14.658] [main] INFO com.caucho.env.service.ResinSystem - Java(TM) SE Runtime Environment 1.8.0_65-b17, UTF-8, ko
[10:54:14.659] [main] INFO com.caucho.env.service.ResinSystem - Java HotSpot(TM) 64-Bit Server VM 25.65-b01, 64, mixed mode, Oracle Corporation
[10:54:14.659] [main] INFO com.caucho.env.service.ResinSystem -
[10:54:14.659] [main] INFO com.caucho.env.service.ResinSystem - user.name =
[10:54:14.805] [main] INFO com.caucho.db.table.Table - Table[stat_name:2,/resin-pro-4.0.51/resin-data/b/stat_name.db] validating indexes due to unclean shutdown.
[10:54:14.816] [main] INFO com.caucho.db.table.Table - Table[stat_data:3,/resin-pro-4.0.51/resin-data/b/stat_data.db] validating indexes due to unclean shutdown.
[10:54:17.125] [main] INFO com.caucho.cloud.network.NetworkClusterSystem -
[10:54:17.126] [main] INFO com.caucho.network.listen.TcpPort - server listening to localhost:26812
[10:54:17.131] [main] INFO com.caucho.cloud.network.NetworkClusterSystem -
[10:54:17.188] [main] INFO com.caucho.db.table.Table - Table[mnode:4,/resin-pro-4.0.51/resin-data/b/distcache/mnode.db] validating indexes due to unclean shutdown.
[10:54:17.190] [main] WARNING com.caucho.db.table.Table - java.lang.IllegalStateException: BinaryColumn[id,index]: invalid index (key=BinaryKeyCompare[76243cf940a3c5f14e5fc4487a18a5dabd57f5e0b4eca28f40b6b66ed7deac98], index value=0, row addr=9249, Table[mnode:4,/resin-pro-4.0.51/resin-data/b/distcache/mnode.db])
                at com.caucho.db.table.Column.validateIndex(Column.java:627)
                at com.caucho.db.table.Table.validateIndexByRow(Table.java:792)
                at com.caucho.db.table.Table.validateIndexes(Table.java:756)
                at com.caucho.db.table.Table.validateIndexesSafe(Table.java:720)
                at com.caucho.db.table.Table.loadFromFile(Table.java:340)
                at com.caucho.db.Database.getTable(Database.java:217)
                at com.caucho.db.sql.Parser.parseFromItem(Parser.java:530)
                at com.caucho.db.sql.Parser.parseFromItems(Parser.java:373)
                at com.caucho.db.sql.Parser.parseSelect(Parser.java:249)
                at com.caucho.db.sql.Parser.parseSelect(Parser.java:213)
                at com.caucho.db.sql.Parser.parse(Parser.java:178)
                at com.caucho.db.sql.Parser.parse(Parser.java:151)
                at com.caucho.db.Database.parseQuery(Database.java:317)
                at com.caucho.db.jdbc.StatementImpl.executeQuery(StatementImpl.java:108)
                at com.caucho.server.distcache.MnodeStore.initDatabase(MnodeStore.java:296)
                at com.caucho.server.distcache.MnodeStore.init(MnodeStore.java:266)
                at com.caucho.server.distcache.CacheDataBackingImpl.start(CacheDataBackingImpl.java:499)
                at com.caucho.server.distcache.CacheStoreManager.start(CacheStoreManager.java:374)
                at com.caucho.server.distcache.DistCacheSystem.start(DistCacheSystem.java:162)
                at com.caucho.env.service.ResinSystem.startServices(ResinSystem.java:544)
                at com.caucho.env.service.ResinSystem.start(ResinSystem.java:511)
                at com.caucho.server.resin.Resin.start(Resin.java:875)
                at com.caucho.server.resin.Resin.initMain(Resin.java:1008)
                at com.caucho.server.resin.Resin.main(Resin.java:1467)

[10:54:17.190] [main] WARNING com.caucho.db.table.Table - resin-pro-4.0.51/resin-data/b/distcache/mnode.db]' because they did not properly validate on startup, total blocks=1024
[10:54:17.193] [main] WARNING com.caucho.db.table.Table - rebuilding indexes for 'Table[mnode:4,/resin-pro-4.0.51/resin-data/b/distcache/mnode.db]' finished.
[10:54:17.197] [main] WARNING com.caucho.db.block.BlockStore - removing /resin-pro-4.0.51/resin-data/b/distcache/data.db due to restart corruption
[10:54:17.199] [main] INFO com.caucho.db.table.Table - Table[data:5,/resin-pro-4.0.51/resin-data/b/distcache/data.db] validating indexes due to unclean shutdown.
[10:54:17.322] [NotifyUpdateWorker[]-10] WARNING com.caucho.server.distcache.LocalDataManager - LocalDataManager[b] cannot load data for key=HashKey[68ffbdd0] from triad (id=464d1 time=1490061183803)
[10:54:17.323] [NotifyUpdateWorker[]-10] WARNING com.caucho.server.distcache.DistCacheEntry - Missing or corrupted data in get for MnodeEntry[value=8259a92398c61bad,flags=0x6,version=1490061183802,lease=-1] DistCacheEntry[key=resin:scaling:app:main,keyHash=68ffbdd0,owner=A_B]
[10:54:17.357] [main] WARNING com.caucho.server.distcache.LocalDataManager - LocalDataManager[b] cannot load data for key=HashKey[023286aa] from triad (id=16c224 time=1489982456278)
[10:54:17.361] [main] WARNING com.caucho.server.distcache.DistCacheEntry - Missing or corrupted data in get for MnodeEntry[value=cccff7b5701b9f41,flags=0x6,version=1489982456278,lease=-1] DistCacheEntry[key=resin/clustering/repository/tagHash/app,keyHash=023286aa,owner=C_A]
[10:54:18.320] [resin-75] WARNING com.caucho.server.distcache.LocalDataManager - LocalDataManager[b] cannot load data for key=HashKey[017226d5] from triad (id=46dea time=1490061184350)
[10:54:18.323] [resin-75] WARNING com.caucho.server.distcache.DistCacheEntry - Missing or corrupted data in get for MnodeEntry[value=2f80c9d6a781aa68,flags=0x6,version=1490061184321,lease=-1] DistCacheEntry[key=resin:scaling:web:main,keyHash=017226d5,owner=A_C]
[11:28:48.696] [resin-port-127.0.0.1:-29] WARNING com.caucho.cloud.network.ClusterServer - ClusterServer[id=front,127.0.0.1:] notify-heartbeat-stop
[11:28:53.343] [resin-port-127.0.0.1:-31] WARNING com.caucho.cloud.network.ClusterServer - ClusterServer[id=a,127.0.0.1:] notify-heartbeat-stop
[11:29:17.249] [resin-26] INFO com.caucho.server.distcache.CacheDataBackingImpl - ReaperListener removed 12 expired items (removed=12,old=0)

Attached Files

Relationships

There are no notes attached to this issue.

Issue History
Date Modified	Username	Field	Change
03-23-17 11:51	wileysaw	New Issue
03-30-17 15:24	ferg	Assigned To	=> ferg
03-30-17 15:24	ferg	Status	new => closed
03-30-17 15:24	ferg	Resolution	open => fixed
03-30-17 15:24	ferg	Fixed in Version	=> 4.0.52

Mantis 1.0.0rc3[^]

28 total queries executed.
26 unique queries executed.