0006047: resin-data db corruption creates large log file, prevents restart

Viewing Issue Simple Details [ Jump to Notes ]

[ View Advanced ] [ Issue History ] [ Print ]

Category

Severity

Reproducibility

Date Submitted

Last Update

0006047

[Resin]

minor

always

03-23-17 11:51

03-30-17 15:24

Reporter

wileysaw

View Status

public

Assigned To

ferg

Priority

normal

Resolution

fixed

Status

closed

Product Version

4.0.51

Summary

0006047: resin-data db corruption creates large log file, prevents restart

Description

ava.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1329)
        at com.caucho.db.block.BlockStore.init(BlockStore.java:545)
        at com.caucho.db.table.Table.loadFromFile(Table.java:332)
        at com.caucho.db.Database.getTable(Database.java:217)
        at com.caucho.db.sql.Parser.parseFromItem(Parser.java:530)
        at com.caucho.db.sql.Parser.parseFromItems(Parser.java:373)
        at com.caucho.db.sql.Parser.parseSelect(Parser.java:249)
        at com.caucho.db.sql.Parser.parseSelect(Parser.java:213)
        at com.caucho.db.sql.Parser.parse(Parser.java:178)
        at com.caucho.db.sql.Parser.parse(Parser.java:151)
        at com.caucho.db.Database.parseQuery(Database.java:317)
        at com.caucho.db.jdbc.StatementImpl.executeQuery(StatementImpl.java:108)
        at com.caucho.server.distcache.DataStore.initDatabase(DataStore.java:205)
        at com.caucho.server.distcache.DataStore.init(DataStore.java:173)
        at com.caucho.server.distcache.CacheDataBackingImpl.start(CacheDataBackingImpl.java:502)
        at com.caucho.server.distcache.CacheStoreManager.start(CacheStoreManager.java:374)
        at com.caucho.server.distcache.DistCacheSystem.start(DistCacheSystem.java:162)
        at com.caucho.env.service.ResinSystem.startServices(ResinSystem.java:544)
        at com.caucho.env.service.ResinSystem.start(ResinSystem.java:511)
        at com.caucho.server.resin.Resin.start(Resin.java:875)
        at com.caucho.server.resin.Resin.initMain(Resin.java:1008)
        at com.caucho.server.resin.Resin.main(Resin.java:1467)
NODATA: 464d1 1490061183803
java.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1329)
        at com.caucho.server.distcache.LocalDataManager.readData(LocalDataManager.java:179)
        at com.caucho.server.distcache.DistCacheEntry.get(DistCacheEntry.java:933)
        at com.caucho.server.distcache.DistCacheEntry.get(DistCacheEntry.java:166)
        at com.caucho.server.distcache.CacheImpl.get(CacheImpl.java:318)
        at com.caucho.cloud.globalcache.GlobalCacheManager.notifyUpdate(GlobalCacheManager.java:301)
        at com.caucho.cloud.globalcache.GlobalCacheActor$NotifyUpdateWorker.runTask(GlobalCacheActor.java:98)
        at com.caucho.env.thread2.AbstractTaskWorker2.run(AbstractTaskWorker2.java:247)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
Suspicious change: 0 1
java.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1329)
        at com.caucho.db.block.BlockStore.setAllocation(BlockStore.java:991)
        at com.caucho.db.block.BlockStore.findFreeBlock(BlockStore.java:778)
        at com.caucho.db.block.BlockStore.allocateBlock(BlockStore.java:725)
        at com.caucho.db.block.BlockStore.allocateRow(BlockStore.java:636)
        at com.caucho.db.table.TableRowAllocator.allocateInsertRowBlock(TableRowAllocator.java:161)
        at com.caucho.db.table.Table.insert(Table.java:983)
        at com.caucho.db.sql.InsertQuery.execute(InsertQuery.java:118)
        at com.caucho.db.jdbc.PreparedStatementImpl.execute(PreparedStatementImpl.java:367)
        at com.caucho.db.jdbc.PreparedStatementImpl.executeUpdate(PreparedStatementImpl.java:337)
        at com.caucho.server.distcache.DataStore.insert(DataStore.java:486)
        at com.caucho.server.distcache.DataStore.save(DataStore.java:459)
        at com.caucho.server.distcache.CacheDataBackingImpl.saveData(CacheDataBackingImpl.java:389)
        at com.caucho.server.distcache.LocalDataManager.writeData(LocalDataManager.java:117)
        at com.caucho.server.distcache.DistCacheEntry.localUpdate(DistCacheEntry.java:1250)
        at com.caucho.server.distcache.DistCacheEntry.putLocalValue(DistCacheEntry.java:1276)
        at com.caucho.server.distcache.DistCacheEntry.putIfNew(DistCacheEntry.java:557)
        at com.caucho.server.distcache.CacheImpl.putIfNew(CacheImpl.java:545)
        at com.caucho.cloud.globalcache.GlobalCacheManager.put(GlobalCacheManager.java:249)
        at com.caucho.cloud.globalcache.GlobalCacheActor.pushData(GlobalCacheActor.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.caucho.bam.proxy.ProxySkeleton$QueryMethodInvoker.invoke(ProxySkeleton.java:488)
        at com.caucho.bam.proxy.ProxySkeleton.query(ProxySkeleton.java:248)
        at com.caucho.bam.proxy.ProxyActor.query(ProxyActor.java:117)
        at com.caucho.bam.query.QueryActorFilter.query(QueryActorFilter.java:95)
        at com.caucho.bam.packet.Query.dispatch(Query.java:86)
        at com.caucho.bam.mailbox.MultiworkerMailbox$PacketProcessor.process(MultiworkerMailbox.java:423)
        at com.caucho.bam.mailbox.MultiworkerMailbox$PacketProcessor.process(MultiworkerMailbox.java:409)
        at com.caucho.env.actor.ValueActorQueue$ValueItemProcessor.process(ValueActorQueue.java:183)
        at com.caucho.env.actor.ValueActorQueue$ValueItemProcessor.process(ValueActorQueue.java:161)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.doConsume(ActorQueue.java:454)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.consumeAll(ActorQueue.java:407)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.access$300(ActorQueue.java:341)
        at com.caucho.env.actor.ActorQueue$ActorWorker.runTask(ActorQueue.java:556)
        at com.caucho.env.thread2.AbstractTaskWorker2.run(AbstractTaskWorker2.java:247)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
Suspicious change: 0 1
java.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1329)
        at com.caucho.db.block.BlockStore.setAllocation(BlockStore.java:991)
        at com.caucho.db.block.BlockStore.findFreeBlock(BlockStore.java:778)
        at com.caucho.db.block.BlockStore.allocateBlock(BlockStore.java:725)
        at com.caucho.db.block.BlockStore.allocateRow(BlockStore.java:636)
        at com.caucho.db.table.TableRowAllocator.allocateInsertRowBlock(TableRowAllocator.java:161)
        at com.caucho.db.table.Table.insert(Table.java:983)
        at com.caucho.db.sql.InsertQuery.execute(InsertQuery.java:118)
        at com.caucho.db.jdbc.PreparedStatementImpl.execute(PreparedStatementImpl.java:367)
        at com.caucho.db.jdbc.PreparedStatementImpl.executeUpdate(PreparedStatementImpl.java:337)
        at com.caucho.server.distcache.DataStore.insert(DataStore.java:486)
        at com.caucho.server.distcache.DataStore.save(DataStore.java:459)
        at com.caucho.server.distcache.CacheDataBackingImpl.saveData(CacheDataBackingImpl.java:389)
        at com.caucho.server.distcache.LocalDataManager.writeData(LocalDataManager.java:117)
        at com.caucho.server.distcache.DistCacheEntry.localUpdate(DistCacheEntry.java:1250)
        at com.caucho.server.distcache.DistCacheEntry.putLocalValue(DistCacheEntry.java:1276)
        at com.caucho.server.distcache.DistCacheEntry.putIfNew(DistCacheEntry.java:557)
        at com.caucho.server.distcache.CacheImpl.putIfNew(CacheImpl.java:545)
        at com.caucho.cloud.globalcache.GlobalCacheManager.put(GlobalCacheManager.java:249)
        at com.caucho.cloud.globalcache.GlobalCacheActor.pushData(GlobalCacheActor.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.caucho.bam.proxy.ProxySkeleton$QueryMethodInvoker.invoke(ProxySkeleton.java:488)
        at com.caucho.bam.proxy.ProxySkeleton.query(ProxySkeleton.java:248)
        at com.caucho.bam.proxy.ProxyActor.query(ProxyActor.java:117)
        at com.caucho.bam.query.QueryActorFilter.query(QueryActorFilter.java:95)
        at com.caucho.bam.packet.Query.dispatch(Query.java:86)
        at com.caucho.bam.mailbox.MultiworkerMailbox$PacketProcessor.process(MultiworkerMailbox.java:423)
        at com.caucho.bam.mailbox.MultiworkerMailbox$PacketProcessor.process(MultiworkerMailbox.java:409)
        at com.caucho.env.actor.ValueActorQueue$ValueItemProcessor.process(ValueActorQueue.java:183)
        at com.caucho.env.actor.ValueActorQueue$ValueItemProcessor.process(ValueActorQueue.java:161)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.doConsume(ActorQueue.java:454)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.consumeAll(ActorQueue.java:407)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.access$300(ActorQueue.java:341)
        at com.caucho.env.actor.ActorQueue$ActorWorker.runTask(ActorQueue.java:556)
        at com.caucho.env.thread2.AbstractTaskWorker2.run(AbstractTaskWorker2.java:247)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
Suspicious change: 0 1
java.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1329)
        at com.caucho.db.block.BlockStore.setAllocation(BlockStore.java:991)
        at com.caucho.db.block.BlockStore.findFreeBlock(BlockStore.java:778)
        at com.caucho.db.block.BlockStore.allocateBlock(BlockStore.java:725)
        at com.caucho.db.block.BlockStore.allocateRow(BlockStore.java:636)
        at com.caucho.db.table.TableRowAllocator.allocateInsertRowBlock(TableRowAllocator.java:161)
        at com.caucho.db.table.Table.insert(Table.java:983)
        at com.caucho.db.sql.InsertQuery.execute(InsertQuery.java:118)
        at com.caucho.db.jdbc.PreparedStatementImpl.execute(PreparedStatementImpl.java:367)
        at com.caucho.db.jdbc.PreparedStatementImpl.executeUpdate(PreparedStatementImpl.java:337)
        at com.caucho.server.distcache.DataStore.insert(DataStore.java:486)
        at com.caucho.server.distcache.DataStore.save(DataStore.java:459)
        at com.caucho.server.distcache.CacheDataBackingImpl.saveData(CacheDataBackingImpl.java:389)
        at com.caucho.server.distcache.LocalDataManager.writeData(LocalDataManager.java:117)
        at com.caucho.server.distcache.DistCacheEntry.localUpdate(DistCacheEntry.java:1250)
        at com.caucho.server.distcache.DistCacheEntry.putLocalValue(DistCacheEntry.java:1276)
        at com.caucho.server.distcache.DistCacheEntry.putIfNew(DistCacheEntry.java:557)
        at com.caucho.server.distcache.CacheImpl.putIfNew(CacheImpl.java:545)
        at com.caucho.cloud.globalcache.GlobalCacheManager.put(GlobalCacheManager.java:249)
        at com.caucho.cloud.globalcache.GlobalCacheActor.pushData(GlobalCacheActor.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.caucho.bam.proxy.ProxySkeleton$QueryMethodInvoker.invoke(ProxySkeleton.java:488)
        at com.caucho.bam.proxy.ProxySkeleton.query(ProxySkeleton.java:248)
        at com.caucho.bam.proxy.ProxyActor.query(ProxyActor.java:117)
        at com.caucho.bam.query.QueryActorFilter.query(QueryActorFilter.java:95)
        at com.caucho.bam.packet.Query.dispatch(Query.java:86)
        at com.caucho.bam.mailbox.MultiworkerMailbox$PacketProcessor.process(MultiworkerMailbox.java:423)
        at com.caucho.bam.mailbox.MultiworkerMailbox$PacketProcessor.process(MultiworkerMailbox.java:409)
        at com.caucho.env.actor.ValueActorQueue$ValueItemProcessor.process(ValueActorQueue.java:183)
        at com.caucho.env.actor.ValueActorQueue$ValueItemProcessor.process(ValueActorQueue.java:161)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.doConsume(ActorQueue.java:454)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.consumeAll(ActorQueue.java:407)
        at com.caucho.env.actor.ActorQueue$ActorConsumer.access$300(ActorQueue.java:341)
        at com.caucho.env.actor.ActorQueue$ActorWorker.runTask(ActorQueue.java:556)
        at com.caucho.env.thread2.AbstractTaskWorker2.run(AbstractTaskWorker2.java:247)
        at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
        at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
Suspicious change: 0 1

Additional Information

[ &8203;log-handler log]
&8203;
[10:54:14.366] [main] CONFIG com.caucho.loader.ClassEntry - In-place class redefinition (HotSwap) is available.
[10:54:14.658] [main] INFO com.caucho.env.service.ResinSystem -
[10:54:14.658] [main] INFO com.caucho.env.service.ResinSystem - Resin Professional 4.0.51 (built Tue, 28 Feb 2017 02:27:46 PST)
[10:54:14.658] [main] INFO com.caucho.env.service.ResinSystem -
[10:54:14.658] [main] INFO com.caucho.env.service.ResinSystem - Linux 2.6.32-642.15.1.el6.x86_64 amd64
[10:54:14.658] [main] INFO com.caucho.env.service.ResinSystem - Java(TM) SE Runtime Environment 1.8.0_65-b17, UTF-8, ko
[10:54:14.659] [main] INFO com.caucho.env.service.ResinSystem - Java HotSpot(TM) 64-Bit Server VM 25.65-b01, 64, mixed mode, Oracle Corporation
[10:54:14.659] [main] INFO com.caucho.env.service.ResinSystem -
[10:54:14.659] [main] INFO com.caucho.env.service.ResinSystem - user.name =
[10:54:14.805] [main] INFO com.caucho.db.table.Table - Table[stat_name:2,/resin-pro-4.0.51/resin-data/b/stat_name.db] validating indexes due to unclean shutdown.
[10:54:14.816] [main] INFO com.caucho.db.table.Table - Table[stat_data:3,/resin-pro-4.0.51/resin-data/b/stat_data.db] validating indexes due to unclean shutdown.
[10:54:17.125] [main] INFO com.caucho.cloud.network.NetworkClusterSystem -
[10:54:17.126] [main] INFO com.caucho.network.listen.TcpPort - server listening to localhost:26812
[10:54:17.131] [main] INFO com.caucho.cloud.network.NetworkClusterSystem -
[10:54:17.188] [main] INFO com.caucho.db.table.Table - Table[mnode:4,/resin-pro-4.0.51/resin-data/b/distcache/mnode.db] validating indexes due to unclean shutdown.
[10:54:17.190] [main] WARNING com.caucho.db.table.Table - java.lang.IllegalStateException: BinaryColumn[id,index]: invalid index (key=BinaryKeyCompare[76243cf940a3c5f14e5fc4487a18a5dabd57f5e0b4eca28f40b6b66ed7deac98], index value=0, row addr=9249, Table[mnode:4,/resin-pro-4.0.51/resin-data/b/distcache/mnode.db])
                at com.caucho.db.table.Column.validateIndex(Column.java:627)
                at com.caucho.db.table.Table.validateIndexByRow(Table.java:792)
                at com.caucho.db.table.Table.validateIndexes(Table.java:756)
                at com.caucho.db.table.Table.validateIndexesSafe(Table.java:720)
                at com.caucho.db.table.Table.loadFromFile(Table.java:340)
                at com.caucho.db.Database.getTable(Database.java:217)
                at com.caucho.db.sql.Parser.parseFromItem(Parser.java:530)
                at com.caucho.db.sql.Parser.parseFromItems(Parser.java:373)
                at com.caucho.db.sql.Parser.parseSelect(Parser.java:249)
                at com.caucho.db.sql.Parser.parseSelect(Parser.java:213)
                at com.caucho.db.sql.Parser.parse(Parser.java:178)
                at com.caucho.db.sql.Parser.parse(Parser.java:151)
                at com.caucho.db.Database.parseQuery(Database.java:317)
                at com.caucho.db.jdbc.StatementImpl.executeQuery(StatementImpl.java:108)
                at com.caucho.server.distcache.MnodeStore.initDatabase(MnodeStore.java:296)
                at com.caucho.server.distcache.MnodeStore.init(MnodeStore.java:266)
                at com.caucho.server.distcache.CacheDataBackingImpl.start(CacheDataBackingImpl.java:499)
                at com.caucho.server.distcache.CacheStoreManager.start(CacheStoreManager.java:374)
                at com.caucho.server.distcache.DistCacheSystem.start(DistCacheSystem.java:162)
                at com.caucho.env.service.ResinSystem.startServices(ResinSystem.java:544)
                at com.caucho.env.service.ResinSystem.start(ResinSystem.java:511)
                at com.caucho.server.resin.Resin.start(Resin.java:875)
                at com.caucho.server.resin.Resin.initMain(Resin.java:1008)
                at com.caucho.server.resin.Resin.main(Resin.java:1467)

[10:54:17.190] [main] WARNING com.caucho.db.table.Table - resin-pro-4.0.51/resin-data/b/distcache/mnode.db]' because they did not properly validate on startup, total blocks=1024
[10:54:17.193] [main] WARNING com.caucho.db.table.Table - rebuilding indexes for 'Table[mnode:4,/resin-pro-4.0.51/resin-data/b/distcache/mnode.db]' finished.
[10:54:17.197] [main] WARNING com.caucho.db.block.BlockStore - removing /resin-pro-4.0.51/resin-data/b/distcache/data.db due to restart corruption
[10:54:17.199] [main] INFO com.caucho.db.table.Table - Table[data:5,/resin-pro-4.0.51/resin-data/b/distcache/data.db] validating indexes due to unclean shutdown.
[10:54:17.322] [NotifyUpdateWorker[]-10] WARNING com.caucho.server.distcache.LocalDataManager - LocalDataManager[b] cannot load data for key=HashKey[68ffbdd0] from triad (id=464d1 time=1490061183803)
[10:54:17.323] [NotifyUpdateWorker[]-10] WARNING com.caucho.server.distcache.DistCacheEntry - Missing or corrupted data in get for MnodeEntry[value=8259a92398c61bad,flags=0x6,version=1490061183802,lease=-1] DistCacheEntry[key=resin:scaling:app:main,keyHash=68ffbdd0,owner=A_B]
[10:54:17.357] [main] WARNING com.caucho.server.distcache.LocalDataManager - LocalDataManager[b] cannot load data for key=HashKey[023286aa] from triad (id=16c224 time=1489982456278)
[10:54:17.361] [main] WARNING com.caucho.server.distcache.DistCacheEntry - Missing or corrupted data in get for MnodeEntry[value=cccff7b5701b9f41,flags=0x6,version=1489982456278,lease=-1] DistCacheEntry[key=resin/clustering/repository/tagHash/app,keyHash=023286aa,owner=C_A]
[10:54:18.320] [resin-75] WARNING com.caucho.server.distcache.LocalDataManager - LocalDataManager[b] cannot load data for key=HashKey[017226d5] from triad (id=46dea time=1490061184350)
[10:54:18.323] [resin-75] WARNING com.caucho.server.distcache.DistCacheEntry - Missing or corrupted data in get for MnodeEntry[value=2f80c9d6a781aa68,flags=0x6,version=1490061184321,lease=-1] DistCacheEntry[key=resin:scaling:web:main,keyHash=017226d5,owner=A_C]
[11:28:48.696] [resin-port-127.0.0.1:-29] WARNING com.caucho.cloud.network.ClusterServer - ClusterServer[id=front,127.0.0.1:] notify-heartbeat-stop
[11:28:53.343] [resin-port-127.0.0.1:-31] WARNING com.caucho.cloud.network.ClusterServer - ClusterServer[id=a,127.0.0.1:] notify-heartbeat-stop
[11:29:17.249] [resin-26] INFO com.caucho.server.distcache.CacheDataBackingImpl - ReaperListener removed 12 expired items (removed=12,old=0)

Attached Files

Relationships

There are no notes attached to this issue.

Issue History
Date Modified	Username	Field	Change
03-23-17 11:51	wileysaw	New Issue
03-30-17 15:24	ferg	Assigned To	=> ferg
03-30-17 15:24	ferg	Status	new => closed
03-30-17 15:24	ferg	Resolution	open => fixed
03-30-17 15:24	ferg	Fixed in Version	=> 4.0.52

Mantis 1.0.0rc3[^]

28 total queries executed.
26 unique queries executed.