0005457: ArrayIndexOutOfBoundsException in BlockStore after Database corruption

Viewing Issue Simple Details [ Jump to Notes ]

[ View Advanced ] [ Issue History ] [ Print ]

Category

Severity

Reproducibility

Date Submitted

Last Update

0005457

[Resin]

minor

always

06-14-13 06:00

07-23-13 11:31

Reporter

cowan

View Status

public

Assigned To

ferg

Priority

normal

Resolution

fixed

Status

closed

Product Version

4.0.36

Summary

0005457: ArrayIndexOutOfBoundsException in BlockStore after Database corruption

Description

[13-06-09 06:12:54.824] {resin-148} java.lang.ArrayIndexOutOfBoundsException: 102708
                           at com.caucho.db.block.BlockStore.getAllocation(BlockStore.java:927)
                           at com.caucho.db.block.BlockStore.getAllocationByAddress(BlockStore.java:917)
                           at com.caucho.db.block.BlockStore.isRowBlock(BlockStore.java:620)
                           at com.caucho.db.table.TableIterator.isValidRow(TableIterator.java:294)
                           at com.caucho.db.sql.IdentityIndexExpr.initRow(IdentityIndexExpr.java:102)
                           at com.caucho.db.sql.Query$TailInitRow.initBlockRow(Query.java:952)
                           at com.caucho.db.sql.Query.start(Query.java:727)
                           at com.caucho.db.sql.SelectQuery.execute(SelectQuery.java:209)
                           at com.caucho.db.sql.SelectQuery.execute(SelectQuery.java:171)
                           at com.caucho.db.jdbc.PreparedStatementImpl.execute(PreparedStatementImpl.java:357)
                           at com.caucho.db.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:325)
                           at com.caucho.server.distcache.DataStore.loadBlob(DataStore.java:289)
                           at com.caucho.server.distcache.DataStore.load(DataStore.java:241)
                           at com.caucho.server.distcache.CacheDataBackingImpl.loadData(CacheDataBackingImpl.java:259)
                           at com.caucho.server.distcache.LocalDataManager.readData(LocalDataManager.java:172)
                           at com.caucho.server.distcache.DistCacheEntry.get(DistCacheEntry.java:905)
                           at com.caucho.server.distcache.DistCacheEntry.get(DistCacheEntry.java:163)
                           at com.caucho.server.distcache.CacheImpl.get(CacheImpl.java:312)
                           at com.caucho.cloud.globalcache.GlobalCacheManager.get(GlobalCacheManager.java:170)
                           at com.caucho.cloud.globalcache.AbstractGlobalCache.get(AbstractGlobalCache.java:97)
                           at com.caucho.cloud.elastic.ScalingManager$PeerPod.getScalingPod(ScalingManager.java:398)
                           at com.caucho.cloud.elastic.ScalingManager.updatePod(ScalingManager.java:248)
                           at com.caucho.cloud.elastic.ScalingManager$ScalingCacheListener.onCacheChange(ScalingManager.java:411)
                           at com.caucho.cloud.globalcache.GlobalCacheManager.notifyListeners(GlobalCacheManager.java:249)
                           at com.caucho.cloud.globalcache.GlobalCacheManager.access$000(GlobalCacheManager.java:61)
                           at com.caucho.cloud.globalcache.GlobalCacheManager$ClusterCacheListener.entryUpdated(GlobalCacheManager.java:398)
                           at com.caucho.server.distcache.CacheImpl$UpdatedListener.entryUpdated(CacheImpl.java:1323)
                           at com.caucho.server.distcache.CacheImpl.entryUpdate(CacheImpl.java:1122)
                           at com.caucho.server.distcache.CacheImpl.access$200(CacheImpl.java:85)
                           at com.caucho.server.distcache.CacheImpl$1.run(CacheImpl.java:1105)
                           at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
                           at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

[13-06-09 06:12:54.825] {global-cache@baa.app-tier.admin.resin-139} Missing or corrupted data in get for MnodeEntry[value=668bf04ad8a4a104,flags=0x6,version=1370780541908,lease=-1] DistCacheEntry[key=resin:scaling:services-tier:main,keyHash=496c390b,owner=A_C]
                        {resin-148} Missing or corrupted data in get for MnodeEntry[value=668bf04ad8a4a104,flags=0x6,version=1370780541908,lease=-1] DistCacheEntry[key=resin:scaling:services-tier:main,keyHash=496c390b,owner=A_C]
[13-06-09 06:12:54.825] {resin-143} Missing or corrupted data in get for MnodeEntry[value=668bf04ad8a4a104,flags=0x6,version=1370780541908,lease=-1] DistCacheEntry[key=resin:scaling:services-tier:main,keyHash=496c390b,owner=A_C]
[13-06-09 06:12:54.826] {global-cache@baa.app-tier.admin.resin-140} Missing or corrupted data in get for MnodeEntry[value=668bf04ad8a4a104,flags=0x6,version=1370780541908,lease=-1] DistCacheEntry[key=resin:scaling:services-tier:main,keyHash=496c390b,owner=A_C]
[13-06-09 06:13:03.835] {main} Table[log_name:6] validating indexes due to unclean shutdown.
[13-06-09 06:13:03.836] {main} java.lang.IllegalStateException: LongColumn[id,index]: invalid index (key=89, index value=0, row addr=8f85, Table[log_name:6])
                           at com.caucho.db.table.Column.validateIndex(Column.java:600)
                           at com.caucho.db.table.Table.validateIndexByRow(Table.java:779)
                           at com.caucho.db.table.Table.validateIndexes(Table.java:743)
                           at com.caucho.db.table.Table.validateIndexesSafe(Table.java:707)
                           at com.caucho.db.table.Table.loadFromFile(Table.java:338)
                           at com.caucho.db.Database.getTable(Database.java:214)
                           at com.caucho.db.sql.Parser.parseFromItem(Parser.java:530)
                           at com.caucho.db.sql.Parser.parseFromItems(Parser.java:373)
                           at com.caucho.db.sql.Parser.parseSelect(Parser.java:249)
                           at com.caucho.db.sql.Parser.parseSelect(Parser.java:213)
                           at com.caucho.db.sql.Parser.parse(Parser.java:178)
                           at com.caucho.db.sql.Parser.parse(Parser.java:151)
                           at com.caucho.db.Database.parseQuery(Database.java:314)
                           at com.caucho.db.jdbc.StatementImpl.executeQuery(StatementImpl.java:108)
                           at com.caucho.env.log.NameDatabase.createDatabases(NameDatabase.java:211)
                           at com.caucho.env.log.NameDatabase.<init>(NameDatabase.java:55)
                           at com.caucho.env.log.ProLogSystem.start(ProLogSystem.java:397)
                           at com.caucho.env.service.ResinSystem.startServices(ResinSystem.java:544)
                           at com.caucho.env.service.ResinSystem.start(ResinSystem.java:511)
                           at com.caucho.server.resin.Resin.start(Resin.java:849)
                           at com.caucho.server.resin.Resin.initMain(Resin.java:981)
                           at com.caucho.server.resin.Resin.main(Resin.java:1437)

[13-06-09 06:13:03.836] {main} rebuilding indexes for 'Table[log_name:6]' because they did not properly validate on startup.

[13-06-09 06:12:52.767] {main} java.lang.IllegalStateException: BinaryColumn[id,index]: invalid index (key=BinaryKeyCompare[da57dac7e0405ea873f92cfd1473069d7882cdeb0442c8f2594af2310df0b377], index value=0, row addr=9a696e, Table[mnode:4])
                           at com.caucho.db.table.Column.validateIndex(Column.java:600)
                           at com.caucho.db.table.Table.validateIndexByRow(Table.java:779)
                           at com.caucho.db.table.Table.validateIndexes(Table.java:743)
                           at com.caucho.db.table.Table.validateIndexesSafe(Table.java:707)
                           at com.caucho.db.table.Table.loadFromFile(Table.java:338)
                           at com.caucho.db.Database.getTable(Database.java:214)
                           at com.caucho.db.sql.Parser.parseFromItem(Parser.java:530)
                           at com.caucho.db.sql.Parser.parseFromItems(Parser.java:373)
                           at com.caucho.db.sql.Parser.parseSelect(Parser.java:249)
                           at com.caucho.db.sql.Parser.parseSelect(Parser.java:213)
                           at com.caucho.db.sql.Parser.parse(Parser.java:178)
                           at com.caucho.db.sql.Parser.parse(Parser.java:151)
                           at com.caucho.db.Database.parseQuery(Database.java:314)
                           at com.caucho.db.jdbc.StatementImpl.executeQuery(StatementImpl.java:108)
                           at com.caucho.server.distcache.MnodeStore.initDatabase(MnodeStore.java:267)
                           at com.caucho.server.distcache.MnodeStore.init(MnodeStore.java:237)
                           at com.caucho.server.distcache.CacheDataBackingImpl.start(CacheDataBackingImpl.java:380)
                           at com.caucho.server.distcache.CacheStoreManager.start(CacheStoreManager.java:364)
                           at com.caucho.server.distcache.DistCacheSystem.start(DistCacheSystem.java:162)
                           at com.caucho.env.service.ResinSystem.startServices(ResinSystem.java:544)
                           at com.caucho.env.service.ResinSystem.start(ResinSystem.java:511)
                           at com.caucho.server.resin.Resin.start(Resin.java:849)
                           at com.caucho.server.resin.Resin.initMain(Resin.java:981)
                           at com.caucho.server.resin.Resin.main(Resin.java:1437)

[13-06-09 06:12:52.768] {main} rebuilding indexes for 'Table[mnode:4]' because they did not properly validate on startup.
[13-06-09 06:12:54.351] {main} java.sql.SQLException: table data is not a Resin DB. Version '7'
[13-06-09 06:12:54.561] {main} removing /var/resin/resin-data/site/distcache/data.db due to restart corruption
java.lang.ArrayIndexOutOfBoundsException: 124774
    at com.caucho.db.block.BlockStore.getAllocation(BlockStore.java:927)
    at com.caucho.db.block.BlockStore.getAllocationByAddress(BlockStore.java:917)
    at com.caucho.db.block.BlockStore.isRowBlock(BlockStore.java:620)
    at com.caucho.db.table.TableIterator.isValidRow(TableIterator.java:294)
    at com.caucho.db.sql.IdentityIndexExpr.initRow(IdentityIndexExpr.java:102)
    at com.caucho.db.sql.LeftOuterIndexExpr.initRow(LeftOuterIndexExpr.java:97)
    at com.caucho.db.sql.Query$TailInitRow.initBlockRow(Query.java:952)
    at com.caucho.db.sql.Query$NonTailInitRow.initBlockRow(Query.java:989)
    at com.caucho.db.sql.Query.start(Query.java:727)
    at com.caucho.db.sql.SelectQuery.execute(SelectQuery.java:209)
    at com.caucho.db.sql.SelectQuery.execute(SelectQuery.java:171)
    at com.caucho.db.jdbc.PreparedStatementImpl.execute(PreparedStatementImpl.java:357)
    at com.caucho.db.jdbc.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:325)
    at com.caucho.server.distcache.DataStore.deleteOrphans(DataStore.java:539)
    at com.caucho.server.distcache.DataStore.access$100(DataStore.java:61)
    at com.caucho.server.distcache.DataStore$DeleteAlarm.handleAlarm(DataStore.java:690)
    at com.caucho.util.Alarm.handleAlarm(Alarm.java:523)
    at com.caucho.util.Alarm.run(Alarm.java:495)
    at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
    at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)

Additional Information

Rep by E Larson

Attached Files

Relationships

There are no notes attached to this issue.

Issue History
Date Modified	Username	Field	Change
06-14-13 06:00	cowan	New Issue
07-23-13 11:31	ferg	Assigned To	=> ferg
07-23-13 11:31	ferg	Status	new => closed
07-23-13 11:31	ferg	Resolution	open => fixed
07-23-13 11:31	ferg	Fixed in Version	=> 4.0.37
10-16-14 21:33	bun_a	Issue Monitored: bun_a

Mantis 1.0.0rc3[^]

29 total queries executed.
26 unique queries executed.