corrupt db / zope? maint not reverting, event not archiving
dusty <zenoss <at> dusty.name>
2009-06-01 00:54:48 GMT
I have an issue where maintenance windows are sometimes not bringing systems back to their original mode
when complete. This is even more difficult because its not every time, just every few days, seemingly randomly.
Here are some things I've seen in my logs
2009-05-18 04:00:03 INFO zen.MaintenanceWindows: MW daily changes xxx's production state from 1000 to 300
2009-05-18 04:00:03 INFO zen.MaintenanceWindows: MW daily changes xxx's production state from 1000 to 300
2009-05-18 04:00:03 INFO zen.MaintenanceWindows: MW daily changes xxx's production state from 1000 to 300
2009-05-18 04:00:05 CRITICAL txn.2475712: A storage error occured during the second phase of the
two-phase commit. Resources may be in an inconsistent state.
Then, at the end that one, I see a bunch of errors like this:
2009-05-18 04:59:59 ERROR ZODB.Connection: Couldn't load state for 0x89fc
Traceback (most recent call last):
File "/usr/local/zenoss/zenoss/lib/python/ZODB/Connection.py", line 704, in setstate
self._setstate(obj)
File "/usr/local/zenoss/zenoss/lib/python/ZODB/Connection.py", line 758, in _setstate
self._reader.setGhostState(obj, p)
File "/usr/local/zenoss/zenoss/lib/python/ZODB/serialize.py", line 495, in setGhostState
state = self.getState(pickle)
File "/usr/local/zenoss/zenoss/lib/python/ZODB/serialize.py", line 488, in getState
return unpickler.load()
File "/usr/local/zenoss/zenoss/lib/python/ZODB/serialize.py", line 436, in _persistent_load
return self._conn.get(oid)
File "/usr/local/zenoss/zenoss/lib/python/ZODB/Connection.py", line 207, in get
p, serial = self._storage.load(oid, self._version)
File "/data/zenoss/zenoss/lib/python/ZEO/ClientStorage.py", line 746, in load
return self.loadEx(oid, version)[:2]
File "/data/zenoss/zenoss/lib/python/ZEO/ClientStorage.py", line 774, in loadEx
self._cache.store(oid, ver, tid, None, data)
(Continue reading)