Re: zeopack error

To debug the pickle, I copied the 4.7GB users.fs to my local machine. I changed the code as proposed by
Martijn, started zeo und zeopack - hours later - everything was packed without any troubles. I was puzzled.

I then realized that I was running the whole thing with Python 2.7.1 on my local machine (OS X 10.7, 64bit),
while the original zeo server was running on Python 2.6.6. (Debian, 32bit).

I will rerun the test on my local machine with Python 2.6 and see if I can reproduce the problem.

Kind regards,
Kaweh

On 31.01.2012, at 14:51, Kaweh Kazemi wrote:

> Thanks Martijn,
> 
> I'll try to dump the pickle, and see what I can find out.
> 
> I'm still puzzled how this/what was pickled in the first place.

_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev <at> zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [X-Post] Figure out bottle neck in a repoze.bfg based web app

Hi,

Last week I started this thread to try and figure out the bottle neck in our
web-app. After trying out different approaches, we decided to profile the app.
using repoze.profile[1] in the middleware and hit the app. using funkload[2].

This is what we saw for a test run: (although my question below is not related
to the specifics of the test run, details of the same can be provided if needed):

http://pastebin.com/pzwA74EN

As you can see, most of the time is being spent, acquiring locks, or (validating
?) cache code.

After looking at ZODB3-3.10.2-py2.6-linux-x86_64.egg/ZODB/Connection.py line 864
(_setstate) (line 13 in that ^ paste), I think most of those acquire()s are
coming from line 900:

            self._inv_lock.acquire()

So, my question here is:
a. Is the invalidation-queue-size just a start-up time optimization or does it
play a part in invalidation of the zeo client cache for every transaction ? (we
use the repoze.zodbconn middleware, using which, every request to the webserver
is treated as a transaction)

b. Does drop_cache_rather_verify() play a part on a 'connected' zeo client --
ie: would the client always drop cache rather than verify each transaction or is
this just a startup optimization ?

Re: [RFC] Allow serial to be returned as late as tpc_finish.

Le jeudi 16 décembre 2010 17:19:33, Jim Fulton a écrit :
> Cool. I'll review this for 3.11 and it or somethink like it will go
> into that release.

bump :)

Attached are:
- a stand-alone change to tests/TransactionalUndoStorage.py (it should be
  acceptable to apply it even without considering the main feature change of
  my patch)
- the main patch, updated a bit:
  - fix case where Connection._creating is not a dict (it was a list on older
    ZODB versions - although it shouldn't matter for trunk inclusion)
  - fix serials of newly created objects (obj._p_changed is None, so serial
    was not set)

I verified there is no regression in "setup.py test", but I get 2 failures 
(both with an without my changes), one on ZEO/tests/zeo-fan-out.test and 
another on ZEO.tests.testZEO.client_has_newer_data_than_server .

Regards,
--

-- 
Vincent Pelletier
ERP5 - open source ERP/CRM for flexible enterprises

From 9d266460b66d6a852c2072eb46c0e5cdd99413d8 Mon Sep 17 00:00:00 2001
From: Vincent Pelletier <vincent <at> nexedi.com>
Date: Fri, 10 Dec 2010 08:30:35 +0000
Subject: [RFC 1/2] The number of entries doesn't matter as long as both OIDs

Re: zeopack error

Hi there,

Recap: last week I examined problems I had packing our 4GB users storage. With Martijn's help I was able to
fix zeo's exception output and write out the first broken pickle that throws an exception. During my
checks I realized that running the pack in a Python 2.7 environment (using the same ZODB version - 3.10.3)
works fine, the pack reduces our 4GB storage to 1GB. But our production server uses Python 2.6 (same
ZODB3.10.3) which yields the problem (though the test had been done on OS X 10.7.3 - 64bit, and the
production server is Debian Squeeze 32bit).

Currently I see only three ways to resolve that problem:

	- Upgrade production server from Python 2.6 to Python 2.7 which would probably solve the current problem
(if it's really a Python 2.6/2.7 problem).

	- Pack the storage with Python 2.7/ZODB3.10.3 on the working system and deploy it back to the Python 2.6
production environment - very time consuming (because I have to download/pack/upload a big storage and
during that time the production server wouldn't be available to avoid changes during that time)/
potentially risky(?).

	- Find out what is broken regarding that pickle and possibly the Python 2.6/ZODB3.10.3 combination and
deploy a fix.

You can download the broken pickle from here: http://www.reversepanda.com/download/brokenpickle

If someone has more experience in parsing and understanding pickles in regards to ZODB3, any help would be appreciated.

Kind regards,
Kaweh
_______________________________________________
For more information about ZODB, see http://zodb.org/

Re: zeopack error

On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote:
> Recap: last week I examined problems I had packing our 4GB users
> storage. With Martijn's help I was able to fix zeo's exception output
> and write out the first broken pickle that throws an exception.
...
> You can download the broken pickle from here:
> http://www.reversepanda.com/download/brokenpickle
> 
> If someone has more experience in parsing and understanding pickles in
> regards to ZODB3, any help would be appreciated.

I don't have much experience here, but I love a puzzle

    >>> import pickletools
    >>> f = open('brokenpickle', 'rb')

A ZODB record consists of two pickles: the first stores the class of
the object, the other stores the state of the object

    >>> pickletools.dis(f)
        0: c    GLOBAL     'rp.odb.containers EntityMapping'
       33: q    BINPUT     1
       35: .    STOP
    highest protocol among opcodes = 1

    >>> pickletools.dis(f)
       36: }    EMPTY_DICT
       37: q    BINPUT     2
       39: U    SHORT_BINSTRING 'data'
       45: q    BINPUT     3

Re: zeopack error

On Thu, Feb 09, 2012 at 01:25:48AM +0200, Marius Gedminas wrote:
> On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote:
> > Recap: last week I examined problems I had packing our 4GB users
> > storage.
...
>     >>> unp = pickle.Unpickler(f)
>     >>> unp.persistent_load = lambda oid: '<persistent reference %r>' % oid
>     >>> pprint.pprint(unp.load())
>     {'data': {"<persistent reference ['m', ('game', '\\x00\\x00\\x00\\x00\\x00\\x00\\tT', <class
'__main__.Tool'>)]>": 1,
>               "<persistent reference ['m', ('game', '\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x03', <class
'__main__.EnergyPack'>)]>": 1}}
> 
> Those look like cross-database references to me.
> 
> The original error (aaaugh Mutt makes it hard for me to look upthread
> while I'm writing a response) was something about non-hashable lists?
> Looks like a piece of code is trying to put persistent references into a
> dict, which can't possibly work in all cases.
...
> > During my checks I realized that running the pack in a Python 2.7
> > environment (using the same ZODB version - 3.10.3) works fine, the
> > pack reduces our 4GB storage to 1GB. But our production server uses
> > Python 2.6 (same ZODB3.10.3) which yields the problem (though the test
> > had been done on OS X 10.7.3 - 64bit, and the production server is
> > Debian Squeeze 32bit).
> 
> I've no idea why running the same ZODB version on Python 2.7 instead of
> 2.6 would make this error go away.

Re: zeopack error

I'm sorry I haven't had time to look at this. Still don't really.

Thanks Marius!!!

On Wed, Feb 8, 2012 at 6:48 PM, Marius Gedminas <marius <at> gedmin.as> wrote:
> On Thu, Feb 09, 2012 at 01:25:48AM +0200, Marius Gedminas wrote:
>> On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote:
>> > Recap: last week I examined problems I had packing our 4GB users
>> > storage.
> ...
>>     >>> unp = pickle.Unpickler(f)
>>     >>> unp.persistent_load = lambda oid: '<persistent reference %r>' % oid
>>     >>> pprint.pprint(unp.load())
>>     {'data': {"<persistent reference ['m', ('game', '\\x00\\x00\\x00\\x00\\x00\\x00\\tT',
<class '__main__.Tool'>)]>": 1,
>>               "<persistent reference ['m', ('game',
'\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x03', <class '__main__.EnergyPack'>)]>": 1}}

Note the reference to __main__. This is almost certainly the root problem.
Classes shouldn't be defined in __main__ (except when experimenting).

At one time, I thought pickle disallowed pickling classes from __main__.
ZODB probably should. It's a bug magnet.

>>
>> Those look like cross-database references to me.
>>
>> The original error (aaaugh Mutt makes it hard for me to look upthread
>> while I'm writing a response) was something about non-hashable lists?
>> Looks like a piece of code is trying to put persistent references into a

Re: [RFC] Allow serial to be returned as late as tpc_finish.

On Tue, Feb 7, 2012 at 12:05 PM, Vincent Pelletier <vincent <at> nexedi.com> wrote:
> Le jeudi 16 décembre 2010 17:19:33, Jim Fulton a écrit :
>> Cool. I'll review this for 3.11 and it or somethink like it will go
>> into that release.
>
> bump :)

Um. Thanks.

Jim

--

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev <at> zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: zeopack error

On 9 February 2012 11:24, Jim Fulton <jim <at> zope.com> wrote:
> I'm sorry I haven't had time to look at this. Still don't really.
>
> Thanks Marius!!!
>
> On Wed, Feb 8, 2012 at 6:48 PM, Marius Gedminas <marius <at> gedmin.as> wrote:
>> On Thu, Feb 09, 2012 at 01:25:48AM +0200, Marius Gedminas wrote:
>>> On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote:
>>> > Recap: last week I examined problems I had packing our 4GB users
>>> > storage.
>> ...
>>>     >>> unp = pickle.Unpickler(f)
>>>     >>> unp.persistent_load = lambda oid: '<persistent reference %r>' % oid
>>>     >>> pprint.pprint(unp.load())
>>>     {'data': {"<persistent reference ['m', ('game', '\\x00\\x00\\x00\\x00\\x00\\x00\\tT',
<class '__main__.Tool'>)]>": 1,
>>>               "<persistent reference ['m', ('game',
'\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x03', <class '__main__.EnergyPack'>)]>": 1}}
>
> Note the reference to __main__. This is almost certainly the root problem.
> Classes shouldn't be defined in __main__ (except when experimenting).
>
> At one time, I thought pickle disallowed pickling classes from __main__.
> ZODB probably should. It's a bug magnet.
>
>
>>>
>>> Those look like cross-database references to me.
>>>
>>> The original error (aaaugh Mutt makes it hard for me to look upthread

Re: zeopack error

>> 
>> I suspect a bug in the application (defining persistent classes in __main__)
>> is the root problem that's aggravated by the cPickle problem.
> 
> The pickle's classes were defined in a normal module, I think Marius
> just aliased those to modules to __main__ and defined the classes
> there in order to load the pickle without the original code:
> 
>>>> sys.modules['game.objects.item'] = sys.modules['__main__'] # hack
>>>> sys.modules['game.objects'] = sys.modules['__main__'] # hack
>>>> sys.modules['game'] = sys.modules['__main__'] # hack
> 

That's correct, we don't define classes in __main__. All our classes are in their own modules and are imported.

Kind regards,
Kaweh
_______________________________________________
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev <at> zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev