cool-RR | 28 May 21:55 2010

Pickling unbound methods on Python 3

Hello,

I've had a problem when porting my project from Python 2.6 to Python 3.x. 

At some point in my project I need to be able to pickle unbound methods.

In Python 2.x it's impossible to pickle unbound methods by default; I managed to do it there in some weird way I don't understand: I wrote a reducer with the `copy_reg` module for the MethodType class, which covers both bound and unbound methods. But the reducer only solved the case of the bound method, because it depended on `my_method.im_self`. Mysteriously it has also caused Python 2.x to be able to pickle unbound methods. This does not happen on Python 3.x.

When trying to pickle a method, I'm getting this error:

    >>> class A:
    ...     def m(self):
    ...         pass
    >>> import pickle
    >>> pickle.dumps(A.m)
    Traceback (most recent call last):
      File "<pyshell#3>", line 1, in <module>
        pickle.dumps(A.m)
      File "C:\Python31\lib\pickle.py", line 1358, in dumps
        Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
    _pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup builtins.function failed

How can this been solved?

One person told me that given an unbound method in Python 3.x, it's *impossible* to tell to which class it belongs. Is it true?

Ram.
_______________________________________________
Python-porting mailing list
Python-porting@...
http://mail.python.org/mailman/listinfo/python-porting
R. David Murray | 29 May 17:46 2010

Re: Pickling unbound methods on Python 3

On Fri, 28 May 2010 21:55:28 +0200, cool-RR wrote:
> One person told me that given an unbound method in Python 3.x, it's
> *impossible* to tell to which class it belongs. Is it true?

I believe that that is true.  In Python3 there is no such thing as an
unbound method as a distinct object type.  There are functions, and
there are bound methods.  See the second sentence in this section:

http://docs.python.org/release/3.0.1/whatsnew/3.0.html#operators-and-special-methods

You'll probably have to explain more about the problem you are
trying to solve in order for us to help you find a solution.

--
R. David Murray                                      www.bitdance.com
cool-RR | 29 May 18:40 2010

Re: Pickling unbound methods on Python 3

On Sat, May 29, 2010 at 5:46 PM, R. David Murray <rdmurray-2P5OLIWfD11Wk0Htik3J/w@public.gmane.org> wrote:
On Fri, 28 May 2010 21:55:28 +0200, cool-RR wrote:
> One person told me that given an unbound method in Python 3.x, it's
> *impossible* to tell to which class it belongs. Is it true?

I believe that that is true.  In Python3 there is no such thing as an
unbound method as a distinct object type.  There are functions, and
there are bound methods.  See the second sentence in this section:

http://docs.python.org/release/3.0.1/whatsnew/3.0.html#operators-and-special-methods

I see. Where would be a good place to discuss this decision? I would want 3.2 to allow pickling of unbound methods.

 
You'll probably have to explain more about the problem you are
trying to solve in order for us to help you find a solution.

--
R. David Murray                                      www.bitdance.com

I found a hacky workaround: When I have a method `my_method` that I want to pickle, I write `my_method = my_class.my_method` in the method's module.

So no need to burden this list with the specifics of my problem.

Now I only worry about the future: I want 3.2 to enable pickling of unbound methods so I could get rid of this workaround in the future.

Ram.
_______________________________________________
Python-porting mailing list
Python-porting@...
http://mail.python.org/mailman/listinfo/python-porting
M.-A. Lemburg | 29 May 18:56 2010

Re: Pickling unbound methods on Python 3

cool-RR wrote:
> On Sat, May 29, 2010 at 5:46 PM, R. David Murray <rdmurray@...>wrote:
> 
>> On Fri, 28 May 2010 21:55:28 +0200, cool-RR wrote:
>>> One person told me that given an unbound method in Python 3.x, it's
>>> *impossible* to tell to which class it belongs. Is it true?
>>
>> I believe that that is true.  In Python3 there is no such thing as an
>> unbound method as a distinct object type.  There are functions, and
>> there are bound methods.  See the second sentence in this section:
>>
>>
>> http://docs.python.org/release/3.0.1/whatsnew/3.0.html#operators-and-special-methods
> 
> 
> I see. Where would be a good place to discuss this decision? I would want
> 3.2 to allow pickling of unbound methods.

Unbound methods don't exist in Python3. You only have functions and
(bound) methods.

You can see that if you try to call an unbound method with a
non-instance first arg:

>>> class X:
...  def test(self): return 42
...
>>> X.test(X())
42
>>> X.test(3)
42

Doing the same in Python2 gives an error:

>>> X.test(X())
42
>>> X.test(3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unbound method test() must be called with X instance as first argument (got int instance
instead)

--

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 29 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________
2010-07-19: EuroPython 2010, Birmingham, UK                50 days to go

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
Lennart Regebro | 29 May 19:41 2010
Picon

Re: Pickling unbound methods on Python 3

On Sat, May 29, 2010 at 18:40, cool-RR <cool-rr@...> wrote:
> I see. Where would be a good place to discuss this decision? I would want
> 3.2 to allow pickling of unbound methods.

That would be python-dev@...

However, it has already been discussed:

  http://mail.python.org/pipermail/python-dev/2005-January/050625.html
  http://mail.python.org/pipermail/python-dev/2007-November/075279.html

It turns out, even the pickling argument was discussed.

  http://mail.python.org/pipermail/python-dev/2005-January/051143.html

Essentially, the argument is that it's easier to use a function
instead of writing a pickler for a method in the first case, so
support for pickling unbound methods isn't a high priority as there
are easier and less magic ways to solve the use case.

I know that's not what you want to hear, but that's likely to be the
answer you'll get.

--

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64
cool-RR | 29 May 19:59 2010

Re: Pickling unbound methods on Python 3

On Sat, May 29, 2010 at 6:56 PM, M.-A. Lemburg <mal-SVD0I98eSHvQT0dZR+AlfA@public.gmane.org> wrote:
cool-RR wrote:
> On Sat, May 29, 2010 at 5:46 PM, R. David Murray <rdmurray-2P5OLIWfD11Wk0Htik3J/w@public.gmane.org>wrote:
>
>> On Fri, 28 May 2010 21:55:28 +0200, cool-RR wrote:
>>> One person told me that given an unbound method in Python 3.x, it's
>>> *impossible* to tell to which class it belongs. Is it true?
>>
>> I believe that that is true.  In Python3 there is no such thing as an
>> unbound method as a distinct object type.  There are functions, and
>> there are bound methods.  See the second sentence in this section:
>>
>>
>> http://docs.python.org/release/3.0.1/whatsnew/3.0.html#operators-and-special-methods
>
>
> I see. Where would be a good place to discuss this decision? I would want
> 3.2 to allow pickling of unbound methods.
 
Unbound methods don't exist in Python3. You only have functions and
(bound) methods.

I know that unbound methods are of the function type in Python 3. I'm calling them unbound methods because they are methods of classes which are not bound to an instance. I'm aware they have no special stance, but I still refer to them as unbound methods.

You can see that if you try to call an unbound method with a
non-instance first arg:

>>> class X:
...  def test(self): return 42
...
>>> X.test(X())
42
>>> X.test(3)
42

Doing the same in Python2 gives an error:

>>> X.test(X())
42
>>> X.test(3)
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: unbound method test() must be called with X instance as first argument (got int instance
instead)

Thanks for the interesting example, Marc-Andre. 

Ram.
_______________________________________________
Python-porting mailing list
Python-porting@...
http://mail.python.org/mailman/listinfo/python-porting
cool-RR | 29 May 20:03 2010

Re: Pickling unbound methods on Python 3

On Sat, May 29, 2010 at 7:41 PM, Lennart Regebro <regebro-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
On Sat, May 29, 2010 at 18:40, cool-RR <cool-rr-4Sw9Ecu7DfZBDgjK7y7TUQ@public.gmane.org> wrote:
> I see. Where would be a good place to discuss this decision? I would want
> 3.2 to allow pickling of unbound methods.

That would be python-dev <at> python.org

However, it has already been discussed:

 http://mail.python.org/pipermail/python-dev/2005-January/050625.html
 http://mail.python.org/pipermail/python-dev/2007-November/075279.html

It turns out, even the pickling argument was discussed.

 http://mail.python.org/pipermail/python-dev/2005-January/051143.html

Essentially, the argument is that it's easier to use a function
instead of writing a pickler for a method in the first case, so
support for pickling unbound methods isn't a high priority as there
are easier and less magic ways to solve the use case.


I know that's not what you want to hear, but that's likely to be the
answer you'll get.

--
Lennart Regebro

Thanks for the info Lennart.

Ram. 

_______________________________________________
Python-porting mailing list
Python-porting@...
http://mail.python.org/mailman/listinfo/python-porting
"Martin v. Löwis" | 29 May 20:43 2010
Picon

Re: Pickling unbound methods on Python 3

> I know that unbound methods are of the function type in Python 3. I'm
> calling them unbound methods because they are methods of classes which
> are not bound to an instance. I'm aware they have no special stance, but
> I still refer to them as unbound methods.

Neglecting reality doesn't help your cause, though. You'll need to 
understand *why* pickling fails, and then see whether there might be
a solution.

A work-around is to put all methods you want to pickle into module-scope
of your module

class Foo:
   def bar(self):
     pass

bar = Foo.bar

Now you can pickle Foo.bar.

Regards,
Martin
cool-RR | 29 May 22:52 2010

Re: Pickling unbound methods on Python 3

Hello Martin, thanks for joining the discussion.

On Sat, May 29, 2010 at 8:43 PM, "Martin v. Löwis" <martin <at> v.loewis.de> wrote:
I know that unbound methods are of the function type in Python 3. I'm
calling them unbound methods because they are methods of classes which
are not bound to an instance. I'm aware they have no special stance, but
I still refer to them as unbound methods.

Neglecting reality doesn't help your cause, though. You'll need to understand *why* pickling fails, and then see whether there might be
a solution.

I think I understand why pickling fails. An unbound method is not differentiated in any way from a function. The `save_global` function tries to look for it in the main module namespace, instead of in the class.

 
A work-around is to put all methods you want to pickle into module-scope
of your module

class Foo:
 def bar(self):
   pass

bar = Foo.bar

Now you can pickle Foo.bar.

Yes, I've already done this, but I'd prefer a less hackish solution. 

 
Regards,
Martin

I can think of a few solutions:

1. Make `save_global` look in the classes inside the module as well.

2. Allow me to specify a reducer for FunctionType. This is currently not working, see here: http://stackoverflow.com/questions/2932742/python-using-copyreg-to-define-reducers-for-types-that-already-have-reducers


Ram.
_______________________________________________
Python-porting mailing list
Python-porting@...
http://mail.python.org/mailman/listinfo/python-porting
Lennart Regebro | 30 May 00:13 2010
Picon

Re: Pickling unbound methods on Python 3

On Sat, May 29, 2010 at 20:43, "Martin v. Löwis" <martin <at> v.loewis.de> wrote:
> bar = Foo.bar
>
> Now you can pickle Foo.bar.

Ah yes, of course. So simple and neat.

--

-- 
Lennart Regebro: Python, Zope, Plone, Grok
http://regebro.wordpress.com/
+33 661 58 14 64

Gmane