headers
Patricio Stegmann | 23 Jun 2011 00:18
Picon
Favicon

freeing the memory of a casted object ?

Hello to all,

I use intensively ctypes for wrapping lot of cdll's and love the module !
However I face a painful problem of memory leak when I cast an object to something other.
I post the following example which running on win32 is perfectly stable in memory when not casting, but leask 10MB per cycle when casting.
I couldnt find help on the internet for this, nor found a free function on ctypes module or whatever.
Please see and test if possible the following code:
"""
import ctypes
import os
import sys
import time

if __name__ == '__main__':
    if sys.argv[1] == '0':
        cast_flag = False
    else:
        cast_flag = True
    while 1:
        print 'creating 1MB buffer'
        tmp_buffer = ctypes.create_string_buffer(10*1024*1024)
        if cast_flag:
            print 'casting it to c_void_p'
            tmp_pointer = ctypes.cast(tmp_buffer, ctypes.POINTER(ctypes.c_void_p))
        time.sleep(1)
"""

running:

python myscript.py 0

it will run without leaking

running:

python myscript.py 1

will leak more than 10MB at each cycle (seen with windows task manager or process explorer).

Can someone please give me a tip on how to free that memory ?
I really thank you for any tip !

Patricio
------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger.
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today.
http://p.sf.net/sfu/quest-sfdev2dev
_______________________________________________
ctypes-users mailing list
ctypes-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ctypes-users
Permalink | Reply | 3 comments |
headers
Diez B. Roggisch | 23 Jun 2011 09:05
Picon
Gravatar

Re: freeing the memory of a casted object ?

 
On Jun 23, 2011, at 12:18 AM, Patricio Stegmann wrote:

Hello to all,

I use intensively ctypes for wrapping lot of cdll's and love the module !
However I face a painful problem of memory leak when I cast an object to something other.
I post the following example which running on win32 is perfectly stable in memory when not casting, but leask 10MB per cycle when casting.
I couldnt find help on the internet for this, nor found a free function on ctypes module or whatever.
Please see and test if possible the following code:

Under OSX I see a similar behavior, so at first I thought you were right.

But you aren't!

Python uses GC, and that's just not kicking in. If you reduce the wait-time to a .1 (so the memory consumption grows much faster) you will see that it peaks - and then a great deal will be lost again.

Alternatively, you can do 

import gc


and then call gc.collect() every now and then. Then there should be no noticeable growth in mem-consumption.

But *don't* do this in real code unless you really have to. It just costs time. Let Python decide when to free memory.

Diez
------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger.
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today.
http://p.sf.net/sfu/quest-sfdev2dev
_______________________________________________
ctypes-users mailing list
ctypes-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ctypes-users
Permalink | Reply | 2 comments |
headers
Diez B. Roggisch | 23 Jun 2011 10:50
Picon
Gravatar

Re: freeing the memory of a casted object ? [SEC=PERSONAL]

On Thursday, June 23, 2011 10:49:04 am Andrew MacIntyre wrote:
> [apologies for top post]
> 
> On Windows 7 32bit, Patricio's code triggers a MemoryError when it exhausts
> the heap. gc is not kicking in here, as the number of deallocations isn't
> sufficient to trigger it; reduce the buffer size to 1MB (and ditch the
> sleep) and gc does in fact kick in.
> 
> I think it comes down to the cast() function doing more under the covers
> than is explained in the docs, as it would appear that cast() results in
> an extra (weak?) reference being kept back to the original buffer.
> 
> This protectionism results in reference cycles, which gc.collect() can
> clean up.  This was probably to make certain operations feasible without
> lots of explicit references needing to be kept; see the type conversions
> section of the ctypes tutorial for an example that wouldn't work as
> intended/desired without this behind-the-scenes reference keeping.
> 
> Rather than routinely calling gc.collect(), tuning the gc thresholds for
> the particular usage pattern might be more attractive.  Either way, this
> is a case where actively using the gc module is required.

I have to admit that I was a little bit premature in my assesment of the 
situation - I later found out that the memory problem actually *did* occur 
without the gc.collect() call - it just took longer. I wanted to post that, 
but got distracted.

Sorry for the confusion.

Diez

------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger.
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today.
http://p.sf.net/sfu/quest-sfdev2dev
Permalink | Reply | 0 comments |
headers
Andrew MacIntyre | 23 Jun 2011 10:49
Picon

Re: freeing the memory of a casted object ? [SEC=PERSONAL]

[apologies for top post]

 

On Windows 7 32bit, Patricio’s code triggers a MemoryError when it exhausts the heap. gc is not kicking in here, as the number of deallocations isn’t sufficient to trigger it; reduce the buffer size to 1MB (and ditch the sleep) and gc does in fact kick in.

 

I think it comes down to the cast() function doing more under the covers than is explained in the docs, as it would appear that cast() results in an extra (weak?) reference being kept back to the original buffer.

 

This protectionism results in reference cycles, which gc.collect() can clean up.  This was probably to make certain operations feasible without lots of explicit references needing to be kept; see the type conversions section of the ctypes tutorial for an example that wouldn’t work as intended/desired without this behind-the-scenes reference keeping.

 

Rather than routinely calling gc.collect(), tuning the gc thresholds for the particular usage pattern might be more attractive.  Either way, this is a case where actively using the gc module is required.

 

-------------------------> "These thoughts are mine alone!" <---------

Andrew MacIntyre           Operations Branch

tel:   +61 2 6219 5356     Communications Infrastructure Division

fax:   +61 2 6253 3277     Australian Communications & Media Authority

email: andrew.macintyre <at> acma.gov.au            http://www.acma.gov.au/

 

From: Diez B. Roggisch [mailto:deets <at> web.de]
Sent: Thursday, 23 June 2011 5:06 PM
To: ctypes users mailing list
Subject: Re: [ctypes-users] freeing the memory of a casted object ?

 

 

On Jun 23, 2011, at 12:18 AM, Patricio Stegmann wrote:



Hello to all,

I use intensively ctypes for wrapping lot of cdll's and love the module !
However I face a painful problem of memory leak when I cast an object to something other.
I post the following example which running on win32 is perfectly stable in memory when not casting, but leask 10MB per cycle when casting.
I couldnt find help on the internet for this, nor found a free function on ctypes module or whatever.
Please see and test if possible the following code:

 

Under OSX I see a similar behavior, so at first I thought you were right.

 

But you aren't!

 

Python uses GC, and that's just not kicking in. If you reduce the wait-time to a .1 (so the memory consumption grows much faster) you will see that it peaks - and then a great deal will be lost again.

 

Alternatively, you can do 

 

import gc

 

 

and then call gc.collect() every now and then. Then there should be no noticeable growth in mem-consumption.

 

But *don't* do this in real code unless you really have to. It just costs time. Let Python decide when to free memory.

 

Diez

NOTICE: This email message is for the sole use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized
review, use, disclosure or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and destroy all
copies of the original message. 

------------------------------------------------------------------------------
Simplify data backup and recovery for your virtual environment with vRanger.
Installation's a snap, and flexible recovery options mean your data is safe,
secure and there when you need it. Data protection magic?
Nope - It's vRanger. Get your free trial download today.
http://p.sf.net/sfu/quest-sfdev2dev
_______________________________________________
ctypes-users mailing list
ctypes-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ctypes-users
Permalink | Reply | 1 comment |
headers
Patricio Stegmann | 24 Jun 2011 01:00
Picon
Favicon

Re: freeing the memory of a casted object ? Solution found !

Thank you all for the interest in my pain :)

I managed, after googling and ircing a bit, a quick solution for this problem. Basically I manually free the memory with the platform's free function.
This, on my WinXP box, is present on MsVcrt libs. In practice, this worked very well:

"""
import ctypes
import ctypes.util
import os
import sys
import time

if __name__ == '__main__':
    vcrt = ctypes.CDLL(ctypes.util.find_msvcrt())
    if sys.argv[1] == '0':
        cast_flag = False
    else:
        cast_flag = True
    while 1:
        print 'creating 1MB buffer'
        tmp_buffer = ctypes.create_string_buffer(10*1024*1024)
        if cast_flag:
            print 'casting it to c_void_p'
            tmp_pointer = ctypes.cast(tmp_buffer, ctypes.POINTER(ctypes.c_void_p))
            vcrt.free(tmp_pointer)
        time.sleep(1)
"""

The vcrt.free stuff is freeing the memory.

Now I wonder if there is a way to find out which memory is unused and free it "automagically". Do someone know of a way to check memory allocation for everything derivated from a python script ?


> From: ctypes-users-request <at> lists.sourceforge.net
> Subject: ctypes-users Digest, Vol 61, Issue 1
> To: ctypes-users <at> lists.sourceforge.net
> Date: Thu, 23 Jun 2011 09:05:17 +0000
>
> Send ctypes-users mailing list submissions to
> ctypes-users <at> lists.sourceforge.net
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.sourceforge.net/lists/listinfo/ctypes-users
> or, via email, send a message with subject or body 'help' to
> ctypes-users-request <at> lists.sourceforge.net
>
> You can reach the person managing the list at
> ctypes-users-owner <at> lists.sourceforge.net
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of ctypes-users digest..."
>
>
> Today's Topics:
>
> 1. freeing the memory of a casted object ? (Patricio Stegmann)
> 2. Re: freeing the memory of a casted object ? (Diez B. Roggisch)
> 3. Re: freeing the memory of a casted object ? [SEC=PERSONAL]
> (Diez B. Roggisch)
> 4. Re: freeing the memory of a casted object ? [SEC=PERSONAL]
> (Andrew MacIntyre)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 22 Jun 2011 19:18:57 -0300
> From: Patricio Stegmann <kpoman <at> hotmail.com>
> Subject: [ctypes-users] freeing the memory of a casted object ?
> To: ctypes users mailing list <ctypes-users <at> lists.sourceforge.net>
> Message-ID: <SNT140-w43A1DD4ED291BF4E718EEFD9500 <at> phx.gbl>
> Content-Type: text/plain; charset="iso-8859-1"
>
>
> Hello to all,
>
> I use intensively ctypes for wrapping lot of cdll's and love the module !
> However I face a painful problem of memory leak when I cast an object to something other.
> I post the following example which running on win32 is perfectly stable in memory when not casting, but leask 10MB per cycle when casting.
> I couldnt find help on the internet for this, nor found a free function on ctypes module or whatever.
> Please see and test if possible the following code:
> """
> import ctypes
> import os
> import sys
> import time
>
> if __name__ == '__main__':
> if sys.argv[1] == '0':
> cast_flag = False
> else:
> cast_flag = True
> while 1:
> print 'creating 1MB buffer'
> tmp_buffer = ctypes.create_string_buffer(10*1024*1024)
> if cast_flag:
> print 'casting it to c_void_p'
> tmp_pointer = ctypes.cast(tmp_buffer, ctypes.POINTER(ctypes.c_void_p))
> time.sleep(1)
> """
>
> running:
>
> python myscript.py 0
>
> it will run without leaking
>
> running:
>
> python myscript.py 1
>
> will leak more than 10MB at each cycle (seen with windows task manager or process explorer).
>
> Can someone please give me a tip on how to free that memory ?
> I really thank you for any tip !
>
> Patricio
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
>
> ------------------------------
>
> Message: 2
> Date: Thu, 23 Jun 2011 09:05:41 +0200
> From: "Diez B. Roggisch" <deets <at> web.de>
> Subject: Re: [ctypes-users] freeing the memory of a casted object ?
> To: ctypes users mailing list <ctypes-users <at> lists.sourceforge.net>
> Message-ID: <83FDA802-CB2F-4BFA-8C8D-AB967C4E88F3 <at> web.de>
> Content-Type: text/plain; charset="us-ascii"
>
>
> On Jun 23, 2011, at 12:18 AM, Patricio Stegmann wrote:
>
> > Hello to all,
> >
> > I use intensively ctypes for wrapping lot of cdll's and love the module !
> > However I face a painful problem of memory leak when I cast an object to something other.
> > I post the following example which running on win32 is perfectly stable in memory when not casting, but leask 10MB per cycle when casting.
> > I couldnt find help on the internet for this, nor found a free function on ctypes module or whatever.
> > Please see and test if possible the following code:
>
> Under OSX I see a similar behavior, so at first I thought you were right.
>
> But you aren't!
>
> Python uses GC, and that's just not kicking in. If you reduce the wait-time to a .1 (so the memory consumption grows much faster) you will see that it peaks - and then a great deal will be lost again.
>
> Alternatively, you can do
>
> import gc
>
>
> and then call gc.collect() every now and then. Then there should be no noticeable growth in mem-consumption.
>
> But *don't* do this in real code unless you really have to. It just costs time. Let Python decide when to free memory.
>
> Diez
> -------------- next part --------------
> An HTML attachment was scrubbed...
>
> ------------------------------
>
> Message: 3
> Date: Thu, 23 Jun 2011 10:50:29 +0200
> From: "Diez B. Roggisch" <deets <at> web.de>
> Subject: Re: [ctypes-users] freeing the memory of a casted object ?
> [SEC=PERSONAL]
> To: "Andrew MacIntyre" <Andrew.MacIntyre <at> acma.gov.au>
> Cc: ctypes users mailing list <ctypes-users <at> lists.sourceforge.net>
> Message-ID: <201106231050.29553.deets <at> web.de>
> Content-Type: Text/Plain; charset="iso-8859-15"
>
> On Thursday, June 23, 2011 10:49:04 am Andrew MacIntyre wrote:
> > [apologies for top post]
> >
> > On Windows 7 32bit, Patricio's code triggers a MemoryError when it exhausts
> > the heap. gc is not kicking in here, as the number of deallocations isn't
> > sufficient to trigger it; reduce the buffer size to 1MB (and ditch the
> > sleep) and gc does in fact kick in.
> >
> > I think it comes down to the cast() function doing more under the covers
> > than is explained in the docs, as it would appear that cast() results in
> > an extra (weak?) reference being kept back to the original buffer.
> >
> > This protectionism results in reference cycles, which gc.collect() can
> > clean up. This was probably to make certain operations feasible without
> > lots of explicit references needing to be kept; see the type conversions
> > section of the ctypes tutorial for an example that wouldn't work as
> > intended/desired without this behind-the-scenes reference keeping.
> >
> > Rather than routinely calling gc.collect(), tuning the gc thresholds for
> > the particular usage pattern might be more attractive. Either way, this
> > is a case where actively using the gc module is required.
>
> I have to admit that I was a little bit premature in my assesment of the
> situation - I later found out that the memory problem actually *did* occur
> without the gc.collect() call - it just took longer. I wanted to post that,
> but got distracted.
>
> Sorry for the confusion.
>
> Diez
>
>
>
> ------------------------------
>
> Message: 4
> Date: Thu, 23 Jun 2011 08:49:04 +0000
> From: "Andrew MacIntyre" <Andrew.MacIntyre <at> acma.gov.au>
> Subject: Re: [ctypes-users] freeing the memory of a casted object ?
> [SEC=PERSONAL]
> To: "Diez B. Roggisch" <deets <at> web.de>, "ctypes users mailing list"
> <ctypes-users <at> lists.sourceforge.net>
> Message-ID:
> <A1E4E9D08D14924498E135FB4441EE8D07193BEC <at> act01exmbx01vp.internal.govt>
>
> Content-Type: text/plain; charset="us-ascii"
>
> [apologies for top post]
>
> On Windows 7 32bit, Patricio's code triggers a MemoryError when it exhausts the heap. gc is not kicking in here, as the number of deallocations isn't sufficient to trigger it; reduce the buffer size to 1MB (and ditch the sleep) and gc does in fact kick in.
>
> I think it comes down to the cast() function doing more under the covers than is explained in the docs, as it would appear that cast() results in an extra (weak?) reference being kept back to the original buffer.
>
> This protectionism results in reference cycles, which gc.collect() can clean up. This was probably to make certain operations feasible without lots of explicit references needing to be kept; see the type conversions section of the ctypes tutorial for an example that wouldn't work as intended/desired without this behind-the-scenes reference keeping.
>
> Rather than routinely calling gc.collect(), tuning the gc thresholds for the particular usage pattern might be more attractive. Either way, this is a case where actively using the gc module is required.
>
> -------------------------> "These thoughts are mine alone!" <---------
> Andrew MacIntyre Operations Branch
> tel: +61 2 6219 5356 Communications Infrastructure Division
> fax: +61 2 6253 3277 Australian Communications & Media Authority
> email: andrew.macintyre <at> acma.gov.au<mailto:andrew.macintyre <at> acma.gov.au> http://www.acma.gov.au/
>
> From: Diez B. Roggisch [mailto:deets <at> web.de]
> Sent: Thursday, 23 June 2011 5:06 PM
> To: ctypes users mailing list
> Subject: Re: [ctypes-users] freeing the memory of a casted object ?
>
>
> On Jun 23, 2011, at 12:18 AM, Patricio Stegmann wrote:
>
>
> Hello to all,
>
> I use intensively ctypes for wrapping lot of cdll's and love the module !
> However I face a painful problem of memory leak when I cast an object to something other.
> I post the following example which running on win32 is perfectly stable in memory when not casting, but leask 10MB per cycle when casting.
> I couldnt find help on the internet for this, nor found a free function on ctypes module or whatever.
> Please see and test if possible the following code:
>
> Under OSX I see a similar behavior, so at first I thought you were right.
>
> But you aren't!
>
> Python uses GC, and that's just not kicking in. If you reduce the wait-time to a .1 (so the memory consumption grows much faster) you will see that it peaks - and then a great deal will be lost again.
>
> Alternatively, you can do
>
> import gc
>
>
> and then call gc.collect() every now and then. Then there should be no noticeable growth in mem-consumption.
>
> But *don't* do this in real code unless you really have to. It just costs time. Let Python decide when to free memory.
>
> Diez
>
> NOTICE: This email message is for the sole use of the intended recipient(s)
> and may contain confidential and privileged information. Any unauthorized
> review, use, disclosure or distribution is prohibited. If you are not the
> intended recipient, please contact the sender by reply email and destroy all
> copies of the original message.
> -------------- next part --------------
> An HTML attachment was scrubbed...
>
> ------------------------------
>
> ------------------------------------------------------------------------------
> Simplify data backup and recovery for your virtual environment with vRanger.
> Installation's a snap, and flexible recovery options mean your data is safe,
> secure and there when you need it. Data protection magic?
> Nope - It's vRanger. Get your free trial download today.
> http://p.sf.net/sfu/quest-sfdev2dev
>
> ------------------------------
>
> _______________________________________________
> ctypes-users mailing list
> ctypes-users <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ctypes-users
>
>
> End of ctypes-users Digest, Vol 61, Issue 1
> *******************************************
------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a 
definitive record of customers, application performance, security 
threats, fraudulent activity and more. Splunk takes this data and makes 
sense of it. Business sense. IT sense. Common sense.. 
http://p.sf.net/sfu/splunk-d2d-c1
_______________________________________________
ctypes-users mailing list
ctypes-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ctypes-users
Permalink | Reply | 1 comment |
headers
Diez B. Roggisch | 24 Jun 2011 11:14
Picon
Gravatar

Re: freeing the memory of a casted object ? Solution found !


On Jun 24, 2011, at 1:00 AM, Patricio Stegmann wrote:

Thank you all for the interest in my pain :)

I managed, after googling and ircing a bit, a quick solution for this problem. Basically I manually free the memory with the platform's free function.
This, on my WinXP box, is present on MsVcrt libs. In practice, this worked very well:

That's the worst solution. You are bound to get segfaults if python eventually tries to get rid of the allocated objects itself - and will cause a double free.

If you can't get the problem solved differently, the least thing you should do is to allocate the memory with malloc yourself - then it's fully up to you to release it.

Now I wonder if there is a way to find out which memory is unused and free it "automagically". Do someone know of a way to check memory allocation for everything derivated from a python script ?

It's called garbage collection, and that's what Python is supposed to do. Why don't you just use the gc.collect()-call Andrew McIntyre & I suggested?

Diez
------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a 
definitive record of customers, application performance, security 
threats, fraudulent activity and more. Splunk takes this data and makes 
sense of it. Business sense. IT sense. Common sense.. 
http://p.sf.net/sfu/splunk-d2d-c1
_______________________________________________
ctypes-users mailing list
ctypes-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ctypes-users
Permalink | Reply | 0 comments |
headers
Dominik George | 26 Jun 2011 15:17
Picon
Gravatar

Strings with null characters

Hi,

I am using ctypes while porting parts of glibc to pure Python (i.e. to
be able to do ifconfig from within Python). The complete code can be
found in my Redmine under
http://git.proparity.net/redmine/projects/pyverkko/repository/revisions/master/show/src/pyverkko/ifconfig
.

In structs.py, I have defined the structs used by glibc to do ioctl(),
which works perfectly in most cases. But, this breaks when I have to
cope with null bytes. For example, set_ip_address("eth0", "192.168.0.1")
breaks, because:

I create an isntance of sockaddr, which is a ctypes Structure (or, a
PackableStructure, which is a ctypes Structure to which I added a pack()
method as described in the FAQ). Now, assigning the binary form of
192.168.0.1 to a member field of sockaddr, breaks because it is
expecting a null-termianted string, so the string I assign is only the
first two bytes because the third octet, 0, terminates the string.

I am not sure if Python does this while assigning, because the field
sa_data I am assigning to is declared as a char[14].

How can I take care that *all* bytes of a string are stored in the
Structure and recognized when I run pack(), i.e. strings aren't cut off
at the first null byte?

Regards,
Nik

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security 
threats, fraudulent activity, and more. Splunk takes this data and makes 
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
Permalink | Reply | 2 comments |
headers
Dominik George | 26 Jun 2011 16:10
Picon
Gravatar

Re: Strings with null characters

Hi,

here is an example code that breaks:

from ctypes import *
 class a(Structure):
  _fields_ = [("aaa", c_char * 14)]

mya = a()
mya.aaa = "a\0b"
print mya.aaa
'a'

I found two workarounds at
http://stackoverflow.com/questions/5082753/how-do-i-build-a-python-string-from-a-ctype-struct
, but those solutions look somehow nasty ...

-nik

Am 26.06.2011 15:17, schrieb Dominik George:
> Hi,
>
> I am using ctypes while porting parts of glibc to pure Python (i.e. to
> be able to do ifconfig from within Python). The complete code can be
> found in my Redmine under
> http://git.proparity.net/redmine/projects/pyverkko/repository/revisions/master/show/src/pyverkko/ifconfig
> .
>
> In structs.py, I have defined the structs used by glibc to do ioctl(),
> which works perfectly in most cases. But, this breaks when I have to
> cope with null bytes. For example, set_ip_address("eth0", "192.168.0.1")
> breaks, because:
>
> I create an isntance of sockaddr, which is a ctypes Structure (or, a
> PackableStructure, which is a ctypes Structure to which I added a pack()
> method as described in the FAQ). Now, assigning the binary form of
> 192.168.0.1 to a member field of sockaddr, breaks because it is
> expecting a null-termianted string, so the string I assign is only the
> first two bytes because the third octet, 0, terminates the string.
>
> I am not sure if Python does this while assigning, because the field
> sa_data I am assigning to is declared as a char[14].
>
> How can I take care that *all* bytes of a string are stored in the
> Structure and recognized when I run pack(), i.e. strings aren't cut off
> at the first null byte?
>
> Regards,
> Nik
>
> ------------------------------------------------------------------------------
> All of the data generated in your IT infrastructure is seriously valuable.
> Why? It contains a definitive record of application performance, security 
> threats, fraudulent activity, and more. Splunk takes this data and makes 
> sense of it. IT sense. And common sense.
> http://p.sf.net/sfu/splunk-d2d-c2
> _______________________________________________
> ctypes-users mailing list
> ctypes-users <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ctypes-users

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security 
threats, fraudulent activity, and more. Splunk takes this data and makes 
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
Permalink | Reply | 1 comment |
headers
Mads Kiilerich | 26 Jun 2011 23:57

Re: Strings with null characters

Dominik George wrote, On 06/26/2011 04:10 PM:
> Hi,
>
> here is an example code that breaks:
>
> from ctypes import *
> class a(Structure):
>    _fields_ = [("aaa", c_char * 14)]
>
> mya = a()
> mya.aaa = "a\0b"
> print mya.aaa
> 'a'
>
>
> I found two workarounds at
> http://stackoverflow.com/questions/5082753/how-do-i-build-a-python-string-from-a-ctype-struct
> , but those solutions look somehow nasty ...

ctypes handles string-like struct fields in a special friendly way that 
often isn't very useful. Char arrays can normally be accessed 
zero-terminated through .value or directly through .raw - but not when 
they are used as struct members. That is a nasty problem that 
unfortunately requires a nasty solution.

One way to work around the issue is to avoid char arrays in structs and 
use byte arrays instead - and cast them to char arrays on demand as 
shown by this example:

from ctypes import *
class a(Structure):
   _fields_ = [("x", c_char), ("aaa", c_byte * 14)]
def chars(x):
   return cast(addressof(x), POINTER(c_char * sizeof(x))).contents
def dump(x):
   print repr(chars(x).raw)
mya = a()
dump(mya)
chars(mya.aaa).raw = "a\0b"
dump(mya)

or to make it a bit more like the stackoverflow example:

class a(Structure):
   _fields_ = [("x", c_char), ("_aaa", c_byte * 14)]
    <at> property
   def aaa(self):
     return chars(self._aaa)
mya = a()
mya.aaa.raw = "a\0b"

/Mads

------------------------------------------------------------------------------
All of the data generated in your IT infrastructure is seriously valuable.
Why? It contains a definitive record of application performance, security 
threats, fraudulent activity, and more. Splunk takes this data and makes 
sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-d2d-c2
Permalink | Reply | 0 comments |

Gmane