Carlos Pita | 9 Jun 19:49 2016
Picon
Gravatar

Pickling ctypes pointer/array

Hi all,

this  question was triggered by the suggestion of Eryk in http://bugs.python.org/issue27274. Currently my way to go in order to pickle/unpickle the underlying array of a pointer is:

ptr --cast--> array --copy--> bytes --dump--> pickle
pickle --load--> bytes --(from_buffer_)copy --> array --cast--> ptr

Eryk suggested a more concise way of achieving the same:

ptr --string_at --> bytes --dump--> pickle
pickle --load--> bytes --> create_string_buffer --> ptr

(where I'm taking create_string_buffer as a kind of inverse of string_at).

But this still makes a couple of copies. There are at least two obstacles on the way to pickling without copies:

1. Pickle is unable to serialize memoryviews.
2. It's not desirable to expose a pointer to the underlying memory of a bytes object, since bytes are supposed to be immutable.

Now, what about bytearrays? If the underlying buffer of ctypes objects were get/settable as (views to) bytearrays no immutability constraint would be violated and we could move between python and c worlds without copying.

Is something like that already possible? If not, is there an essential limitation preventing its implementation?

Thank you in advance
--
Carlos




------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
ctypes-users mailing list
ctypes-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ctypes-users
jeff.esquivel@gmail.com | 8 Jun 00:06 2016
Picon

Using ctypes inside a C application embedding the CPython runtime

Hi,

We have a python server application that will load a shared library (using ctypes) and make its functions available to a client application on another host.

Now we're been asked to embedded that application into a larger C application that also loads the same library (the goal is to let the C application turn on and off the server as needed). We used the CPython runtime to embed the python application inside the C application successfully, using the documentation available here[0].

The problem is that right now we make sure that our python application is using the same library (in memory) as the C application by passing None as the name parameter to the CDLL constructor, which means it ultimately calls dlopen() with Null as it's filename parameter, having the following effect:

If filename is NULL, then the returned handle is for the main program. When given to dlsym(), this handle causes a search for a symbol in the main program, followed by all shared objects loaded at program startup, and then all shared objects loaded by dlopen() with the flag RTLD_GLOBAL.
As documented on dlopen's manpage[1], and this is working right now. What we're worried about is that passing None to the CDLL constructor is not documented anywhere (at least that I could find), so we're not sure if we can rely on this functionality (if so, the documentation should probably be updated and I can help with that if needed) or if there's an alternate recommended and documented way of achieving the same result.

Thank you very much for your time.


Regards,
------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic
patterns at an interface-level. Reveals which users, apps, and protocols are 
consuming the most bandwidth. Provides multi-vendor support for NetFlow, 
J-Flow, sFlow and other flows. Make informed decisions using capacity 
planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e
_______________________________________________
ctypes-users mailing list
ctypes-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ctypes-users
Glenn Pierce | 3 Mar 09:32 2016
Picon

Passing self though to callbacks

Hi I am having trouble getting self passed through to callbacks.
I know ctypes usually takes care of this but in my case it does not
because the c API I am wrapping requires the function pointers to be
placed into a structure
which then is passed to the function that sets up the connection. I
believe this stops the ctypes self propagation from working.
Is there a way to get it working in this case  ?

An example is below.

The functions I have wrap except a context which is just callback
data. So I have tried passing id through this
like

self.handle = self._Create(C.byref(self.callback_structure),
C.c_void_p(id(self)))

in the callback I cast back like

self = C.cast(context, C.py_object).value

This does not seem to work well though. With one object it is find but
when adding different callback objects of the same style the cast back
sometimes returns the wrong object.
Not sure why though. Neither object gets garbage collected so id
should be valid unique.

I would try to pass a unique int / dict mapping though context but how
can I pass an int through a void*   ?

If anyone has any suggestions that would be great. Thanks

if sys.platform == 'win32':
    CallbackType = C.WINFUNCTYPE
else:
    CallbackType = C.CFUNCTYPE

SessionListenerSessionLostFuncType = CallbackType(None, C.c_void_p,
C.c_int, C.c_int) # context sessionId reason

class SessionListenerCallBacks(C.Structure):
    _fields_ = [
                ("SessionLost", SessionListenerSessionLostFuncType)
               ]

class SessionListener(object):

    def __init__(self, callback_data=None):

        super(SessionListener, self).__init__()

        self.callback_structure = SessionListenerCallBacks()
        self.callback_data = callback_data

        self.callback_structure.SessionLost =
SessionListenerSessionLostFuncType(SessionListener._OnSessionLostCallBack)
        self.callback_structure.SessionMemberAdded =
SessionListenerSessionMemberAddedFuncType(SessionListener._OnSessionMemberAddedCallback)

        # I have remove the wrapper for this for simplicaity
        # The prototype is # _create(const sessionlistener_callbacks*
callbacks, const void* context);
        self.handle = self._Create(C.byref(self.callback_structure),
C.c_void_p(id(self)))

    def __del__(self):
        if self.handle:
            return self._Destroy(self.handle)

     <at> staticmethod
    def  _OnSessionLostCallBack(context, sessionId, reason):
        self = C.cast(context, C.py_object).value
        self.OnSessionLostCallBack(self.callback_data, sessionId, reason)

    def OnSessionLostCallBack(self, context, sessionId, reason):
        pass

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
Yang Liu | 22 Feb 08:52 2016
Picon

why sturct size different between C and Python ctypes?

Hi,all

I am using ctypes today and a sturct size is differet in C and Python ctypes code
here is my code

C: 
#include<linux/netlink.h>
#include<linux/inet_diag.h>

int main(int argc, char *argv[])
{
    struct {
        struct nlmsghdr      nlh;
        struct inet_diag_req req;
     } nlreq;
    printf("nlreq: %d\n", sizeof(nlreq));
    return 0;
}

out:
nlreq: 76


Python:
class nlmsghdr(ctypes.Structure):
    _fields_ = [
        ("len", ctypes.c_uint32),
        ("type", ctypes.c_uint16),
        ("flags", ctypes.c_uint16),
        ("seq", ctypes.c_uint32),
        ("pid", ctypes.c_uint32)]

class Inet_diag_sockid(ctypes.Structure):
    _fields_ = [
        ("idiag_sport", ctypes.c_uint16.__ctype_be__),
        ("idiag_dport", ctypes.c_uint16.__ctype_be__),
        ("idiag_src", ctypes.c_uint32.__ctype_be__ * 4),
        ("idiag_dst", ctypes.c_uint32.__ctype_be__ * 4),
        ("idiag_if", ctypes.c_uint32),
        ("idiag_cookie", ctypes.c_uint32 * 2),
    ]

class inet_diag_req(ctypes.Structure):
    _fields_ = [
        ("idiag_family", ctypes.c_uint8),
        ("idiag_src_len", ctypes.c_uint8),
        ("idiag_dst_len", ctypes.c_uint8),
        ("idiag_ext", ctypes.c_uint8),
        ("id", ctypes.POINTER(Inet_diag_sockid)),
        ("idiag_states", ctypes.c_uint32),
        ("idiag_dbs", ctypes.c_uint32)
    ]

class nlmsg(ctypes.Structure):
    _fields_ = [
        ("nlh", nlmsghdr),
        ("req", Inet_diag_req)
    ]

print ctypes.sizeof(nlmsg)

out: 
24


why the same struct size is 76 inC and 24 in ctypes ?

Thanks
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
ctypes-users mailing list
ctypes-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ctypes-users
Carlos Pita | 6 Feb 15:25 2016
Picon
Gravatar

Structure inheritance and tail padding

Hi all,

just from observation I arrived to the unconfirmed conclusion that
Structure inheritance adds tail padding. For example:

class A(Structure):
  _fields_ = [('a', c_double), ('b', c_char)]

class B(A):
  _fields_ = [('c', c_int)]

will not show the same layout than:

class C(Structure):
  _fields_ = [('a', c_double), ('b', c_char), ('c', c_int)]

Supposing my conclusion is right, what is the intention of this feature:

i. To support c++ pod subclassing? (most compilers reuse the tail
padding for non-pod subclassing).

ii. To support a more oo view of c struct composition? (btw, it
complements the ms-extensions very well).

Thank you in advance
--
Carlos

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
Carlos Pita | 26 Jan 03:37 2016
Picon
Gravatar

resize content, not type

Hi all,

for the resize function the ctypes tutorial states:

> Another way to use variable-sized data types with ctypes is to use the dynamic
> nature of Python, and (re-)define the data type after the required size is already
> known, on a case by case basis.

What does this mean? I tried monkey-patching the __class__ attribute,
also changing the _length_ attribute, but to no avail. This is not
surprising as the relevant length attribute is part of the C-side
object state, which doesn't seem to be exposed to the python API and
won't change just because I changed the type after construction.

Another option is to resize and then create a new array from_address,
but it won't own the memory so it's a problematic approach.

Cheers
--
Carlos

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
Andrew Robert Whalley | 21 Dec 13:48 2015
Picon

Library Import

Hi All,

I would like to use Python / ctypes to test a C library that contains 
some dependencies. Is it possible to satisfy those dependencies from 
within the Python interpreter, by using ctypes to define callables 
(similar to the callback function mechanism)? Or do any dependencies 
have to be defined in other, loadable, libraries?

Regards,

Rob Whalley

------------------------------------------------------------------------------
Cristian Badescu | 28 Oct 13:31 2015
Picon

NtQuerySystemInformation

Hello,

I am trying to get handles information by using NtQuerySystemInformation but my code fails for some reason. I am using the following code to call NtQuerySystemInformation:

from ctypes import *
from ctypes.util import *

class SYSTEM_HANDLE(Structure):

_fields_ = [("ProcessId", c_ulong),
("ObjetTypeNumber", c_byte),
("Flags", c_byte),
("Handle", c_ushort),
("Object", c_void_p ),
("GrantedAccess", c_ulong)
]

class SYSTEM_HANDLE_INFORMATION(Structure):

_fields_ = [("HandleCount", c_ulong),
("Handles", SYSTEM_HANDLE * 1)
]

handleInfoSize = 65536

SystemHandleInformation = 16

c_lib=CDLL(find_library("c"))
malloc=c_lib.malloc
malloc.restype=c_void_p
handleInfo = malloc(handleInfoSize)

pHandleInfo = cast(handleInfo, POINTER(SYSTEM_HANDLE_INFORMATION))

retVal = cdll.ntdll.NtQuerySystemInformation(SystemHandleInformation, pHandleInfo, handleInfoSize, None)
------------------------------------------------------------------------------
_______________________________________________
ctypes-users mailing list
ctypes-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ctypes-users
TuX RaceR | 19 Aug 22:18 2015
Picon

Python ctypes program works on 3.2 but fails on 3.4

Hello List

I suspect a bug in ctypes;
I asked the question below on stackoverflow:
http://stackoverflow.com/questions/32100067/python-ctypes-program-works-on-3-2-but-incompatiable-on-3-4

Any idea of why the same code would fail (return a blank string) on 3.4?

Thanks
TR

------------------------------------------------------------------------------
Jan Wagner | 13 Aug 07:19 2015

contribute ctypeslib patch?

Hi list, it seems this project is quite dead and I'm not sure how to contribute a feature fix (not even to https://pypi.python.org/pypi/ctypeslib/). Perhaps someone on the list is able to do it? There is an issue in ctypeslib codegenerator.py that affects xml2py.py option "-r" for which usage says that "-r EXPRESSION regular expression for symbols to include". The expression is actually never evaluated as a regular expression. Only exact name matches are selected. The below patch changes the behaviour to be consistent with the usage instructions (and makes -r behaviour more useful). cheers, Jan --- /usr/lib/pymodules/python2.5/ctypeslib/codegen/codegenerator.py 2009-01-17 03:50:18.000000000 +0900 +++ codegenerator.py 2015-04-08 18:34:06.000000000 +0900 <at> <at> -859,6 +859,12 <at> <at> if match and match.group() == i.name: todo.append(i) break + # above selects only *exact* matches, now include partial/regexp name matches: + match = s.search(i.name) + if match: + todo.append(i) + break + if symbols or expressions: items = todo
------------------------------------------------------------------------------
_______________________________________________
ctypes-users mailing list
ctypes-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ctypes-users
Timothy W. Grove | 19 Mar 17:11 2015

Calling Objective C functions with ctypes

I have been trying to get the following code copied from another forum to work on my system (OSX 10.6, 10.8, Python 3.3 installed via MacPorts), and it just won't. It seems to be finding the 'AppKit' and 'objc' libraries okay, but it is not getting the classes 'NSSpeechSynthesizer' or 'NSAutoreleasePool', so 'count' ends up as '0' and everything falls over from there.

I did notice from the "print voiceNames" statement that this code must be pre-python 3, so are there any obvious changes needed to bring this up to date? Thanks for any help or suggestions.

Best regards,
Tim Grove
[from http://stackoverflow.com/questions/1490039/calling-objective-c-functions-from-python/1490644#1490644]
import ctypes
import ctypes.util

# Need to do this to load the NSSpeechSynthesizer class, which is in AppKit.framework
appkit = ctypes.cdll.LoadLibrary(ctypes.util.find_library('AppKit'))
objc = ctypes.cdll.LoadLibrary(ctypes.util.find_library('objc'))

objc.objc_getClass.restype = ctypes.c_void_p
objc.sel_registerName.restype = ctypes.c_void_p
objc.objc_msgSend.restype = ctypes.c_void_p
objc.objc_msgSend.argtypes = [ctypes.c_void_p, ctypes.c_void_p]

# Without this, it will still work, but it'll leak memory
NSAutoreleasePool = objc.objc_getClass('NSAutoreleasePool')
pool = objc.objc_msgSend(NSAutoreleasePool, objc.sel_registerName('alloc'))
pool = objc.objc_msgSend(pool, objc.sel_registerName('init'))

NSSpeechSynthesizer = objc.objc_getClass('NSSpeechSynthesizer')
availableVoices = objc.objc_msgSend(NSSpeechSynthesizer, objc.sel_registerName('availableVoices'))

count = objc.objc_msgSend(availableVoices, objc.sel_registerName('count'))
voiceNames = [
  ctypes.string_at(
    objc.objc_msgSend(
      objc.objc_msgSend(availableVoices, objc.sel_registerName('objectAtIndex:'), i),
      objc.sel_registerName('UTF8String')))
  for i in range(count)]
print voiceNames

objc.objc_msgSend(pool, objc.sel_registerName('release'))
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the 
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
ctypes-users mailing list
ctypes-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ctypes-users

Gmane