Frank Goenninger | 2 Dec 18:36 2011

defcallback with &rest

Hi all!

Question:

I have a Lisp function as follows:

(defun frgo (fmt &rest args)
 (format t fmt args))

Now I want to define a C call back for that function:

(defcallback frgoc :void ((fmt :string) (args ???????

How do I define the callback such that I can have a C function with varargs being defined?

Thx!

Cheers
    Frank
Luís Oliveira | 2 Dec 19:46 2011
Picon

Re: defcallback with &rest

On Fri, Dec 2, 2011 at 5:36 PM, Frank Goenninger <frgo <at> me.com> wrote:
> (defun frgo (fmt &rest args)
>  (format t fmt args))
>
> Now I want to define a C call back for that function:
>
> (defcallback frgoc :void ((fmt :string) (args ???????
>
> How do I define the callback such that I can have a C function with varargs being defined?

Sorry, no way to do that right now. Last time I checked, no Lisp's got
support for such a feature. We might start using libffi to add extra
(low-level) features in the near feature, but not even libffi supports
this particular feature.

--

-- 
Luís Oliveira
http://r42.eu/~luis/
Frank Goenninger | 2 Dec 19:55 2011

Re: defcallback with &rest

Am 02.12.2011 um 19:46 schrieb Luís Oliveira:

> On Fri, Dec 2, 2011 at 5:36 PM, Frank Goenninger <frgo <at> me.com> wrote:
>> (defun frgo (fmt &rest args)
>>  (format t fmt args))
>> 
>> Now I want to define a C call back for that function:
>> 
>> (defcallback frgoc :void ((fmt :string) (args ???????
>> 
>> How do I define the callback such that I can have a C function with varargs being defined?
> 
> Sorry, no way to do that right now. Last time I checked, no Lisp's got
> support for such a feature. We might start using libffi to add extra
> (low-level) features in the near feature, but not even libffi supports
> this particular feature.

Understood. So I 'm back to square 1 with just a pointer to a struct that I have to de-structure myself. Not
beautiful but ok. Thanks for the quick help.

Cheers
    Frank
Frank Goenninger | 5 Dec 00:21 2011

Dynamically call a certain function in a lib

Hi, 

I have:

(defun load-and-open-module (pathname)
  (let ((lib (cffi:load-foreign-library pathname)))
    (if lib
	(progn
	  (format t "~%LOAD-AND-OPEN-MODULE: ~s LOADED." lib)
	  (let ((handle (cffi:foreign-funcall `("open" :library ,lib) :pointer))) ;; <<--- Here's the
challenge ...
	    (if (not (eql handle (cffi:null-pointer)))
	        (let ((module (make-instance 'pib-module
					     :lib (c-in lib)
					     :handle (c-in handle))))
		  (format t "~%LOAD-AND-OPEN-MODULE: module = ~s." module)
		  module)
		nil)
	  nil)))))

The marked line should load the "open" function in the just loaded foreign lib. As I intend to load multiple
so-called modules where each module has an "open" function I /think/ I need to say which open() function to
call (the loaded libs are C DLLs, mostly).

How do I name a certain foreign lib in the call to foreign-funcall when the library given is not evaluated ?

Thanks for any hints!

Cheers
    Frank
(Continue reading)

Anton Kovalenko | 5 Dec 00:42 2011

Re: Dynamically call a certain function in a lib

Frank Goenninger <frgo <at> me.com> writes:

> The marked line should load the "open" function in the just loaded
> foreign lib. As I intend to load multiple so-called modules where each
> module has an "open" function I /think/ I need to say which open()
> function to call (the loaded libs are C DLLs, mostly).
>
> How do I name a certain foreign lib in the call to foreign-funcall
> when the library given is not evaluated ?

Many CFFI backends have CFFI-FEATURES:FLAT-NAMESPACE, which you can (or
can't) see in *features*. If you see it present after loading CFFI in
your CL implementation, it means that underlying /native/ FFI is unable
to work with multiple functions having the same name in different
modules. If there is no CFFI-FEATURES:FLAT-NAMESPACE in *FEATURES*, you
can use something like (CFFI:FOREIGN-FUNCALL ("open" :library
my-module)), but it's fantastically unportable, and you'll have to
DEFINE-FOREIGN-LIBRARY for each module in use.

Fortunately, CFFI:FOREIGN-FUNCALL-POINTER is widely available, so you
may solve it the following way:

* define foreign interface to dlopen [or LoadLibrary] and dlsym [or
  GetProcAddress]

* load your module by calling dlopen [or LoadLibrary]

* call dlsym(<your-module-handle>,"open") [or GetProcAddress]

* dlsym/GetProcAddress returns a pointer to function, which you may call
(Continue reading)

Liam Healy | 12 Dec 06:26 2011
Picon

Re: First stab at reviewing cffi-fsbv

Hi Luis,

On Sun, Dec 11, 2011 at 8:36 PM, Luís Oliveira <luismbo <at> gmail.com> wrote:
Hello,

I went through all the code and here are my review notes. (In org-mode
format.) I suggest we tackle them incrementally, not necessarily in
this order. Meanwhile, I'll be tackling the missing bits in the new
convert-into-foreign-memory. Should we be having this discussion in
cffi-devel btw?

Sure.  I've copied the list for anyone who's interested.


* cffi-fsbv.asd
 What happens on #-unix?
 (:cffi-grovel-file "libffi" :pathname #+unix "libffi-unix")

Absolutely nothing.  Someone named "CRLF0710" sent me an email claiming to have a libffi-win32 for the old standalone version of FSBV, and I wrote back saying I was interested but I've heard nothing.


* fsbv/package.lisp
 1. Regarding SIZET, I'd say we can define this type using a
    keyword :SIZET.
 
Yes, if you think that's OK.  That was me totally sneaking something in that GSLL needed that was awkward to generate there but that CFFI could generate easily.
 
 2. cffi-fsbv calls and hooks into so many of CFFI internals. Should
    we just use the CFFI package? I mean, the code is full of ::s.

I'm fine with that.  I don't think there'll be any symbol conflicts.
 

* fsbv/libffi-unix.lisp
 1. OSX ships with libffi. Is (cc-flags "-I/opt/local/include/")
    necessary? (I guess it doesn't hurt. Just curious. Perhaps it
    didn't ship with libffi in the past?)

Ugh, OSX.  It apparently does ship with libffi, except when it doesn't.  There are so many ways to install system software, I just gave up.  I'm all for your approach because it does simplify the code and make it match the other OSes; when the complaints come in to cffi-devel, you can answer them :-)
 

 2. Aren't :uint and :ushort equivalent to:
      (ctype ushort "unsigned short")
      (ctype unsigned "unsigned")?
 
Um, I don't know?  What are you advocating here?  Removing something that I put in?  :uint and :ushort don't ring a bell for me.
 

 3. Do we really need to grovel this stuff? Grovelling requires a C
    compiler which is particularly tricky on Windows. It'd be nice to
    avoid that requirement if possible. Didn't see anything in here
    that seemed to *require* grovelling. Did I miss something?

"Here" means libffi-unix?  The constants at the end aren't necessary, but I think the rest is.  In particular, FFI_DEFAULT_ABI (abi) and FFI_OK (status) are pretty critical, as is the struct ffi-type.  I'm a little confused; I thought the point of cffi-grovel was to provide access to definitions in .h files, and that's what we need for ffi.h and ffi-target.h.  How do you propose defining these without groveling?  Just hardwiring into lisp and hoping no one ever changes those values?

By the way I have FFI_STDCALL commented out and didn't try to mate it with CFFI's definition on Windows, perhaps CRLF0710 or another Windows user can figure that out.


* fsbv/build-in-type.lisp
 1. Could we use DEFCVAR here to declare the various ffi_type_*
    globals? (I think it'd be slightly clearer.)
 
I kind of like the automatic way of doing it; manually means that every time something changes (new type added or old one removed) you have to remember to do it in two places.  That's a recipe for breakage.
 
 2. Why do we have to cache pointers for FOREIGN-TYPE-ALIAS and
    FOREIGN-ENUM? (But see my comments on fsbv/cstruct.lisp)

We probably don't.
 
 3. Why different LIBFFI-TYPE-POINTER for those two types?
    FOREIGN-ENUM inherits from FOREIGN-TYPE-ALIAS.

Is it really different, or is it just getting the pointer from the underlying type?


* fsbv/cstruct.lisp
 1. slots-in-order should be moved near FOREIGN-STRUCT-TYPE.
 2. I think we should move libffi-type-pointer out of FOREIGN-TYPE
    and centralize storage in an hash-table.
    a) this simplifies implementation a little bit: it's weird that
       pretty much all libffi-type-pointer methods have :AROUND
       qualifiers.
    b) makes it easy to avoid memory leaks when redefining/reloading
       structure types. (Although this sort of leak is probably not a
       huge issue, it's certainly inelegant.)
 3. The code that creates ffi_types for structs could use some
    refactoring. :-) I'll give it a go.

Sure.
 

* fsbv/functions.lisp
 1. I think this file can benefit from a little bit of refactoring.
 2. I *think* we can move the indirection logic from the translation
    methods onto FFCALL-BODY-LIBFFI, but it's not yet clear how that
    can happen.

Interesting thought, I'd like to hear your ideas.
 

* fsbv/example.lisp
 1. Need to clear the dead code and move it to examples/.

examples.lisp?  It's probably all dead code now, just get rid of the whole thing.
 

* src/functions.lisp
 1. We can add a neat restart to *foreign-structures-by-value*.

Yes I think you mentioned this idea on the todo list.  I expect the restart is "load cffi-libffi"?
 

* src/structures.lisp
 1. As I've mentioned before, we should discuss how we can implement
    this sort of functionality in a backwards-compatible way.

I'd prefer not defaulting to previous assumptions (that is, no translation of structures), if that's what backwards-compatible means.  But I'm interested in how you'd approach this.

* src/early-types.lisp
 1. missing expand-into-foreign-memory, and all the hooking up that
    implies. Started working on that.
 2. I think the :indirect keyword is delegating the responsibility to
    the wrong place. All the type translations (in CFFI or in other
    libraries) would have to be updated accordingly! As mentioned in
    the fsbv/functions.lisp notes, I think we can handle the
    indirection there. (It might require a slightly different way of
    hooking up ffcall-body-libffi with foreign-funcall.)

No doubt.  I would like to avoid special-casing though, e.g. "here's an enum, got to indirect it".  I think that kind of thing belongs in a method associated with the type.
 
* Nomenclature
 I can forsee using libffi to deal with stuff like callbacks, long
 long, varargs, etc. Should we rename cffi-fsbv to cffi-libffi?

Sure.  Also, you mentioned that some implementations have their own call-by-value mechanism, so it makes more sense to rename it, since they would have "fsbv" but no libffi.
 

* Wishlist
*** foreign-funcall-varargs and friends could use libffi
   ... to do proper varargs.
*** defcallbacks with structures...

Yup.

Liam

--
Luís Oliveira
http://r42.eu/~luis/

_______________________________________________
cffi-devel mailing list
cffi-devel <at> common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/cffi-devel
Stelian Ionescu | 18 Dec 05:13 2011

CFFI foreign library error pathname patch

Xach requested better error reporting from load-foreign-library, so
please review this patch

--

-- 
Stelian Ionescu a.k.a. fe[nl]ix
Quidquid latine dictum sit, altum videtur.
http://common-lisp.net/project/iolib

_______________________________________________
cffi-devel mailing list
cffi-devel <at> common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/cffi-devel
Luís Oliveira | 18 Dec 12:21 2011
Picon

Re: CFFI foreign library error pathname patch

On Sun, Dec 18, 2011 at 4:13 AM, Stelian Ionescu <sionescu <at> cddr.org> wrote:
> Xach requested better error reporting from load-foreign-library, so
> please review this patch

So now we consistently get a load-foreign-library-error instead of
getting a simple-error for some of the errors. Storing the pathname in
the error seems like a good idea too. Is there a specific use-case
that motivated that change? I'm asking because it's not populated
correctly in the try-foreign-library-alternatives case, and it's not
clear what should go there in that case. Perhaps we should store
something else? (The foreign-library object perhaps?)

--

-- 
Luís Oliveira
http://r42.eu/~luis/
Luís Oliveira | 18 Dec 13:35 2011
Picon

Re: CFFI foreign library error pathname patch

On Sun, Dec 18, 2011 at 12:17 PM, Zach Beane <xach <at> xach.com> wrote:
> The use-case I had in mind was an environment that caught the error and
> offered a useful action, e.g. "install the library via
> apt/homebrew/macports/etc and continue".

OK, and so you'd like to get a hold of
"/maybe/some/path/to/libfoo.so.x" or whatever. Hmm, so maybe in
try-foreign-library-alternatives we could store the first alternative
since that should usually be something like "libfoo.so.x".

What do you think Stelian? (Also, did I miss any other improvements
introduced by this patch?)

--

-- 
Luís Oliveira
http://r42.eu/~luis/
Luís Oliveira | 18 Dec 14:02 2011
Picon

Re: CFFI foreign library error pathname patch

On Sun, Dec 18, 2011 at 12:40 PM, Zach Beane <xach <at> xach.com> wrote:
>> OK, and so you'd like to get a hold of
>> "/maybe/some/path/to/libfoo.so.x" or whatever. Hmm, so maybe in
>> try-foreign-library-alternatives we could store the first alternative
>> since that should usually be something like "libfoo.so.x".
>
> It would be nice to have all the alternatives.

We could store the definition s-exp.

;;; (define-foreign-library opengl
;;;   (:darwin  (:framework "OpenGL"))
;;;   (:unix    (:or "libGL.so" "libGL.so.1"
;;;                  #p"/myhome/mylibGL.so"))
;;;   (:windows "opengl32.dll")
;;;   ;; an hypothetical example of a particular platform
;;;   ((:and :some-system :some-cpu) "libGL-support.lib")
;;;   ;; if no other clauses apply, this one will and a type will be
;;;   ;; automagically appended to the name passed to :default
;;;   (t (:default "libGL")))

In this case, the error's LIBRARY-SPECIFICATION slot (or something
along those lines), could hold a string, a pathname, or s-exps whose
CARs are one of :OR, :DEFAULT, or :FRAMEWORK.

But it's a bit messy. Wouldn't it be easier to deal with a single
string? At first sight, it seems unlikely that some of the
alternatives in an :OR clause would hold more project information than
others. (Except perhaps for the version, which should be in the first
alternative, but sometimes isn't.)

In this example, said string would be
"/path/to/OpenGL.framework/something/something.dylib", "libGL.so",
"opengl32.dll" on OS X, Unix, and Windows, respectively.

--

-- 
Luís Oliveira
http://r42.eu/~luis/

Gmane