James Bielman | 1 Jan 02:37 2006

Re: Re: CFFI Callbacks on SBCL

On Sat, 2005-12-31 at 17:59 +0000, Luís Oliveira wrote:
> I'm Cc-ing cffi-devel and sbcl-devel, since I'm sure more people will  
> be interested in this and hopefully others will help in figuring this  
> out.

Here's another example I cooked up before I read Luis's e-mail:

This is SBCL 0.9.8, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information. 
* (sb-alien::define-alien-callback foo int () 42)
; [snip unused var warning]
; compilation unit finished 
;   caught 1 STYLE-WARNING condition

FOO
* (alien-funcall foo)

42
* (save-lisp-and-die "foo.core")
[undoing binding stack and other enclosing state... done]
[saving current Lisp image into /home/jamesjb/foo.core:
writing 1664 bytes from the read-only space at 0x01000000
writing 1776 bytes from the static space at 0x05000000
writing 23977984 bytes from the dynamic space at 0x09000000
sdone]
(Continue reading)

James Bielman | 1 Jan 11:11 2006

Type Translator Branch Update

Hi all,

I've updated the cffi-new-translators branch, improving the exported
interface and fixing a few bugs.  The branch is available via Darcs (or
a browser to browse the source) at:

   http://slacknet.com/~jamesjb/cffi-new-translators

The exported interface for defining type translators is by specializing
the following generic functions:

Generic Function: TRANSLATE-TO-FOREIGN value type-name

Convert the Lisp object VALUE to a foreign object of the type named by
TYPE-NAME, which is a symbol naming a type defined by DEFCTYPE.  Methods
on this generic function should EQL-specialize TYPE-NAME for the type to
translate.

Returns the translated foreign object and an optional second value that
will be passed as the PARAM argument to FREE-TRANSLATED-OBJECT.

Generic Function: TRANSLATE-FROM-FOREIGN value type-name

Convert the foreign object VALUE of the type named by TYPE-NAME, which
is a symbol naming a type defined by DEFCTYPE, to a Lisp object.
Methods on this generic function should EQL-specialize TYPE-NAME for the
type to translate.

Generic Function: FREE-TRANSLATED-OBJECT value type-name param

(Continue reading)

Thomas F. Burdick | 1 Jan 12:04 2006
Picon

Re: Re: CFFI Callbacks on SBCL

On 12/31/05, Juho Snellman <jsnell <at> iki.fi> wrote:
> <luismbo <at> gmail.com> wrote:
> > We're not even testing if the callback moves or not. Our libtest is
> > not saving the callbacks' address.
> >
> > Here's the a testcase, using sb-alien (sort of):
>
> Thanks. The problem is that the callback support assumes that
> #'SB-VM::ENTER-ALIEN-CALLBACK can never move [*]. Which is no
> longer true on x86 and x86-64 as of a few releases ago, since
> everything is in the dynamic space by default, and will get
> relocated when the core is saved [**].

Okay, that makes sense now.  Any pointers to the discussion of why
purify doesn't happen by default anymore on those platforms? 
(curious).

> A workaround is to run PURIFY once before creating any callbacks. The
> proper solution would be adding a level of indirection into the
> generated callback wrappers. Maybe make 'SB-VM::ENTER-ALIEN-CALLBACK a
> static symbol, so that it will be guaranteed to not move, and then
> emit instructions to peek into its symbol-function slot.

The layout of symbols is unlikely to change soon, so that will work
fine.  But another option would be to cater to what the trampolines do
now and move the ENTER-ALIEN-CALLBACK function to static space.

-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
(Continue reading)

Juho Snellman | 1 Jan 12:29 2006
Picon
Picon

Re: Re: CFFI Callbacks on SBCL

On Sun, Jan 01, 2006 at 12:04:40PM +0100, Thomas F. Burdick wrote:
> On 12/31/05, Juho Snellman <jsnell <at> iki.fi> wrote:
> > Thanks. The problem is that the callback support assumes that
> > #'SB-VM::ENTER-ALIEN-CALLBACK can never move [*]. Which is no
> > longer true on x86 and x86-64 as of a few releases ago, since
> > everything is in the dynamic space by default, and will get
> > relocated when the core is saved [**].
> 
> Okay, that makes sense now.  Any pointers to the discussion of why
> purify doesn't happen by default anymore on those platforms? 
> (curious).

It's a performance thing. The static space doesn't have a write
barrier, and so needs to be scavenged completely on every GC.

<http://ww.telent.net/diary/2003/12/#21.34500>
<http://www.iki.fi/jsnell/blog/archive/2005-05-12.html>

> The layout of symbols is unlikely to change soon, so that will work
> fine. But another option would be to cater to what the trampolines do
> now and move the ENTER-ALIEN-CALLBACK function to static space.

I don't think we have any simple way of moving a single function into
the static space. Attached patch implements my suggestion on x86 and
x86-64 (with the exception that it uses the symbol-value slot, since
the symbol-function slot doesn't actually exist).

--

-- 
Juho Snellman
(Continue reading)

Hoehle, Joerg-Cyril | 2 Jan 11:26 2006

Re: Calling function expecting char**

James Bielman wrote:
>I wonder if MEM-AREF might be
>better given a more generic name and documented as the default accessor
>for foreign pointers.
I think mem-AREF is fine, since (apropos "AREF") will find it, and it nicely fits array access.
Yet I also thought MEM-REF's offset-as-bytes needs better visibility (presence in the user's mind &
documentation). That's precisely why I submitted one patch to the documentation on 2005-12-23. Let's
hope it will be enough and this issue will not become a FAQ or a FMB (frequently made bug).

I think it's perfectly fine for MEM-REF to count in bytes. Such a low-level operator is needed.

>  I don't think user code will need to use the more
>primitive features of MEM-REF directly very often...
What about dereferencing a pointer (not as a slot in a struct)? mem-ref is the only candidate for that.

BTW, thanks to Yaroslav for checking whether the documentation works. Every project needs such contributions.

Regards,
	Jorg Hohle.
Hoehle, Joerg-Cyril | 2 Jan 11:44 2006

Re: Providing access to errno in CFFI-SYS

James Bielman wrote:
>While working through my fancy gettimeofday CFFI example, I've added a
>type translator used as the return type for standard C functions that
>return a negative integer on error and set 'errno'.

A very useful type indeed. I'd like to hear about its design.
Would a result of -1 raise a Lisp error?

[ok, I'm checking the sources myself, "darcs pull" ...]
I think you did TRT by throwing an error in that case. Yet possibly some people might object throwing errors
out of "low-level" FFI code, as it requires them to be very careful about freeing all resources...

May I suggest you defined a CLCS condition, so there's
a) a known slot where to get the errno value (instead of pulling it out of a string)
b) a known condition and a possibility to hook in with HANDLER-BIND etc.?

I believe exporting an interface to errno from CFFI will avoid many headaches to users (writing unportable
or otherwise broken code to get its value). Please go ahead and include this stuff in the main tree, not examples/.

>(minusp value)
I think you should specifically check for -1.  Some functions could return a legal value such large that it
overflows an int and becomes negative.
Actually, I wonder whether the DWIM pattern of all these functions shouldn't have been: "it returns an
unsigned int, except for (unsigned-int)(-1) which is taken to mean `error'".
E.g. lseek() etc. (although I don't know whether Linux supports files of length >2GB, but lseek is just an example).

BTW, I find the name syscall-result misleading. What about int-or-errno? (Or rather, to keep with my own
advice: uint-or-errno)?
Or, like the boolean generic type, write a
(define-foreign-type maybe-errno (&optional (base-type int))?
(Continue reading)

Hoehle, Joerg-Cyril | 2 Jan 12:29 2006

Re: Branch: New Type Translator Interface

James Bielman wrote:
>I've done yet-another rewrite of the type translator 
>interface.  It does
>everything through generic functions at run-time, which allows us to
>specialize on both the Lisp object being converted, and the 
>foreign type we are converting to.

How does this interact with compiler macros and the wish for partial evaluation at compile-time?
This remembers me of the AMOP book which carefully distinguishes things possibly known at compile or
macro-expansion time from the non-optimizable run-time path.  It's design seems at odd which your
approach of mixing both: the AMOP is very careful to separate the possibly known paths (depending on the
type) from the unknown (depending on the value) and as a result, separates functions that operate on types only.

Regards,
	Jorg Hohle.
James Bielman | 2 Jan 12:47 2006

Re: a thought on string encodings

On Thu, 2005-12-22 at 18:50 +0100, Hoehle, Joerg-Cyril wrote:

> I hope encoding stuff will be the next great addition to CFFI. Here's some vague idea I once had.
> I got the impression that there are (at least) two types of functions:
>  - one where the conversion depends on whatever dynamic calling    context
>  - another where the conversion is fixed, i.e. depends on the function only, not on the caller (but possibly
on the library).
> 
> Given CFFI's post transformers, I suspect that there's an opportunity to model both kinds of functions, i.e.
>  - some where defcfun expands to defaults of custom:*foreign-encoding* (in CLISP speak)
>  - some where the wrappers within defcfun impose a given encoding, e.g. ASCII, ISO-8859-1, UTF-8 or UTF16.

Hi Jörg,

I've started thinking about this.  To demonstrate the new type
translator interface, I'm working on (to begin with), a UTF8-STRING type
which converts Lisp strings to/from UTF-8 on Unicode Lisps.

I want to implement this efficiently in CLISP, so I want to be sure I
use optimized C primitives as much as possible.  I think I have a fairly
efficient method for conversion to a foreign string:

#+clisp
(defmethod translate-to-foreign ((s string) (name (eql 'utf8-string)))
  (ffi:with-foreign-string (ptr chars bytes s :encoding charset:utf-8)
    (declare (ignore chars))
    (let ((buf (foreign-alloc :unsigned-char :count bytes)))
      (memcpy buf ptr bytes)
      (values buf t))))

(Continue reading)

James Bielman | 2 Jan 12:53 2006

Re: Branch: New Type Translator Interface

On Mon, 2006-01-02 at 12:29 +0100, Hoehle, Joerg-Cyril wrote:
> James Bielman wrote:
> >I've done yet-another rewrite of the type translator 
> >interface.  It does
> >everything through generic functions at run-time, which allows us to
> >specialize on both the Lisp object being converted, and the 
> >foreign type we are converting to.
> 
> How does this interact with compiler macros and the wish for partial
> evaluation at compile-time?

If the (compiler-)macros that are responsible for calling these generic
functions are able to parse the type at macroexpansion-time (because it
is constant), they can avoid the generic function overhead if the type
is a built-in type (because the user is not allowed to define
translators on built-in types).  I think this handles the most critical
path for optimization.

If the type must be evaluated, or it is a non-built-in type, there isn't
an easy way to avoid the call to the translator GF, even if it would end
up simply returning the value unchanged.

James
James Bielman | 2 Jan 13:02 2006

Re: Providing access to errno in CFFI-SYS

On Mon, 2006-01-02 at 11:44 +0100, Hoehle, Joerg-Cyril wrote:

> May I suggest you defined a CLCS condition, so there's
> a) a known slot where to get the errno value (instead of pulling it
> out of a string)
> b) a known condition and a possibility to hook in with HANDLER-BIND
> etc.?
> 
> I believe exporting an interface to errno from CFFI will avoid many
> headaches to users (writing unportable or otherwise broken code to get
> its value). Please go ahead and include this stuff in the main tree,
> not examples/.

Yeah, I would like to do this as well, but I'm still figuring out how to
get access to 'errno' in all the supported Lisps.

> >haven't found anything in Allegro CL or CLISP (Jörg?) for this yet.
> See clisp/modules/bindings/glibc/linux.lisp as a source of inspiration
> & read the comments about errno there.  It's hairy indeed.

I'm not an expert CLISP user---is this not built as a part of the
standard image?  I went APROPOSing around for "errno" and didn't find
anything.

James

Gmane