Bruno Daniel | 18 Sep 13:51 2007
Picon
Picon

Is it safe to use alien-lambda with closures?

Dear developers,

I see that the function alien-lambda used by CFFI in the file
http://common-lisp.net/project/cffi/darcs/cffi/src/cffi-sbcl.lisp
is undocumented in both the CMUCL and the SBCL manuals. 
Is it safe to use this function with closures, say, in the following way?

(defmacro alien-lambda-with-cffi-types
              (rettype arg-names-and-types &body body)
  (let ((arg-names (mapcar #'first arg-names-and-types))
        (arg-types (mapcar #'second arg-names-and-types)))
  `(alien-sap
      (sb-alien::alien-lambda ,(cffi::convert-foreign-type rettype)
          ,(mapcar (lambda (sym type)
                     (list sym (cffi::convert-foreign-type type)))
                   arg-names arg-types)
        , <at> body)))

(let ((ptr nil))
  (let ((a 3))
    (setq ptr (cffi-like-alien-lambda :int ((b :int))
                (+ a b))))
  (some-c-function ptr))

Best regards
  Bruno Daniel

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
(Continue reading)

Jonathon McKitrick | 18 Sep 18:56 2007
Picon

get-mutex waitp behavior


What should happen when calling get-mutex on an unavailable mutex without
waitp?  The docstring does not seem to be clear on this, and I seem to
be getting different behaviour on darwin with sb-thread and linux.

Jonathon McKitrick
--
My other computer is your Windows box.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Bruno Daniel | 18 Sep 20:57 2007
Picon
Picon

Re: Is it safe to use alien-lambda with closures?

Dear developers,

I just tried it out and it seems to work. Here's the code with some
corrections (CFFI has to be installed and loaded):

library-test.cc:
------------------------------------------------------------
// compile with
//  g++ -fPIC -c library-test.cc && g++ -shared -Wl,-soname,library-test.so -o library-test.so library-test.o

#include <iostream>
using namespace std;

extern "C" {
  int caller_func(int a, int (*f)(int)) {
    return (*f)(a) + 5;
  }
}
------------------------------------------------------------
in Lisp:
------------------------------------------------------------
(let ()
  (define-foreign-library library-test
    (:unix "/home/daniel/pj/lisp/library-test.so"))
  (use-foreign-library library-test))

(defcfun "caller_func" :int (a :int) (f :pointer))

(defmacro alien-lambda-with-cffi-types
              (rettype arg-names-and-types &body body)
(Continue reading)

Bruno Daniel | 18 Sep 23:05 2007
Picon
Picon

Re: Is it safe to use alien-lambda with closures?

> Dear developers,
> I just tried it out and it seems to work. Here's the code with some
> corrections (CFFI has to be installed and loaded):

I thought it over and now I understand why this function is undocumented:
Using closures as callbacks will result in a serious memory leak: Since the
C code might have stored the function pointer anywhere, Lisp will never be
able to garbage collect the closure.

I must find another solution for my application.

Best regards
  Bruno Daniel

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Nikodemus Siivola | 19 Sep 13:17 2007
Picon

Re: Is it safe to use alien-lambda with closures?

On 9/18/07, Bruno Daniel <bruno.daniel <at> gmx.net> wrote:

> I thought it over and now I understand why this function is undocumented:
> Using closures as callbacks will result in a serious memory leak: Since the
> C code might have stored the function pointer anywhere, Lisp will never be
> able to garbage collect the closure.

Quite so. Constructing the callback for a closure is also
horrendously expensive compared to non-closures:

(defun foo (x)
  (alien-lambda () ... x ...))

(defun bar ()
  (alien-lambda () ...))

FOO needs to round-trip through the compiler every time, whereas
BAR does that only on the first call.

...but the reason for this being undocumented is just a feature
of me having run out of steam & time while working on this.

> I must find another solution for my application.

Most C libraries using callbacks have a separate "data" argument
to the callback you can use.

Cheers,

 -- Nikodemus
(Continue reading)

Bruno Daniel | 19 Sep 14:04 2007
Picon
Picon

Re: Is it safe to use alien-lambda with closures?

Dear Nikodemus,

thank you very much for your clarification.

> Most C libraries using callbacks have a separate "data" argument
> to the callback you can use.

Unfortunately this isn't an option for my application. What I want to do is
to improve Swig's support for Common Lisp (using CFFI). I already fixed the
overloaded functions and methods problem. Now I'm working on enabling Lisp to
subclass C++ classes in a way that makes the resulting objects usable in 
C++ libraries just as if they had been suclassed in C++ itself, and the
solution shouldn't be too inefficient.

My solution would be the following: Let Swig add a C++ subclass that receives
some Lisp callback pointers for the methods to override in the constructor.
The subclass will have overriding methods that just delegate to the callbacks
in case the pointers are non-zero and to the upper-class methods otherwise.

This might be fine, but what about additional slots in the subclass? As the
overriding methods, these are to be defined in Lisp, but how do I get the
overriding methods to reach them when they are called? Since the methods must
keep the signatures of the overridden methods, I can't add a data argument, so
only the self-argument ("this") remains. But this is a C++ object and it can't
keep references to Lisp objects since these may be moved around by the garbage
collector. I don't want to switch off the garbage collector, so I turned to
closures which also don't work out as we have seen.

In the meantime I had another idea that is not too inefficient: Instead of a
reference I'll keep an index in the C++ object. It's an index into a
(Continue reading)

Tobias C. Rittweiler | 19 Sep 14:39 2007
Picon

Re: Is it safe to use alien-lambda with closures?

Bruno Daniel <bruno.daniel <at> gmx.net> writes:

> (defmacro with-dslots (vars (obj class) &body body)
>    " ... "
>   (with-gensyms (obj-evaluated)
>     (labels
>         ((slot-ref (obj1 slot class)
>            `(,(intern (concatenate 'string (string class) "-" (string slot)))
>               ,obj1))
>         ...))))

This doesn't work for structs that are defined with :conc-name. Better
use something like SWANK::WITH-STRUCT where you have to explicitely pass
the conc-name.

> ;; Example:
> (defstruct test-struct
>   (a 0 :type mword)
>   (b 0 :type mword))
>
> (let ((teststr (make-test-struct :a 3 :b 4)))
>   (with-dslots (a b) (teststr test-struct)
>     (print (list a b))))

(with-struct (test-struct- a b)
  ..)

For (defstruct (matching :conc-name matching.)
      prefix infix suffix)

(Continue reading)

Bruno Daniel | 19 Sep 16:59 2007
Picon
Picon

Re: Is it safe to use alien-lambda with closures?

Dear Tobias,

thanks for your advice! You convinced me and I changed it everywhere in my
code.

> This doesn't work for structs that are defined with :conc-name. Better
> use something like SWANK::WITH-STRUCT where you have to explicitely pass
> the conc-name.
>
> (with-struct (test-struct- a b)
>   ..)
> 
> For (defstruct (matching :conc-name matching.)
>       prefix infix suffix)
> 
> (with-struct (matching. prefix infix suffix)
>   ...)
> 
> You may find a syntax like
> 
>   (with-struct-slots matching. (prefix infix suffix)
>     ...)
> 
> more pleasing. (I'm still undecided.)

Best regards
  Bruno Daniel

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
(Continue reading)

Bruno Daniel | 25 Sep 15:49 2007
Picon
Picon

Thread-safety of hash tables? Thread-safety and the garbage collector?

Dear developers,

is it safe to create a hash table in one thread and allow it to self-expand
(e.g. by (make-hash-table :test #'eql :size 10000
                   :rehash-size 2d0 :rehash-threshold 0.5))
and then use it from another thread? What happens if it is expanded while some
new entry is added from another thread?

What happens if the garbage collector is moving a data structure that is
simultaneously accessed from another thread? Say you have created an array 
by (setq arr (make-array 100)) and the garbage collector is moving
it while another thread executes (setf (aref arr 3) (list 1 2))?

If you don't pass an array to any other thread, will it stay untouched by the
garbage collector runs of other threads?

What is the answer to these questions if a foreign library creates threads on
its own and then calls some Lisp functions via callbacks?

Best regards
  Bruno Daniel

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Juho Snellman | 25 Sep 16:59 2007
Picon
Picon

Re: Thread-safety of hash tables? Thread-safety and the garbage collector?

Bruno Daniel <bruno.daniel <at> gmx.net> writes:

> Dear developers,
> 
> is it safe to create a hash table in one thread and allow it to self-expand
> (e.g. by (make-hash-table :test #'eql :size 10000
>                    :rehash-size 2d0 :rehash-threshold 0.5))
> and then use it from another thread? What happens if it is expanded while some
> new entry is added from another thread?

Currently hash-tables are protected by spinlocks, so two threads can't
be accessing the same table at the same time. This is however not a
documented property of the public interface, and is likely to change
in the very near future to a situation where: multiple concurrent
readers on the same hash-table are safe, but having a writer thread +
another concurrent thread doing either reads or writes is unsafe. Thus
application level locks will be needed in the latter case.

> What happens if the garbage collector is moving a data structure that is
> simultaneously accessed from another thread? Say you have created an array 
> by (setq arr (make-array 100)) and the garbage collector is moving
> it while another thread executes (setf (aref arr 3) (list 1 2))?

That can't happen. There will be a reference to ARR in a register of
thread 2, which the gc will treat as a conservative pointer, and thus
not move the array.

> If you don't pass an array to any other thread, will it stay untouched by the
> garbage collector runs of other threads?

(Continue reading)


Gmane