Brian Perry | 2 May 2012 20:39
Picon
Favicon

Lisp Server Disconnect and Threading Issues

Greetings,

I've been working on adapting existing Lisp code to communicate with a socket server with the end goal of allowing a couple of different web based clients (one using Flash, another in Processing) to communicate with the Lisp server.  I've run into a couple of specific roadblocks and was wondering if anyone could help.

The first issue is that the lisp server is crashing when a client disconnects.  I'm seeing an error similar to the following:

-------
debugger invoked on a END-OF-FILE in thread #<THREAD RUNNING {1003707001}>:
  end of file on #<SB-SYS:FD-STREAM for "a socket" {10037EFDE1}>
-------

If we could get past this one, we'd be able to move forward quite a bit.

The second issue is that the Lisp server currently blocks when one connection is made, preventing other clients from connecting simultaneously.  I've assumed that sbcl's threading might help here, but I didn't have any luck with it myself (most likely due to lack of experience).

I based the server on the following example:


I'd be happy to provide more info if it helps.

Thanks,

Brian
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help
Grant Rettke | 3 May 2012 04:48
Picon
Favicon
Gravatar

Creating a sandbox

Hi,


Although I've not yet learned Lisp and SBCL well, I'm curious about whether preparing an eval sandbox so that it would be restricted in it's bindings, file system access, memory usage, network access, and CPU time, something like this


would be possible?

Best wishes,

Grant

--
http://www.wisdomandwonder.com/
ACM, AMA, COG, IEEE
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help
Nikodemus Siivola | 3 May 2012 08:54
Gravatar

Re: Creating a sandbox

On 3 May 2012 05:48, Grant Rettke <grettke <at> acm.org> wrote:

> Although I've not yet learned Lisp and SBCL well, I'm curious about whether
> preparing an eval sandbox so that it would be restricted in it's bindings,
> file system access, memory usage, network access, and CPU time, something
> like this
>
> http://docs.racket-lang.org/reference/Sandboxed_Evaluation.html
>
> would be possible?

/Possible/, yes. Straightforward, actually, mostly yes. Easy to get right? No.

Racket's language abstraction is pretty cool, and we don't have
anything like that. So you'd be rolling up some infrastructure to
start with.

Cheers,

 -- nikodemus

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Robert Bebop | 3 May 2012 12:35
Picon

Callback from C

Hi all,

Is it actually possible to callback from C with SBCL. According to the 
manual it's tricky but the CFFI manual claims that it's possible on some 
platforms with SBCL.

I am a musician and enjoy messing around with audio DSP. I currently use 
SBCL on a GNU/Linux platform and have written a C wrapper for the JACK 
audio server. I am getting pretty good real-time performance with the 
current blocking method I am using (about 10 -15 ms latency) but it 
should improve if I can get the server to callback into LISP.

I have written the FFI interface using SBCL's interface but if CFFI 
works I will recode to use it.

Another question I have is whether there could be some unpleasant side 
effects of having a real-time thread callback into LISP.

Thanks,

Robert.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Leslie P. Polzer | 3 May 2012 12:07
Picon

Re: Lisp Server Disconnect and Threading Issues


Brian Perry wrote:
> Greetings,
>
> I've been working on adapting existing Lisp code to communicate with a socket server
> with the end goal of allowing a couple of different web based clients (one using Flash,
> another in Processing) to communicate with the Lisp server.  I've run into a couple of
> specific roadblocks and was wondering if anyone could help.
>
> The first issue is that the lisp server is crashing when a client disconnects.  I'm
> seeing an error similar to the following:
>
> -------
> debugger invoked on a END-OF-FILE in thread #<THREAD RUNNING {1003707001}>:
>   end of file on #<SB-SYS:FD-STREAM for "a socket" {10037EFDE1}>

You want to catch that exception, or check for end of file before you read,
or disable the condition throw on EOF (if possible).

> The second issue is that the Lisp server currently blocks when one connection is made,
> preventing other clients from connecting simultaneously.  I've assumed that sbcl's
> threading might help here, but I didn't have any luck with it myself (most likely due to
> lack of experience).

I haven't looked at your code, but I suppose some kind of non-blocking
or event-based I/O might be possible. On the other hand it might still
be easier to get used to threads since it's not that hard...

  Leslie

>
> I based the server on the following example:
>
> http://jsnell.iki.fi/tmp/echo-server.lisp
>
> I'd be happy to provide more info if it helps.
>
> Thanks,
>
> Brian
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats.
> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/_______________________________________________
> Sbcl-help mailing list
> Sbcl-help <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sbcl-help
>

--

-- 

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Kalyanov Dmitry | 3 May 2012 12:47
Picon
Gravatar

Re: Callback from C

0On Thu, 2012-05-03 at 12:35 +0200, Robert Bebop wrote:
> Hi all,
> 
> Is it actually possible to callback from C with SBCL. According to the 
> manual it's tricky but the CFFI manual claims that it's possible on some 
> platforms with SBCL.
> 
> I am a musician and enjoy messing around with audio DSP. I currently use 
> SBCL on a GNU/Linux platform and have written a C wrapper for the JACK 
> audio server. I am getting pretty good real-time performance with the 
> current blocking method I am using (about 10 -15 ms latency) but it 
> should improve if I can get the server to callback into LISP.
> 
> I have written the FFI interface using SBCL's interface but if CFFI 
> works I will recode to use it.
> 
> Another question I have is whether there could be some unpleasant side 
> effects of having a real-time thread callback into LISP.
> 
> Thanks,
> 
> Robert.

It is possible to call back from C to SBCL (with either SBCL's FFI or
with CFFI). The big caveat is that callback invocation has to happen on
a Lisp thread (an initial Lisp thread or thread created with
sb-thread:create-thread).

As far as I know, many audio APIs invoke callbacks in their own threads
which conflicts with the requirement of invoking callback in Lisp
thread. If JACK does that, one option is to have callback written in C
that notifies waiting Lisp thread (for example, via a condition
variable) and marshalls the data between Lisp and JACK.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Robert Bebop | 3 May 2012 13:57
Picon

Re: Callback from C

El 03/05/12 12:47, Kalyanov Dmitry escribió:
> 0On Thu, 2012-05-03 at 12:35 +0200, Robert Bebop wrote:
>> Hi all,
>>
>> Is it actually possible to callback from C with SBCL. According to the
>> manual it's tricky but the CFFI manual claims that it's possible on some
>> platforms with SBCL.
>>
>> I am a musician and enjoy messing around with audio DSP. I currently use
>> SBCL on a GNU/Linux platform and have written a C wrapper for the JACK
>> audio server. I am getting pretty good real-time performance with the
>> current blocking method I am using (about 10 -15 ms latency) but it
>> should improve if I can get the server to callback into LISP.
>>
>> I have written the FFI interface using SBCL's interface but if CFFI
>> works I will recode to use it.
>>
>> Another question I have is whether there could be some unpleasant side
>> effects of having a real-time thread callback into LISP.
>>
>> Thanks,
>>
>> Robert.
> It is possible to call back from C to SBCL (with either SBCL's FFI or
> with CFFI). The big caveat is that callback invocation has to happen on
> a Lisp thread (an initial Lisp thread or thread created with
> sb-thread:create-thread).
>
> As far as I know, many audio APIs invoke callbacks in their own threads
> which conflicts with the requirement of invoking callback in Lisp
> thread. If JACK does that, one option is to have callback written in C
> that notifies waiting Lisp thread (for example, via a condition
> variable) and marshalls the data between Lisp and JACK.
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Sbcl-help mailing list
> Sbcl-help <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/sbcl-help
Hi Kalyanov,

That's exactly what happens, JACK calls back in it's own thread which is 
running in hard real-time.

Signaling a condition sounds like the best way to go although I'm not 
sure as to how to signal a Lisp condition from C and can't see anything 
in the manual about it?

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Leslie P. Polzer | 3 May 2012 14:18
Picon

Re: Callback from C

On Thu, May 03, 2012 at 01:57:19PM +0200, Robert Bebop wrote:
> Signaling a condition sounds like the best way to go although I'm not 
> sure as to how to signal a Lisp condition from C and can't see anything 
> in the manual about it?

He was talking about condition *variables*. They help with thread
coordination and don't really have anything to do with Common Lisp
conditions.

  Leslie

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Robert Bebop | 3 May 2012 15:06
Picon

Re: Callback from C

El 03/05/12 14:18, Leslie P. Polzer escribió:
> On Thu, May 03, 2012 at 01:57:19PM +0200, Robert Bebop wrote:
>> Signaling a condition sounds like the best way to go although I'm not
>> sure as to how to signal a Lisp condition from C and can't see anything
>> in the manual about it?
> He was talking about condition *variables*. They help with thread
> coordination and don't really have anything to do with Common Lisp
> conditions.
>
>    Leslie
Ok, thanks Leslie, I misunderstood.

That is actually what I'm doing now with pthread conditions, I thought 
Kalyanov was suggesting that a LISP condition would be a good way to go. 
It seems that the limitations on performance depend on the GC and lack 
of priority scheduling on the sb-thread. I guess I'll have to wait until 
SBCL develops some kind of scheduling mechanism for different 
sb-threads. I am too nervous about running SBCL top-level in hard real-time.

Cheers.

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
Brian Perry | 3 May 2012 22:21
Picon
Favicon

Re: Lisp Server Disconnect and Threading Issues

Thanks for the reply Leslie.  I dug a little deeper into what's possible regarding exception handling, and it does seem likely that would allow us to work around / more gracefully handle this error case.

As for the second issue - I'm still relatively new to Lisp, but I'm always willing to learn.  Is there any particular resource you (or others reading) would suggest as a crash course on threading for scbl?

Brian

On Thursday, May 3, 2012 at 6:07 AM, Leslie P. Polzer wrote:


Brian Perry wrote:
Greetings,

I've been working on adapting existing Lisp code to communicate with a socket server
with the end goal of allowing a couple of different web based clients (one using Flash,
another in Processing) to communicate with the Lisp server. I've run into a couple of
specific roadblocks and was wondering if anyone could help.

The first issue is that the lisp server is crashing when a client disconnects. I'm
seeing an error similar to the following:

-------
debugger invoked on a END-OF-FILE in thread #<THREAD RUNNING {1003707001}>:
end of file on #<SB-SYS:FD-STREAM for "a socket" {10037EFDE1}>

You want to catch that exception, or check for end of file before you read,
or disable the condition throw on EOF (if possible).


The second issue is that the Lisp server currently blocks when one connection is made,
preventing other clients from connecting simultaneously. I've assumed that sbcl's
threading might help here, but I didn't have any luck with it myself (most likely due to
lack of experience).

I haven't looked at your code, but I suppose some kind of non-blocking
or event-based I/O might be possible. On the other hand it might still
be easier to get used to threads since it's not that hard...

Leslie



I based the server on the following example:


I'd be happy to provide more info if it helps.

Thanks,

Brian

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats.
Sbcl-help mailing list


--

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help

Gmane