headers
Sam Steingold | 1 Nov 2010 04:35
Picon

Re: gensym, gentemp with threads

> * Vladimir Tzankov <igmnaxbi <at> tznvy.pbz> [2010-10-31 16:51:29 +0200]:
>
> On 10/31/10, Don Cohen <don-sourceforge-xxz <at> isis.cs3-inc.com> wrote:
>> BTW, I'm guessing that gensym and gentemp are not protected in any way,
>> so that two different threads could end up using the same names,
>> and in the case of gentemp, returning the same symbols?
>
> gensym is not guarded in any way and it's possible to get same name in
> different treads (by default every thread binds *gensym-counter* via
> *default-special-bindings*). In presence of possible conflicts - may
> be use unique thread prefix?
>
> gentemp is more problematic (have not looked on it till now). Again -
> per thread prefix usage is fine but the internal counter is shared
> among all threads.
>
> Another option is to use a form of atomic increment of counters (but
> how exactly to do is problematic - cpu supported atomic increment does
> not fit well clisp arithmetic, may be use mutex?).

I think using a mutex in getsym & gentems (and not binding
*gensym-counter* in every thread) is a good idea.
these functions are usually called in macroexpansion, so they are not
too speed-critical and not particularly prone to deadlocks.

--

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 10.04 (lucid)
http://thereligionofpeace.com http://www.memritv.org http://pmw.org.il
http://camera.org http://mideasttruth.com http://palestinefacts.org
I'm out of my mind, but feel free to leave a message...
(Continue reading)

Permalink | Reply | 2 comments |
headers
Don Cohen | 1 Nov 2010 04:57
Favicon

Re: exemption-wait question

I'm only gradually beginning to understand what exemptions
are supposed to do.  If my interpretation below is correct then
perhaps some of it should be added to the doc.

Looking at doc for exemption-wait
  On return mutex is acquired again.
This is true even on timeout?

So you're supposed to do something like this, right?
 (with-mutext-lock m
   (if (exemption-wait e m (lambda () <condition>))
      ;; condition should be something that can only become true/false
      ;; due to something executed by a thread owning mutex m, right?
      ;; that's why we expect that evaluating the condition while 
      ;; holding m will not be affected by race conditions
      (;; now I have the lock and condition is true
       ...)
      (;; exemption-wait returns false 
       ;; I have the lock but condition is (probably still) false
       ;; This can happen only if I supply a timeout?
       ;; Or can it occur if this thread is interrupted?
       ...))
    ) 

If exemption wait returned false and this does not re-acquire the lock
then I presume that the with-mutex-lock does not unlock m since
unlocking m when it's not locked is supposed to be an error.

Now that I think I understand what exemptions are supposed to mean,
it occurs to me that they could be implemented in lisp if we only had
(Continue reading)

Permalink | Reply | 19 comments |
headers
Vladimir Tzankov | 1 Nov 2010 08:24
Picon

Re: exemption-wait question

On 11/1/10, Don Cohen <don-sourceforge-xxz <at> isis.cs3-inc.com> wrote:
> I'm only gradually beginning to understand what exemptions
> are supposed to do.  If my interpretation below is correct then
> perhaps some of it should be added to the doc.
>
> Looking at doc for exemption-wait
>   On return mutex is acquired again.
> This is true even on timeout?
Yes, in all cases. When the control returns from exemption-wait - the
mutex is owned by the thread regardless what caused exemption-wait to
return.

> So you're supposed to do something like this, right?
>  (with-mutext-lock m
>    (if (exemption-wait e m (lambda () <condition>))
>       ;; condition should be something that can only become true/false
>       ;; due to something executed by a thread owning mutex m, right?
yes
>       ;; that's why we expect that evaluating the condition while
>       ;; holding m will not be affected by race conditions
>       (;; now I have the lock and condition is true
>        ...)
>       (;; exemption-wait returns false
>        ;; I have the lock but condition is (probably still) false
>        ;; This can happen only if I supply a timeout?
only on timeout (or on some obscure error but if happens - it is error
in clisp implementation).
>        ;; Or can it occur if this thread is interrupted?
>        ...))
>     )
(Continue reading)

Permalink | Reply | 18 comments |
headers
Don Cohen | 1 Nov 2010 09:11
Favicon

Re: exemption-wait question

Vladimir Tzankov writes:

 > > So you're supposed to do something like this, right?
 > >  (with-mutext-lock m
 > >    (if (exemption-wait e m (lambda () <condition>))
 > >       ;; condition should be something that can only become true/false
 > >       ;; due to something executed by a thread owning mutex m, right?
 > yes
 > >       ;; that's why we expect that evaluating the condition while
 > >       ;; holding m will not be affected by race conditions
 > >       (;; now I have the lock and condition is true
 > >        ...)
 > >       (;; exemption-wait returns false
 > >        ;; I have the lock but condition is (probably still) false
 > >        ;; This can happen only if I supply a timeout?
 > only on timeout (or on some obscure error but if happens - it is error
 > in clisp implementation).
 > >        ;; Or can it occur if this thread is interrupted?
 > >        ...))
 > >     )
 > 
 > If thread is interrupted while in exemption-wait the interrupt
 > function will be executed with mutex held.
So if you interrupt the thread to go into the debugger, then all other
threads that need to get the mutex (like to return write access) are
stuck while you debug.  And if you then interrupt another to go into
the debugger you can't even get into the debugger cause you're stuck
waiting for the mutex before entering the debugger. 
It seems to me that if you interrupt exemption-wait you should not
wait to get the lock.  That's supposed to happen only before you exit
(Continue reading)

Permalink | Reply | 17 comments |
headers
Vladimir Tzankov | 1 Nov 2010 11:56
Picon

Re: exemption-wait question

On 11/1/10, Don Cohen <don-sourceforge-xxz <at> isis.cs3-inc.com> wrote:
>  > If thread is interrupted while in exemption-wait the interrupt
>  > function will be executed with mutex held.
> So if you interrupt the thread to go into the debugger, then all other
> threads that need to get the mutex (like to return write access) are
> stuck while you debug.  And if you then interrupt another to go into
> the debugger you can't even get into the debugger cause you're stuck
> waiting for the mutex before entering the debugger.
> It seems to me that if you interrupt exemption-wait you should not
> wait to get the lock.  That's supposed to happen only before you exit
> exemption-wait and you should be able to interrupt and go into the
> debugger before that.

If you are brave enough - unlock the mutex from interrupt function -
but be careful to lock it again on exit.
btw: debugging in presence of many threads is quite adventurous. I
find it mostly useful for fixing deadlocks - for everything else I
prefer logging.

Let's explain why it works this way:
Suppose you are stuck in pthread_cond_wait() and you want to interrupt
it. According to POSIX standard this function never returns EINTR (if
it gets signal). Rather it calls signal handler and continues to wait
as if nothing happened. Within signal handler no lisp code can be
executed (and actually very limited set of C functions are allowed) .
The only way to implement thread-interrupt while we are waiting in
pthread_cond_wait() is to signal the condition/exemption, run the
interrupt function and continue to wait. After we signal the exemption
- mutex is reacquired automatically by the OS. As final result - lock
is held when the interrupt functions is executed.
(Continue reading)

Permalink | Reply | 16 comments |
headers
Don Cohen | 1 Nov 2010 19:12
Favicon

Re: exemption-wait question

Vladimir Tzankov writes:
 > On 11/1/10, Don Cohen <don-sourceforge-xxz <at> isis.cs3-inc.com> wrote:
 > >  > If thread is interrupted while in exemption-wait the interrupt
 > >  > function will be executed with mutex held.
 > > So if you interrupt the thread to go into the debugger, then all other
 > > threads that need to get the mutex (like to return write access) are
 > > stuck while you debug.  And if you then interrupt another to go into
 > > the debugger you can't even get into the debugger cause you're stuck
 > > waiting for the mutex before entering the debugger.
I gather you agree with all above.
I think you'll admit that this is not a good thing for cases where you
want most of your threads to keep working (like in a web server) while
you look around in the debugger to see what's going wrong with one thread.

 > > It seems to me that if you interrupt exemption-wait you should not
 > > wait to get the lock.  That's supposed to happen only before you exit
 > > exemption-wait and you should be able to interrupt and go into the
 > > debugger before that.

 > If you are brave enough - unlock the mutex from interrupt function -
 > but be careful to lock it again on exit.
This actually seems reasonable - I interrupt a thread, go into the
debugger, find that it's in exemption-wait and unlock the mutex.
Now other threads can continue to run while I examine the state of
this one.
I'd like to automate the first step so that other threads do not have
to wait while I figure out that I'm in exemption-wait and unlock the
mutex.  Where is doc that tells me how to do that? 
When you interrupt with function NIL to debug, is that the same as
some function that I can call?  I'm guessing something like
(Continue reading)

Permalink | Reply | 15 comments |
headers
Vladimir Tzankov | 1 Nov 2010 19:41
Picon

Re: exemption-wait question

On 11/1/10, Don Cohen <don-sourceforge-xxz <at> isis.cs3-inc.com> wrote:
>  > Let's explain why it works this way:
>  > Suppose you are stuck in pthread_cond_wait() and you want to interrupt
>  > it. According to POSIX standard this function never returns EINTR (if
>  > it gets signal). Rather it calls signal handler and continues to wait
>  > as if nothing happened. Within signal handler no lisp code can be
>  > executed (and actually very limited set of C functions are allowed) .
>  > The only way to implement thread-interrupt while we are waiting in
>  > pthread_cond_wait() is to signal the condition/exemption, run the
>  > interrupt function and continue to wait. After we signal the exemption
>  > - mutex is reacquired automatically by the OS. As final result - lock
>  > is held when the interrupt functions is executed.
> Ok, so you can never look up the stack and find that you're in
> exemption-wait.  If you interrupt while in that function you find
> yourself ready to execute the thing after it.  And at that point
> exemption-wait has returned T.

Have you tried it? exemption-wait does not return when it handles
interrupts - interrupt function is called from it.

>  > > > On non-local exit from the function - all unwind-protect forms will
>  > > > be executed and mutex in the above example will be unlocked. In
>  > > > case of normal exit from interrupt function - exemption-wait will
>  > > > continue to wait.
>  > > "The function" means the one running the code above?
>  > "interrupt function" means the function passed as :function to
>  > thread-interrupt to be executed in the context of the thread.
> So I interrupt the thread with function NIL, which does something like
> a call to cerror after the return from exemption-wait.
before exemption-wait returns.
(Continue reading)

Permalink | Reply | 14 comments |
headers
Don Cohen | 2 Nov 2010 00:33
Favicon

Re: exemption-wait question

Vladimir Tzankov writes:
 > On 11/1/10, Don Cohen <don-sourceforge-xxz <at> isis.cs3-inc.com> wrote:
 > > > The only way to implement thread-interrupt while we are waiting in
 > > > pthread_cond_wait() is to signal the condition/exemption, run the
 > > > interrupt function and continue to wait. After we signal the exemption
 > > > - mutex is reacquired automatically by the OS. As final result - lock
 > > > is held when the interrupt functions is executed.
 > > Ok, so you can never look up the stack and find that you're in
 > > exemption-wait.  If you interrupt while in that function you find
 > > yourself ready to execute the thing after it.  And at that point
 > > exemption-wait has returned T.
 > 
 > Have you tried it? exemption-wait does not return when it handles
 > interrupts - interrupt function is called from it.
Then I misinterpreted your text above.
Let me try again.
Exemption-wait releases the lock and then calls pthread_cond_wait?
Or perhaps pthread_cond_wait releases the lock.
We then try to interrupt.
At that point pthread_cond_wait gets the lock (might have to wait for
it!) and exits but we're still in exemption-wait?  So then we call
the interrupt function, e.g., debugger.  And we see
- exemption-wait on the stack
- we have the lock
If I then do continue, the condition will be retested if :test was
supplied and that will probably go back to pthread_cond_wait.
If no test was supplied or if I return from exemption-wait then
if I have supplied my own loop, it will retest, call exemption-wait
again and that will call pthread_cond_wait again.
Otherwise (return from exemption-wait and no test or loop), it's as
(Continue reading)

Permalink | Reply | 13 comments |
headers
Vladimir Tzankov | 2 Nov 2010 08:04
Picon

Re: exemption-wait question

On 11/2/10, Don Cohen <don-sourceforge-xxz <at> isis.cs3-inc.com> wrote:
> Let me try again.
> Exemption-wait releases the lock and then calls pthread_cond_wait?
> Or perhaps pthread_cond_wait releases the lock.
pthread_cond_wait atomically releases the lock and starts to wait on
condition variable (exemption in clisp).
> We then try to interrupt.
Which actually signals the condition in order to get out of pthread_cond_wait.
> At that point pthread_cond_wait gets the lock (might have to wait for
> it!) and exits but we're still in exemption-wait?  So then we call
> the interrupt function, e.g., debugger.  And we see
> - exemption-wait on the stack
> - we have the lock
yes
> If I then do continue, the condition will be retested if :test was
> supplied and that will probably go back to pthread_cond_wait.
yes
> If no test was supplied or if I return from exemption-wait then
> if I have supplied my own loop, it will retest, call exemption-wait
> again and that will call pthread_cond_wait again.
that's should be the usage in case of no :test argument.
> Otherwise (return from exemption-wait and no test or loop), it's as
> if exemption-wait returned (whatever value I chose).
Hmm, how are you going to return from exemption-wait from interrupt function?
The only way to get out of it is to perform non-local exit.

>  > When you interrupt a thread and do not perform non-local exit from
>  > interrupt function - the thread resumes normally. If you inspect the
>  > stack from the interrupt - you will see all
>  > mutex-lock/exemption-wait/read/etc there.
(Continue reading)

Permalink | Reply | 12 comments |
headers
Don Cohen | 2 Nov 2010 21:02
Favicon

Re: exemption-wait question

Vladimir Tzankov writes:
 > > Otherwise (return from exemption-wait and no test or loop), it's as
 > > if exemption-wait returned (whatever value I chose).

 > Hmm, how are you going to return from exemption-wait from interrupt
 > function?  The only way to get out of it is to perform non-local
 > exit.
I had in mind using the debugger to do something like return or redo.
I agree that these are all non-local exits.

 > > few functions on the stack and then resume?
 > (thread-interrupt some-thread :function (lambda () (show-stack 1 10)))
 > second argument of show-stack is number of frames to be shown.
[A little off topic, but
Is this documented?  I don't see it in impnotes.  I notice that a
version of impnotes all in one page would be useful for searching.
]

I do this
(loop for x in (mt:LIST-THREADS) do
 (format t "-- ~a --~&" x)
 (mt:thread-interrupt x :function (lambda () (ext:show-stack 1 3)))) 
and I see a backtrace only for the current thread.
I see the output of the format for other threads but no backtrace.
I do see the backtrace for another thread in its listener.
It appears that there's no stream argument.  I guess now (after losing
contact with all threads) that I should have done something like
(loop for x in (mt:LIST-THREADS) do
 (format t "-- ~a --~&" x)
 (mt:thread-interrupt x :function
(Continue reading)

Permalink | Reply | 11 comments |

Gmane