Juan Jose Garcia-Ripoll | 3 Apr 09:51 2012

Re: Locking strategy (best so far)

On Sat, Mar 31, 2012 at 12:13 PM, Matthew Mondor <mm_lists <at> pulsar-zone.net> wrote:
It appears that it locks before all initial threads of the HTTPd can be
started.  The REPL doesn't come back while that thread appears to be
waiting on a pthread_cond_wait(3) forever without getting an event.
This happens inside the GC, invoked sometime during HTTPD-INIT (that
happens in the first/main thread):

#0  0x00007f7ff6a76eda in ___lwp_park50 () from /usr/lib/libc.so.12
#1  0x00007f7ff72088f4 in pthread_cond_timedwait (abstime=0x0, mutex=0x7f7ff7b010e8, cond=0x7f7ff7b01118) at /usr/src/lib/libpthread/pthread_cond.c:148
#2  pthread_cond_wait (cond=0x7f7ff7b01118, mutex=0x7f7ff7b010e8) at /usr/src/lib/libpthread/pthread_cond.c:193
#3  0x00007f7ff7205cc1 in sem_wait (sem=0x7f7ff60408f0) at /usr/src/lib/libpthread/sem.c:323
#4  0x00007f7ff5e187d8 in GC_stop_world () from /usr/pkg/lib/libgc.so.1

This is where the garbage collector tries to suspend all threads that were running and registered with it. It is waiting probably for the ok from every thread but...

As for the other successfully-started threads (there are 9 of these,
8 of which are waiting in sigsuspend(2)) after an attempt to obtain the
global accept-lock unsuccessfully (as the first thread acquired it).

#0  0x00007f7ff6a38eca in _sys___sigsuspend14 () from /usr/lib/libc.so.12
#1  0x00007f7ff7206732 in __sigsuspend14 (sigmask=<optimized out>) at /usr/src/lib/libpthread/pthread_cancelstub.c:567
#2  0x00007f7ff777f341 in ecl_wait_on (env=0x7f7ff7ed8000, condition=0x7f7ff777105a <get_lock_inner>, o=0x28b5f90)

... somehow these threads did not respond to the signal. I do not really understand it. Was it before a change I did recently? ECL now _only_ blocks its communication signal in sigsuspend(). It should not be blocking the signal from the garbage collector.

Juanjo

--
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://juanjose.garciaripoll.googlepages.com
------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second 
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
Ecls-list mailing list
Ecls-list@...
https://lists.sourceforge.net/lists/listinfo/ecls-list
Matthew Mondor | 3 Apr 12:17 2012
Picon

Re: Locking strategy (best so far)

On Tue, 3 Apr 2012 09:51:59 +0200
Juan Jose Garcia-Ripoll <juanjose.garciaripoll@...> wrote:

> ... somehow these threads did not respond to the signal. I do not really
> understand it. Was it before a change I did recently? ECL now _only_ blocks
> its communication signal in sigsuspend(). It should not be blocking the
> signal from the garbage collector.

The issue started to occur when I imported your new changes after
reading your message that started this thread (Message-ID:
<CANejTzqbReyz6dgpqXCvpKtCh+7DbjN+Ldb7LG5qD8QK8wQT_w@...>).
The new implementation before that thread, including where you
implemented a FIFO, were not blocking.  Perhaps that the issue is with
the GC, as it's similar, although much less random (this always locks
now), than an older problem with the stable release ECL locks
implementation...

I could try to check the masks and signals of the threads the next time
I give it a try.
--

-- 
Matt

------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second 
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
Daniel Herring | 5 Apr 04:42 2012

slowdown using Slime

Hi,

I'm running Slime on ECL, and both were updated about a week old right 
now.  There are often noticeable pauses when typing in a source buffer. 
I'm not sure whether this is a new behavior; I had been doing most 
interactive development under SBCL for the past year, and my Slime was 
almost two years old.

One of the pauses occurs when editing a make-instance form in a source 
buffer.  Every time I type a key in a make-instance form, there is a pause 
and a string prints in the slime-repl buffer.  Cursor key or text doesn't 
matter.

SWANK-MOP:COMPUTE-APPLICABLE-METHODS-USING-CLASSES

>From a cursory read of swank-ecl.lisp, it appears that the 
swank-mop:c-a-m-u-c normally calls clos:c-a-m-u-c, but this has been 
disabled in ECL.  I uninterned this symbol from swank-mop in my Slime 
session; either that or something else I did "fixed" the problem for the 
moment.

Is there a proper solution?

Thanks,
Daniel

------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second 
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
Daniel Herring | 5 Apr 04:47 2012

Re: slowdown using Slime

On Wed, 4 Apr 2012, Daniel Herring wrote:

> I'm running Slime on ECL, and both were updated about a week old right
> now.  There are often noticeable pauses when typing in a source buffer.
> I'm not sure whether this is a new behavior; I had been doing most
> interactive development under SBCL for the past year, and my Slime was
> almost two years old.
>
> One of the pauses occurs when editing a make-instance form in a source
> buffer.  Every time I type a key in a make-instance form, there is a pause
> and a string prints in the slime-repl buffer.  Cursor key or text doesn't
> matter.
>
> SWANK-MOP:COMPUTE-APPLICABLE-METHODS-USING-CLASSES
>
>> From a cursory read of swank-ecl.lisp, it appears that the
> swank-mop:c-a-m-u-c normally calls clos:c-a-m-u-c, but this has been
> disabled in ECL.  I uninterned this symbol from swank-mop in my Slime
> session; either that or something else I did "fixed" the problem for the
> moment.

Arg.  Minor enlightenment always comes after publication.

For some reason, the "fix" was actually changing the repl out of the 
:cl-user package (where my code currently lives) and into any other 
package.  Somehow that changes Slime's behavior.

If you think this looks like a Slime bug, I'll try to send them a bug 
report later this week.  Right now, I have a workaround, and there's some 
real code I want to write in my remaining time for today.

- Daniel

------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second 
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
Daniel Herring | 5 Apr 04:51 2012

Re: slowdown using Slime

On Wed, 4 Apr 2012, Daniel Herring wrote:

> On Wed, 4 Apr 2012, Daniel Herring wrote:
>
>> I'm running Slime on ECL, and both were updated about a week old right
>> now.  There are often noticeable pauses when typing in a source buffer.
>> I'm not sure whether this is a new behavior; I had been doing most
>> interactive development under SBCL for the past year, and my Slime was
>> almost two years old.
>>
>> One of the pauses occurs when editing a make-instance form in a source
>> buffer.  Every time I type a key in a make-instance form, there is a pause
>> and a string prints in the slime-repl buffer.  Cursor key or text doesn't
>> matter.
>>
>> SWANK-MOP:COMPUTE-APPLICABLE-METHODS-USING-CLASSES
>>
>>> From a cursory read of swank-ecl.lisp, it appears that the
>> swank-mop:c-a-m-u-c normally calls clos:c-a-m-u-c, but this has been
>> disabled in ECL.  I uninterned this symbol from swank-mop in my Slime
>> session; either that or something else I did "fixed" the problem for the
>> moment.
>
> Arg.  Minor enlightenment always comes after publication.
>
> For some reason, the "fix" was actually changing the repl out of the
> :cl-user package (where my code currently lives) and into any other
> package.  Somehow that changes Slime's behavior.
>
> If you think this looks like a Slime bug, I'll try to send them a bug
> report later this week.  Right now, I have a workaround, and there's some
> real code I want to write in my remaining time for today.

Really hope this is the last update tonight.

The repl's package matters because my source file didn't have a proper 
in-package form.  Thus slime doesn't see the actual class, and the pause 
doesn't occur.

So my current workaround is to put (in-package :cl) before a form 
containing make-instance, edit it, then remove the in-package kludge.

- Daniel

------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second 
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
Juan Jose Garcia-Ripoll | 5 Apr 10:00 2012

Re: slowdown using Slime

On Thu, Apr 5, 2012 at 4:42 AM, Daniel Herring <dherring <at> tentpost.com> wrote:
One of the pauses occurs when editing a make-instance form in a source
buffer.  Every time I type a key in a make-instance form, there is a pause
and a string prints in the slime-repl buffer.  Cursor key or text doesn't
matter.

Looks like it is trying to compute the signature of the applicable method to tell you what keywords are allowed. What I do not understand is why the package matters. Might be that if it cannot find the class then it simply resorts to T and ECL has to compute a list with all the possible MAKE-INSTANCE for all classes... Would be worth further investigating.

Juanjo

--
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://juanjose.garciaripoll.googlepages.com
------------------------------------------------------------------------------
Better than sec? Nothing is better than sec when it comes to
monitoring Big Data applications. Try Boundary one-second 
resolution app monitoring today. Free.
http://p.sf.net/sfu/Boundary-dev2dev
_______________________________________________
Ecls-list mailing list
Ecls-list@...
https://lists.sourceforge.net/lists/listinfo/ecls-list
Andrey G. Grozin | 6 Apr 08:46 2012
Picon

maxima 5.27.0

maxima-5.27.0 is in Gentoo. Testsuite full times on my 32-bit x86 box (seconds) 
and failed tests are:

sbcl-1.0.55    166
cmucl-20c      181
gcl-2.6.8_pre  184
clozurecl-1.8  253  rtest15 (37, 193, 196)
ecl-12.2.1     345  rtest8 (126, 127)
clisp-2.49     569

Details of failed tests:

clozurecl:
----------------------------------------------------------------
********************** Problem 37 ***************
Input:
     2         2 - %pi
  sin (3 a) sin (----- + a + b)
                   3
(-----------------------------
             2     %pi
          sin (a - ---)
                    3
                                 - %pi
    2 sin(a) sin(3 a) cos(b) sin(----- + a + b) sin(3 a + 3 b)
                                   3
  - ----------------------------------------------------------
                             %pi
                     sin(a - ---) sin(a + b)
                              3
       2       2
    sin (a) sin (3 a + 3 b)
  + -----------------------, result : trigrat(%%),
             2
          sin (a + b)
expected : (- (- 9 + 10 cos(2 a) - 2 sqrt(3) sin(2 a) - cos(4 a)
  + sqrt(3) sin(4 a) + 10 cos(2 b) - 2 sqrt(3) sin(2 b) - cos(4 b)
  + sqrt(3) sin(4 b) - 4 cos(2 b - 2 a) - 8 cos(2 a + 2 b)
  + 4 sqrt(3) sin(2 a + 2 b) + 2 cos(4 a + 2 b) - 2 sqrt(3) sin(4 a + 2 b)
  + 2 cos(2 a + 4 b) - 2 sqrt(3) sin(2 a + 4 b) - cos(4 a + 4 b)
  + sqrt(3) sin(4 a + 4 b)))/4, ratsimp(result - expected))

Result:
Polynomial quotient is not exact
error-catch

This differed from the expected result:
0

********************** Problem 193 ***************
Input:
                2
               t  log(t)
integrate(-----------------, t, 0, 1)
             2            4
           (t  - 1) (1 + t )

Result:
                    2
   (sqrt(2) - 2) %pi
- ------------------
           32

This differed from the expected result:
                  2
(sqrt(2) - 1) %pi
------------------
         9/2
        2

********************** Problem 196 ***************
Input:
                                         1
factor(expand(sqrtdenest(integrate(------------, x, 0, 1))))
                                    1          4
                                    - + (x - 3)
                                    2
                                     3/4      9/4
                     73 + sqrt(2) + 2    - 3 2
  - factor((- (- log(----------------------------)
                                  33
                        3/4      9/4              3/4    5/2    13/4
        73 + sqrt(2) - 2    + 3 2              - 2    + 2    - 2
  + log(----------------------------) + 2 atan(---------------------)
                     33                                3/4    13/4
                                                 98 + 2    - 2
            3/4    5/2    13/4
           2    + 2    + 2         7/4
  + 2 atan(-------------------)))/2   )
                   3/4    13/4
           - 98 + 2    - 2

Result:
          13/4    5/2    3/4              13/4    5/2    3/4
         2     + 2    + 2              - 2     + 2    - 2
(2 atan(-------------------) + 2 atan(---------------------)
            13/4    3/4                    13/4    3/4
         - 2     + 2    - 98            - 2     + 2    + 98
           9/4    3/4                            9/4    3/4
        3 2    - 2    + sqrt(2) + 73        - 3 2    + 2    + sqrt(2) + 73
  + log(----------------------------) - log(------------------------------))
                     33                                   33
                 9/4    3/4
   7/4        3 2    - 2    + sqrt(2) + 73
/2    - (log(----------------------------)
                           33
             9/4    3/4
        - 3 2    + 2    + sqrt(2) + 73
  - log(------------------------------) - 2
                      33
           3/4                      1/4
       79 2    + 365 sqrt(2) + 877 2    + 5
  atan(------------------------------------)
                      10657
                 3/4                      1/4
           - 79 2    + 365 sqrt(2) - 877 2    + 5    7/4
  + 2 atan(--------------------------------------))/2
                           10657

This differed from the expected result:
0
----------------------------------------------------------------

ecl:
----------------------------------------------------------------
********************** Problem 126 ***************
Input:
                  1                                       2
(f : diff(----------------, a), g : quad_qags(f b (1 - b) , b, 0, 1) ,
                  2                                                  1
           (a - b)  + 1 + 1
                                                      find_root(g = 0, a, 0, 1))

Result:
                                          2
                                 2 (1 - b)  (a - b) b
find_root(subscript(quad_qags(- --------------------, b, 0, 1, epsrel = 1.e-8,
                                           2     2
                                   ((a - b)  + 2)
                epsabs = 0.0, limit = 200), 1) = 0, a, .3980373668760611, 0.375)

This differed from the expected result:
.3978613590133817

********************** Problem 127 ***************
Input:
                  1                                       2
(f : diff(----------------, a), g : quad_qags(f b (1 - b) , b, 0, 1) ,
                  2                                                  1
           (a - b)  + 1 + 1
                                                      find_root(g = 0, a, 0, 1))

Result:
                                2
                       2 (1 - b)  (a - b) b
find_root(quad_qags(- --------------------, b, 0, 1)  = 0, a,
                                 2     2
                         ((a - b)  + 2)
                                                     1
                                                       .3980373668760611, 0.375)

This differed from the expected result:
.3978613590133817
----------------------------------------------------------------

Andrey
_______________________________________________
Maxima mailing list
Maxima <at> math.utexas.edu
http://www.math.utexas.edu/mailman/listinfo/maxima
Juan Jose Garcia-Ripoll | 6 Apr 13:02 2012

Re: Locking strategy (best so far)

On Sat, Mar 31, 2012 at 12:13 PM, Matthew Mondor <mm_lists <at> pulsar-zone.net> wrote:
It appears that it locks before all initial threads of the HTTPd can be
started.  The REPL doesn't come back while that thread appears to be
waiting on a pthread_cond_wait(3) forever without getting an event.
This happens inside the GC, invoked sometime during HTTPD-INIT (that
happens in the first/main thread):

There were a bunch of problems with the previous code. From a wrong order in the arguments to pthread_sigmask() to the choice of signals. I have fixed them together with the following things

* Implemented semaphores
* Implemented barriers (copied part of the API from Lispworks, though I feel it is overengineered)
* mp:process-join relies on barriers (no trivial thing, the original code was a mess because of the lack of this tool)
* fixes in external processes, locking, etc.

In particular the changes to mp:process-join are quite critical for ECL being able to gracefully quit when there are running threads.

I can confirm that I am able to run your server and get decent connection times, slightly better than on OS X (see below). However, please bear in mind that I am running this all inside a virtualbox which may improve the stability, as processors are not really independent, but bound by the emulator.

Requests per second:    725.38 [#/sec] (mean)
Time per request:       22.057 [ms] (mean)
Time per request:       1.379 [ms] (mean, across all concurrent requests)
Transfer rate:          452.72 [Kbytes/sec] received

Juanjo

--
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://juanjose.garciaripoll.googlepages.com
------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
Ecls-list mailing list
Ecls-list@...
https://lists.sourceforge.net/lists/listinfo/ecls-list
Matthew Mondor | 6 Apr 14:32 2012
Picon

Re: Locking strategy (best so far)

On Fri, 6 Apr 2012 13:02:16 +0200
Juan Jose Garcia-Ripoll <juanjose.garciaripoll@...> wrote:

> There were a bunch of problems with the previous code. From a wrong order
> in the arguments to pthread_sigmask() to the choice of signals. I have
> fixed them together with the following things
> 
> * Implemented semaphores
> * Implemented barriers (copied part of the API from Lispworks, though I
> feel it is overengineered)
> * mp:process-join relies on barriers (no trivial thing, the original code
> was a mess because of the lack of this tool)
> * fixes in external processes, locking, etc.
> 
> In particular the changes to mp:process-join are quite critical for ECL
> being able to gracefully quit when there are running threads.
> 
> I can confirm that I am able to run your server and get decent connection
> times, slightly better than on OS X (see below). However, please bear in
> mind that I am running this all inside a virtualbox which may improve the
> stability, as processors are not really independent, but bound by the
> emulator.

I confirm that these changes allow the HTTPd to run here as well,
thanks!  However, I didn't stress-test it much or try performance tests
with it yet, but that should happen soon.
--

-- 
Matt

------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
Juan Jose Garcia-Ripoll | 6 Apr 16:24 2012

Re: Locking strategy (best so far)

On Fri, Apr 6, 2012 at 2:32 PM, Matthew Mondor <mm_lists <at> pulsar-zone.net> wrote:
I confirm that these changes allow the HTTPd to run here as well,
thanks!  However, I didn't stress-test it much or try performance tests
with it yet, but that should happen soon.

I am now running some stress tests and trying to hit some corner cases. There are already some subtle race conditions in the lisp part of the code. Hopefully will be easy to fix...

Juanjo

--
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://juanjose.garciaripoll.googlepages.com
------------------------------------------------------------------------------
For Developers, A Lot Can Happen In A Second.
Boundary is the first to Know...and Tell You.
Monitor Your Applications in Ultra-Fine Resolution. Try it FREE!
http://p.sf.net/sfu/Boundary-d2dvs2
_______________________________________________
Ecls-list mailing list
Ecls-list@...
https://lists.sourceforge.net/lists/listinfo/ecls-list

Gmane