[Chaitanya Gupta] Fwd: (CMUCL) 1000 socket connections send GC in an infinite loop

Just forwarding to the list.  The lists should be working now.

Ray

From: Chaitanya Gupta <mail <at> chaitanyagupta.com>
Subject: Fwd: (CMUCL) 1000 socket connections send GC in an infinite loop
Date: 2006-09-05 15:01:01 GMT

Hello,

I tried posting this to cmucl-help, but it wasn't delivered for some
reason. Since you are the author of code/internet.lisp and
fd-stream.lisp, I thought I might forward it to you. Sorry for sending
it directly to your address.

Chaitanya

---------- Forwarded message ----------
From: Chaitanya Gupta <mail <at> chaitanyagupta.com>
Date: Sep 5, 2006 7:34 PM
Subject: 1000 socket connections send GC in an infinite loop
To: cmucl-help <at> cons.org

test

test

Is this a bug in format?

In CMUCL 19c , the expression (format nil "~,2f" 0.0001) returns "0.000".
SBCL returns "0.00" as I'd expect.

cheers,

Alex

;;;;;;;;; CMUCL ;;;;;;;;;;;

>: cmucl
CMU Common Lisp CVS 19c 19c-release + minimal debian patches (19C), running
on pcalex
With core: /usr/lib/cmucl/lisp.core
Dumped on: Mon, 2006-02-20 15:02:51Z on pcalex
For support see http://www.cons.org/cmucl/support.html Send bug reports to
the debian BTS.
or to pvaneynd <at> debian.org
type (help) for help, (quit) to exit, and (demo) to see the demos

Loaded subsystems:
    Python 1.1, target Intel x86
    CLOS based on Gerd's PCL 2004/04/14 03:32:47
* (format nil "~,2f" 0.0001)

"0.000"
*

;;;;;;;;;;;; SBCL ;;;;;;;;;;;;;;;;

>: sbcl

Re: 1000 socket connections send GC in an infinite loop

I am trying to post this for the third time here. Lets hope it works.

I am using CMUCL 19c on an x86-64 server (running Redhat Linux). We
are running a website using TBNL (running on top of apache2/mod_lisp2)
to generate dynamic pages. Also, we use trivial-http to send GET
requests to another server. The problem used to be that whenever we
had 1000 connections to that server, the GC would go into an infinite
loop.

Part of the problem was with the http-get function in trivial-http (as
highlighted by John Wiseman here -
http://lemonodor.com/archives/001145.html) -

(defun http-get (url)
 (let* ((host (url-host url))
        (port (url-port url))
        (stream (open-stream host port)))
   (format stream "GET ~A HTTP/1.0~AHost: ~A~AUser-Agent: Trivial
HTTP for Common Lisp~A~A"
       url +crlf+ host +crlf+ +crlf+ +crlf+)
   (force-output stream)
   (list
    (response-read-code stream)
    (response-read-headers stream)
    stream)))

If no exception occurred while sending the request or reading the
response (which was done by another function), the connection would
close normally. However, if an excepton occurred, the stream would not
be closed, and the connection would remain forever in the CLOSE_WAIT

Re: 1000 socket connections send GC in an infinite loop

Good day,

mail> If no exception occurred while sending the request or reading the
mail> response (which was done by another function), the connection would
mail> close normally. However, if an excepton occurred, the stream would not
mail> be closed, and the connection would remain forever in the CLOSE_WAIT
mail> state. And when the no. of connections reaches 1000, the site would
mail> become completely unresponsive and on connecting to lisp, we see the
mail> GC occuring repeatedly (as if in an infinite loop). Also, lisp's CPU
mail> usage goes to 99%. On quitting lisp, all those connections are closed.

Look at FORCIBLY-CLOSE-STREAM function in Araneida source,
file compat/compat-cmucl.lisp:

(defun forcibly-close-stream (s)
  (let ((fd (sys:fd-stream-fd s)))
    (multiple-value-bind (r e) (ignore-errors (close s) t)
      (unless r
        (format t "Unable to close fd ~A: ~A, trying harder ~%" fd e)
        (multiple-value-bind (r e) (ignore-errors (close s :abort t) t)
          (unless r
            (format t "still unable to close ~A: ~A, try harder ~%" fd e)
            (multiple-value-bind (r e)
                (ignore-errors (unix:unix-close fd) t)
              (unless r
                (format t "Even unix-close failed on ~A:~A, giving up~%"
                        fd e)))))))))

It helped me in the same situation.

Re: 1000 socket connections send GC in an infinite loop

On 9/12/06, Vladimir Sekissov <svg <at> surnet.ru> wrote:
> Good day,
>
> Look at FORCIBLY-CLOSE-STREAM function in Araneida source,
> file compat/compat-cmucl.lisp:

The fix I made to thttp:http-get also works fine. Whether there is any
error or not, the socket is closed now. The number of sockets open
remain quite low now.

> It helped me in the same situation.

So its a cmucl problem? i.e. too many open sockets (1000 in my case),
or is it something else?

Thanks,
Chaitanya

Re: Is this a bug in format?

>>>>> "Alex" == Alex McGuire <cage433 <at> gmail.com> writes:

    Alex> In CMUCL 19c , the expression (format nil "~,2f" 0.0001) returns "0.000".
    Alex> SBCL returns "0.00" as I'd expect.

This was fixed in May or so.  Please try a snapshot if you can.

Ray

Re: 1000 socket connections send GC in an infinite loop

>>>>> "Chaitanya" == Chaitanya Gupta <mail <at> chaitanyagupta.com> writes:

    Chaitanya> On 9/12/06, Vladimir Sekissov <svg <at> surnet.ru> wrote:
    >> Good day,
    >> 
    >> Look at FORCIBLY-CLOSE-STREAM function in Araneida source,
    >> file compat/compat-cmucl.lisp:

    Chaitanya> The fix I made to thttp:http-get also works fine. Whether there is any
    Chaitanya> error or not, the socket is closed now. The number of sockets open
    Chaitanya> remain quite low now.

Cool!

    >> It helped me in the same situation.

    Chaitanya> So its a cmucl problem? i.e. too many open sockets (1000 in my case),
    Chaitanya> or is it something else?

I think it's the user's responsibility to release resources.  CMUCL
will usually free up files eventually, assuming you can wait that
long.  Don't know how sockets are handled though.

Ray

newbie question.

I am a complete newbie to lips and CMUCL.  I have two problems that I need
your help with.

First, I'm having problems with launching the debugger on my ubuntu 64 box.

Second, I'm having problems with loading uffi library.

See the output below.

Can any of you give me some clues as how to fix both these problems?

Thanks in advance.

Yarek

yarek <at> ubuntu:/usr/share/common-lisp/systems$ lisp
CMU Common Lisp 19c (19C), running on ubuntu
With core: /usr/bin/lisp.core
Dumped on: Wed, 2006-08-23 11:17:38-07:00 on ubuntu
See <http://www.cons.org/cmucl/> for support information.
Loaded subsystems:
    Python 1.1, target Intel x86
    CLOS based on Gerd's PCL 2004/04/14 03:32:47
    Gray Streams Protocol Support
    CLX X Library MIT R5.02
    Motif toolkit and graphical debugger 1.0
    Hemlock 3.5
*  (asdf:oos `asdf:load-op :uffi)

; loading system definition from uffi.asd into #<The ASDF1788 package>

Re: 1000 socket connections send GC in an infinite loop

"Chaitanya Gupta" <mail <at> chaitanyagupta.com> writes:
> So its a cmucl problem? i.e. too many open sockets (1000 in my case),
> or is it something else?

I think it's a more general problem; most unixes don't handle more
than about 1000 open fd's well with select. (At least I /think/
that's the problem.)

',mr

--

-- 
[Emacs] is written in Lisp, which is the only computer language that is
beautiful.  -- Neal Stephenson, _In the Beginning was the Command Line_