Development mailing list for the j editor

Alan Ruttenberg | 22 May 2013 22:35

[armedbear-devel] Performance of eq hash tables

Hi,

Has anyone looked at this? I'm traversing a large structure and using an eq (tried eql too) hash table to record where I've been so that I don't get caught in circularities. There are about 1.5M nodes, but what I see is that adding entries to the hash table slows down (a lot) over time, even when I give an initial size of 4,000,000. That would be max occupancy of 30%, lower than anyone would typically rehash.

What I see is that it rapidly starts to fill. At 300k entries I'm getting about 2.5K new entries per second. By the time I get to 600k entries it is down to 750 new entries/sec. At 900k (where it stands now) it's about 500 entries/sec.

That makes me suspect that the hash isn't distributing well over the size of the underlying vector and that the slowdown is due to buckets getting long.

I'm not sure how to probe this - any ideas?

Or to fix it. I suppose I could look for an alternative hash implementation, but I thought I would check first with the list to see if anyone else can shed some light.

Thanks,

Alan

headers

Erik Huelsmann | 11 Apr 2013 10:49

Are we back?

Test.

sent from my phone

headers

Erik Huelsmann | 3 Apr 2013 23:55

[armedbear-devel] REMOVE CACHES: r14460 changes FASL entry point name

Hi all,

Commits between r14456 and r14460 change the FASL entry point name. Everybody tracking trunk will need to blow their caches away.

Bye,

Erik.

_______________________________________________
armedbear-devel mailing list
armedbear-devel <at> common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel

headers

Erik Huelsmann | 3 Apr 2013 23:50

[armedbear-devel] Patch for ASDF2.26.32 to support concatenated fasls for ABCL

Hi!

In order for these commands to generate a fasl per ASDF system, I had to apply the patch below to ASDF.

$ ./abcl

CL-USER(1): :ld c:/users/erik/quicklisp/setup

CL-USER(2): (ql:quickload :bordeaux-threads)

CL-USER(3): (asdf:oos 'asdf:binary-op :bordeax-threads)

The commands above work with current trunk (r14460) and produces 2 concatenated fasls in my cache directory: 1 for alexandria and 1 for bordeaux-threads.

This patch is required (also attached, to prevent line wrapping):

diff --git a/bundle.lisp b/bundle.lisp

index 568f894..24fbf90 100644

--- a/bundle.lisp

+++ b/bundle.lisp

(or #+ecl (or (equalp type (compile-file-type :type :object))

(equalp type (compile-file-type :type :static-library)))

#+mkcl (equalp type (compile-file-type :fasl-p nil))

- #+(or allegro clisp clozure cmu lispworks sbcl scl xcl) (equalp type (compile-file-type)))))

+ #+(or abcl allegro clisp clozure cmu lispworks sbcl scl xcl) (equalp type (compile-file-type)))))

(defgeneric* (trivial-system-p) (component))

diff --git a/uiop/lisp-build.lisp b/uiop/lisp-build.lisp

index e2e376d..4b7e819 100644

--- a/uiop/lisp-build.lisp

+++ b/uiop/lisp-build.lisp

<at> <at> -669,11 +669,12 <at> <at> it will filter them appropriately."

;;; Links FASLs together

(with-upgradability ()

(defun combine-fasls (inputs output)

- #-(or allegro clisp clozure cmu lispworks sbcl scl xcl)

+ #-(or abcl allegro clisp clozure cmu lispworks sbcl scl xcl)

(error "~A does not support ~S~%inputs ~S~%output ~S"

(implementation-type) 'combine-fasls inputs output)

#+clozure (ccl:fasl-concatenate output inputs :if-exists :supersede)

#+(or allegro clisp cmu sbcl scl xcl) (concatenate-files inputs output)

+ #+abcl (sys:concatenate-fasls inputs output)

#+lispworks

(let (fasls)

(unwind-protect

Attachment (asdf.patch): application/octet-stream, 1429 bytes

_______________________________________________
armedbear-devel mailing list
armedbear-devel <at> common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel

headers

Erik Huelsmann | 3 Apr 2013 22:39

[armedbear-devel] Support for concatenated fasls

Hi!

As we discussed, I'd write FASL concatenation support for ABCL. I'm nearly done, however I'm running into a snag caused by ASDF:

Normally, a file "package.lisp" is compiled into a fasl "package.abcl" which is basically a zip with at least one entry: "package._". The "._" file contains instructions for loading the fasl and may contain references to other files in the same zip.

My FASL concatenation code currently uses the assumption that the "._" file has exactly the same name as the enclosing package. However, FASLs generated using ASDF violate that assumption: the ._ file in the fasl isn't called "package._" but it's called "package-ASDF_TMP._".

How does ABCL know about the suffix when ASDF is loading the FASL "package.abcl" with the suffixed file in it in the regular situation?

Bye,

Erik.

_______________________________________________
armedbear-devel mailing list
armedbear-devel <at> common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel

headers

Erik Huelsmann | 2 Apr 2013 21:40

[armedbear-devel] Other findings (was Re: Questions about r14452 (Fix cl-cont))

Well, to some extent I can see your problem. Indeed there seem to be no slots in the 'MOP:FUNCALLABLE-STANDARD-OBJECT class. The layoutFuncallableStandardClass is what confused me, but that's indeed the layout of the metaclass.

So, you're correct that the fix works by accident. The solution should be to add a slot by the name 'MOP::NAME to the FUNCALLABLE-STANDARD-OBJECT. That slot can contain the name of the object (ie function) for which the item is defined. In that case, we should probably remove the direct slot MOP::NAME on the standard generic function, since it's already defined on the superclass if we decide to do that.

While studying this problem using DESCRIBE, it seems our current implementation dispatches (DESCRIBE (MAKE-INSTANCE 'CL-CONT::FUNCALLABLE/CC)) the wrong way: to the T way instead of the STANDARD-OBJECT way...

Yet more to investigate...

Bye,

Erik.

_______________________________________________
armedbear-devel mailing list
armedbear-devel <at> common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel

headers

Rudolf Schlatte | 2 Apr 2013 20:08

Re: [armedbear-devel] Questions about r14452 (Fix cl-cont)

What makes me suspicious is the output of

(describe (make-instance 'funcallable-standard-object))

vs

(describe (make-instance 'standard-generic-function))

.

There's a layout called layoutFuncallableStandardClass, defined in StandardClass.java, which has NAME as first slot, but that's for class objects with metaclass funcallable-standard-class, not for instances of these classes. I'll think a bit in the next days and try to either resolve my doubts or come up with another fix for the problem. Maybe I'm simply confused; wouldn't be the first time. :)

Rudi

On Apr 2, 2013, at 19:52, Erik Huelsmann <ehuels <at> gmail.com> wrote:

Hi Rudi,

Basically the expectation we violated was the fact that %set-lambda-name didn't want to take a funcallable instance as its function argument. The commit changed that and used the first slot of the funcallable instance class layout (called NAME) to store the name. I guess we forget to finalize the class somewhere, so the spots don't appear?

Bye,

Erik.

sent from my phone

On Apr 2, 2013 7:32 PM, "Rudolf Schlatte" <rudi <at> constantly.at> wrote:
Hi,

I suspect that the changes in commit r14452 only work by accident.

The definions of SLOT_INDEX_NAME in FuncallableStandardClass vs FuncallableStandardObject are no problem, since the latter simply isn't used anywhere. But the new methods getName() and setName() assume that a) the FuncallableStandardObject instance has at least one slot, and b) that slot holds a name. Since
(class-slots (find-class 'funcallable-standard-object))
returns NIL, these assumptions hold only when the objects are in fact generic functions.

Amusingly,
(function-lambda-expression (make-instance 'funcallable-standard-object))
runs into the type error on line 2666 of Primitives.lisp ("The value #<FUNCALLABLE-STANDARD-OBJECT {665C5A83}> is not of type FUNCTION.") even though
(typep (make-instance 'funcallable-standard-object) 'function) => T
so I can't trigger the array-out-of-bounds error right now.

Could I have a recipe for triggering the bug? I'd like to see how cl-cont uses funcallable-standard-objects, and what expectations we violate. I believe I can fix this properly.

Cheers,

Rudi

_______________________________________________
armedbear-devel mailing list
armedbear-devel <at> common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel

headers

Erik Huelsmann | 2 Apr 2013 19:55

Re: [armedbear-devel] Questions about r14452 (Fix cl-cont)

Forgot to mention: Pre 14452 it was enough to just quickload cl-cont to trigger the bug.

Bye, Erik

sent from my phone

On Apr 2, 2013 7:52 PM, "Erik Huelsmann" <ehuels <at> gmail.com> wrote:

Hi Rudi,

Basically the expectation we violated was the fact that %set-lambda-name didn't want to take a funcallable instance as its function argument. The commit changed that and used the first slot of the funcallable instance class layout (called NAME) to store the name. I guess we forget to finalize the class somewhere, so the spots don't appear?

Bye,

Erik.

sent from my phone

On Apr 2, 2013 7:32 PM, "Rudolf Schlatte" <rudi <at> constantly.at> wrote:
Hi,

I suspect that the changes in commit r14452 only work by accident.

The definions of SLOT_INDEX_NAME in FuncallableStandardClass vs FuncallableStandardObject are no problem, since the latter simply isn't used anywhere. But the new methods getName() and setName() assume that a) the FuncallableStandardObject instance has at least one slot, and b) that slot holds a name. Since
(class-slots (find-class 'funcallable-standard-object))
returns NIL, these assumptions hold only when the objects are in fact generic functions.

Amusingly,
(function-lambda-expression (make-instance 'funcallable-standard-object))
runs into the type error on line 2666 of Primitives.lisp ("The value #<FUNCALLABLE-STANDARD-OBJECT {665C5A83}> is not of type FUNCTION.") even though
(typep (make-instance 'funcallable-standard-object) 'function) => T
so I can't trigger the array-out-of-bounds error right now.

Could I have a recipe for triggering the bug? I'd like to see how cl-cont uses funcallable-standard-objects, and what expectations we violate. I believe I can fix this properly.

Cheers,

Rudi

_______________________________________________
armedbear-devel mailing list
armedbear-devel <at> common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel

headers

Erik Huelsmann | 2 Apr 2013 19:52

[armedbear-devel] Fwd: Re: Questions about r14452 (Fix cl-cont)

sent from my phone

---------- Forwarded message ----------
From: ehuels <at> gmail.com
Date: Apr 2, 2013 7:51 PM
Subject: Re: Questions about r14452 (Fix cl-cont)
To: Rudolf Schlatte <rudi <at> constantly.at>
Cc:

Hi Rudi,

Basically the expectation we violated was the fact that %set-lambda-name didn't want to take a funcallable instance as its function argument. The commit changed that and used the first slot of the funcallable instance class layout (called NAME) to store the name. I guess we forget to finalize the class somewhere, so the spots don't appear?

Bye,

Erik.

sent from my phone

On Apr 2, 2013 7:32 PM, "Rudolf Schlatte" <rudi <at> constantly.at> wrote:
Hi,

I suspect that the changes in commit r14452 only work by accident.

The definions of SLOT_INDEX_NAME in FuncallableStandardClass vs FuncallableStandardObject are no problem, since the latter simply isn't used anywhere. But the new methods getName() and setName() assume that a) the FuncallableStandardObject instance has at least one slot, and b) that slot holds a name. Since
(class-slots (find-class 'funcallable-standard-object))
returns NIL, these assumptions hold only when the objects are in fact generic functions.

Amusingly,
(function-lambda-expression (make-instance 'funcallable-standard-object))
runs into the type error on line 2666 of Primitives.lisp ("The value #<FUNCALLABLE-STANDARD-OBJECT {665C5A83}> is not of type FUNCTION.") even though
(typep (make-instance 'funcallable-standard-object) 'function) => T
so I can't trigger the array-out-of-bounds error right now.

Could I have a recipe for triggering the bug? I'd like to see how cl-cont uses funcallable-standard-objects, and what expectations we violate. I believe I can fix this properly.

Cheers,

Rudi

_______________________________________________
armedbear-devel mailing list
armedbear-devel <at> common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel

headers

Rudolf Schlatte | 2 Apr 2013 19:32

[armedbear-devel] Questions about r14452 (Fix cl-cont)

Hi,

I suspect that the changes in commit r14452 only work by accident.

The definions of SLOT_INDEX_NAME in FuncallableStandardClass vs FuncallableStandardObject are no
problem, since the latter simply isn't used anywhere.  But the new methods getName() and setName() assume
that a) the FuncallableStandardObject instance has at least one slot, and b) that slot holds a name.  Since 
    (class-slots (find-class 'funcallable-standard-object))
returns NIL, these assumptions hold only when the objects are in fact generic functions.

Amusingly,
    (function-lambda-expression (make-instance 'funcallable-standard-object))
runs into the type error on line 2666 of Primitives.lisp ("The value #<FUNCALLABLE-STANDARD-OBJECT
{665C5A83}> is not of type FUNCTION.") even though
    (typep (make-instance 'funcallable-standard-object) 'function) => T
so I can't trigger the array-out-of-bounds error right now.

Could I have a recipe for triggering the bug?  I'd like to see how cl-cont uses
funcallable-standard-objects, and what expectations we violate.  I believe I can fix this properly.

Cheers,

Rudi

headers

Erik Huelsmann | 1 Apr 2013 14:15

[armedbear-devel] ASDF-JAR vs ASDF3 deployment support

Hi all,

Some years ago, Mark wrote ASDF-JAR; a solution to package ASDF systems and their dependencies in JARs.

As it turns out, the solution chosen has some problems with correctly packaging ASDs for systems which use the #. reader macro. As an example: bordeaux-threads uses a version.lisp-sexp file to canonically define the version number and the system definition reads that file in a #. form. However, this file isn't packaged because it's not part of the file listing in the system definition. This means asdf can't load the definition after deployment.

With the advent of ASDF3, its developers have created infrastructure to combine fasls of a single system and even fasls of a system and all its dependencies into a single large combined FASL. Among others, this new infrastructure does not run into the issue ASDF-JAR is running into. While not completely equivalent to the JAR solution, using this infrastructure would help us a huge step along that way of getting better deployment. My current thinking is that if the single-packed-FASL needs to end up as a jar, we can work on that as a next step.

The first step to take would be to create "concatenate fasls" functionality, which would do the following:

* Detect all the file entries in all the passed-in fasls

* Create a new output zip file with directories for each fasl being packed, populated with the files and directories of that fasl

* create a toplevel ._ file which loads the sub-fasls' ._ files

Comments?

Bye,

Erik.

_______________________________________________
armedbear-devel mailing list
armedbear-devel <at> common-lisp.net
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/armedbear-devel

6	May 2013
30	April 2013
56	March 2013
70	February 2013
34	January 2013
49	December 2012
18	November 2012
55	October 2012
26	September 2012
100	August 2012
44	July 2012
36	June 2012
55	May 2012
41	April 2012
22	March 2012
27	February 2012
59	January 2012
11	December 2011
23	November 2011
45	October 2011
15	September 2011
42	August 2011
50	July 2011
30	June 2011
41	May 2011
20	April 2011
67	March 2011
51	February 2011
115	January 2011
41	December 2010
93	November 2010
47	October 2010
34	September 2010
39	August 2010
29	July 2010
74	June 2010
135	May 2010
135	April 2010
164	March 2010
116	February 2010
157	January 2010
64	December 2009
93	November 2009
93	October 2009
140	September 2009
54	August 2009
8	July 2009
32	June 2009
112	May 2009
89	April 2009
121	March 2009
99	February 2009
181	January 2009
143	December 2008
81	November 2008
136	October 2008
141	September 2008
89	August 2008
4	July 2008
7	June 2008
2	May 2008
16	February 2008
34	January 2008
7	December 2007
11	November 2007
24	October 2007
8	September 2007
22	August 2007
10	July 2007
18	June 2007
35	May 2007
5	April 2007
28	March 2007
51	February 2007
6	January 2007
4	December 2006
5	November 2006
18	October 2006
13	September 2006
13	August 2006
2	July 2006
16	June 2006
12	May 2006
10	April 2006
19	March 2006
13	February 2006
39	January 2006
6	December 2005
20	November 2005
64	October 2005
24	September 2005
12	August 2005
15	July 2005
40	June 2005
25	May 2005
15	April 2005
11	March 2005
31	February 2005
41	January 2005
17	December 2004
36	November 2004
36	October 2004
51	September 2004
164	August 2004
31	July 2004
46	June 2004
13	May 2004
12	April 2004
9	March 2004
1	February 2004
21	January 2004
109	December 2003
66	November 2003
70	October 2003
11	September 2003
33	August 2003
35	July 2003
2	June 2003

Mon	Tue	Wed	Thu	Fri	Sat	Sun

		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31