General discussion of Steel Bank Common Lisp

Michael J. Barillier | 3 Feb 2011 18:28

Compiler implementation

I've been curious about how the SBCL compiler works (but not enough to
code-dive).  Docs say that SBCL compiles to native code, and
disassembling a function shows native instructions, but how does SBCL
load that into an executable area of memory?  Intel processors and
others don't allow execution of memory segments (as I recall from
assembly hacking in a previous life), so how does SBCL generate a stream
of machine instructions and call the entry point?

--

-- 
Michael J. Barillier   ///   http://www.blackwolfinfosys.net/~blackwolf/
_O_|  ``Ignorance breeds monsters to fill up the vacancies of the soul
__O|  that are unoccupied by the verities of knowledge.''
OOO|      -- Horace Mann

------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires 
February 28th, so secure your free ArcSight Logger TODAY! 
http://p.sf.net/sfu/arcsight-sfd2d

headers

Roman Marynchak | 3 Feb 2011 21:52

Re: Compiler implementation

In general, a process of the generated code execution is not SBCL-specific. One of the possible scenarios is (on Windows):

1. Allocate a memory region with VirtualAllocEx, using the appropriate protection constant. See
http://msdn.microsoft.com/en-us/library/aa366890%28v=vs.85%29.aspx

2. Write the function's machine code to the allocated region.

3. To execute the function, pass the arguments using registers/stack, and jump to the machine code (see CALL instruction).

I don't know whether the above approach is used by SBCL, but it works. Maybe there are some other ways.

Roman

2011/2/3 Michael J. Barillier <blackwolf <at> blackwolfinfosys.net>

I've been curious about how the SBCL compiler works (but not enough to
code-dive). Docs say that SBCL compiles to native code, and
disassembling a function shows native instructions, but how does SBCL
load that into an executable area of memory? Intel processors and
others don't allow execution of memory segments (as I recall from
assembly hacking in a previous life), so how does SBCL generate a stream
of machine instructions and call the entry point?

--
Michael J. Barillier /// http://www.blackwolfinfosys.net/~blackwolf/
_O_| ``Ignorance breeds monsters to fill up the vacancies of the soul
__O| that are unoccupied by the verities of knowledge.''
OOO| -- Horace Mann

------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help

------------------------------------------------------------------------------
The modern datacenter depends on network connectivity to access resources
and provide services. The best practices for maximizing a physical server's
connectivity to a physical network are well understood - see how these
rules translate into the virtual world? 
http://p.sf.net/sfu/oracle-sfdevnlfb

_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help

headers

Martin Cracauer | 3 Feb 2011 22:01

Re: Compiler implementation

Michael J. Barillier wrote on Thu, Feb 03, 2011 at 12:28:35PM -0500: 
> I've been curious about how the SBCL compiler works (but not enough to
> code-dive).  Docs say that SBCL compiles to native code, and
> disassembling a function shows native instructions, but how does SBCL
> load that into an executable area of memory?  Intel processors and
> others don't allow execution of memory segments (as I recall from
> assembly hacking in a previous life), so how does SBCL generate a stream
> of machine instructions and call the entry point?

You get memory from the OS via sbrk or mmap.  In the case of the
latter nothing prevents you from using the PROT_EXEC flag on that
region.  Whether a region originated from mmap or sbrk (or the stack
for that matter), you can change the page protection later to make it
executable. 

You can just dump instructions into memory and jump right into it.

Martin
--

-- 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Martin Cracauer <cracauer <at> cons.org>   http://www.cons.org/cracauer/

------------------------------------------------------------------------------
The modern datacenter depends on network connectivity to access resources
and provide services. The best practices for maximizing a physical server's
connectivity to a physical network are well understood - see how these
rules translate into the virtual world? 
http://p.sf.net/sfu/oracle-sfdevnlfb

headers

Peter Keller | 3 Feb 2011 21:56

Re: Compiler implementation

On Thu, Feb 03, 2011 at 12:28:35PM -0500, Michael J. Barillier wrote:
> I've been curious about how the SBCL compiler works (but not enough to
> code-dive).  Docs say that SBCL compiles to native code, and
> disassembling a function shows native instructions, but how does SBCL
> load that into an executable area of memory?  Intel processors and
> others don't allow execution of memory segments (as I recall from
> assembly hacking in a previous life), so how does SBCL generate a stream
> of machine instructions and call the entry point?

Under unix, the system level calls of mmap() and mprotect() are your
friends....

-pete

------------------------------------------------------------------------------
The modern datacenter depends on network connectivity to access resources
and provide services. The best practices for maximizing a physical server's
connectivity to a physical network are well understood - see how these
rules translate into the virtual world? 
http://p.sf.net/sfu/oracle-sfdevnlfb

headers

Jim Wise | 3 Feb 2011 23:32

Re: Compiler implementation

Peter Keller <psilord <at> cs.wisc.edu> writes:

> On Thu, Feb 03, 2011 at 12:28:35PM -0500, Michael J. Barillier wrote:
>> I've been curious about how the SBCL compiler works (but not enough to
>> code-dive).  Docs say that SBCL compiles to native code, and
>> disassembling a function shows native instructions, but how does SBCL
>> load that into an executable area of memory?  Intel processors and
>> others don't allow execution of memory segments (as I recall from
>> assembly hacking in a previous life), so how does SBCL generate a stream
>> of machine instructions and call the entry point?
>
> Under unix, the system level calls of mmap() and mprotect() are your
> friends....

... and it's worth noting that ld.so / rtld, the shared library loader,
is itself a user-space program whose source is often available for
comparison.

--

-- 
				Jim Wise
				jwise <at> draga.com

------------------------------------------------------------------------------
The modern datacenter depends on network connectivity to access resources
and provide services. The best practices for maximizing a physical server's
connectivity to a physical network are well understood - see how these
rules translate into the virtual world? 
http://p.sf.net/sfu/oracle-sfdevnlfb

_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help

headers

Roman Marynchak | 14 Feb 2011 19:43

How to debug the dynamically generated code?

Hello,

I want to debug the code which is generated and compiled during the program execution. While there is a workaround (I can dump the code into a file and load it on the fly), I would like to ask for a more elegant solution (if any).

The next very simplified code demonstrates the real-life problem:

(declaim (optimize (debug 3) (speed 0) (safety 3)))

(defun generate-lambda (x)
`(lambda ()
     (let ((y ,(+ x 7))
            (z ,(* x x)))
       (break)
       (print y)
       (print z))))

(defun test ()
(let ((lm (generate-lambda 8)))
    (funcall (compile nil lm))))

After this code is compiled and loaded, I evaluate (test) and SLIME shows that there is no information about the locals in the generated lambda:

Backtrace:
0: (BREAK "break")
1: ((LAMBDA ()))
      [No Locals]
2: (TEST)
      Locals:
        LM = (LAMBDA () ..)
3: (SB-INT:SIMPLE-EVAL-IN-LEXENV (TEST) #<NULL-LEXENV>)
4: ((LAMBDA ()))

Is there any way to make these local variables visible?

Thank you,
Roman

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb

_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help

headers

Pascal J. Bourguignon | 14 Feb 2011 19:53

Re: How to debug the dynamically generated code?

Roman Marynchak <roman.marynchak <at> gmail.com> writes:

> Hello,
>
>  I want to debug the code which is generated and compiled during the program execution. While there is a
workaround (I can dump the code into a file and load it on the fly), I would like to ask for a more
> elegant solution (if any).
>
>   The next very simplified code demonstrates the real-life problem:
>
> (declaim (optimize (debug 3) (speed 0) (safety 3)))
>
> (defun generate-lambda (x)
>   `(lambda ()
>      (let ((y ,(+ x 7))
>             (z ,(* x x)))
>        (break)
>        (print y)
>        (print z))))
>
> (defun test ()
>   (let ((lm (generate-lambda 8)))
>     (funcall (compile nil lm))))
>
> After this code is compiled and loaded, I evaluate (test) and SLIME
> shows that there is no information about the locals in the generated
> lambda:
>
> Backtrace:
>   0: (BREAK "break")
>   1: ((LAMBDA ()))
>       [No Locals]
>   2: (TEST)
>       Locals:
>         LM = (LAMBDA () ..)
>   3: (SB-INT:SIMPLE-EVAL-IN-LEXENV (TEST) #<NULL-LEXENV>)
>   4: ((LAMBDA ()))
>
> Is there any way to make these local variables visible?

My guess is that despite (debug 3), the compiler optimize out the
useless variables. 

My advice would be to make use of the variable, ie. to mutate them:

 (defun generate-lambda (x)
   `(lambda ()
      (let ((y ,(+ x 7))
            (z ,(* x x)))
        (break)
        (print y)
        (print z)
        (setf y (+ y z)
              z (- y z)
              y (- y z)))))

--

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb

headers

Roman Marynchak | 14 Feb 2011 20:02

Re: How to debug the dynamically generated code?

I have changed the example to

(declaim
(optimize (debug 3) (speed 0) (safety 3)))

(defun provide-runtime-value ()
(sxhash (random 100000)))

(defun generate-lambda (x)
`(lambda ()
     (let ((y ,(+ x 7))
       (z (provide-runtime-value)))
       (break)
       (print y)
       (print (+ z 4)))))

(defun test ()
(let ((lm (generate-lambda 8)))
    (funcall (compile nil lm))))

but it still shows no local variables.

Regards,
Roman

2011/2/14 Pascal J. Bourguignon <pjb <at> informatimago.com>

Roman Marynchak <roman.marynchak <at> gmail.com> writes:

> Hello,
>
> I want to debug the code which is generated and compiled during the program execution. While there is a workaround (I can dump the code into a file and load it on the fly), I would like to ask for a more
> elegant solution (if any).
>
> The next very simplified code demonstrates the real-life problem:
>
> (declaim (optimize (debug 3) (speed 0) (safety 3)))
>
> (defun generate-lambda (x)
> `(lambda ()
> (let ((y ,(+ x 7))
> (z ,(* x x)))
> (break)
> (print y)
> (print z))))
>
> (defun test ()
> (let ((lm (generate-lambda 8)))
> (funcall (compile nil lm))))
>
> After this code is compiled and loaded, I evaluate (test) and SLIME
> shows that there is no information about the locals in the generated
> lambda:
>
> Backtrace:
> 0: (BREAK "break")
> 1: ((LAMBDA ()))
> [No Locals]
> 2: (TEST)
> Locals:
> LM = (LAMBDA () ..)
> 3: (SB-INT:SIMPLE-EVAL-IN-LEXENV (TEST) #<NULL-LEXENV>)
> 4: ((LAMBDA ()))
>
> Is there any way to make these local variables visible?

My guess is that despite (debug 3), the compiler optimize out the
useless variables.

My advice would be to make use of the variable, ie. to mutate them:

(defun generate-lambda (x)
`(lambda ()
(let ((y ,(+ x 7))
(z ,(* x x)))
(break)
(print y)
(print z)

(setf y (+ y z)
z (- y z)
y (- y z)))))

--
__Pascal Bourguignon__ http://www.informatimago.com/
A bad day in () is better than a good day in {}.

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb
_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb

_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help

headers

Nikodemus Siivola | 14 Feb 2011 20:02

Re: How to debug the dynamically generated code?

;;; In your example the constants get optimized despite
;;; any debug setting.
(defun generate-lambda ()
  `(lambda (x)
     (let ((y (+ x 7))
           (z (* x x)))
       (break)
       (print y)
       (print z))))

;;; The DECLAIM is file local, and doesn't affect COMPILE.
(defun test (&optional policy)
  (let ((lm (generate-lambda)))
    (funcall (funcall (compile nil
                               `(lambda ()
                                  (declare (optimize , <at> policy))
                                  ,lm)))
             8)))

(test '(debug 3))

...gives:

break
   [Condition of type SIMPLE-CONDITION]

Restarts:
 0: [CONTINUE]         Return from BREAK.
 1: [RETRY]            Retry SLIME interactive evaluation request.
 2: [*ABORT]           Return to SLIME's top level.
 3: [TERMINATE-THREAD] Terminate this thread (#<THREAD "worker"
RUNNING {100312CAA1}>)

Backtrace:
  0: (BREAK "break")
  1: ((LAMBDA (X)) 8)
      Locals:
        X = 8
        Y = 15
        Z = 64

Cheers,

 -- Nikodemus

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb

headers

Roman Marynchak | 14 Feb 2011 20:13

Re: How to debug the dynamically generated code?

Yes, I see it now.

Thanks a lot for the help!

Roman

2011/2/14 Nikodemus Siivola <nikodemus <at> random-state.net>

;;; In your example the constants get optimized despite
;;; any debug setting.
(defun generate-lambda ()
`(lambda (x)
(let ((y (+ x 7))
(z (* x x)))
(break)
(print y)
(print z))))

;;; The DECLAIM is file local, and doesn't affect COMPILE.
(defun test (&optional policy)
(let ((lm (generate-lambda)))
(funcall (funcall (compile nil
`(lambda ()
(declare (optimize , <at> policy))
,lm)))
8)))

(test '(debug 3))

...gives:

break
[Condition of type SIMPLE-CONDITION]

Restarts:
0: [CONTINUE] Return from BREAK.
1: [RETRY] Retry SLIME interactive evaluation request.
2: [*ABORT] Return to SLIME's top level.
3: [TERMINATE-THREAD] Terminate this thread (#<THREAD "worker"
RUNNING {100312CAA1}>)

Backtrace:
0: (BREAK "break")
1: ((LAMBDA (X)) 8)
Locals:
X = 8
Y = 15
Z = 64

Cheers,

-- Nikodemus

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb

_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help

10	April 2013
17	March 2013
8	February 2013
9	January 2013
4	December 2012
14	November 2012
10	October 2012
10	September 2012
12	August 2012
40	July 2012
13	June 2012
58	May 2012
16	April 2012
17	March 2012
22	February 2012
9	January 2012
21	December 2011
21	November 2011
25	October 2011
21	September 2011
38	August 2011
15	July 2011
28	June 2011
17	May 2011
31	April 2011
5	March 2011
31	February 2011
17	January 2011
19	December 2010
41	November 2010
33	October 2010
43	September 2010
31	August 2010
43	July 2010
24	June 2010
8	May 2010
59	April 2010
55	March 2010
44	February 2010
26	January 2010
31	December 2009
46	November 2009
59	October 2009
52	September 2009
35	August 2009
48	July 2009
58	June 2009
29	May 2009
15	April 2009
18	March 2009
31	February 2009
33	January 2009
83	December 2008
42	November 2008
52	October 2008
19	September 2008
52	August 2008
24	July 2008
25	June 2008
1	May 2008
13	April 2008
28	March 2008
17	February 2008
36	January 2008
59	December 2007
9	November 2007
16	October 2007
11	September 2007
24	August 2007
31	July 2007
31	June 2007
60	May 2007
40	April 2007
50	March 2007
37	February 2007
41	January 2007
69	December 2006
20	November 2006
39	October 2006
51	September 2006
23	August 2006
11	July 2006
29	June 2006
19	May 2006
26	April 2006
19	March 2006
37	February 2006
45	January 2006
24	December 2005
30	November 2005
17	October 2005
28	September 2005
26	August 2005
33	July 2005
37	June 2005
69	May 2005
65	April 2005
39	March 2005
32	February 2005
24	January 2005
27	December 2004
80	November 2004
20	October 2004
38	September 2004
14	August 2004
23	July 2004
15	June 2004
18	May 2004
69	April 2004
21	March 2004
15	February 2004
32	January 2004
19	December 2003
14	November 2003
3	October 2003
27	September 2003
28	August 2003
3	July 2003
10	June 2003
27	May 2003
14	March 2003
4	February 2003
5	January 2003
8	December 2002
3	September 2002
2	August 2002
5	May 2002

Mon	Tue	Wed	Thu	Fri	Sat	Sun

	1	2	3	4	5	6
7	8	9	10	11	12	13
14	15	16	17	18	19	20
21	22	23	24	25	26	27
28