Michael J. Barillier | 3 Feb 18:28 2011
Picon

Compiler implementation

I've been curious about how the SBCL compiler works (but not enough to
code-dive).  Docs say that SBCL compiles to native code, and
disassembling a function shows native instructions, but how does SBCL
load that into an executable area of memory?  Intel processors and
others don't allow execution of memory segments (as I recall from
assembly hacking in a previous life), so how does SBCL generate a stream
of machine instructions and call the entry point?

--

-- 
Michael J. Barillier   ///   http://www.blackwolfinfosys.net/~blackwolf/
_O_|  ``Ignorance breeds monsters to fill up the vacancies of the soul
__O|  that are unoccupied by the verities of knowledge.''
OOO|      -- Horace Mann

------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires 
February 28th, so secure your free ArcSight Logger TODAY! 
http://p.sf.net/sfu/arcsight-sfd2d
Roman Marynchak | 3 Feb 21:52 2011
Picon

Re: Compiler implementation

In general, a process of the generated code execution is not SBCL-specific. One of the possible scenarios is (on Windows):

1. Allocate a memory region with VirtualAllocEx, using the appropriate protection constant. See
    http://msdn.microsoft.com/en-us/library/aa366890%28v=vs.85%29.aspx

2. Write the function's machine code to the allocated region.

3. To execute the function, pass the arguments using registers/stack, and jump to the machine code (see CALL instruction).

I don't know whether the above approach is used by SBCL, but it works. Maybe there are some other ways.


Roman


2011/2/3 Michael J. Barillier <blackwolf <at> blackwolfinfosys.net>
I've been curious about how the SBCL compiler works (but not enough to
code-dive).  Docs say that SBCL compiles to native code, and
disassembling a function shows native instructions, but how does SBCL
load that into an executable area of memory?  Intel processors and
others don't allow execution of memory segments (as I recall from
assembly hacking in a previous life), so how does SBCL generate a stream
of machine instructions and call the entry point?

--
Michael J. Barillier   ///   http://www.blackwolfinfosys.net/~blackwolf/
_O_|  ``Ignorance breeds monsters to fill up the vacancies of the soul
__O|  that are unoccupied by the verities of knowledge.''
OOO|      -- Horace Mann

------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help

------------------------------------------------------------------------------
The modern datacenter depends on network connectivity to access resources
and provide services. The best practices for maximizing a physical server's
connectivity to a physical network are well understood - see how these
rules translate into the virtual world? 
http://p.sf.net/sfu/oracle-sfdevnlfb
_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help
Martin Cracauer | 3 Feb 22:01 2011

Re: Compiler implementation

Michael J. Barillier wrote on Thu, Feb 03, 2011 at 12:28:35PM -0500: 
> I've been curious about how the SBCL compiler works (but not enough to
> code-dive).  Docs say that SBCL compiles to native code, and
> disassembling a function shows native instructions, but how does SBCL
> load that into an executable area of memory?  Intel processors and
> others don't allow execution of memory segments (as I recall from
> assembly hacking in a previous life), so how does SBCL generate a stream
> of machine instructions and call the entry point?

You get memory from the OS via sbrk or mmap.  In the case of the
latter nothing prevents you from using the PROT_EXEC flag on that
region.  Whether a region originated from mmap or sbrk (or the stack
for that matter), you can change the page protection later to make it
executable. 

You can just dump instructions into memory and jump right into it.

Martin
--

-- 
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Martin Cracauer <cracauer <at> cons.org>   http://www.cons.org/cracauer/

------------------------------------------------------------------------------
The modern datacenter depends on network connectivity to access resources
and provide services. The best practices for maximizing a physical server's
connectivity to a physical network are well understood - see how these
rules translate into the virtual world? 
http://p.sf.net/sfu/oracle-sfdevnlfb
Peter Keller | 3 Feb 21:56 2011
Picon

Re: Compiler implementation

On Thu, Feb 03, 2011 at 12:28:35PM -0500, Michael J. Barillier wrote:
> I've been curious about how the SBCL compiler works (but not enough to
> code-dive).  Docs say that SBCL compiles to native code, and
> disassembling a function shows native instructions, but how does SBCL
> load that into an executable area of memory?  Intel processors and
> others don't allow execution of memory segments (as I recall from
> assembly hacking in a previous life), so how does SBCL generate a stream
> of machine instructions and call the entry point?

Under unix, the system level calls of mmap() and mprotect() are your
friends....

-pete

------------------------------------------------------------------------------
The modern datacenter depends on network connectivity to access resources
and provide services. The best practices for maximizing a physical server's
connectivity to a physical network are well understood - see how these
rules translate into the virtual world? 
http://p.sf.net/sfu/oracle-sfdevnlfb
Jim Wise | 3 Feb 23:32 2011

Re: Compiler implementation

Peter Keller <psilord <at> cs.wisc.edu> writes:

> On Thu, Feb 03, 2011 at 12:28:35PM -0500, Michael J. Barillier wrote:
>> I've been curious about how the SBCL compiler works (but not enough to
>> code-dive).  Docs say that SBCL compiles to native code, and
>> disassembling a function shows native instructions, but how does SBCL
>> load that into an executable area of memory?  Intel processors and
>> others don't allow execution of memory segments (as I recall from
>> assembly hacking in a previous life), so how does SBCL generate a stream
>> of machine instructions and call the entry point?
>
> Under unix, the system level calls of mmap() and mprotect() are your
> friends....

... and it's worth noting that ld.so / rtld, the shared library loader,
is itself a user-space program whose source is often available for
comparison.

--

-- 
				Jim Wise
				jwise <at> draga.com
------------------------------------------------------------------------------
The modern datacenter depends on network connectivity to access resources
and provide services. The best practices for maximizing a physical server's
connectivity to a physical network are well understood - see how these
rules translate into the virtual world? 
http://p.sf.net/sfu/oracle-sfdevnlfb
_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help
Roman Marynchak | 14 Feb 19:43 2011
Picon

How to debug the dynamically generated code?

Hello,

  I want to debug the code which is generated and compiled during the program execution. While there is a workaround (I can dump the code into a file and load it on the fly), I would like to ask for a more elegant solution (if any).

  The next very simplified code demonstrates the real-life problem:

(declaim (optimize (debug 3) (speed 0) (safety 3)))

(defun generate-lambda (x)
  `(lambda ()
     (let ((y ,(+ x 7))
            (z ,(* x x)))
       (break)
       (print y)
       (print z))))

(defun test ()
  (let ((lm (generate-lambda 8)))
    (funcall (compile nil lm))))

After this code is compiled and loaded, I evaluate (test) and SLIME shows that there is no information about the locals in the generated lambda:


Backtrace:
  0: (BREAK "break")
  1: ((LAMBDA ()))
      [No Locals]
  2: (TEST)
      Locals:
        LM = (LAMBDA () ..)
  3: (SB-INT:SIMPLE-EVAL-IN-LEXENV (TEST) #<NULL-LEXENV>)
  4: ((LAMBDA ()))

Is there any way to make these local variables visible?

Thank you,
Roman

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb
_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help
Pascal J. Bourguignon | 14 Feb 19:53 2011
Face

Re: How to debug the dynamically generated code?

Roman Marynchak <roman.marynchak <at> gmail.com> writes:

> Hello,
>
>  I want to debug the code which is generated and compiled during the program execution. While there is a
workaround (I can dump the code into a file and load it on the fly), I would like to ask for a more
> elegant solution (if any).
>
>   The next very simplified code demonstrates the real-life problem:
>
> (declaim (optimize (debug 3) (speed 0) (safety 3)))
>
> (defun generate-lambda (x)
>   `(lambda ()
>      (let ((y ,(+ x 7))
>             (z ,(* x x)))
>        (break)
>        (print y)
>        (print z))))
>
> (defun test ()
>   (let ((lm (generate-lambda 8)))
>     (funcall (compile nil lm))))
>
> After this code is compiled and loaded, I evaluate (test) and SLIME
> shows that there is no information about the locals in the generated
> lambda:
>
> Backtrace:
>   0: (BREAK "break")
>   1: ((LAMBDA ()))
>       [No Locals]
>   2: (TEST)
>       Locals:
>         LM = (LAMBDA () ..)
>   3: (SB-INT:SIMPLE-EVAL-IN-LEXENV (TEST) #<NULL-LEXENV>)
>   4: ((LAMBDA ()))
>
> Is there any way to make these local variables visible?

My guess is that despite (debug 3), the compiler optimize out the
useless variables. 

My advice would be to make use of the variable, ie. to mutate them:

 (defun generate-lambda (x)
   `(lambda ()
      (let ((y ,(+ x 7))
            (z ,(* x x)))
        (break)
        (print y)
        (print z)
        (setf y (+ y z)
              z (- y z)
              y (- y z)))))

--

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb
Roman Marynchak | 14 Feb 20:02 2011
Picon

Re: How to debug the dynamically generated code?

I have changed the example to

(declaim
 (optimize (debug 3) (speed 0) (safety 3)))

(defun provide-runtime-value ()
  (sxhash (random 100000)))

(defun generate-lambda (x)
  `(lambda ()
     (let ((y ,(+ x 7))
       (z (provide-runtime-value)))
       (break)
       (print y)
       (print (+ z 4)))))

(defun test ()
  (let ((lm (generate-lambda 8)))
    (funcall (compile nil lm))))

but it still shows no local variables.

Regards,
Roman

2011/2/14 Pascal J. Bourguignon <pjb <at> informatimago.com>
Roman Marynchak <roman.marynchak <at> gmail.com> writes:

> Hello,
>
>  I want to debug the code which is generated and compiled during the program execution. While there is a workaround (I can dump the code into a file and load it on the fly), I would like to ask for a more
> elegant solution (if any).
>
>   The next very simplified code demonstrates the real-life problem:
>
> (declaim (optimize (debug 3) (speed 0) (safety 3)))
>
> (defun generate-lambda (x)
>   `(lambda ()
>      (let ((y ,(+ x 7))
>             (z ,(* x x)))
>        (break)
>        (print y)
>        (print z))))
>
> (defun test ()
>   (let ((lm (generate-lambda 8)))
>     (funcall (compile nil lm))))
>
> After this code is compiled and loaded, I evaluate (test) and SLIME
> shows that there is no information about the locals in the generated
> lambda:
>
> Backtrace:
>   0: (BREAK "break")
>   1: ((LAMBDA ()))
>       [No Locals]
>   2: (TEST)
>       Locals:
>         LM = (LAMBDA () ..)
>   3: (SB-INT:SIMPLE-EVAL-IN-LEXENV (TEST) #<NULL-LEXENV>)
>   4: ((LAMBDA ()))
>
> Is there any way to make these local variables visible?

My guess is that despite (debug 3), the compiler optimize out the
useless variables.

My advice would be to make use of the variable, ie. to mutate them:

 (defun generate-lambda (x)
  `(lambda ()
     (let ((y ,(+ x 7))
           (z ,(* x x)))
       (break)
       (print y)
       (print z)
       (setf y (+ y z)
             z (- y z)
             y (- y z)))))



--
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.


------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb
_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb
_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help
Nikodemus Siivola | 14 Feb 20:02 2011
Picon

Re: How to debug the dynamically generated code?

;;; In your example the constants get optimized despite
;;; any debug setting.
(defun generate-lambda ()
  `(lambda (x)
     (let ((y (+ x 7))
           (z (* x x)))
       (break)
       (print y)
       (print z))))

;;; The DECLAIM is file local, and doesn't affect COMPILE.
(defun test (&optional policy)
  (let ((lm (generate-lambda)))
    (funcall (funcall (compile nil
                               `(lambda ()
                                  (declare (optimize , <at> policy))
                                  ,lm)))
             8)))

(test '(debug 3))

...gives:

break
   [Condition of type SIMPLE-CONDITION]

Restarts:
 0: [CONTINUE]         Return from BREAK.
 1: [RETRY]            Retry SLIME interactive evaluation request.
 2: [*ABORT]           Return to SLIME's top level.
 3: [TERMINATE-THREAD] Terminate this thread (#<THREAD "worker"
RUNNING {100312CAA1}>)

Backtrace:
  0: (BREAK "break")
  1: ((LAMBDA (X)) 8)
      Locals:
        X = 8
        Y = 15
        Z = 64

Cheers,

 -- Nikodemus

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb
Roman Marynchak | 14 Feb 20:13 2011
Picon

Re: How to debug the dynamically generated code?

Yes, I see it now.

Thanks a lot for the help!

Roman

2011/2/14 Nikodemus Siivola <nikodemus <at> random-state.net>
;;; In your example the constants get optimized despite
;;; any debug setting.
(defun generate-lambda ()
 `(lambda (x)
    (let ((y (+ x 7))
          (z (* x x)))
      (break)
      (print y)
      (print z))))

;;; The DECLAIM is file local, and doesn't affect COMPILE.
(defun test (&optional policy)
 (let ((lm (generate-lambda)))
   (funcall (funcall (compile nil
                              `(lambda ()
                                 (declare (optimize , <at> policy))
                                 ,lm)))
            8)))

(test '(debug 3))

...gives:

break
  [Condition of type SIMPLE-CONDITION]

Restarts:
 0: [CONTINUE]         Return from BREAK.
 1: [RETRY]            Retry SLIME interactive evaluation request.
 2: [*ABORT]           Return to SLIME's top level.
 3: [TERMINATE-THREAD] Terminate this thread (#<THREAD "worker"
RUNNING {100312CAA1}>)

Backtrace:
 0: (BREAK "break")
 1: ((LAMBDA (X)) 8)
     Locals:
       X = 8
       Y = 15
       Z = 64

Cheers,

 -- Nikodemus

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb
_______________________________________________
Sbcl-help mailing list
Sbcl-help <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sbcl-help

Gmane