Ned Batchelder | 1 Jan 2011 18:18
Favicon
Gravatar

Re: Possible optimization for LOAD_FAST ?

On 12/31/2010 12:51 PM, Cesare Di Mauro wrote:
2010/12/31  <skip <at> pobox.com>

   >> Another example. I can totally remove the variable i, just using the
   >> stack, so a debugger (or, in general, having the tracing enabled)
     >> cannot even find something to change about it.
   Ethan> -1

   Ethan> Debugging is challenging enough as it is -- why would you want to
   Ethan> make it even more difficult?

<snarky>
I don't know.  Maybe he wants his program to run faster.
 </snarky>

:D

"Aggressive" optimizations can be enabled with explicit options, in order to leave normal "debugger-prone" code.
I wish the Python compiler would adopt a strategy of being able to disable optimizations.  I wrote a bug about a "leaky abstraction" optimization messing up coverage testing 2.5 years ago, and it was closed as won't fix: http://bugs.python.org/issue2506.  The debate there centered around, "but that line isn't executed, because it's been optimized away."  It's common in sophisticated compilers (as in, any C compiler) to be able to choose whether you want optimizations for speed, or disabling optimizations for debugging and reasoning about the code.  Python would benefit from the same choice.

--Ned.
 
If you use print statements for the bulk of your debugging (many people do),
unrolling loops doesn't affect your debugging ability.

 Skip

It's a common practice. Also IDEs helps a lot, and advanced interactive shells too (such as DreamPie).

Cesare
_______________________________________________ Python-Dev mailing list Python-Dev <at> python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com
<div>
    On 12/31/2010 12:51 PM, Cesare Di Mauro wrote:
    <blockquote cite="mid:AANLkTimmTOSqc43+qhei=C=p7i8Ry7ufiOZziYMZRpsB <at> mail.gmail.com" type="cite">2010/12/31 &nbsp;<span dir="ltr">&lt;<a moz-do-not-send="true" href="mailto:skip <at> pobox.com">skip <at> pobox.com</a>&gt;</span><br><blockquote class="gmail_quote">
        <div class="im">
<br>
          &nbsp; &nbsp;&gt;&gt; Another example. I can totally remove the variable
          i, just using the<br>
          &nbsp; &nbsp;&gt;&gt; stack, so a debugger (or, in general, having the
          tracing enabled)<br>
          &nbsp; &nbsp; &nbsp;&gt;&gt; cannot even find something to change about it.</div>
      </blockquote>
      <blockquote class="gmail_quote">
        &nbsp; &nbsp;Ethan&gt; -1<br><br>
        &nbsp; &nbsp;Ethan&gt; Debugging is challenging enough as it is -- why
        would you want to<br>
        &nbsp; &nbsp;Ethan&gt; make it even more difficult?<br><br>
        &lt;snarky&gt;<br>
        I don't know. &nbsp;Maybe he wants his program to run faster.<br>
        &nbsp;&lt;/snarky&gt;<br>
</blockquote>
      <div><br></div>
      <div>:D</div>
      <div><br></div>
      <div>"Aggressive" optimizations can be enabled with explicit
        options, in order to leave normal "debugger-prone" code.</div>
    </blockquote>
    I wish the Python compiler would adopt a strategy of being able to
    disable optimizations.&nbsp; I wrote a bug about a "leaky abstraction"
    optimization messing up coverage testing 2.5 years ago, and it was
    closed as won't fix: <a class="moz-txt-link-freetext" href="http://bugs.python.org/issue2506">http://bugs.python.org/issue2506</a>.&nbsp; The debate
    there centered around, "but that line isn't executed, because it's
    been optimized away."&nbsp; It's common in sophisticated compilers (as
    in, any C compiler) to be able to choose whether you want
    optimizations for speed, or disabling optimizations for debugging
    and reasoning about the code.&nbsp; Python would benefit from the same
    choice.<br><br>
    --Ned.<br><blockquote cite="mid:AANLkTimmTOSqc43+qhei=C=p7i8Ry7ufiOZziYMZRpsB <at> mail.gmail.com" type="cite">
      <div>&nbsp;</div>
      <blockquote class="gmail_quote">
        If you use print statements for the bulk of your debugging (many
        people do),<br>
        unrolling loops doesn't affect your debugging ability.<br><br>
        &nbsp;Skip</blockquote>
      <div><br></div>
      <div>It's a common practice. Also IDEs helps a lot, and advanced
        interactive shells too (such as DreamPie).</div>
      <div>
<br>
        Cesare<br>
</div>

_______________________________________________
Python-Dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Python-Dev <at> python.org">Python-Dev <at> python.org</a>
<a class="moz-txt-link-freetext" href="http://mail.python.org/mailman/listinfo/python-dev">http://mail.python.org/mailman/listinfo/python-dev</a>
Unsubscribe: <a class="moz-txt-link-freetext" href="http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com">http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com</a>

    </blockquote>
  </div>
Terry Reedy | 1 Jan 2011 22:07
Picon
Favicon

Re: [Python-checkins] r87603 - python/branches/py3k/Misc/NEWS


On 1/1/2011 5:07 AM, georg.brandl wrote:
> Author: georg.brandl
> Date: Sat Jan  1 11:07:30 2011
> New Revision: 87603
>
> Log:
> Fix issue references.

(add '#' to issue numbers). Whoops, two of those are mine. I am still 
learning and will try to remember to include it in both log messages and 
NEWS entries.

Terry

Victor Stinner | 2 Jan 2011 14:09

Re: Issue #10348: concurrent.futures doesn't work on BSD

Le jeudi 30 décembre 2010 à 17:05 +0100, "Martin v. Löwis" a écrit :
> > I really don't think it is our job to maintain a list of OS/versions
> > which work and don't work.
> 
> Of course not. I would propose a dynamic test: check how many POSIX
> semaphores the installation supports, and fail if it's less than
> 200 (say).

I added informations about FreeBSD, NetBSD, Darwin and OpenBSD to the
issue #10348:
http://bugs.python.org/issue10348#msg125042

The maximum number of POSIX semaphores can be read with sysctl:
 - FreeBSD: "p1003_1b.sem_nsems_max"
 - NetBSD: "kern.posix.semmax"
 - Darwin: "kern.posix.sem.max"
 - OpenBSD: (no support)

Victor

_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
Victor Stinner | 2 Jan 2011 14:17

Re: Possible optimization for LOAD_FAST ?

Le mercredi 29 décembre 2010 à 14:20 +0100, "Martin v. Löwis" a écrit :
> Am 28.12.2010 18:08, schrieb Lukas Lueg:
> > Also, the
> > load_fast in lne 22 to reference x could be taken out of the loop as x
> > will always point to the same object....
> 
> That's not true; a debugger may change the value of x.

That's why Python has the following option:

-O     : optimize generated bytecode slightly; also PYTHONOPTIMIZE=x

I regulary recompile programs with gcc -O0 -g to debug them. It is very
difficult to debug (with gdb) a program compiled with gcc -O2: many
variables are stored in registers, and gdb doesn't support that
correctly.

Victor

_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
R. David Murray | 2 Jan 2011 17:02
Gravatar

Re: [Python-checkins] r87603 - python/branches/py3k/Misc/NEWS

On Sat, 01 Jan 2011 16:07:02 -0500, Terry Reedy <tjreedy <at> udel.edu> wrote:
> On 1/1/2011 5:07 AM, georg.brandl wrote:
> > Author: georg.brandl
> > Date: Sat Jan  1 11:07:30 2011
> > New Revision: 87603
> >
> > Log:
> > Fix issue references.
> 
> (add '#' to issue numbers). Whoops, two of those are mine. I am still 
> learning and will try to remember to include it in both log messages and 
> NEWS entries.

Heh.  I think two of them were mine, and I'm supposed to know better by now.

--
R. David Murray                                      www.bitdance.com
Georg Brandl | 2 Jan 2011 19:13
Picon
Gravatar

Re: r87577 - in python/branches/py3k: Makefile.pre.in configure configure.in pyconfig.h.in

Am 30.12.2010 15:55, schrieb martin.v.loewis:
> Author: martin.v.loewis
> Date: Thu Dec 30 15:55:47 2010
> New Revision: 87577
> 
> Log:
> Build and install libpython3.so.

> Modified: python/branches/py3k/configure.in
> ==============================================================================
> --- python/branches/py3k/configure.in	(original)
> +++ python/branches/py3k/configure.in	Thu Dec 30 15:55:47 2010

>  <at>  <at>  -737,6 +738,10  <at>  <at> 
>  	  BLDLIBRARY='-Wl,-R,$(LIBDIR) -L. -lpython$(LDVERSION)'
>  	  RUNSHARED=LD_LIBRARY_PATH=`pwd`:${LD_LIBRARY_PATH}
>  	  INSTSONAME="$LDLIBRARY".$SOVERSION
> +	  if test $with_pydebug == no
> +	  then
> +	      PY3LIBRARY=libpython3.so
> +	  fi
>            ;;
>      Linux*|GNU*|NetBSD*|FreeBSD*|DragonFly*)
>  	  LDLIBRARY='libpython$(LDVERSION).so'
>  <at>  <at>  -748,6 +753,11  <at>  <at> 
>  		;;
>  	  esac
>  	  INSTSONAME="$LDLIBRARY".$SOVERSION
> +	  PY3LIBRARY=libpython3.so
> +	  if test $with_pydebug == no
> +          then
> +	      PY3LIBRARY=libpython3.so
> +	  fi
>  	  ;;
>      hp*|HP*)
>  	  case `uname -m` in

These changes do not work as written: if --with-pydebug is not given,
$with_pydebug is empty, not "no".  Also, in the second case the
unconditional PY3LIBRARY assignment should probably be deleted.

cheers,
Georg

Martin v. Löwis | 3 Jan 2011 00:59
Picon
Gravatar

Re: Issue #10348: concurrent.futures doesn't work on BSD

> I added informations about FreeBSD, NetBSD, Darwin and OpenBSD to the
> issue #10348:
> http://bugs.python.org/issue10348#msg125042
> 
> The maximum number of POSIX semaphores can be read with sysctl:
>  - FreeBSD: "p1003_1b.sem_nsems_max"
>  - NetBSD: "kern.posix.semmax"
>  - Darwin: "kern.posix.sem.max"
>  - OpenBSD: (no support)

I've been using os.sysconf("SC_SEM_NSEMS_MAX"), which seems to have
worked fine (it's also what POSIX mandates). See #10798.

Regards,
Martin
Ned Batchelder | 3 Jan 2011 02:25
Favicon
Gravatar

Re: Possible optimization for LOAD_FAST ?

On 1/2/2011 8:17 AM, Victor Stinner wrote:
> Le mercredi 29 décembre 2010 à 14:20 +0100, "Martin v. Löwis" a écrit :
>> Am 28.12.2010 18:08, schrieb Lukas Lueg:
>>> Also, the
>>> load_fast in lne 22 to reference x could be taken out of the loop as x
>>> will always point to the same object....
>> That's not true; a debugger may change the value of x.
> That's why Python has the following option:
>
> -O     : optimize generated bytecode slightly; also PYTHONOPTIMIZE=x
>
> I regulary recompile programs with gcc -O0 -g to debug them. It is very
> difficult to debug (with gdb) a program compiled with gcc -O2: many
> variables are stored in registers, and gdb doesn't support that
> correctly.
>
Victor, you seem to be equating the gcc -O flag with the Python -O 
flag.  They are described similarly, but can't be used the same way.  In 
particular, there is no Python equivalent to gcc's -O0: there is no way 
to disable the Python peephole optimizer.

--Ned.
> Victor
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev <at> python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
Alex Gaynor | 3 Jan 2011 02:50
Picon
Gravatar

Re: Possible optimization for LOAD_FAST ?

Cesare Di Mauro <cesare.di.mauro <at> gmail.com> writes:

> 
> 
> 2010/12/28 Lukas Lueg <lukas.lueg <at> googlemail.com>
> 
> Consider the following code:
> def foobar(x):
>     for i in range(5):
>         x[i] = i
> The bytecode in python 2.7 is the following:
>   2           0 SETUP_LOOP              30 (to 33)
>               3 LOAD_GLOBAL              0 (range)
>               6 LOAD_CONST               1 (5)
>               9 CALL_FUNCTION            1
>              12 GET_ITER
>         >>   13 FOR_ITER                16 (to 32)
>              16 STORE_FAST               1 (i)
>   3          19 LOAD_FAST                1 (i)
>              22 LOAD_FAST                0 (x)
>              25 LOAD_FAST                1 (i)
>              28 STORE_SUBSCR
>              29 JUMP_ABSOLUTE           13
>         >>   32 POP_BLOCK
>         >>   33 LOAD_CONST               0 (None)
>              36 RETURN_VALUE
> Can't we optimize the LOAD_FAST in lines 19 and 25 to a single load
> and put the reference twice on the stack? There is no way that the
> reference of i might change in between the two lines. Also, the
> load_fast in lne 22 to reference x could be taken out of the loop as x will 
always point to the same object....
> 
> Yes, you can, but you need:
> - a better AST evaluator (to mark symbols/variables with proper attributes);
> - a better optimizer (usually located on compile.c) which has a "global 
vision" (not limited to single instructions and/or single expressions).
> 
> It's not that simple, and the results aren't guaranteed to be good.
> 
> Also, consider that Python, as a dynamic-and-not-statically-compiled language 
need to find a good trade-off between compilation time and execution.
> 
> Just to be clear, a C program is usually compiled once, then executed, so you 
can spend even *hours* to better optimize the final binary code.
> 
> With a dynamic language, usually the code is compiled and the executed as 
needed, in "realtime". So it isn't practical neither desirable having to wait 
too much time before execution begins (the "startup" problem).
> 
> Python stays in a "gray area", because modules are usually compiled once (when 
they are first used), and executed many times, but it isn't the only case.
> 
> You cannot assume that optimization techniques used on other (static) 
languages can be used/ported in Python.
> 
> Cesare
> 

No, it's singularly impossible to prove that any global load will be any given 
value at compile time.  Any optimization based on this premise is wrong.

Alex

_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
Guido van Rossum | 3 Jan 2011 04:18
Favicon

Re: Possible optimization for LOAD_FAST ?

On Sun, Jan 2, 2011 at 5:50 PM, Alex Gaynor <alex.gaynor <at> gmail.com> wrote:
> No, it's singularly impossible to prove that any global load will be any given
> value at compile time.  Any optimization based on this premise is wrong.

True.

My proposed way out of this conundrum has been to change the language
semantics slightly so that global names which (a) coincide with a
builtin, and (b) have no explicit assignment to them in the current
module, would be fair game for such optimizations, with the
understanding that the presence of e.g. "len = len" anywhere in the
module (even in dead code!) would be sufficient to disable the
optimization.

But barring someone interested in implementing something based on this
rule, the proposal has languished for many years.

FWIW, this is reminiscent of Fortran's rules for "intrinsics" (its
name for builtins), which have a similar optimization behavior (except
there the potential overrides that the compiler doesn't need to take
into account are load-time definitions).

--

-- 
--Guido van Rossum (python.org/~guido)

Gmane