Jameson "Chema" Quinn | 7 Aug 20:52 2007
Picon

An idea and a bug

I'm hacking at my own version of IDLE to try to implement language support in Python. I found a bug.

1. The bug.
I'm using IDLE to work on a copy of IDLE in my home directory. I copy the idlelib directory to ~/jqidlelib , add "jq" to lines 2 and 22 of idle.py, open it in IDLE, and start hacking on the other files. If a hacked version of some other module is not open in IDLE and I press F5 to run idle.py, I get the desired behaviour (the modules are loaded from my hacked copies) but the error display is wacky. Errors refer to the line numbers of my hacked versions, but they give the path and code snippets from the actual IDLE directory.

2. My idea
I have started a discussion on the OLPC wiki and over email with some interested parties within OLPC about the way to do i18n (internat...ion) on Python. My idea is to have code on-disk in "real", english-based python, but to allow displaying and editing of that code in the user's native language. This means that there are systems for invisible, real-time, two-way translation of keywords, identifiers, and docstrings/comments. This is primarily aimed at a young demographic; I realize that there are plenty of "real programmers" who learn the keywords of a programming language in their abstract meaning, without knowing the everyday English meaning of those terms. (after all, "string", "hash", and "for" have meanings i n programming quite distinct from their everyday meanings anyway).

Keywords: dynamic two-way translation is easy.
Identifiers: supports initially creating identifiers in a native (unicode) language. When a translation is created later (using easy context menus and dictionary-based guessing support), the .py file is changed to include the English version, and the translation is stored in a parallel .p4n file. Translations of included modules are imported from their .p4n files. All clashes and character set issues are dealt with (using prefixes to escape, for instance) in order to keep the translation reversible at all times. This requires some minor modifications to python itself for the case when an imported module has added a translation but the calling code has not been edited since then so it still uses the non-english version of the identif ier. This case would be treated with extreme care to prevent security hazards.
Docstrings and comments: These are either escaped as "code" (using simple ascii markup) and translated using the same system as above, or translated on a by-line basis. Support would exist to do initial translation using some on-line babelfish-like tool.

I think the best way to explain all this is to implement it, so I'm doing so. Ideally, this implementation will be clean, simple, and unobtrusive enough that it will become a standard feature of IDLE, and even inspire inclusion in other IDEs - becoming standard within Python, and available for other languages and human-editable data formats.

Cheers,
Jameson Quinn

ps. Here's how Mike Fletcher put his proposed solution, my emphasis added, the visual stuff he mentions is a separate issue:


My idea for internationalization is as follows:

   * assume a progression from graphical, to local language, to
     subset-of-English for those who want to become proficient programmers
         o provide tools that let the children put blocks together in a
           reasonably constrained environment
         o show the children what putting the blocks together does in
           their native language
         o let the children edit the source-code in their own language
           to start with
         o allow the children to flip back and forth between
           native-language keywords and English keywords
         o support the children in translating to/from English for code
           they want to publish
   * for each component in the programming language (e.g. for loops,
     while loops, variable assignment, etceteras)
         o provide a visual form which implies the operation and allows
           for drag-and-drop operation
         o provide a canonical translation (which is also reserved
           words/symbols/punctuation in the native language)
         o provide a localized description of the operation (preferably
           with some simple lessons on usage, maybe some demos with
           sample code for each one, hook these up in the UI so that
           the child can explore the idea behind the operation)
         o when used, show the (standard) Python code generated by the
           usage
   * for the special case of identifiers in the user's
     (non-ascii-compatible) language
         o store all identifiers as transcoded[1] ascii identifiers in
           the source code on initial writing (before translation)
         o display all identifiers as their non-transcoded versions in
           the UI
         o store the translation matrix in a separate file (to allow
           for multiple translations)
   * allow for a localization option that also translates keywords and
     builtins
         o uses the canonical translation
         o allow the user to input any Unicode identifier as an
           identifier which is not in the canonical translation
         o typing in the localized version of "for" or "class" would
           insert "for" and "class" into the stored code
   * identifiers/keywords which have no translations are presented in
     ascii (i.e. all current Python code would show up in English,
     potentially with keywords in Arabic or Thai)
   * have a tool that allows for querying e.g. BabelFish to get
     translations from/to a given language for given identifiers
         o allow for exporting any pure-ascii translation (e.g.
           English) to the base file (updating the translation tables
           as we do so)
         o should provide both localization and internationalization
           support (i.e. the same tool would allow the country to load
           a common Python module and produce a translation for it)

_______________________________________________
IDLE-dev mailing list
IDLE-dev <at> python.org
http://mail.python.org/mailman/listinfo/idle-dev
Tal Einat | 7 Aug 21:19 2007
Picon

Re: An idea and a bug

On 8/7/07, Jameson Chema Quinn <jquinn <at> cs.oberlin.edu> wrote:
> I'm hacking at my own version of IDLE to try to implement language support
> in Python. I found a bug.
>
> 1. The bug.
> I'm using IDLE to work on a copy of IDLE in my home directory. I copy the
> idlelib directory to ~/jqidlelib , add "jq" to lines 2 and 22 of idle.py,
> open it in IDLE, and start hacking on the other files. If a hacked version
> of some other module is not open in IDLE and I press F5 to run idle.py, I
> get the desired behaviour (the modules are loaded from my hacked copies) but
> the error display is wacky. Errors refer to the line numbers of my hacked
> versions, but they give the path and code snippets from the actual IDLE
> directory.

If you do a directory-wide search for 'idlelib' you'll find that it is
mentioned in PyShell.py as well. I haven't tried this myself, so
please inform us whether or not this works :)

> 2. My idea
> [snip]

Awesome! I'll be glad to answer any questions you might have regarding
IDLE's internals.

An early pointer: IDLE doesn't currently allow unicode input, so
you'll have to rework that bit if you want unicode input support.

Good luck, and please keep us posted on developments,
- Tal
Jameson "Chema" Quinn | 7 Aug 21:48 2007
Picon

Re: An idea and a bug



On 8/7/07, Tal Einat <taleinat <at> gmail.com> wrote:
On 8/7/07, Jameson Chema Quinn <jquinn <at> cs.oberlin.edu> wrote:
> I'm hacking at my own version of IDLE to try to implement language support
> in Python. I found a bug.
>
> 1. The bug.
> I'm using IDLE to work on a copy of IDLE in my home directory. I copy the
> idlelib directory to ~/jqidlelib , add "jq" to lines 2 and 22 of idle.py,
> open it in IDLE, and start hacking on the other files. If a hacked version
> of some other module is not open in IDLE and I press F5 to run idle.py, I
> get the desired behaviour (the modules are loaded from my hacked copies) but
> the error display is wacky. Errors refer to the line numbers of my hacked
> versions, but they give the path and code snippets from the actual IDLE
> directory.

If you do a directory-wide search for 'idlelib' you'll find that it is
mentioned in PyShell.py as well. I haven't tried this myself, so
please inform us whether or not this works :)

Changing it there has no effect. In the intervening time, I'm getting even weirder behavior: with my hacked EditorWindow open, I fix the bug in my code and save it. Then I press F5 in the idle.py window again. I get:

Traceback (most recent call last):
  File "/home/compaq/jqidlelib/idle.py", line 22, in <module>
    jqidlelib.PyShell.main()
  File "/usr/lib/python2.5/idlelib/PyShell.py", line 1406, in main
    return # couldn't open shell
  File "/usr/lib/python2.5/idlelib/PyShell.py", line 275, in open_shell
    self.pyshell = PyShell(self)
  File "/usr/lib/python2.5/idlelib/PyShell.py", line 813, in __init__
    OutputWindow.__init__(self, flist, None, None)
  File "idlelib/OutputWindow.py", line 16, in __init__
    EditorWindow.__init__(self, *args)
  File "/home/compaq/jqidlelib/EditorWindow.py", line 56, in __init__
    if EditorWindow.help_url is None:
NameError: global name 'trns' is not defined

The "trns" is from the previous version of EditorWindow which NO LONGER EXISTS on the disk, which is why the code snippet doesn't mention it. Also, note that the paths for the call stack show a mix between idlelib and jqidlelib, depending on which files I have open. This is just FALSE - all modules are actually being loaded from jqidlelib, as I can easily tell if I introduce a syntax error and then close the file.

Meanwhile, I can work around by just closing and reopening idle. Annoying, though.

> 2. My idea
> [snip]

Awesome! I'll be glad to answer any questions you might have regarding
IDLE's internals.

An early pointer: IDLE doesn't currently allow unicode input, so
you'll have to rework that bit if you want unicode input support.

I can afford to wait on that, as my first test case is Spanish, and Idle only has minor complaints about accents. Thanks for the heads up, though - this will need to be fixed for py3k anyway, what's the status on that?

Good luck, and please keep us posted on developments,
- Tal
Thanks,
Jameson

ps. here's my idle version stuff:
Python 2.5.1 (r251:54863, May  2 2007, 16:56:35)
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "copyright", "credits" or "license()" for more information.

    ****************************************************************
    Personal firewall software may warn about the connection IDLE
    makes to its subprocess using this computer's internal loopback
    interface.  This connection is not visible on any external
    interface and no data is sent to or received from the Internet.
    ****************************************************************
   
IDLE 1.2.1      ==== No Subprocess ====
_______________________________________________
IDLE-dev mailing list
IDLE-dev <at> python.org
http://mail.python.org/mailman/listinfo/idle-dev
Tal Einat | 8 Aug 18:54 2007
Picon

Re: An idea and a bug

On 8/8/07, Jameson Chema Quinn <jquinn <at> cs.oberlin.edu> wrote:
>
>
> >
> > This is all very strange behavior.
> >
> > Please try the following:
> > 1. Replace all instances of 'idlelib' into 'jqidlelib', including
> > appearances inside strings. There should be two of these in PyShell.py
> > and two in idle.py. Also, make sure it doesn't appear in run.py either
> > (if it does, change it there too.)
> > 2. Rename/move the normal idlelib directory (temporarily)
> > 3. Close all existing instances of IDLE (and make sure no zombie
> > Python processes are still running in the background)
> > 4. Try running your version of IDLE again
> >
> > If IDLE still crashes, please send the tracebacks.
> >
> > - Tal
> >
>
> I think you misunderstand. The problem occurs, not when I run my script from
> the shell or from some other IDE, but when I run it from within IDLE. This
> makes it more understandable - but it is still a bug. Perhaps it's OK if you
> can't edit IDLE from within IDLE, but you should not get funny behaviour
> every time you load a module that happens to share a name with an IDLE
> module.
>
> Jameson
>

I think I understand the problem now - you're editing your version of
IDLE with the existing version, and you're having module importing
issues.

The problem is that when you run a Python script, the interpreter adds
the script's directory to the beginning of sys.path. This means the
when you do "import <module_name>", Python will go through the
directories in sys.path searching for a file by that name, and so it
will find first and use anything in the directory from where the
script was run. This is true for IDLE as well, and I can see now how
this can cause problems.

What you should do is run IDLE without adding its directory to
sys.path. Here's a quick way to do so from the command line:
python.exe -c "import idlelib.PyShell; idlelib.PyShell.main()"

Just run this from a directory which has nothing Python-related in it,
and you'll have no such importing issues. To completely ignore the
directory from which Python was run, use:
python.exe -c "import sys; del sys.path[0]; import idlelib.PyShell;
idlelib.PyShell.main()"

Another way would be to remove the directory from sys.path after IDLE
has started, inside IDLE's shell. However, it may not be the first
item in the sys.path list, because IDLE sometimes adds more
directories to sys.path.

Anyways, sorry for taking a while to understand the issue and hoping
the above helps,
- Tal
Tal Einat | 11 Aug 14:40 2007
Picon

Cleaner Percolator.py implementation

I took the time to clean up the implementation of Percolator and add
some comments and doc-strings. I plan to do the same for Delegator
soon. I have found these to be very confusing and hard to understand,
although they could be a good piece of code to learn from. I hope I've
made the code more Pythonic :)

This version of Percolator inherits from Delegator (since a Percolator
"is a" Delegator, after all). I've also refactored the
Tk-Text-widget-specific stuff into a subclass, TkTextPercolator, which
is functionally equivalent to the original Percolator.

To use this in IDLE, use this file instead of Percolator.py (either
replace it or import this file instead of the original), and use the
TkTextPercolator class instead of Percolator (in EditorWindow.py).

I'm willing to post this as a patch on SF if there is any interest.

Comments are most welcome :)
- Tal
Attachment (MyPercolator.py): text/x-python, 5750 bytes
_______________________________________________
IDLE-dev mailing list
IDLE-dev <at> python.org
http://mail.python.org/mailman/listinfo/idle-dev
Douglas S. Blank | 16 Aug 18:07 2007
Picon

Re: IDLE getpass?

Tal Einat wrote:
>> 3) If I have to open a Tk window, how can I get it to appear on top of
>> the IDLE window?
> 
> IIRC this depends on your window manager. Tk's 'raise' command
> ('tkraise' in Python's Tkinter, because 'raise' is a Python keyword)
> just doesn't work on windows - this is a well known bug. I seem to
> recall that the TopLevel widgets' 'wm_deiconify' method has something
> to do with it... Try a Google Groups search on this issue + Tk, and
> you should find an answer.

Ok, I can't find anything to fix this. I've tried combinations of lower, 
lift, deiconify, iconify, update, overridedirect, ... but nothing I do 
can get a new Tk window to appear on top of IDLE.

This is very confusing for new students (and experts). I'll do whatever 
is necessary even it requires plugin or whatever. Can I lower IDLE?

Any ideas?

-Doug

Here's some code to help anyone see the problem:

# in IDLE on Windows XP:
import Tkinter
tl = Tkinter.Toplevel()
tl.update()
tl.raise()
...

I can't get the tl window to appear on top of IDLE.

>> Thanks for any additional hints,
>>
>> -Doug
> 
> Sorry for the unusually delayed reply!
> - Tal
> 
Douglas S. Blank | 16 Aug 22:31 2007
Picon

Extension to handle Cancel

I'm working on a plugin extension for IDLE that can run a test when 
Control+C is pressed. Unfortunately, Control+C (or <Cancel> in Tkinter 
bindings) is not handled as other key bindings are. Instead there is a 
cancel_callback in the pyshell instance that is called.

So, I've tried to replace the cancel_callback method from inside the 
extension (I have a working version from an external main---but it is 
really hacky). However, it looks like editwin.flist.pyshell doesn't 
exist yet when the constructor of the extension is called.

Is there any way for my extension module to have some code that gets 
called after the pyshell has been created?

-Doug

Sample code:

class ExtensionTest:
    def __init__(self, editwin):
       # works for other windows, after the shell is made, but:
       print editwin.flist.pyshell.cancel_callback, "does not exist yet"
       # when constructed for the shell
Douglas S. Blank | 16 Aug 23:10 2007
Picon

Running with subprocesses?

[Sorry for the barrage of questions... a new semester starting soon... :(]

It would be nice if an extension, or even a general running program, 
could look somewhere to see if IDLE is running with subprocesses. 
Something like PyShell.using_subprocess = 0. Is there a hack that would 
currently work to tell?

-Doug
Tal Einat | 17 Aug 00:31 2007
Picon

Re: Running with subprocesses?

On 8/17/07, Douglas S. Blank <dblank <at> brynmawr.edu> wrote:
> [Sorry for the barrage of questions... a new semester starting soon... :(]
>
> It would be nice if an extension, or even a general running program,
> could look somewhere to see if IDLE is running with subprocesses.
> Something like PyShell.using_subprocess = 0. Is there a hack that would
> currently work to tell?
>

You can check flist.pyshell.interp.rpcclt. The debugging code in
PyShell.py checks that it is not None. The extra-safe version (used in
AutoComplete.py) is:

try:
    rpcclt = self.editwin.flist.pyshell.interp.rpcclt
except:
    rpcclt = None
if rpcclt:
    <with process>
else:
    <without subprocess>

This would be a nice addition to PyShell (or perhaps flist?). I'll add
this to my to-do list.

- Tal
Tal Einat | 17 Aug 00:42 2007
Picon

Re: Extension to handle Cancel

On 8/16/07, Douglas S. Blank <dblank <at> brynmawr.edu> wrote:
> I'm working on a plugin extension for IDLE that can run a test when
> Control+C is pressed. Unfortunately, Control+C (or <Cancel> in Tkinter
> bindings) is not handled as other key bindings are. Instead there is a
> cancel_callback in the pyshell instance that is called.
>
> So, I've tried to replace the cancel_callback method from inside the
> extension (I have a working version from an external main---but it is
> really hacky). However, it looks like editwin.flist.pyshell doesn't
> exist yet when the constructor of the extension is called.
>
> Is there any way for my extension module to have some code that gets
> called after the pyshell has been created?
>
> -Doug
>
> Sample code:
>
> class ExtensionTest:
>     def __init__(self, editwin):
>        # works for other windows, after the shell is made, but:
>        print editwin.flist.pyshell.cancel_callback, "does not exist yet"
>        # when constructed for the shell

Hi Doug,

First off - Why Ctrl+C? It seems the simplest solution would be to
bind to a different hotkey, unless you deliberately want to override
the ability to use Ctrl+C to raise KeyboardInterrupt exceptions.

The answer is to just use editwin, and not flist.pyshell. When PyShell
is initialized, it will load the extension (the extensions are loaded
once per EditorWindow instance). You can set the 'enable_editor'
config option to False (or zero) for the extension to cause it to be
loaded only for the PyShell window and not for file editing windows.

If you want to be 100% sure, you can do 'isinstance(PyShell.PyShell,
editwin)' to ensure that the window is a shell window (so that things
don't go wrong if a user changes the 'enable_editor' config option).

- Tal

Gmane