Discussion of Emacs support for multi-directional text

headers Gregg Reynolds | 26 Nov 15:58

_______________________________________________
emacs-bidi mailing list
emacs-bidi <at> gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-bidi

Hello Eli, Gregg,

On 2009/11/27 1:10, Eli Zaretskii wrote:
>> Date: Thu, 26 Nov 2009 08:58:39 -0600
>> From: Gregg Reynolds<gar <at> arabink.com>

>> It's also a major pain to edit XML in a bidi editor.  And I do mean
>> major.
>
> Again, an explicit list of main problems would be useful (although I
> deliberately decided not to handle bidirectional editing for meta
> documents yet, so this specific class of problems is still a long way
> from being dealt with).

This is a well-known problem. For some more background and some attempt 
at a solution, please see:

http://www.sw.it.aoyama.ac.jp/2005/pub/IUC28-bidi/ and
http://www.sw.it.aoyama.ac.jp/2008/pub/IUC32-bidi/

Unfortunately, the test prototype available at
http://www.sw.it.aoyama.ac.jp/2008/pub/IUC32-bidi/bidi-source.html
is not very usable currently because the carret jumps around and has to 
be repositioned. That problem has been solved, but I didn't get around 
to make the update publicly available. Anyway, both the simulation and 
the Web-based editor should make it possible to test your problem cases 
and tell us whether our solutions help (we know they aren't perfect; 
there is no such thing as perfect bidi, unfortunately).

Questions and comments are very welcome!

In a discussion with Ken'ichi Handa and Naoto Takahashi in 2005, they 
pointed out that using bidi marks in overlays should make it possible in 
bidi emacs to address this problem. Unfortunately, nobody here is 
familiar enough with emacs lisp to implement this, but I'd be very glad 
to help somebody with some emacs lisp skills to work on such a project.

Regards,    Martin.

--

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:duerst <at> it.aoyama.ac.jp

> Date: Fri, 27 Nov 2009 20:11:47 +0900
> From: "Martin J. Dürst" <duerst <at> it.aoyama.ac.jp>
> CC: Gregg Reynolds <gar <at> arabink.com>, emacs-bidi <at> gnu.org, emacs-devel <at> gnu.org
> 
> On 2009/11/27 1:10, Eli Zaretskii wrote:
> >> Date: Thu, 26 Nov 2009 08:58:39 -0600
> >> From: Gregg Reynolds<gar <at> arabink.com>
> 
> >> It's also a major pain to edit XML in a bidi editor.  And I do mean
> >> major.
> >
> > Again, an explicit list of main problems would be useful (although I
> > deliberately decided not to handle bidirectional editing for meta
> > documents yet, so this specific class of problems is still a long way
> > from being dealt with).
> 
> This is a well-known problem. For some more background and some attempt 
> at a solution, please see:
> 
> http://www.sw.it.aoyama.ac.jp/2005/pub/IUC28-bidi/ and
> http://www.sw.it.aoyama.ac.jp/2008/pub/IUC32-bidi/

Thanks.

As I pointed out elsewhere, this class of problems is currently beyond
the scope of what I'm trying to do in Emacs.  That is because the
display engine basically examines characters one by one, so correct
display of XML/HTML using just UAX#9 algorithm is impossible, as you
and others have found out.

I think that support for bidi editing of documents that use markup
languages should be based on the features that exist as part of the
markup: &lrm;/&rlm;, the dir= directive etc.  Using these, a Lisp
application should place overlays or maybe text properties on the
portions of text that need to be reordered.  For example, one could
place an overlay on a portion of the buffer with `display' property
whose value is the reordered text of that portion (it will have to be
enclosed in LRO..PDF embedding to avoid the automatic reordering when
the display string is delivered to the glass).  Or we could add a new
overlay property that would just tell the display engine "reorder this
part".  Then the display engine will DTRT guided by these overlays and
text properties.

However, please note that the above is not something I thought about
long enough to be convinced that it is the right solution.  The only
thing I convinced myself in at this point is that bidi support for
markup languages should not be on the basic display engine level,
which is what I'm currently hacking.  Rather, it should use the basic
reordering capabilities of the display engine in some way, but do some
additional work on higher levels to implement the equivalent of the
UAX#9 ``higher protocols''.

> In a discussion with Ken'ichi Handa and Naoto Takahashi in 2005, they 
> pointed out that using bidi marks in overlays should make it possible in 
> bidi emacs to address this problem. Unfortunately, nobody here is 
> familiar enough with emacs lisp to implement this, but I'd be very glad 
> to help somebody with some emacs lisp skills to work on such a project.

Thanks for the offer.  I don't know when I will get to those parts.
For now, text properties and overlays are not yet supported in the
modified display engine -- that is the very next problem on my agenda.
When these parts are done, hopefully others could come on board and
implement markup support on top of that.

Hello Eli,

Many thanks for your comments. I very much agree with what you say, 
namely that good display for HTML/XML (and similar stuff) should be at a 
level above the basic bidi support, and that it may be implemented by 
somebody else than you.

Nevertheless, I very much look forward to hear from Gregg (and others) 
about what they see as the current problems with bidi editing (such as 
the issue with carret placement that Gregg mentioned).

Regards,   Martin.

On 2009/11/27 21:24, Eli Zaretskii wrote:
>> Date: Fri, 27 Nov 2009 20:11:47 +0900
>> From: "Martin J. Dürst"<duerst <at> it.aoyama.ac.jp>
>> CC: Gregg Reynolds<gar <at> arabink.com>, emacs-bidi <at> gnu.org, emacs-devel <at> gnu.org
>>
>> On 2009/11/27 1:10, Eli Zaretskii wrote:
>>>> Date: Thu, 26 Nov 2009 08:58:39 -0600
>>>> From: Gregg Reynolds<gar <at> arabink.com>
>>
>>>> It's also a major pain to edit XML in a bidi editor.  And I do mean
>>>> major.
>>>
>>> Again, an explicit list of main problems would be useful (although I
>>> deliberately decided not to handle bidirectional editing for meta
>>> documents yet, so this specific class of problems is still a long way
>>> from being dealt with).
>>
>> This is a well-known problem. For some more background and some attempt
>> at a solution, please see:
>>
>> http://www.sw.it.aoyama.ac.jp/2005/pub/IUC28-bidi/ and
>> http://www.sw.it.aoyama.ac.jp/2008/pub/IUC32-bidi/
>
> Thanks.
>
> As I pointed out elsewhere, this class of problems is currently beyond
> the scope of what I'm trying to do in Emacs.  That is because the
> display engine basically examines characters one by one, so correct
> display of XML/HTML using just UAX#9 algorithm is impossible, as you
> and others have found out.
>
> I think that support for bidi editing of documents that use markup
> languages should be based on the features that exist as part of the
> markup:&lrm;/&rlm;, the dir= directive etc.  Using these, a Lisp
> application should place overlays or maybe text properties on the
> portions of text that need to be reordered.  For example, one could
> place an overlay on a portion of the buffer with `display' property
> whose value is the reordered text of that portion (it will have to be
> enclosed in LRO..PDF embedding to avoid the automatic reordering when
> the display string is delivered to the glass).  Or we could add a new
> overlay property that would just tell the display engine "reorder this
> part".  Then the display engine will DTRT guided by these overlays and
> text properties.
>
> However, please note that the above is not something I thought about
> long enough to be convinced that it is the right solution.  The only
> thing I convinced myself in at this point is that bidi support for
> markup languages should not be on the basic display engine level,
> which is what I'm currently hacking.  Rather, it should use the basic
> reordering capabilities of the display engine in some way, but do some
> additional work on higher levels to implement the equivalent of the
> UAX#9 ``higher protocols''.
>
>> In a discussion with Ken'ichi Handa and Naoto Takahashi in 2005, they
>> pointed out that using bidi marks in overlays should make it possible in
>> bidi emacs to address this problem. Unfortunately, nobody here is
>> familiar enough with emacs lisp to implement this, but I'd be very glad
>> to help somebody with some emacs lisp skills to work on such a project.
>
> Thanks for the offer.  I don't know when I will get to those parts.
> For now, text properties and overlays are not yet supported in the
> modified display engine -- that is the very next problem on my agenda.
> When these parts are done, hopefully others could come on board and
> implement markup support on top of that.
>

--

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:duerst <at> it.aoyama.ac.jp

Mon	Tue	Wed	Thu	Fri	Sat	Sun

						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30

1	November 2011
14	October 2011
1	July 2011
4	June 2011
8	April 2011
10	March 2011
4	February 2011
3	November 2010
47	September 2010
38	August 2010
62	July 2010
61	June 2010
46	May 2010
12	April 2010
1	March 2010
34	February 2010
33	January 2010
4	November 2009
14	October 2009
4	July 2009
1	June 2008
7	May 2008
2	December 2007
13	March 2007
32	December 2005
75	November 2005
13	April 2004
17	March 2004
16	October 2003
9	September 2003
60	August 2003
1	July 2003
2	June 2003
15	March 2003
3	February 2003
10	December 2002
2	November 2002
1	October 2002
3	September 2002
2	August 2002
31	July 2002
24	June 2002
1	May 2002

Re: Bidi Emacs: R2L paragraphs and the direction of glyph_row

Re: Bidi Emacs: R2L paragraphs and the direction of glyph_row

Re: Bidi Emacs: R2L paragraphs and the direction of glyph_row

Re: Bidi Emacs: R2L paragraphs and the direction of glyph_row