Eli Zaretskii | 18 Apr 16:54 2011
Picon

Re: Bidirectional editing in Emacs -- main design decisions

> Date: Fri, 09 Oct 2009 23:18:00 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> Cc: 
> 
> While there's a lot of turf to be covered yet, I thought I'd publish
> the main design decisions up to this point.

I don't know if someone follows these design decisions, but in case
people are interested, here's an update:

> 6. Reordering of strings from `display' properties
>    ...
>    Another, perhaps more serious implication of this design decision
>    is that strings from `display' properties are reordered separately
>    from the surrounding buffer text.  IOW, production of glyphs from
>    reordered buffer text is stopped when a `display' property is
>    found, the string that is the property's value is reordered and
>    displayed, and then the rest of text is reordered and its glyphs
>    produced.

After careful thought, I changed my mind about this part.  Whenever
the redisplay iterator finds text that is covered by a `display' text
property or by an overlay with a `display', `before-string', or
`after-string' property, it will not stop reordering.  Instead, it
will treat the entire run of text covered by the text property or
overlay as a single atomic entity, and will reorder it as if it were a
single special character whose name in Unicode is OBJECT REPLACEMENT
CHARACTER (u+FFFC).  This character's bidirectional category (Other
Neutral) and other properties are designed so that it can stand for
display features such as embedded images, and in particular it is
(Continue reading)

Stefan Monnier | 19 Apr 15:11 2011
Picon

Re: Bidirectional editing in Emacs -- main design decisions

> After careful thought, I changed my mind about this part.  Whenever
> the redisplay iterator finds text that is covered by a `display' text
> property or by an overlay with a `display', `before-string', or
> `after-string' property, it will not stop reordering.  Instead, it
> will treat the entire run of text covered by the text property or
> overlay as a single atomic entity, and will reorder it as if it were a
> single special character whose name in Unicode is OBJECT REPLACEMENT
> CHARACTER (u+FFFC).  This character's bidirectional category (Other
> Neutral) and other properties are designed so that it can stand for
> display features such as embedded images, and in particular it is
> reordered as appropriate for such embedded objects.

> This will reorder images and display strings in the same way wrt
> surrounding text, which I think is reasonable.  It is also in line
> with the pre-Emacs 24 unidirectional display engine, which skipped the
> text covered by such properties in one go, after displaying the image
> or a display string specified by the property.

It sounds very reasonable, but at the same time I don't understand in
which way it differs from your earlier opinion that it should "stop
reordering" (whose meaning is very unclear to me).

        Stefan

Eli Zaretskii | 19 Apr 18:02 2011
Picon

Re: Bidirectional editing in Emacs -- main design decisions

> From: Stefan Monnier <monnier <at> iro.umontreal.ca>
> Cc: emacs-devel <at> gnu.org,  emacs-bidi <at> gnu.org
> Date: Tue, 19 Apr 2011 10:11:33 -0300
> 
> It sounds very reasonable, but at the same time I don't understand in
> which way it differs from your earlier opinion that it should "stop
> reordering" (whose meaning is very unclear to me).

Well, that's my fault: "stop reordering" is simply misleading.

What I meant by that was that the reordering will completely process
all the text before (in the logical order) the part covered by a
display property, then process the display spec, and only then reorder
the text after the property.

So this buffer text

  ABCDE display XYZ

will be reordered as

  EDCBA display ZYX

instead of more sensible

  ZYX display EDCBA

under the new plan.
Stefan Monnier | 20 Apr 05:15 2011
Picon

Re: Bidirectional editing in Emacs -- main design decisions

> Well, that's my fault: "stop reordering" is simply misleading.

> What I meant by that was that the reordering will completely process
> all the text before (in the logical order) the part covered by a
> display property, then process the display spec, and only then reorder
> the text after the property.

> So this buffer text

>   ABCDE display XYZ

> will be reordered as

>   EDCBA display ZYX

> instead of more sensible

>   ZYX display EDCBA

> under the new plan.

Thanks.  That makes sense,

        Stefan

Mohsen BANAN | 25 Apr 19:31 2011
Picon

Re: Bidirectional editing in Emacs -- main design decisions


First let me preface that I am new to this list
(through gmane) and that I am not on top of the
subject matter at hand.

If I am understaning it right, I think your new
decision is right also from a different
perspective.

In that it then becomes consistent with how bidi
works in mozilla/firefox.

Being consistent with the browser is important in
practice. Let me cite an example.

I have been using build GNU Emacs 24.0.50.1 for a
while to exchange email (Gnus) in Farsi and mixed
Farsi/Latin with family and friends in Iran who
use browser based MUAs.

My mixed language text generated with emacs
displays inconsistently in firefox vs emacs24.

For example when I write:

اسم کوچک من محسن Mohsen و اسم خانوادگى من بنان
Banan است.

My first name is Mohsen محسن and my last name is
Banan بنان.
(Continue reading)

Eli Zaretskii | 25 Apr 19:58 2011
Picon

Re: Bidirectional editing in Emacs -- main design decisions

> From: Mohsen BANAN <list-general <at> mohsen.1.banan.byname.net>
> Date: Mon, 25 Apr 2011 10:31:57 -0700
> Cc: emacs-devel <at> gnu.org
> 
> My mixed language text generated with emacs
> displays inconsistently in firefox vs emacs24.
> 
> For example when I write:
> 
> اسم کوچک من محسن Mohsen و اسم خانوادگى من بنان
> Banan است.
> 
> 
> My first name is Mohsen محسن and my last name is
> Banan بنان.

Did you (setq bidi-display-reordering t) in the buffer where this is
displayed?  Without that, the bidirectional reordering does not
happen.  (Some day, this will be on by default, but not yet.)

_______________________________________________
emacs-bidi mailing list
emacs-bidi <at> gnu.org
https://lists.gnu.org/mailman/listinfo/emacs-bidi
Eli Zaretskii | 25 Apr 20:59 2011
Picon

Re: Bidirectional editing in Emacs -- main design decisions

> From: Mohsen BANAN <list-general <at> mohsen.1.banan.byname.net>
> Cc: Mohsen BANAN <list-general <at> mohsen.1.banan.byname.net>,  emacs-bidi <at> gnu.org,  emacs-devel <at> gnu.org
> Date: Mon, 25 Apr 2011 11:44:06 -0700
> 
> The problem is not with display of a pure Farsi
> line or a pure Latin line or a mixed Latin+Farsi
> line.
> 
> The problem is with Fasri+Latin line. 
> In that it displays one way in emacs and a 
> differnt way in firefox.
> 
> Your citation above of my email has somehow
> reproduced the problem.
> 
> So, right here we have it captured.
> 
> Look at the display of my original message 
> below:
> 
> اسم کوچک من محسن Mohsen و اسم خانوادگى من بنان
> Banan است.
> 
> Now look at the display of citation line above
> starting on the left with ' >>'  after the 
>   >> For example when I write:
> 
> You see how the Farsi piece (word sequence -- not
> character sequence) is flipped around "Mohsen".

(Continue reading)

Eli Zaretskii | 26 Apr 00:00 2011
Picon

Re: Now: Paragraph Direction Detection and Harmonization -- Was: Re: Bidirectional editing in Emacs -- main design decisions

> From: Mohsen BANAN <list-general <at> mohsen.1.banan.byname.net>
> Cc: Mohsen BANAN <list-general <at> mohsen.1.banan.byname.net>,  emacs-bidi <at> gnu.org,  emacs-devel <at> gnu.org
> Date: Mon, 25 Apr 2011 14:31:22 -0700
> 
> I am saying that emacs display is correct but that
> there are interoperability problems.

That could be, but it sounds like the solution to those problems
should be on the Firefox side.  Or maybe Firefox also has some
customization feature, like bidi-paragraph-direction in Emacs.

> For example, I think that it is worthwhile for
> emacs24 to have a good Conformance Statement for 
>  http://unicode.org/reports/tr9/

We already do, see etc/NEWS:

  Reordering of bidirectional text for display in Emacs is a "Full
  bidirectionality" class implementation of the Unicode Bidirectional
  Algorithm.

Is that what you meant by "conformance statement"?  If not, what is
it?

> The existence of -- Unicode Standard Annex #9 --
> Unicode Bidirectional Algorithm -- speaks to that
> requirement for harmonization.

If every application out there implements UAX#9 to the letter, there
shouldn't be interoperability problems.
(Continue reading)

Mohsen BANAN | 25 Apr 20:44 2011
Picon

Re: Bidirectional editing in Emacs -- main design decisions

>>>>> On Mon, 25 Apr 2011 20:58:26 +0300, Eli Zaretskii <eliz <at> gnu.org> said:

  >> From: Mohsen BANAN <list-general <at> mohsen.1.banan.byname.net>
  >> Date: Mon, 25 Apr 2011 10:31:57 -0700
  >> Cc: emacs-devel <at> gnu.org
  >> 
  >> My mixed language text generated with emacs
  >> displays inconsistently in firefox vs emacs24.
  >> 
  >> For example when I write:
  >> 
  >> اسم کوچک من محسن Mohsen و اسم خانوادگى من بنان
  >> Banan است.
  >> 
  >> 
  >> My first name is Mohsen محسن and my last name is
  >> Banan بنان.

  Eli> Did you (setq bidi-display-reordering t) in the buffer where this is
  Eli> displayed?  Without that, the bidirectional reordering does not
  Eli> happen.  (Some day, this will be on by default, but not yet.)

Of course Eli! (setq bidi-display-reordering t) is
there when I compose and when I read.

The problem is not with display of a pure Farsi
line or a pure Latin line or a mixed Latin+Farsi
line.

The problem is with Fasri+Latin line. 
(Continue reading)

Mohsen BANAN | 25 Apr 23:31 2011
Picon

Now: Paragraph Direction Detection and Harmonization -- Was: Re: Bidirectional editing in Emacs -- main design decisions


>>>>> On Mon, 25 Apr 2011 21:59:43 +0300, Eli Zaretskii <eliz <at> gnu.org> said:
  >> You see how the Farsi piece (word sequence -- not
  >> character sequence) is flipped around "Mohsen".

  Eli> Which one of the two is correct, in the original text or in the
  Eli> citations?  (I don't read Farsi.)

Of course, the original is correct.
Because I wrote it and I wrote it with emacs.

  Eli> I think the issue here is paragraph direction, which in Emacs is
  Eli> dynamically determined by default.  See below.

I agree. That is the problem.  

I have therefore changed the subject line to:
 Paragraph Direction Detection and Harmonization

Please see below as I expand on what parts I see
as non-problem and what parts I think can be
improved.

  >> In the browser my original text appears as it does
  >> in the citation.

  Eli> Does the browser display it flushed to the left or to the right?

Everything is flushed to the left. Which is
consistent your diagnosis of paragraph direction problem.
(Continue reading)


Gmane