headers
Martin Duerst | 1 Dec 10:14
Picon
Gravatar

Re: state of BIDI within stable GNU emacs (UTF8)nikud

At 05:48 05/12/01, Eli Zaretskii wrote:

 >So some real debugging is, in fact, in order.  The good news are that
 >the bidi reordering code is not invoked unless you turn on the bidi
 >display flag (the buffer-specific variable enable-bidi-display, which
 >is nil by default).

Okay, from your explanations, it sounds as if the following
would work: Do some edits with enable-bidi-display nil, then
switch that on to trigger display, then switch it off again,
and continue editing. Not exactly sure I'd like to work like
this, I just want to know whether that would avoid the bugs.

Also, have you considered calling whatever corrresponds to
cntl-L internally instead of abort when you find an inconsistent
situation, or log that inconsistency in some kind of 'dump'?

Regards,   Martin.
Permalink | Reply | 1 comment |
headers
Martin Duerst | 1 Dec 12:21
Picon
Gravatar

Re: Callbacks?

At 06:14 05/12/01, Eli Zaretskii wrote:
 >> Date: Wed, 30 Nov 2005 16:04:18 +0900
 >> From: Martin Duerst <duerst <at> it.aoyama.ac.jp>
 >> Cc: emacs-bidi <at> gnu.org
 >>
 >>  >2. When the created file is read (again) for editing, these strings
 >>  >   should be seen (on screen) EXACTLY as it was when it was entered.
 >>
 >> If using the same editor, definitely. But we also found that there
 >> can be some personal preferences, so if you changed your preferences,
 >> the display would change.
 >
 >That's actually a bad idea, IMHO: the text should be saved so that any
 >other bidi-enabled editor will display it the same.

For plain running text, this is definitely true. I'm not sure
this applies also for structured stuff such as XML.

 >That is why
 >fiddling with Unicode character properties is something I feel we
 >should not do: those properties are the only common denominator of all
 >bidi editors.

I definitely understand your feeling. But I just want to mention
that the Unicode bidi algorithm explicitly allows things like this.
If you look e.g. at:
http://www.unicode.org/reports/tr9/#HL3 and
http://www.unicode.org/reports/tr9/#HL5,
you won't see changing of properties mentioned explicitly,
but you'll be able to figure out that "Provide artificial
(Continue reading)

Permalink | Reply | 2 comments |
headers
Martin Duerst | 1 Dec 12:35
Picon
Gravatar

Re: Re: Callbacks?

At 06:28 05/12/01, Eli Zaretskii wrote:
 >> Date: Wed, 30 Nov 2005 07:13:08 -0600
 >> From: Gregg Reynolds <gar <at> arabink.com>
 >> Cc: emacs-bidi <at> gnu.org
 >>
 >> > - In our current approach, we not only change the properties of some
 >> >   characters, but also introduce additional embedding (or occasionally
 >> >   override) levels. Is this possible in your implementation?
 >> >
 >> Have you taken a look at the source for nxml mode?  You might be able to
 >> hack it up do do something like this.  As I understand it, in Emacs you
 >> can set whatever properties you like on each character in a buffer.  See
 >> node 32.19, "Text Properties", of the Elisp manual.
 >
 >There's a confusion here, I think: the term ``properties'' is used in
 >two different meanings (sorry, it's partly my blame, since I started
 >to talk about ``directional properties'' or some such).  The text
 >properties mentioned in the ELisp manuals have nothing to do with
 >directionality, they are a way to specify the typeface (color, font,
 >etc.,) used to display a portion of a buffer.

Yes. And what we are thinking about is making directionality
something that can be influenced in the same way as color
or font. Of course, if that's used the wrong way, it will
create a total mess, but if used the right way, e.g. from
a mode, it can clean things up.

 >With ingenious enough
 >usage of text properties, you can even display one character as
 >something else, e.g., display `a' as `b';
(Continue reading)

Permalink | Reply | 3 comments |
headers
Martin Duerst | 1 Dec 12:26
Picon
Gravatar

Re: Re: Callbacks?

At 22:13 05/11/30, Gregg Reynolds wrote:
 >Have you taken a look at the source for nxml mode?  You might be able to 
hack it up do do something like this.  As I understand it, in Emacs you can 
set whatever properties you like on each character in a buffer.  See node 
32.19, "Text Properties", of the Elisp manual.

I haven't looked at the actual nxml source, but otherwise,
that's very much what I have thought about.

 >Now the interesting possibility presents itself, which is to add 
Unicode-related "special" (ie. built-in) text properties.  In particular, 
directional class.

Exactly!

 >Then allow the user to override the Unicode-defined class for any 
particular character or set of chars in a buffer.  So in your example 
above, the nxml parser identifies weakly directional chars in the XML 
syntax, and switches their "unicode-directionality" property to strong LTR. 
Voila!  The bidi reordering obeys the user-set directionality and does the 
right thing.

Sorry I wasn't able to express this as well as you did.
This is exactly the direction we are working on.
The problem is that in some cases, only e.g. making '<', '>',
and so on (the markup characters) strong LTR isn't enough.
(but it gets a long way). Sometimes embeddings or
overrides are needed, and they are a bit more difficult
to do with properties, in particular if they are nested.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Kenichi Handa | 1 Dec 14:02

Re: Re: Callbacks?

In article <6.0.0.20.2.20051201202734.0510fac0 <at> localhost>, Martin Duerst
<duerst <at> it.aoyama.ac.jp> writes:

>> but there's no way to use
>> text properties to reorder characters for display.

One idea is to use overlay.  If you setup an overlay with
before-string and/or after-string property on a certain
region, I think you can reorder character displaying without
modifying Eli's bidi code.  In this case, the property value
is such bidi controlling characters as "LRE", "RLE", "PDF".

---
Kenichi Handa
handa <at> m17n.org
Permalink | Reply | 1 comment |
headers
Tomas Frydrych | 1 Dec 21:00
Picon
Favicon

using fribidi and Pango

>> Date: Mon, 21 Nov 2005 02:56:37 +0330
>> From: Hossein Noorikhah <address <at> hidden>
>>
>> I want to know more about using Pango for text rendering that can
>> solve these kind of problems. I read in mailing list achieves that
>> Emacs developers have tried Pango for text rendering, but they found
>> it not suitable for Emcas. I want to know why?

> I once wrote here the reason: Pango, like fribidi, is a batch-mode
> reordering library: you hand it a buffer with text and it returns it
> reordered for display.

This is not really to do with Pango or fribidi, but with the nature of
the bidirectional problem. The Unicode bidirectional algorithm requires
text of entire paragraph as an input, and you will not get satisfactory
results otherwise.

> By contrast, Emacs needs a reordering function that could be called
> repeatedly, and will on every call return the next character in the
> visual order.  This is because the Emacs display engine walks the
> buffer one character at a time and decides how to display it based on
> data structures built by the application.

You can do that easily with fribidi; the fribidi_log2vis() function
generates a map that allows you to translate logical index into a visual
one, and you can use it to walk over your buffer. (AbiWord, which too
has complex data structures associated with its text buffer, uses
fribidi without any great difficulties.).

Pango is a slightly different story; its primary api is not suited to an
(Continue reading)

Permalink | Reply | 506 comments |
headers
Eli Zaretskii | 2 Dec 08:52
Picon

Re: state of BIDI within stable GNU emacs (UTF8)nikud

> Date: Thu, 01 Dec 2005 18:14:13 +0900
> From: Martin Duerst <duerst <at> it.aoyama.ac.jp>
> Cc: gar <at> arabink.com, emacs-bidi <at> gnu.org
> 
> At 05:48 05/12/01, Eli Zaretskii wrote:
> 
>  >So some real debugging is, in fact, in order.  The good news are that
>  >the bidi reordering code is not invoked unless you turn on the bidi
>  >display flag (the buffer-specific variable enable-bidi-display, which
>  >is nil by default).
> 
> Okay, from your explanations, it sounds as if the following
> would work: Do some edits with enable-bidi-display nil, then
> switch that on to trigger display, then switch it off again,
> and continue editing.

No, what I meant was that, should some grave bugs be uncovered that
could cause you lose work, you could use the editor with the bidi
display flag switched off until the bug is fixed or until you save
your work and exit (whichever happens first).

> Also, have you considered calling whatever corrresponds to
> cntl-L internally instead of abort when you find an inconsistent
> situation, or log that inconsistency in some kind of 'dump'?

It's not clear to me that C-l will fix the display, but that's one
known way of dealing with display problems.

However, the calls to `abort' are intentional: they make sure that
bugs will not go unnoticed.  As I wrote, Emacs will autosave before it
(Continue reading)

Permalink | Reply | 0 comments |
headers
Eli Zaretskii | 2 Dec 09:32
Picon

Re: using fribidi and Pango

> Date: Thu, 01 Dec 2005 20:00:50 +0000
> From: Tomas Frydrych <tomasfrydrych <at> yahoo.co.uk>
> 
> > I once wrote here the reason: Pango, like fribidi, is a batch-mode
> > reordering library: you hand it a buffer with text and it returns it
> > reordered for display.
> 
> This is not really to do with Pango or fribidi, but with the nature of
> the bidirectional problem. The Unicode bidirectional algorithm requires
> text of entire paragraph as an input, and you will not get satisfactory
> results otherwise.

Well, you may wish to look at the code I wrote for Emacs: modulo any
remaining bugs, it gives satisfactory results and implements the full
UAX#9 algorithm.  It is true that the Unicode algorithm's description
is heavily biased in favor of batch reordering, but that doesn't mean
it couldn't be serialized.  The fact is that I succeeded doing that.

> > By contrast, Emacs needs a reordering function that could be called
> > repeatedly, and will on every call return the next character in the
> > visual order.  This is because the Emacs display engine walks the
> > buffer one character at a time and decides how to display it based on
> > data structures built by the application.
> 
> You can do that easily with fribidi; the fribidi_log2vis() function
> generates a map that allows you to translate logical index into a visual
> one, and you can use it to walk over your buffer.

The initial code to support bidi, written by Handa-san and others at
m17n.org, did something like that.  However, Gerd Moellmann, the
(Continue reading)

Permalink | Reply | 505 comments |
headers
Eli Zaretskii | 2 Dec 11:44
Picon

Re: Callbacks?

> Date: Thu, 01 Dec 2005 20:21:07 +0900
> From: Martin Duerst <duerst <at> it.aoyama.ac.jp>
> Cc: ehud <at> unix.mvs.co.il, emacs-bidi <at> gnu.org
> 
>  >That's actually a bad idea, IMHO: the text should be saved so that any
>  >other bidi-enabled editor will display it the same.
> 
> For plain running text, this is definitely true. I'm not sure
> this applies also for structured stuff such as XML.

This applies in general, IMO.  We could make exceptions in specific
cases, but only if there's no other reasonable way to solve the
problem.  Interoperability with other software is a very important
goal, otherwise we wouldn't be considering strict compliance with
UAX#9 (which clearly made wrong choices at some strategic points).

>  >That is why
>  >fiddling with Unicode character properties is something I feel we
>  >should not do: those properties are the only common denominator of all
>  >bidi editors.
> 
> I definitely understand your feeling. But I just want to mention
> that the Unicode bidi algorithm explicitly allows things like this.
> If you look e.g. at:
> http://www.unicode.org/reports/tr9/#HL3 and
> http://www.unicode.org/reports/tr9/#HL5,
> you won't see changing of properties mentioned explicitly,
> but you'll be able to figure out that "Provide artificial
> context" does very much equivalent things.
(Continue reading)

Permalink | Reply | 1 comment |
headers
Eli Zaretskii | 2 Dec 12:23
Picon

Re: Re: Callbacks?

> Date: Thu, 01 Dec 2005 20:35:42 +0900
> From: Martin Duerst <duerst <at> it.aoyama.ac.jp>
> Cc: emacs-bidi <at> gnu.org
> 
> There is definitely a need for some
> code that does the actual reordering, even if the directionality
> can be defined as some text property. Same for colors; there
> needs to be some C code that actually takes the color property
> and produces the color on the screen.

If you are thinking about using text properties to store the
character's directional properties, the ones specified in the Unicode
database, then that sounds like a waste of resources: the Unicode
version of Emacs already reads that database, so why repeating the
data in another data structure?

>  >> Now the interesting possibility presents itself, which is to add
>  >> Unicode-related "special" (ie. built-in) text properties.  In
>  >> particular, directional class.
>  >
>  >Not a good idea: text properties are considered part of the text, so
>  >they are, e.g., copied together with the characters when you
>  >copy/paste text.
> 
> If this is used with a mode, this should not be a problem:
> The mode reanalyzes the pasted text and fixes things up.

That's not how text properties are handled now in Emacs.  It would
slow down Emacs, for one thing.  In any case, this is using text
properties for something they were not designed to do.  (There are
(Continue reading)

Permalink | Reply | 0 comments |

Gmane