Re: Re: Broken unicode handling in unison 2.27.57
Martin von Gagern <Martin.vGagern <at> gmx.net>
2009-05-01 17:56:31 GMT
Benjamin Pierce wrote:
> I'm not an expert on unicode, character set, or internationalization
> issues, so I'm afraid I can't be much use here.
I guess I could provide the required input here. I mainly need a guide
around the Unison codebase and some OCaml features, so that I can find
out points of interest quickly without understanding the bulk of Unison.
Crash course for the issues at hand, not only for you, but for future
reference as well. SKip this for now if you want.
There are a lot of character sets out there. Some of them interpret one
byte as one character, according to some table, while others use more
complicated encodings to express a larger number of characters.
In the past, different encodings have often caused trouble. Unicode is
supposed to be universal, making all other character sets obsolete in
the long run, as they are all ideally subsets of unicode.
In its current form, a unicode codepoint (roughly a character, but the
term "codepoint" is more precise) is an integer of at most 20 or so
bits. There are several possible encodings of Unicode using units of
less bits, most notably UTF-8 using units of 8 bits (and up to six bytes
for a codepoint) and UTF-16 using units of 16 bits (and up to two such
words for a codepoint). Any of these encodings can express an arbitrary
sequence of codepoints, so they are equally expressive.
Now to normalization. Unicode does provide things called "combining
characters", which don't usually produce a glyph by themselves, but are
instead combined with the preceding character to form a single glyph.
(Continue reading)
I can understand your concerns. But how do you propose to continue? Neither the original author nor I myself are too fluent in OCaml, and I haven't even had a close enough look to understand most parts of the Unison code base. So I guess for us improving the patch on our own, without further input, would be pretty hard indeed. If you were to review the patch, see how ugly it really is, and point out things that would require improvement, that might be a good first step. Things that I can think of might require improvement: 1. The position of the change. Is Case.normalize the correct place? 2. The depndency. Is using camomile acceptable, or do we require our own implementation of unicode normalization? 3. Use of findlib. While I guess the use of findlib for camomile makes the build more portable, it might be cleaner to switch the whole unison build to findlib. On the other hand, if you want to keep build time deps to a minimum, findlib shouldn't be used at all. 4. The handling of compilation alternatives. Is providing two files "unicode.ml" in two different directories an acceptable way to provide and link to optional code?
> >
> >
> >
> > Ok, thanks a lot for your help in advance, and best regards,
> >
> > Stefan.
> >
> Hi Stefan...
>
RSS Feed