Re: Data update from ISO 639-5 to draft-4645bis

Re: Data update from ISO 639-5 to draft-4645bis

Peter Constable scripsit:

> "Dayak" is not a linguonym but rather an ethnonym or, even,
> a socionym. What was explained to me some years back is that its
> history is a pejorative term used by the "civilized" Muslim majority
> on the island of Borneo to refer to the many uncivilized, non-Muslim
> "tribal" minorities.

They are, in fact, the wild men from Borneo who have just (more or less)
come to town.

--

-- 
A poetical purist named Cowan           [that's me: cowan <at> ccil.org]
Once put the rest of us dowan.          [on xml-dev]
    "Your verse would be sweeter        http://www.ccil.org/~cowan
    If it only had metre
And rhymes that didn't force me to frowan."     [overpacked line!] --Michael Kay

draft-ietf-ltru-4645bis-10.txt issue with preferred value for YU

Re: draft-ietf-ltru-4645bis-10.txt issue with preferred value for YU

Shepherding writeup for draft-ietf-ltru-4646bis

[template from http://www.ietf.org/IESG/content/Doc-Writeup.html]

    (1.a) Who is the Document Shepherd for this document?

Martin Duerst (duerst <at> it.aoyama.ac.jp, LTRU WG co-chair)

          Has the 
          Document Shepherd personally reviewed this version of the 
          document and, in particular, does he or she believe this 
          version is ready for forwarding to the IESG for publication?

Yes, I have personally reviewed it, and concluded that this
version is ready for forwarding to the IESG for

    (1.b) Has the document had adequate review both from key WG members 
          and from key non-WG members?

Yes.

                                       Does the Document Shepherd have 
          any concerns about the depth or breadth of the reviews that 
          have been performed?

No.

    (1.c) Does the Document Shepherd have concerns that the document 
          needs more review from a particular or broader perspective, 
          e.g., security, operational complexity, someone familiar with 
          AAA, internationalization or XML?

No.

    (1.d) Does the Document Shepherd have any specific concerns or 
          issues with this document that the Responsible Area Director 
          and/or the IESG should be aware of? For example, perhaps he 
          or she is uncomfortable with certain parts of the document, or 
          has concerns whether there really is a need for it. In any 
          event, if the WG has discussed those issues and has indicated 
          that it still wishes to advance the document, detail those 
          concerns here. Has an IPR disclosure related to this document 
          been filed? If so, please include a reference to the 
          disclosure and summarize the WG discussion and conclusion on 
          this issue.

There are no specific concerns or issues that I would know of.

    (1.e) How solid is the WG consensus behind this document? Does it 
          represent the strong concurrence of a few individuals, with 
          others being silent, or does the WG as a whole understand and 
          agree with it?

Overall, I think the whole WG understands it and agrees with it.
There were a few points where there was long-lasting disagreement
and discussion, but for which we found solutions acceptable in
all scenarios. The main such point was the treatment of extlangs.

    (1.f) Has anyone threatened an appeal or otherwise indicated extreme 
          discontent? If so, please summarise the areas of conflict in 
          separate email messages to the Responsible Area Director. (It 
          should be in a separate email because this questionnaire is 
          entered into the ID Tracker.)

Yes.

    (1.g) Has the Document Shepherd personally verified that the 
          document satisfies all ID nits? (See 
          http://www.ietf.org/ID-Checklist.html and 
          http://tools.ietf.org/tools/idnits/). Boilerplate checks are 
          not enough; this check needs to be thorough. Has the document 
          met all formal review criteria it needs to, such as the MIB 
          Doctor, media type and URI type reviews?

Yes for ID nits. MIB, media type, URI considerations don't apply.

    (1.h) Has the document split its references into normative and 
          informative? Are there normative references to documents that 
          are not ready for advancement or are otherwise in an unclear 
          state? If such normative references exist, what is the 
          strategy for their completion? Are there normative references 
          that are downward references, as described in [RFC3967]? If 
          so, list these downward references to support the Area 
          Director in the Last Call procedure for them [RFC3967].

Yes for the split. No for unclear or downward references.

    (1.i) Has the Document Shepherd verified that the document IANA 
          consideration section exists and is consistent with the body 
          of the document? If the document specifies protocol 
          extensions, are reservations requested in appropriate IANA 
          registries? Are the IANA registries clearly identified? If 
          the document creates a new registry, does it define the 
          proposed initial contents of the registry and an allocation 
          procedure for future registrations? Does it suggest a 
          reasonable name for the new registry? See [RFC5226]. If the 
          document describes an Expert Review process has Shepherd 
          conferred with the Responsible Area Director so that the IESG 
          can appoint the needed Expert during the IESG Evaluation?

The document updates the procedures for the IANA Language Subtag Registry
(and the Language Tag Extension registry, which is mostly dormant).
The document contains the following language:

   The Language Subtag Reviewer is appointed by the IESG for an
   indefinite term, subject to removal or replacement at the IESG's
   discretion.  The IESG will solicit nominees for the position (upon
   adoption of this document or upon a vacancy) and then solicit
   feedback on the nominees' qualifications.  Qualified candidates
   should be familiar with BCP 47 and its requirements; be willing to
   fairly, responsively, and judiciously administer the registration
   process; and be suitably informed about the issues of language
   identification so that the reviewer can assess the claims and draw
   upon the contributions of language experts and subtag requesters.

The IESG will therefore have to solicit nominees for this position
again when this document is adopted.

    (1.j) Has the Document Shepherd verified that sections of the 
          document that are written in a formal language, such as XML 
          code, BNF rules, MIB definitions, etc., validate correctly in 
          an automated checker?

The ABNF for language tags in Figure 1 in the previous draft (-20.txt)
has passed http://www.fenron.com/~fenner/abnf.cgi without errors.
There were no changes in the ABNF between -20 and -21, but
www.fenron.com currently doesn't seem to be reachable.

    (1.k) The IESG approval announcement includes a Document 
          Announcement Write-Up. Please provide such a Document 
          Announcement Write-Up?

[there is no need for a question mark here 

                                 Recent examples can be found in the
          "Action" announcements for approved documents. The approval 
          announcement contains the following sections: 
          Technical Summary 
             Relevant content can frequently be found in the abstract 
             and/or introduction of the document. If not, this may be 
             an indication that there are deficiencies in the abstract 
             or introduction. 
          Working Group Summary 
             Was there anything in WG process that is worth noting? For 
             example, was there controversy about particular points or 
             were there decisions where the consensus was particularly 
             rough? 
          Document Quality 
             Are there existing implementations of the protocol? Have a 
             significant number of vendors indicated their plan to 
             implement the specification? Are there any reviewers that 
             merit special mention as having done a thorough review, 
             e.g., one that resulted in important changes or a 
             conclusion that the document had no substantive issues? If 
             there was a MIB Doctor, Media Type or other expert review, 
             what was its course (briefly)? In the case of a Media Type 
             review, on what date was the request posted?

This document describes the structure, content, construction, and
semantics of language tags for use in cases where it is desirable to
indicate the language used in an information object. It also
describes how to register values for use in language tags and the
creation of user-defined extensions for private interchange.
This document is an update of RFC4646. The main change is the
addition of thousands of three-letter language subtags for languages
for which tagging was not possible up to now. Also, the registry
format and procedures were adjusted to deal with this change,
and to reflect experience from current practice.

The WG process for this document was mostly smooth and revolving 
around details. There were some highly contentious issues, but
for all of them, a solution was found that was acceptable to
the involved parties and works for all scenarios identified.

The IANA Language Subtag Registry, and the language tags that can
be formed according to this document and its predecessor, are widely
used across the Internet to identify languages, both in implementations
(code) and in a wide range of data.

Regards,     Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst <at> it.aoyama.ac.jp     

LAST CALL REQUESTED

Hello Chris,

[I'm assuming you are the acting AD currently.
I'm cc'ing Alex so he knows what might be comming up for him.]

I'm glad to finally request IETF LAST CALL for
draft-ietf-ltru-4645bis(-10) and draft-ietf-ltru-4646bis(-21),
the two current WG documents of the LTRU WG.
The intended status of 4645bis is informational,
the intended status of 4646bis is BCP (BCP 47 together
with RFC 4647, which stays as it is).

I'm sending the shepherding documents in separate mails,
but I'm using this mail as a trigger; that's what Ted Hardie
told me to do when I did this thing last time.

If I need to do anything else, please don't hesitate to tell me.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst <at> it.aoyama.ac.jp     

Shepherding writeup for draft-ietf-ltru-4645bis

[template from http://www.ietf.org/IESG/content/Doc-Writeup.html]

    (1.a) Who is the Document Shepherd for this document?

Martin Duerst (duerst <at> it.aoyama.ac.jp, LTRU WG co-chair)

          Has the 
          Document Shepherd personally reviewed this version of the 
          document and, in particular, does he or she believe this 
          version is ready for forwarding to the IESG for publication?

Yes, I have personally reviewed it, and concluded that this
version is ready for forwarding to the IESG for

    (1.b) Has the document had adequate review both from key WG members 
          and from key non-WG members?

Yes. Due to its nature, the bulk of the document has mostly been
'reviewed' by cross-checking programs.

                                       Does the Document Shepherd have 
          any concerns about the depth or breadth of the reviews that 
          have been performed?

No.

    (1.c) Does the Document Shepherd have concerns that the document 
          needs more review from a particular or broader perspective, 
          e.g., security, operational complexity, someone familiar with 
          AAA, internationalization or XML?

No. The document mainly contains public data, which isn't security-
relevant. The considerable increase in size of the IANA registry has
been discussed directly with IANA.

    (1.d) Does the Document Shepherd have any specific concerns or 
          issues with this document that the Responsible Area Director 
          and/or the IESG should be aware of? For example, perhaps he 
          or she is uncomfortable with certain parts of the document, or 
          has concerns whether there really is a need for it. In any 
          event, if the WG has discussed those issues and has indicated 
          that it still wishes to advance the document, detail those 
          concerns here. Has an IPR disclosure related to this document 
          been filed? If so, please include a reference to the 
          disclosure and summarize the WG discussion and conclusion on 
          this issue.

There are no specific concerns or issues that I would know of.

    (1.e) How solid is the WG consensus behind this document? Does it 
          represent the strong concurrence of a few individuals, with 
          others being silent, or does the WG as a whole understand and 
          agree with it?

I don't have any doubt that the WG as a whole understands it and
agrees it, because it makes available several thousand new language
subtags for languages which up to now cannot be tagged. For some
issues (e.g. the treatment of extlangs), there has been intense
discussion with diametrally opposing positions, but we were able
to find a solution that was acceptable to everybody.

    (1.f) Has anyone threatened an appeal or otherwise indicated extreme 
          discontent? If so, please summarise the areas of conflict in 
          separate email messages to the Responsible Area Director. (It 
          should be in a separate email because this questionnaire is 
          entered into the ID Tracker.)

Yes.

    (1.g) Has the Document Shepherd personally verified that the 
          document satisfies all ID nits? (See 
          http://www.ietf.org/ID-Checklist.html and 
          http://tools.ietf.org/tools/idnits/). Boilerplate checks are 
          not enough; this check needs to be thorough. Has the document 
          met all formal review criteria it needs to, such as the MIB 
          Doctor, media type and URI type reviews?

Yes for ID nits. MIB, media type, URI considerations don't apply.

    (1.h) Has the document split its references into normative and 
          informative? Are there normative references to documents that 
          are not ready for advancement or are otherwise in an unclear 
          state? If such normative references exist, what is the 
          strategy for their completion? Are there normative references 
          that are downward references, as described in [RFC3967]? If 
          so, list these downward references to support the Area 
          Director in the Last Call procedure for them [RFC3967].

Yes for the split. No for unclear or downward references.

    (1.i) Has the Document Shepherd verified that the document IANA 
          consideration section exists and is consistent with the body 
          of the document? If the document specifies protocol 
          extensions, are reservations requested in appropriate IANA 
          registries? Are the IANA registries clearly identified? If 
          the document creates a new registry, does it define the 
          proposed initial contents of the registry and an allocation 
          procedure for future registrations? Does it suggest a 
          reasonable name for the new registry? See [RFC5226]. If the 
          document describes an Expert Review process has Shepherd 
          conferred with the Responsible Area Director so that the IESG 
          can appoint the needed Expert during the IESG Evaluation?

The document provides content for an actual update of the IANA Language
Subtag Registry. The details are clearly spelled out in the IANA section,
and the most crucial aspects of this update have already been discussed
with IANA.

    (1.j) Has the Document Shepherd verified that sections of the 
          document that are written in a formal language, such as XML 
          code, BNF rules, MIB definitions, etc., validate correctly in 
          an automated checker?

The bulk of the document is in a format defined in draft-ietf-ltru-4646bis.
I haven't verified this myself, but several members of the WG have,
with their own tools.

    (1.k) The IESG approval announcement includes a Document 
          Announcement Write-Up. Please provide such a Document 
          Announcement Write-Up?

[there is no need for a question mark here 

                                 Recent examples can be found in the
          "Action" announcements for approved documents. The approval 
          announcement contains the following sections: 
          Technical Summary 
             Relevant content can frequently be found in the abstract 
             and/or introduction of the document. If not, this may be 
             an indication that there are deficiencies in the abstract 
             or introduction. 
          Working Group Summary 
             Was there anything in WG process that is worth noting? For 
             example, was there controversy about particular points or 
             were there decisions where the consensus was particularly 
             rough? 
          Document Quality 
             Are there existing implementations of the protocol? Have a 
             significant number of vendors indicated their plan to 
             implement the specification? Are there any reviewers that 
             merit special mention as having done a thorough review, 
             e.g., one that resulted in important changes or a 
             conclusion that the document had no substantive issues? If 
             there was a MIB Doctor, Media Type or other expert review, 
             what was its course (briefly)? In the case of a Media Type 
             review, on what date was the request posted?

This memo defines the procedure used to update the IANA Language
Subtag Registry in conjunction with the publication of
draft-ietf-ltru-4646bis [NOTE: replace with actual RFC number],
for use in forming tags for identifying languages. As an Internet-Draft,
it also contained a complete replacement of the contents of the Registry
to be used by IANA in updating it. This update adds several thousand
language codes to the registry, which will allow these languages to
be identified appropriately on the Internet. To prevent confusion, the
actual registry contents was removed before publication as an RFC.

The WG process for this document was mostly smooth and revolving 
around details. This document also reflects changes defined in
draft-ietf-ltru-4646bis, which are disussed in a separate writeup.

The data contained in the document when it was an Internet Draft
has been read and processed by several tools the implement parsing
of the registry format and additional operations on this data.

Regards,    Martin.

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst <at> it.aoyama.ac.jp     

Re: draft-ietf-ltru-4645bis-10.txt issue with preferred valuefor YU

Hi -

> From: "Tex Texin" <textexin <at> xencraft.com>
> To: <ltru <at> ietf.org>; <ietf <at> ietf.org>
> Sent: Monday, March 02, 2009 1:05 AM
> Subject: [Ltru] draft-ietf-ltru-4645bis-10.txt issue with preferred valuefor YU
>
> With respect to the proposed update to the Language Subtag Registry draft-ietf-ltru-4645bis-10:
>
> I would like to lodge an objection to the deletion of the Preferred-Value for language subtag YU.

As ltru co-chair: it's exceedingly late for such an objection - the issue was
discussed at length in the working group over a year ago.  A recent
revisiting of the question arrived at the same conclusion.

> This change breaks the equivalence class relation between YU and CS.
> It detrimentally changes the behavior of existing implementations.

As a technical contributor:

The main reason that CS does not make sense as a preferred value for
YU is that there is *not* an "equivalence class relation" between them.
There are pieces of what was YU that are not covered by CS.  To treat
them as an "equivalence class" ignores linguistic, geographic, and
political reality.

> The loss of the relationship between YU and CS makes documents that were
> believed to be tagged equivalently, to no longer be equivalent.

In my opinion, regarding them as equivalent is an error, since
CS and YU don't encompass the same regions.

> There is also no benefit to this change.

I disagree.  The change removes an error.

> To be concrete, assume a user attempts to find documents for languages from Yugoslavia.

Language tags do *not* pretend to be able to answer this sort of query.

Using a region subtag (e.g. 'CS') says that the data subtag uses a specific
variety of the primary language, and that the party tagging the data believes
that this distinction is useful.  For example, I could tag this paragraph with
'en' or with 'en-US'.  Is that extra distinction necessary or useful?  In this
case, no.  Consequently, the "retrieving documents by region subtag" use case,
although technically permitted by RFC 4647, is not realistic, and in many
ways contrary to the basic "tag wisely" principle.

> Using the then current registry data, a query tool noting the preferred
> value relationship, matches either xx-YU and xx-CS.
>
> Another user searches for documents for Serbia.
>
> A query tool using the current registry data noting the preferred value
> relationship, matches either xx-YU and xx-CS.
>
> The results are in some sense accurate and complete, given the history of the subtag.

No, they are not.
  (1) there is no requirement, much less a guarantee, that the data will
      bear a region subtag at all
  (2) there are many bits and pieces of YU not covered by CS - 
      even if data always bore a region subtag, the YU->CS mapping
      would miss all the other territory that once belonged to YU.
  (3) blindly replacing all YU subtags with CS subtags would in fact
      falsify some data, since the language could well be of a variety
      covered by YU but not by CS.

> After this change in the preferred value relationship, the query
> tool does not know to search for both xx-YU and xx-CS, since the
> registry does not indicate a relationship. Only one or the other
> subtag is used for each query. However, the query results are now
> incomplete since some documents for xx-YU have been tagged with
> the one-time preferred tag of xx-CS.

The relationship cannot be adequately automated with a simple
one-way pointer like "preferred-value".  The former YU also
encompassed BA, HR, ME, MK, RS, and SI.

> Comments in the registry are not a solution. Comments are a good
> thing for recording rationale and tangential history. However,
> implementers are not going to go thru and read the comments on any
> or all tags in order to make a correct implementation. They are going
> to implement based on the schema and operate with the data values.

If someone (or something) is applying region subtags, they'd better
have sufficient knowledge of the language varieties to do so meaningfully.
This effectively requires *understanding* those comments and much more.
The Language Subtag Registry does *not* attempt to record all the
information needed to recognize language varieties.  Rather, once
someone (or something) has made a distinction, the LSR provides
the bits needed to encode a tag for that language variety.

In the particular case of the languages of the former YU, the
region subtags now available (such as BA, HR, ME, MK, RS, and SI)
are arguably far more useful, if someone needs to distinguish
regional variations in their Croatian-language data, than just YU.
(It's unclear to me whether YU would ever have been terribly useful,
since it would allow the distinction of Croatian as spoken there
from Croatian spoken somewhere (where?) else.)

> The registry should stay as it is with respect to YU and retain
> CS as the preferred value.
>
> As CS is now being used as a preferred value, deprecated or not,
> there isn't a compelling motivation to remove the preferred value for YU.

Please, let's look at the actual tagged language data.
What corpora out there have employed YU (correctly) as a subtag?
To what extent would replacing that subtag with 'CS' (rather than
with BA, HR, ME, MK, RS, or SI) be correct, for Serbian, Croatian,
or any of the other languages of that region?

> Please eliminate this needless change that breaks applications
> relying on the relationship between YU and CS.

I would argue rather than an application that relies on an
equivalence relation between YU and CS is already in some sense
broken, in the same way as one assuming that Russia and the
Soviet Union are somehow equivalent.

> tex

Randy

RE: draft-ietf-ltru-4645bis-10.txt issue with preferred value for YU

Hi guys,

I sent the message to this list (as well as LTRU) in the belief I was
following the instructions given to me...
(My mail actually preceded Martin's request to the AD.)

I admit to confusion about the suggestion I can register the change after
4645bis is accepted.
4645bis changes a code that exists already. I shouldn't have to reregister
it to restore it.
If anything, have 4645bis go forward without the change, and the change can
be discussed separately.

I'll pursue the discussion on the ltru list as requested and respond to
Randy's other comments there.

tex

-----Original Message-----
From: ietf-bounces <at> ietf.org [mailto:ietf-bounces <at> ietf.org] On Behalf Of
Randy Presuhn
Sent: Monday, March 02, 2009 5:36 PM
To: ietf <at> ietf.org
Subject: Re: draft-ietf-ltru-4645bis-10.txt issue with preferred value for
YU

Hi -

> From: "Phillips, Addison" <addison <at> amazon.com>
> To: <ietf <at> ietf.org>
> Sent: Monday, March 02, 2009 9:49 AM
> Subject: RE: draft-ietf-ltru-4645bis-10.txt issue with preferred value for
YU
>
> Hi Tex,
> 
> I don't think this is probably appropriate, at least for this list to
consider.

Tex's posting came after the document shepherd (co-chair Martin Duerst)
had sent the information to our AD requesting that the IESG consider
publishing it.  So although the IESG has not yet (AFAIK) acted on the
request, much less issued an IETF last call, I can understand why
ietf <at> ietf.org might be included.

I have already responded to it on both lists, even though I think the
issue is probably of little interest to most on the ietf <at> ietf.org list.
Unless instructed to do otherwise by our AD, I would suggest that
all follow-on discussion be directed to ltru <at> ietf.org

> 1. You haven't posted to LTRU's mailing list, only ietf-languages@, yet.

Tex's message was posted to *both* lists.

> 2. Even if draft-4645bis is approved, the process for language tags
> (in either RFC 4646 or its proposed successor) allow you to register
> the information you want, if you think it was inappropriately omitted.
...

Correct.

Randy

_______________________________________________
Ietf mailing list
Ietf <at> ietf.org
https://www.ietf.org/mailman/listinfo/ietf

Re: draft-ietf-ltru-4645bis-10.txt issue with preferred value for YU