Development of the lxml library

headers Robert Pierce | 1 Jun 18:21

_______________________________________________
lxml-dev mailing list
lxml-dev <at> codespeak.net
http://codespeak.net/mailman/listinfo/lxml-dev

Hi,

This is a bit late, but thanks for the response.

I am playing around with iterparse() and am following the advice you  
gave.

I have a question though: I could not find a way to consume an element  
and replace it with just text. For example <i18n:text>hello</ 
i18n:text> when found in the middle of a paragraph will be replaced by  
text. The replace() method requires the replacement to be an element.

Is this possible?

Thanks!

Alexis Georges

On 28-Apr-09, at 1:59 PM, Stefan Behnel wrote:

> Hi,
>
> Alexis Georges wrote:
>> I am maintaining a multilingual website which works with XML, XSLT to
>> generate XHTML.
>>
>> I am working with Apache Cocoon (http://cocoon.apache.org/2.1/) using
>> (among other things) their I18NTransformer. Basically I can use  
>> elements
>> in the I18N (http://apache.org/cocoon/i18n/2.1) namespace, and then  
>> tell
>> Cocoon to apply the I18NTransfomer to the document; this replaces the
>> I18N elements with a localized value (eg. a formatted date/number, a
>> translated label/attribute, etc...).
>>
>> I have been looking at lxml a little bit to see if I could move to a
>> Python-based framework for the website. I am not quite sure how to go
>> about the I18N part though.
>>
>> Using the Babel library (http://babel.edgewall.org/) along with  
>> request
>> headers to generate localized data, I have everything I need. What is
>> missing is the "parser" for the I18N elements. All I can think of  
>> right
>> now is to implement a SAX parser, the way Cocoon does (in Java).
>
> There is a SAX-like interface in lxml.etree, called "target parser".
>
> However, if your documents fit into memory, using iterparse() is a lot
> simpler (and likely not even much slower).
>
> Something like this might work:
>
>     context = etree.iterparse(
>              "somefile.xml",
>              tag = "{http://apache.org/cocoon/i18n/2.1}*")
>
>     for event, i18n_element in context:
>         new_element = get_i18n_replacement_for(i18n_element)
>         i18n_element.getparent().replace(i18n_element, new_element)
>
>     context.getroottree().write("newfile.xml")
>
> See here for some documentation:
>
> http://codespeak.net/lxml/parsing.html
>
> You can also achieve the same thing in XSLT, or using XPath, or ...
>
> Stefan

_______________________________________________
lxml-dev mailing list
lxml-dev <at> codespeak.net
http://codespeak.net/mailman/listinfo/lxml-dev

Hi,

> The nil node <Fubar/> is not deannotated as I would expect in the
> following
> snippet.  I could not find a reference to this behaviour in the archives
> or
> documentation.  Is this a design feature for which there is a work around,
> or a bug?  I'm using lxml-2.2-py2.5-linux-i686.

Design feature. Only py:pytype/xsi:type attributes get removed by deannotate():

>>> print etree.__version__
2.1.5
>>> help(objectify.deannotate)

Help on built-in function deannotate in module lxml.objectify:

deannotate(...)
    deannotate(element_or_tree, pytype=True, xsi=True)

    Recursively de-annotate the elements of an XML tree by removing 'pytype'
    and/or 'type' attributes.

    If the 'pytype' keyword argument is True (the default), 'pytype' attributes
    will be removed. If the 'xsi' keyword argument is True (the default),
    'xsi:type' attributes will be removed.

IMHO the xsi:nil concept in XML Schema pretty much corresponds to NULL values in databases, i.e. a typed
element/column may (or may not) be xsi:nil/NULL, but it does not so directly translate to the distinct
Python None object. OTOH I think mapping xsi:nil to None very much captures the meaning of xsi:nil/NULL,
because in most use cases you'd test if a value has been set (!=None) or not (==None).

Or course, you can always easily get rid of xsi:nil if you wish so:

>>> for elt in root.iter(): elt.attrib.pop('{http://www.w3.org/2001/XMLSchema-instance}nil', None)

Holger
--

-- 
Nur bis 31.05.: GMX FreeDSL Komplettanschluss mit DSL 6.000 Flatrate und
Telefonanschluss nur 17,95 Euro/mtl.!* http://portal.gmx.net/de/go/dsl02

_______________________________________________
lxml-dev mailing list
lxml-dev <at> codespeak.net
http://codespeak.net/mailman/listinfo/lxml-dev

qhlonline wrote:
> When I used the lxml with self defined  Target Parser, There is a
> function that can be redefined-- data . def data (self, data): When can
> we use it?

when you want to receive character content from the document you parse.

> and what it will do when we simply write a single line: "return " ?

nothing? actually, a "pass" will do in that case, as will not implementing
the method (IIRC).

> Is there any encoding conversion?

You will get either ASCII encoded byte strings or unicode strings, just
like everywhere else.

BTW, it's sometimes faster to try these things out than to ask a mailing list.

Stefan

Hi,

qhlonline wrote:
> There are instances that when an HTML file has meta tags, the
> charset declared in  tag is not right, because the HTML content next is
> using a different encoding. But lxml will parse accroding to what  said.
> In this situation, it may report error information of error decoding,
> but some times it can parse, and generate a DOM that is not complete.

By default, the HTML parser will ignore errors and try to keep parsing
regardless. Pass "recover=False" if you want to get an exception instead.

Note that character decoding errors cannot always be detected, as they may
lead to valid (although unreadable) characters even when the wrong encoding
is assumed. Latin-1 is a good example, which uses a plain 8-bit encoding.
It will work perfectly well to read a UTF-8 encoded document with a Latin-1
decoder. It just won't give you readable output in most cases.

> eg. I have a WEB file has  while the following content is encoded with
> GBK(which is a Supper set of GB2312). We have got a result with only
> part of the HTML tags parsed out. I wan't to know, if lxml have any
> warning or error information reported for this situation? What it is?

See the error_log property on the parser.

http://codespeak.net/lxml/parsing.html#error-log

> Is there any common
> method? I have also seen some HTML files have  tag attributes "lang", I
> don't know whether this attribute is used in the HTML parsing process.

I don't think so.

> In meta tag like , there are also language statement, But in the
> htmlCheckMeta method of libxml2 library source, I didn't find any
> processing with the http-equiv attribute value "Content-Language".

The "language" is not relevant to the parser. The charset is. Just think of
UTF-8, which can encode any written language that uses characters defined
in Unicode.

Stefan

Hi,

Holger wrote:
>> The nil node <Fubar/> is not deannotated as I would expect in the
>> following
>> snippet.  I could not find a reference to this behaviour in the archives
>> or
>> documentation.  Is this a design feature for which there is a work around,
>> or a bug?  I'm using lxml-2.2-py2.5-linux-i686.
> 
> Design feature.

I'd be a little more careful with such a big word. ;)

> Only py:pytype/xsi:type attributes get removed by deannotate():
> 
>>>> print etree.__version__
> 2.1.5
>>>> help(objectify.deannotate)
> 
> Help on built-in function deannotate in module lxml.objectify:
>  
> deannotate(...)
>     deannotate(element_or_tree, pytype=True, xsi=True)
>  
>     Recursively de-annotate the elements of an XML tree by removing 'pytype'
>     and/or 'type' attributes.
>  
>     If the 'pytype' keyword argument is True (the default), 'pytype' attributes
>     will be removed. If the 'xsi' keyword argument is True (the default),
>     'xsi:type' attributes will be removed.

Yes, so it's even implicitly documented. :)

Anyway, I'm not sure it's always a good idea to leave this special case in
instead of cleaning everything up. I think if you remove it, you'd get an
empty string result, which may be surprising - but more surprising than not
getting it cleaned up? After all, deannotate() means deannotate()...

Stefan

Hi all,

I just pushed lxml 2.2.1 to PyPI as a minor maintenance release.

Changelog follows below.

This release was built with Cython 0.11.2.

Have fun,

Stefan

2.2.1 (2009-06-02)
Features added

    * Injecting default attributes into a document during XML Schema
      validation (also at parse time).
    * Pass huge_tree parser option to disable parser security restrictions
      imposed by libxml2 2.7.

Bugs fixed

    * The script for statically building libxml2 and libxslt didn't work
      in Py3.
    * XMLSchema() also passes invalid schema documents on to libxml2 for
      parsing (which could lead to a crash before release 2.6.24).

_______________________________________________
lxml-dev mailing list
lxml-dev <at> codespeak.net
http://codespeak.net/mailman/listinfo/lxml-dev

Mon	Tue	Wed	Thu	Fri	Sat	Sun

1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30

9	February 2012
46	January 2012
29	December 2011
33	November 2011
44	October 2011
29	September 2011
26	August 2011
28	July 2011
44	June 2011
42	May 2011
45	April 2011
46	March 2011
82	February 2011
8	January 2011
27	December 2010
36	November 2010
39	October 2010
68	September 2010
29	August 2010
80	July 2010
51	June 2010
72	May 2010
38	April 2010
83	March 2010
48	February 2010
81	January 2010
73	December 2009
88	November 2009
58	October 2009
60	September 2009
83	August 2009
100	July 2009
132	June 2009
51	May 2009
45	April 2009
54	March 2009
58	February 2009
68	January 2009
45	December 2008
104	November 2008
43	October 2008
42	September 2008
101	August 2008
136	July 2008
120	June 2008
110	May 2008
66	April 2008
75	March 2008
165	February 2008
75	January 2008
43	December 2007
30	November 2007
100	October 2007
111	September 2007
84	August 2007
146	July 2007
175	June 2007
95	May 2007
76	April 2007
55	March 2007
94	February 2007
20	January 2007
36	December 2006
63	November 2006
84	October 2006
121	September 2006
117	August 2006
61	July 2006
286	June 2006
281	May 2006
42	April 2006
136	March 2006
32	February 2006
23	January 2006
96	December 2005
263	November 2005
50	October 2005
41	September 2005
126	August 2005
29	July 2005
59	June 2005
78	May 2005
48	April 2005
22	January 2005
5	December 2004
28	November 2004
19	October 2004
6	September 2004

lxml.objectify.deannotate refuses to clean nil nodes

Re: XML Documents & I18N (the way Cocoon does it)

lxml deconding problem caused by <meta> tag specification

Re: lxml.objectify.deannotate refuses to clean nil nodes

lxml about Target Parser

Re: lxml about Target Parser

Re: lxml deconding problem caused by <meta> tag specification

Re: lxml.objectify.deannotate refuses to clean nil nodes

lxml 2.2.1 released

Re: lxml.objectify.deannotate refuses to clean nil nodes