Re: lxml Mac installation idea

Hi Ian,

Ian Bicking wrote:
> Stefan Behnel wrote:
> but I'd like it to work without buildout, and there's also several
> buildout recipes and configurations out there and not one clear
> canonical way to build lxml.

I wouldn't mind to ship lxml with a buildout recipe. I think the current
problem is that if we wait to find one that works well for as many people as
possible, we'll wait forever. So I'm fine with adding any script that's
somewhat tested and in use. Even a set of scripts that you can try is better
than none if people can't manage to build lxml themselves.

>>> Another option would be simply a different tarball that contains the
>>> libxml2/libxslt source, and its setup.py would always build those. 
>>> It could be versioned like 2.1static or something, which should keep
>>> it from being implicitly used by easy_install, etc. (since 2.1static
>>> is considered an earlier version than 2.1).  This might be more
>>> reasonable?
>>
>> The problem with this (and with the static Windows builds) is that
>> libxml2/libxslt both have their release cycles, which are independent of
>> lxml's releases. If you want to upgrade your libxml2 in a static
>> build, you'll have to copy it to the right place anyway.
> 
> Another option is yet another environmental variable to set the
> libxml2/libxslt versions, which are set to defaults.  If you chose a
> version that didn't exist in the tarball it'd download that version.

Re: lxml Mac installation idea

On Nov 2, 2008, at 2:55 PM, Stefan Behnel wrote:

> Hi Ian,
>
> Ian Bicking wrote:
>> Stefan Behnel wrote:
>> but I'd like it to work without buildout, and there's also several
>> buildout recipes and configurations out there and not one clear
>> canonical way to build lxml.
>
> I wouldn't mind to ship lxml with a buildout recipe. I think the  
> current
> problem is that if we wait to find one that works well for as many  
> people as
> possible, we'll wait forever. So I'm fine with adding any script  
> that's
> somewhat tested and in use. Even a set of scripts that you can try  
> is better
> than none if people can't manage to build lxml themselves.

FWIW, we have a very lean buildout that only focuses on the problem of  
building lxml.  I could contribute it if you'd like.

>>>> Another option would be simply a different tarball that contains  
>>>> the
>>>> libxml2/libxslt source, and its setup.py would always build those.
>>>> It could be versioned like 2.1static or something, which should  
>>>> keep
>>>> it from being implicitly used by easy_install, etc. (since  

Re: lxml Mac installation idea

Paul Everitt wrote:
>> Ian Bicking wrote:
>>> Stefan Behnel wrote:
>>> but I'd like it to work without buildout, and there's also several
>>> buildout recipes and configurations out there and not one clear
>>> canonical way to build lxml.
>>
>> I wouldn't mind to ship lxml with a buildout recipe. I think the current
>> problem is that if we wait to find one that works well for as many 
>> people as
>> possible, we'll wait forever. So I'm fine with adding any script that's
>> somewhat tested and in use. Even a set of scripts that you can try is 
>> better
>> than none if people can't manage to build lxml themselves.
> 
> FWIW, we have a very lean buildout that only focuses on the problem of 
> building lxml.  I could contribute it if you'd like.

How does this compare to plone.recipe.lxml 
(http://pypi.python.org/pypi/plone.recipe.lxml)?  I notice that package 
just had a release.  Is this the config you've been using? 
https://svn.plone.org/svn/plone/PloneOrg/sandbox/xdv/new.plone.org/osx.cfg 
(if so, in your run-deliverance script, it'd be best to use exec to 
avoid the intermediate shell process and make it easier to kill the script)

One problem with buildout recipes is that they aren't, AFAIK, very 
compatible with other systems.  That is, they can build the egg, but the 
egg isn't on the path of any interpreter, so unless you use buildout 
entirely you'll have to do some further manipulation to make lxml available.

Re: lxml Mac installation idea

On Sun, 02 Nov 2008 19:55:09 -0000, Stefan Behnel <stefan_ml <at> behnel.de>  
wrote:

> Hi Ian,
>
> Ian Bicking wrote:
>> Stefan Behnel wrote:
>> but I'd like it to work without buildout, and there's also several
>> buildout recipes and configurations out there and not one clear
>> canonical way to build lxml.
>
> I wouldn't mind to ship lxml with a buildout recipe. I think the current
> problem is that if we wait to find one that works well for as many  
> people as
> possible, we'll wait forever. So I'm fine with adding any script that's
> somewhat tested and in use. Even a set of scripts that you can try is  
> better
> than none if people can't manage to build lxml themselves.
>
>

>
>> * Macs come with bad versions of libxml2 and libxslt (depending on the
>> version of the OS, you get either very bad, or not as bad, but bad
>> enough that you'll eventually get a segfault but not immediately, which
>> is actually much worse)
>>
>> * People keep getting the wrong runtime linking, and DYLD_LIBRARY_PATH
>> seems to be necessary.
>

Re: lxml Mac installation idea

On Mon, Nov 3, 2008 at 11:38 AM, Paul Everitt <paul <at> agendaless.com> wrote:
>>> I do now have ssh access to a Mac.  Unfortunately it's behind our
>>> VPN,
>>> so I'm not sure if I can get you access to it too, but I'll ask
>>> about that.
>>
>> I'm still waiting for the usual "we can pay you the hardware, so
>> that you can
>> do the testing yourself" offers. ;o)
>
> I'll chip in.  I wonder if there are OS X hosting companies where we
> could rent a login.

Sourceforge.net Build Farm used to have OSX hosts. They discontinued
the service somewhere in 2007 though. :(

--

-- 
Sidnei da Silva
Enfold Systems
http://enfoldsystems.com
Fax +1 832 201 8856
Office +1 713 942 2377 Ext 214
Skype zopedc

Re: [XML-SIG] Messages when installing xml

Hi,

first of all: is there any reason you are not using the latest binary packages
of lxml 2.1?

Also: the mailing list of lxml is a better place to ask these questions than
the more general XML-SIG list.

Colin J. Williams wrote:
> The log of my install is below:
> 
>     Building lxml version 2.2.alpha1-59220.
>     Building with Cython 0.9.8.1.1.
>     ERROR: 'xslt-config' is not recognized as an internal or external command,
>     operable program or batch file.
> 
>     ** make sure the development packages of libxml2 and libxslt are installed **

You need to install libxml2 and libxslt and add the xslt-config script that
ships with libxslt to your program search path when building lxml.

>     [...]
>     building 'lxml.etree' extension
>     C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\python25\include
>     -IC:\python25\PC -c src/lxml\lxml.etree.c -o
>     build\temp.win32-2.5\Release\src\lxml\lxml.etree.o -w

It's generally recommended to build lxml statically against libxml2/libxslt on
Windows. There are some build instructions on the web page.

Re: lxml Mac installation idea

_______________________________________________
lxml-dev mailing list
lxml-dev <at> codespeak.net
http://codespeak.net/mailman/listinfo/lxml-dev

setupinfo.py that downloads/builds libxml2/libxslt

I've attached a version of setupinfo.py that will download and build 
libxml2 and libxslt.  There are some problems with the script, but 
hopefully this is a start.

If you give the build option --build-libxml2xslt then it will download 
the packages and build them, and set the options to use the xml2-config 
and xslt-config scripts.  I also made it always use --static if you do 
this.  I'm not sure if that's a good idea, but if they aren't built 
statically then I'm not sure where to install the libraries... though I 
guess they could be installed in some subdirectory of the lxml package? 
  That would be fine, unless there were problems if those libraries were 
then moved around (as they are often during the installation process), 
it might mean that the linker won't find them...?  Also adding files to 
the installation process means some more work.  (It just occurred to 
me... do I have to run "make; make install" for those libraries if it is 
a static build, or would ./configure be sufficient?)

The script will download the newest versions of these libraries, or you 
can give a specific version with --libxml2-version/--libxslt-version. 
Also BUILD_LIBXML2XSLT=true, LIBXML2_VERSION, and LIBXSLT_VERSION 
environmental variables work for these settings.  Maybe there can also 
be a fallback to archives distributed directly with lxml?  The tarballs 
are about 8Mb for the two of them.  Or I suppose just some more flags to 
point to existing source code or tarballs.

The biggest problem is that I'm not sure at all where to put this stuff 
in the setup.py build process.  Right now the options to setup() include 
information about the built versions of these libraries, but simply 
calling "python setup.py" should not cause those libraries to be built. 
  (Right now if you use --build-libxml2xslt it will build those 
libraries at setupinfo/setup.py import time.)  This is the biggest 
problem I see with this at the moment.

Lastly, I'm having problems building lxml trunk right now, with this error:

> cythoning src/lxml/lxml.etree.pyx to src/lxml/lxml.etree.c
> 
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
>     cdef readonly object localname
>     cdef readonly object namespace
>     def __init__(self, text_or_uri_or_element, tag=None):
>         if not _isString(text_or_uri_or_element):
>             if isinstance(text_or_uri_or_element, _Element):
>                 text_or_uri_or_element = (<_Element>text_or_uri_or_element).tag
>                                                   ^
> ------------------------------------------------------------
> 
> /home/ianb/src/lxml/src/lxml/lxml.etree.pyx:259:51: Declarator should be empty

But I got this working with the last lxml release, so that's fine.

--

-- 
Ian Bicking : ianb <at> colorstudy.com : http://blog.ianbicking.org

_______________________________________________
lxml-dev mailing list
lxml-dev <at> codespeak.net
http://codespeak.net/mailman/listinfo/lxml-dev

lxml XSD Schema validation problem

Hi guys,

I'm getting an error using the lxml objectification API and XSD Validation.

Am using lxml and from distribution on standard hardy lib versions.

Here's the boilerplate version info from the FAQ:

lxml.etree:        (2, 1, 2, 0)
libxml used:       (2, 6, 31)
libxml compiled:   (2, 6, 31)
libxslt used:      (1, 1, 22)
libxslt compiled:  (1, 1, 22)

The code I'm testing with is as follows:

############################ >8 ############################
#!/usr/bin/python

from lxml import objectify
from lxml import etree

SCHEMA_FILE = 'nvdcve.xsd'
DATA_FILE = 'nvdcve-recent.xml'

schema_file = open(SCHEMA_FILE)
data_file = open(DATA_FILE)

schema = etree.XMLSchema(file=schema_file)
parser = objectify.makeparser(schema=schema)

tree = objectify.parse(data_file, parser)
############################ 8< ############################

The XSD I'm validating against is the XSD provided by NIST for CVE posts and
the XML doc is the recent NVDs:

http://nvd.nist.gov/schema/nvdcve.xsd
http://nvd.nist.gov/download/nvdcve-recent.xml

The error I'm getting from lxml is:

lxml.etree.XMLSyntaxError: Element '{http://nvd.nist.gov/feeds/cve/1.2}ref',
attribute 'url': [facet 'pattern'] The value
'http://www.securitytracker.com/id?1021107' is not accepted by the pattern
'(news|(ht|f)tp(s)?)://.+'.

The problem is that on every implementation of regex patterns I can think of
(Python, Perl, egrep etc) the regex validates... Indeed on my win32 xml
editors (LiquidXML and OxygenXML) the data validates with no problems.

$ egrep "(news|(ht|f)tp(s)?)://.+" nvdcve-recent.xml | wc -l
497

pcretest comes back fine too:

$ pcretest
PCRE version 7.4 2007-09-21

  re> #(news|(ht|f)tp(s)?)://.+#
data> http://www.securitytracker.com/id?1021107
 0: http://www.securitytracker.com/id?1021107
 1: http
 2: ht

Is this a libxml2 problem, my code or, dare I say, an evil bug?

Thanks

--

-- 
Richard

Re: setupinfo.py that downloads/builds libxml2/libxslt

Hi Ian,

Ian Bicking wrote:
> I've attached a version of setupinfo.py that will download and build
> libxml2 and libxslt.  There are some problems with the script, but
> hopefully this is a start.

thanks a lot for doing this. It does not yet build a static lxml for me (the
static stuff was meant for Windows and it doesn't really work on other
systems), but I'm fixing it up to do that.

> If you give the build option --build-libxml2xslt then it will download
> the packages and build them, and set the options to use the xml2-config
> and xslt-config scripts.  I also made it always use --static if you do
> this.

That's definitely the right thing to do. If you use a complete build, it's
best to build statically against your libraries to make sure they are really used.

> do I have to run "make; make install" for those libraries if it is
> a static build, or would ./configure be sufficient?)

"make" is sufficient, no installation required. libxslt even takes a
"--with-libxml-src=DIR" option to build against an uninstalled libxml2.

> Maybe there can also be a fallback to archives distributed directly with lxml?

Yes, that would be great. If they are there, use them, otherwise, do a normal
build.

> The biggest problem is that I'm not sure at all where to put this stuff
> in the setup.py build process.  Right now the options to setup() include
> information about the built versions of these libraries, but simply
> calling "python setup.py" should not cause those libraries to be built.
>  (Right now if you use --build-libxml2xslt it will build those libraries
> at setupinfo/setup.py import time.)  This is the biggest problem I see
> with this at the moment.

I think an option for the download and a check if the library archives lie
around somewhere to trigger the build is best.

> Lastly, I'm having problems building lxml trunk right now, with this error:
> 
>> cythoning src/lxml/lxml.etree.pyx to src/lxml/lxml.etree.c
>>
>> Error converting Pyrex file to C:
>> ------------------------------------------------------------
>> ...
>>     cdef readonly object localname
>>     cdef readonly object namespace
>>     def __init__(self, text_or_uri_or_element, tag=None):
>>         if not _isString(text_or_uri_or_element):
>>             if isinstance(text_or_uri_or_element, _Element):
>>                 text_or_uri_or_element =
>> (<_Element>text_or_uri_or_element).tag
>>                                                   ^
>> ------------------------------------------------------------
>>
>> /home/ianb/src/lxml/src/lxml/lxml.etree.pyx:259:51: Declarator should
>> be empty

Looks like your Cython is somehow broken. It works for me here.

Thanks again,

Stefan