headers
Phillip J. Eby | 5 Jul 2006 17:36
Gravatar

Re: Specification for package indexes?

At 04:04 AM 7/5/2006 -0400, Jim Fulton wrote:

>On Jun 23, 2006, at 4:51 PM, Jim Fulton wrote:
>...
>>
>>That's a lot of screen scraping. :)
>>
>>It would be good to capture this as part of the documentation IMO
>>
>>>I'm considering adding XML-RPC support to easy_install in 0.7,
>>>though.  PyPI now has a nice XML-RPC API that is more responsive
>>>than the web UI, and it supports case-insensitive partial match
>>>searches, making it suitable for easy_install to query when a typed-
>>>in name doesn't exactly match the spelling of a PyPI entry.
>>
>>I think that would be much better.
>
>I just wanted to emphasize that I think this would be a good
>idea.

Patches welcome.  :)  Note that there should still be a fallback to the 
screen scraping code in case of a problem with the XML-RPC, to allow people 
to continue using static mirrors of PyPI or imitation PyPIs without needing 
to support XML-RPC.

>   I was just talking to Richard, and he pointed out that the
>current approach is a problem for him, because it means he can't
>evolve the pypi UI without risking breaking setuptools.

What I would suggest is creating a "microformat" for marking up web pages
(Continue reading)

Permalink | Reply | 0 comments |
headers
richardjones | 6 Jul 2006 11:09
Picon

Re: [Distutils] Specification for package indexes?

Phillip J. Eby <pje <at> telecommunity.com> wrote:
> Patches welcome.  :)  Note that there should still be a fallback to the 
> screen scraping code in case of a problem with the XML-RPC, to allow 
> people 
> to continue using static mirrors of PyPI or imitation PyPIs without 
> needing 
> to support XML-RPC.

Why?

     Richard
Permalink | Reply | 4 comments |
headers
Fred Drake | 6 Jul 2006 15:58
Picon

Re: [Distutils] Specification for package indexes?

On 7/6/06, richardjones <at> optusnet.com.au <richardjones <at> optusnet.com.au> wrote:
> Phillip J. Eby <pje <at> telecommunity.com> wrote:
> > Patches welcome.  :)  Note that there should still be a fallback to the
> > screen scraping code in case of a problem with the XML-RPC, to allow
> > people
> > to continue using static mirrors of PyPI or imitation PyPIs without
> > needing
> > to support XML-RPC.
>
> Why?

So we can easily have alternate or additional package repositories
implemented simply as (simple) HTML files and the downloadable
packages.  We want to be able to have an internal repository that
plays the easy_install game without running our own PyPI.

  -Fred

--

-- 
Fred L. Drake, Jr.    <fdrake at gmail.com>
"Every sin is the result of a collaboration." --Lucius Annaeus Seneca
Permalink | Reply | 0 comments |
headers
Phillip J. Eby | 6 Jul 2006 16:03
Gravatar

Re: [Distutils] Specification for package indexes?

At 07:09 PM 7/6/2006 +1000, richardjones <at> optusnet.com.au wrote:
>Phillip J. Eby <pje <at> telecommunity.com> wrote:
> > Patches welcome.  :)  Note that there should still be a fallback to the
> > screen scraping code in case of a problem with the XML-RPC, to allow
> > people
> > to continue using static mirrors of PyPI or imitation PyPIs without
> > needing
> > to support XML-RPC.
>
>Why?

Why not?  ;)

 From easy_install's point of view, PyPI is just a place to find links for 
a given package name.  Preferably links that go directly to downloads, but 
also to pages that might contain downloads.  If someone doesn't want to use 
PyPI as the source of download links, shouldn't they be able to use their 
own, without having to implement an XML-RPC interface?

Actually, the question of "how do I get easy_install to use something other 
than PyPI?" has been becoming somewhat of a FAQ recently.  Well, two people 
have asked about it in the last couple of weeks, anyway.  And it would've 
sucked to have to say "well, first you need an XML-RPC server..."  :)

Nonetheless, there are various aspects of easy_install's behavior and 
performance that could be significantly improved by using XML-RPC, so I 
definitely want it to do that in 0.7.  I'm just wary of removing the 
existing behavior until it's clear that it's unnecessary for it to.
Permalink | Reply | 0 comments |
headers
richardjones | 6 Jul 2006 17:03
Picon

Re: [Catalog-sig] Specification for package indexes?

> Phillip J. Eby <pje <at> telecommunity.com> wrote:
> Why not?  ;)

That was actually what I was afraid the reasoning was ;)

I guess I just go all wobbly in the knees at the thought of having to maintain a "screen scraping" interface.

Funnily enough, Johannes Gisjbers, Andrew Dalke and I were talking about this very issue last night. I
proposed that we detect the user-agent of the setuptools client, and in response send back really
minimalist HTML (no surrounding page template). Probably overkill, but this may have been after we'd had
beer :) 

Could you provide a clear list of all the specific changes you wish for us to make at the Sprint?

> Nonetheless, there are various aspects of easy_install's behavior and 
> performance that could be significantly improved by using XML-RPC, so I 
> definitely want it to do that in 0.7.  I'm just wary of removing the 
> existing behavior until it's clear that it's unnecessary for it to.

Oh - another thing that occurred to me -- does setuptools auto update itself?

     Richard
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG <at> python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Permalink | Reply | 11 comments |
headers
richardjones | 6 Jul 2006 17:39
Picon

PyPI XML-RPC changes

The XML-RPC interface is being reviewed at present, and some quite reasonable changes are being
suggested. These will alter the actual method names, so I'd like to know whether anyone on this list is
already actually *using* the methods as posted to this list. If so, then I'll leave aliases in the code. If
not, I'll just do the rename.

The changes, BTW, are:

package_urls -> release_urls
package_data -> release_data
package_stable_version -> package_stable_release

     Richard
Permalink | Reply | 2 comments |
headers
Jean-Paul Calderone | 6 Jul 2006 17:43

Re: [Distutils] Specification for package indexes?

On Fri, 07 Jul 2006 01:03:33 +1000, richardjones <at> optusnet.com.au wrote:
>> Phillip J. Eby <pje <at> telecommunity.com> wrote:
>> Why not?  ;)
>
>That was actually what I was afraid the reasoning was ;)
>
>I guess I just go all wobbly in the knees at the thought of having to maintain a "screen scraping" interface.
>
>Funnily enough, Johannes Gisjbers, Andrew Dalke and I were talking about this very issue last night. I
proposed that we detect the user-agent of the setuptools client, and in response send back really
minimalist HTML (no surrounding page template). Probably overkill, but this may have been after we'd had
beer :)

Making this explicit actually makes it a good idea.

  http://host/pypiurl?format=simple

It doesn't even have to be html at this point, either.  Return a nicely
structured xml or csv or whatever document.  And then you can completely
omit setuptools hints from the actual markup, since setuptools won't
ever care about that.

Jean-Paul
Permalink | Reply | 0 comments |
headers
Fred Drake | 6 Jul 2006 17:44
Picon

Re: PyPI XML-RPC changes

On 7/6/06, richardjones <at> optusnet.com.au <richardjones <at> optusnet.com.au> wrote:
> The XML-RPC interface is being reviewed at present, and some quite reasonable changes are being
suggested. These will alter the actual method names, so I'd like to know whether anyone on this list is
already actually *using* the methods as posted to this list.

Do calls to the XML -RPC methods get logged?  That would be a good indicator.

  -Fred

--

-- 
Fred L. Drake, Jr.    <fdrake at gmail.com>
"Every sin is the result of a collaboration." --Lucius Annaeus Seneca
Permalink | Reply | 0 comments |
headers
richardjones | 6 Jul 2006 17:52
Picon

Re: [Distutils] Specification for package indexes?

> Jean-Paul Calderone <exarkun <at> divmod.com> wrote:
> Making this explicit actually makes it a good idea.
> 
>   http://host/pypiurl?format=simple
> 
> It doesn't even have to be html at this point, either.  Return a nicely
> structured xml or csv or whatever document.  And then you can completely
> omit setuptools hints from the actual markup, since setuptools won't
> ever care about that.

I've not actually looked into it, but perhaps DOAP is useful here?

     Richard
Permalink | Reply | 0 comments |
headers
richardjones | 6 Jul 2006 17:50
Picon

Re: PyPI XML-RPC changes

> Fred Drake <fdrake <at> gmail.com> wrote:
> Do calls to the XML -RPC methods get logged?  That would be a good 
> indicator.

Yep, they do, and someone here also suggested this approach. There is a chance though that someone has
distributed code that hasn't been exercised yet. Slim, of course.

     Richard
Permalink | Reply | 1 comment |

Gmane