headers
Phillip J. Eby | 1 May 2007 02:29
Gravatar

PEP 0365: Adding the pkg_resources module

I wanted to get this in before the Py3K PEP deadline, since this is a 
Python 2.6 PEP that would presumably impact 3.x as well.  Feedback welcome.

PEP: 365
Title: Adding the pkg_resources module
Version: $Revision: 55032 $
Last-Modified: $Date: 2007-04-30 20:24:48 -0400 (Mon, 30 Apr 2007) $
Author: Phillip J. Eby <pje <at> telecommunity.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 30-Apr-2007
Post-History: 30-Apr-2007

Abstract
========

This PEP proposes adding an enhanced version of the ``pkg_resources``
module to the standard library.

``pkg_resources`` is a module used to find and manage Python
package/version dependencies and access bundled files and resources,
including those inside of zipped ``.egg`` files.  Currently,
``pkg_resources`` is only available through installing the entire
``setuptools`` distribution, but it does not depend on any other part
of setuptools; in effect, it comprises the entire runtime support
library for Python Eggs, and is independently useful.

In addition, with one feature addition, this module could support
easy bootstrap installation of several Python package management
(Continue reading)

Permalink | Reply | 10 comments |
headers
Andrew Bennetts | 1 May 2007 02:50
Favicon

Re: os.rename on windows

Raghuram Devarakonda wrote:
> Hi,
> 
> I have submitted a patch (http://www.python.org/sf/1704547) that
> allows os.rename to replace the destination file if it exists, on
> windows. As part of discussion in the tracker, Martin suggested that
> python-dev should discuss the change.

Does MOVEFILE_REPLACE_EXISTING mean the rename over an existing file is actually
atomic?  I cannot find any MSDN docs that say so (and I've seen some that
suggest to me that it probably isn't).

If it's not atomic, then this doesn't offer any advantage over shutil.move that
I can see (and in fact would damage the usefulness of os.rename, which is
currently atomic on all platforms AFAIK, even though it cannot succeed all the
time).

If MOVEFILE_REPLACE_EXISTING miraculously turns out to be atomic, then my
opinion is:

  * this feature would be very useful, and I would like a cross-platform way to
    do this in Python.
  * this feature should not be called "os.rename", which for years has done
    something else on some platforms, and so changing it will invite unnecessary
    breakage.  A new function would be better, or perhaps an optional flag.  I
    propose "os.atomic_rename".

Also, I assume this cannot replace files that are in use?

> Currently, os.rename() on windows uses the API MoveFile() which fails
(Continue reading)

Permalink | Reply | 3 comments |
headers
Greg Ewing | 1 May 2007 04:23
Picon
Picon
Favicon

Re: (no subject)

JOSHUA ABRAHAM wrote:
> I was hoping you guys would consider creating  function in os.path or 
> otherwise that would find the full path of a file when given only it's base 
> name and nothing else.I have been made to understand that this is not 
> currently possible.

Does os.path.abspath() do what you want?

If not, what exactly *do* you want?

--
Greg
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
Permalink | Reply | 1 comment |
headers
Raghuram Devarakonda | 1 May 2007 06:14
Picon

Re: os.rename on windows

On 4/30/07, Andrew Bennetts <andrew-pythondev <at> puzzling.org> wrote:

> Does MOVEFILE_REPLACE_EXISTING mean the rename over an existing file is actually
> atomic?  I cannot find any MSDN docs that say so (and I've seen some that
> suggest to me that it probably isn't).

Even though MSDN docs do not say it explicitly, I found some
discussions claiming that MOVEFILE_REPLACE_EXISTING is atomic.
However, after seeing your comment, I did a more thorough search and I
too found some references claiming otherwise. As a last resort, I
checked cygwin documentation which claims that it's rename() is
POSIX.1 compliant. If I am not mistaken, POSIX.1 does require
atomicity so I am curious how rename() is implemented there. I checked
out the sources and I will try to find more about their
implementation.

I completely agree that without positive proof of atomicity, there is
no point in making this code change.

> Also, I assume this cannot replace files that are in use?

A simple test shows that it can indeed replace files that are open.

Thanks,
Raghu
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
(Continue reading)

Permalink | Reply | 2 comments |
headers
Mike Klaas | 1 May 2007 06:49
Picon

Re: (no subject)

On 4/30/07, Greg Ewing <greg.ewing <at> canterbury.ac.nz> wrote:
> JOSHUA ABRAHAM wrote:
> > I was hoping you guys would consider creating  function in os.path or
> > otherwise that would find the full path of a file when given only it's base
> > name and nothing else.I have been made to understand that this is not
> > currently possible.
>
> Does os.path.abspath() do what you want?
>
> If not, what exactly *do* you want?

probably:

def find_in_path(filename):
     for path in os.environ['PATH'].split(os.pathsep):
           if os.path.exists(filename):
                    return os.path.abspath(os.path.join(path, filename))

-Mike
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
Permalink | Reply | 0 comments |
headers
Neal Norwitz | 1 May 2007 08:38
Picon

head crashing (was: Fwd: [Python-checkins] buildbot warnings in x86 mvlgcc trunk)

This is the third time I've seen a crash on 2 different machines.
This is the first time I noticed this unexplained crash:

http://python.org/dev/buildbot/all/amd64%20gentoo%20trunk/builds/1983/step-test/0

That was at r54982.

I tried to reproduce this: with a non-debug build, with a debug build,
with valgrind with both types of build.  I could never reproduce it.
Valgrind did not report any errors either.

Here is the third failure:

http://python.org/dev/buildbot/all/amd64%20gentoo%20trunk/builds/1986/step-test/0

The failure below prints:
python: Objects/obmalloc.c:746: PyObject_Malloc: Assertion `bp !=
((void *)0)' failed.

which probably doesn't really help since the corruption has already
occurred.  See http://python.org/dev/buildbot/all/x86%20mvlgcc%20trunk/builds/497/step-test/0

Anyone have ideas what might have caused this?

n
--

---------- Forwarded message ----------
From: buildbot <at> python.org <buildbot <at> python.org>
Date: Apr 30, 2007 11:17 PM
(Continue reading)

Permalink | Reply | 9 comments |
headers
Neal Norwitz | 1 May 2007 09:27
Picon

Re: head crashing (was: Fwd: [Python-checkins] buildbot warnings in x86 mvlgcc trunk)

In rev 54982 (the first time this crash was seen), I see something
which might create a problem.  In python/trunk/Modules/posixmodule.c
(near line 6300):

+       PyMem_FREE(mode);
       Py_END_ALLOW_THREADS

Can you call PyMem_FREE() without the GIL held?  I couldn't find it
documented either way.

Of the 3 failures I know of, below is the intersection of the tests
that were run prior to crashing:

set(['test_threadedtempfile', 'test_cgi', 'test_dircache', 'test_set',
'test_binascii', 'test_imp', 'test_multibytecodec', 'test_weakref',
'test_ftplib', 'test_posixpath', 'test_xmlrpc', 'test_urllibnet',
'test_old_mailbox', 'test_distutils', 'test_site', 'test_runpy',
'test_fork1', 'test_traceback'])

n
--

On 4/30/07, Neal Norwitz <nnorwitz <at> gmail.com> wrote:
> This is the third time I've seen a crash on 2 different machines.
> This is the first time I noticed this unexplained crash:
>
> http://python.org/dev/buildbot/all/amd64%20gentoo%20trunk/builds/1983/step-test/0
>
> That was at r54982.
>
(Continue reading)

Permalink | Reply | 8 comments |
headers
Alexey Borzenkov | 1 May 2007 09:36
Picon
Gravatar

Re: Python 2.5.1

On 5/1/07, "Martin v. Löwis" <martin <at> v.loewis.de> wrote:
> > After doing some research I found that it seems to be impossible to
> > use CreateFile for a file that doesn't have SHARE_READ. I played with
> > different combinations and with FLAG_BACKUP_SEMANTICS and nothing
> > helped. However on Windows there's still a possibility to read
> > attributes: use FindFirstFile. _WIN32_FIND_DATA structure seems to
> > have all the necessary fields (attributes, file times, size and
> > full/short filename), and FindFirstFile doesn't care about sharing at
> > all...
> So what about GetFileAttributesEx? What are the conditions under which
> I can successfully invoke it?

Seems to have the same problems as with CreateFile(...):

// 1.cc
#include <windows.h>
#include <stdio.h>

int main(int argc, char** argv)
{
	WIN32_FILE_ATTRIBUTE_DATA data;
	if(!GetFileAttributesEx("pagefile.sys", GetFileExInfoStandard, (LPVOID)&data))
	{
		char buffer[1024];
		FormatMessage(FORMAT_MESSAGE_FROM_SYSTEM, NULL, GetLastError(), 0,
buffer, 1024, NULL);
		printf("Error %d: %s\n", GetLastError(), buffer);
		return 1;
	}
	printf("Success\n");
(Continue reading)

Permalink | Reply | 6 comments |
headers
Scott Dial | 1 May 2007 09:33
Favicon

Re: os.rename on windows

Raghuram Devarakonda wrote:
> As a last resort, I
> checked cygwin documentation which claims that it's rename() is
> POSIX.1 compliant. If I am not mistaken, POSIX.1 does require
> atomicity so I am curious how rename() is implemented there.

The cygwin implementation of rename goes like this:

1) Try to use MoveFile
2) Try to use MoveFileEx(..., MOVEFILE_REPLACE_EXISTING)
3) Try to unlink destination, then try to use MoveFile

And as you say, Cygwin claims it meets POSIX.1. And, POSIX.1 says, "If 
newpath already exists it will be atomically replaced (subject to
a few conditions; see ERRORS below), so that there is no point at which 
another process attempting to access newpath will find it missing." 
Clearly, unliking and then calling MoveFile is not atomic. So, cygwin is 
not being honest here because in these less frequent cases, the rename 
will not be atomic.

Also note, MVCRT only tries step 1 of cygwin's version. Which I believe 
also suggests that it's the only version that is atomic.

-Scott

--

-- 
Scott Dial
scott <at> scottdial.com
scodial <at> cs.indiana.edu
_______________________________________________
(Continue reading)

Permalink | Reply | 1 comment |
headers
Martin v. Löwis | 1 May 2007 10:13
Picon
Gravatar

Re: Python 2.5.1

Alexey Borzenkov schrieb:
> On 5/1/07, "Martin v. Löwis" <martin <at> v.loewis.de> wrote:
>> > After doing some research I found that it seems to be impossible to
>> > use CreateFile for a file that doesn't have SHARE_READ.
>> So what about GetFileAttributesEx? What are the conditions under which
>> I can successfully invoke it?
> 
> Seems to have the same problems as with CreateFile(...):

That code only tests it for pagefile.sys. My question was about open
handles in general. Both Calvin Spealman and I found that you cannot
reproduce the problem when you, in Python 2.5.0, open a file, and then
try to os.stat() it - even though, in Python 2.5.0, os.stat() will
perform GetFileAttributesEx. So even though we opened the file with
not passing any sharing flags, we could still do GetFileAttributesEx
on it.

I now studied the CRT sources, and it seems that if you use a regular
open() call, the CRT will pass FILE_SHARE_READ | FILE_SHARE_WRITE to
CreateFile. You would have to use _sopen in the CRT to create any
kind of sharing conflict, and that isn't exposed in Python.

So I guess we need continue using pagefile.sys as a test case.

Regards,
Martin
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
(Continue reading)

Permalink | Reply | 5 comments |

Gmane