Matt Joiner | 20 Feb 01:34 2012
Picon

Re: channel (synchronous queue)

Your implementation is incomplete.

On Feb 20, 2012 2:01 AM, "Sturla Molden" <sturla-PujZ2ClILZlhl2p70BpVqQ@public.gmane.org> wrote:
Den 19.02.2012 18:44, skrev Guido van Rossum:
I may be taking this out of context, but I have a really hard time understanding what you were trying to say. What does it mean for a complaint to be simple? Did you leave out a word in haste? (I know that happens a lot to me. :-)

Sorry for the rude language. I ment I think  it is a problem that does not belong in the standard library, but perhaps in a cookbook. It is ~20 lines of trivial code with objects already in the standard library. Well, one could say the same thing about a queue too (it's just deque and a lock), but it is very useful and commonly used, so there is a difference.

Sturla




_______________________________________________
Python-ideas mailing list
Python-ideas <at> python.org
http://mail.python.org/mailman/listinfo/python-ideas
_______________________________________________
Python-ideas mailing list
Python-ideas@...
http://mail.python.org/mailman/listinfo/python-ideas
Sturla Molden | 20 Feb 01:40 2012
Picon

Re: channel (synchronous queue)

Den 20.02.2012 00:38, skrev Massimo Di Pierro:
> It would be very useful to have something like these channels 
> built-in. Notice that using OS pipes have the problem of a OS 
> dependent size. send is non-blocking for small data-size but becomes 
> blocking for large data sizes. Using OS mkfifo or multiprocessing 
> Queue is better but the OS limits the number of files open by one 
> program. 

Most MPI implementations use shared memory on localhost. In theory one 
could implement a queue (deque and lock) using a shared memory region (a 
file on /tmp or Windows equivalent). It would be extremely fast and 
could contain any number of "pipes" of arbitrary size.

Sturla
Sturla Molden | 20 Feb 01:43 2012
Picon

Re: channel (synchronous queue)

Den 20.02.2012 01:34, skrev Matt Joiner:
>
> Your implementation is incomplete.
>

It does the basic communication you asked for. I know it is a 
featureless proof-of-concept, why don't you fill in the rest? (I don't 
really care.)

Sturla
Massimo Di Pierro | 20 Feb 01:44 2012
Picon

Re: channel (synchronous queue)

+1

On Feb 19, 2012, at 6:40 PM, Sturla Molden wrote:

> Den 20.02.2012 00:38, skrev Massimo Di Pierro:
>> It would be very useful to have something like these channels built-in. Notice that using OS pipes have
the problem of a OS dependent size. send is non-blocking for small data-size but becomes blocking for
large data sizes. Using OS mkfifo or multiprocessing Queue is better but the OS limits the number of files
open by one program. 
> 
> Most MPI implementations use shared memory on localhost. In theory one could implement a queue (deque and
lock) using a shared memory region (a file on /tmp or Windows equivalent). It would be extremely fast and
could contain any number of "pipes" of arbitrary size.
> 
> Sturla
> _______________________________________________
> Python-ideas mailing list
> Python-ideas@...
> http://mail.python.org/mailman/listinfo/python-ideas
Matt Joiner | 20 Feb 01:57 2012
Picon

Re: channel (synchronous queue)

I've created http://bugs.python.org/issue14059 for the
multiprocessing.Barrier. I suggest a new thread be started to continue
discussion on that.

On Mon, Feb 20, 2012 at 8:44 AM, Massimo Di Pierro
<massimo.dipierro@...> wrote:
> +1
>
> On Feb 19, 2012, at 6:40 PM, Sturla Molden wrote:
>
>> Den 20.02.2012 00:38, skrev Massimo Di Pierro:
>>> It would be very useful to have something like these channels built-in. Notice that using OS pipes have
the problem of a OS dependent size. send is non-blocking for small data-size but becomes blocking for
large data sizes. Using OS mkfifo or multiprocessing Queue is better but the OS limits the number of files
open by one program.
>>
>> Most MPI implementations use shared memory on localhost. In theory one could implement a queue (deque
and lock) using a shared memory region (a file on /tmp or Windows equivalent). It would be extremely fast
and could contain any number of "pipes" of arbitrary size.
>>
>> Sturla
>> _______________________________________________
>> Python-ideas mailing list
>> Python-ideas@...
>> http://mail.python.org/mailman/listinfo/python-ideas
>
> _______________________________________________
> Python-ideas mailing list
> Python-ideas@...
> http://mail.python.org/mailman/listinfo/python-ideas
Matt Joiner | 20 Feb 01:59 2012
Picon

Re: channel (synchronous queue)

I've created http://bugs.python.org/issue14060 for the possibility of
a channel implementation.

On Mon, Feb 20, 2012 at 1:44 AM, Guido van Rossum <guido@...> wrote:
> On Sun, Feb 19, 2012 at 9:36 AM, Sturla Molden <sturla@...> wrote:
>> Den 19.02.2012 18:27, skrev Sturla Molden:
>>
>>> Den 19.02.2012 18:18, skrev Antoine Pitrou:
>>>>
>>>> This begs the question: what does it achieve? You know that the data has
>>>> been "received" on the other side (i.e. get() has been called), but this
>>>> doesn't tell you anything was done with the data, so: why is this an useful
>>>> way to synchronize?
>>>
>>>
>>> I think it achieves nothing, except making deadlocks more likely.
>>
>>
>> Which is to say, I just wanted to prove how ridiculously simple Matt
>> Joiner's complaint about a "channel" was.
>
> I may be taking this out of context, but I have a really hard time
> understanding what you were trying to say. What does it mean for a
> complaint to be simple? Did you leave out a word in haste? (I know
> that happens a lot to me. :-)
>
>> The multiprocessing barrier on the other hand is quite useful. (Though the
>> butterfly method is not the most efficient implementation of a barrier.)
>
> Glad to see some real code. It's probably time to move the code
> samples to the bug tracker where they can be reviewed and have a
> chance of getting incorporated into the next release.
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-ideas mailing list
> Python-ideas@...
> http://mail.python.org/mailman/listinfo/python-ideas
anatoly techtonik | 20 Feb 10:47 2012
Picon

sys.path is a hack - bringing it back under control

Hi,


I often find this in my scripts/projects, that I run directly from checkout:

DEVPATH = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, DEVPATH)

This seems like a hack to me, because the process of sys.path modification is completely out of control for Python application developer, which means it is easy to break an application and get lost. I don't remember the exact user story for that bad association with sys.path (perhaps Django issue #1908), but something makes me feel that I am not alone: 

What I'd like to propose is some control/info over what modified sys.path. The simplest case:
1. make sys.path a list of pairs   (path, file-that-added-the-path)
2. make sys.path read-only
3. add sys.path.add() method for modification
4. logger for sys.path.add() events (or recipe how to implement it in documentation)

This will help a lot.

Limiting sys.path may cause a loss of some functionality if you need to remove some or replace it completely, but I don't know where the ability to reset() sys.path can be useful.
--
anatoly t.
_______________________________________________
Python-ideas mailing list
Python-ideas@...
http://mail.python.org/mailman/listinfo/python-ideas
Simon Sapin | 20 Feb 11:06 2012
Picon

Re: sys.path is a hack - bringing it back under control

Le 20/02/2012 10:47, anatoly techtonik a écrit :
>
> I often find this in my scripts/projects, that I run directly from
> checkout:
>
> DEVPATH =os.path.dirname(os.path.abspath(__file__))
> sys.path.insert(0,DEVPATH)
>

Hi,

You shouldn’t have to do that if you’re running 'python something.py'

> As initialized upon program startup, the first item of this list,
> path[0], is the directory containing the script that was used to
> invoke the Python interpreter. If the script directory is not
> available (e.g. if the interpreter is invoked interactively or if the
> script is read from standard input), path[0] is the empty string,
> which directs Python to search modules in the current directory
> first.

http://docs.python.org/py3k/library/sys.html#sys.path

The trick is to place the script in the directory that you want in the 
path, ie. next to top-level packages. But from your code above this 
seems to be the case already...

Regards,
--

-- 
Simon Sapin
anatoly techtonik | 20 Feb 11:31 2012
Picon

Re: sys.path is a hack - bringing it back under control

On Mon, Feb 20, 2012 at 1:06 PM, Simon Sapin <simon.sapin <at> kozea.fr> wrote:
I often find this in my scripts/projects, that I run directly from
checkout:

DEVPATH =os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0,DEVPATH)


You shouldn’t have to do that if you’re running 'python something.py'

But I did for some reason, and right now I can't even say if it was Windows, Linux, FreeBSD, PyPy, IPython, gdb or debugging from IDE.

As initialized upon program startup, the first item of this list,
path[0], is the directory containing the script that was used to
invoke the Python interpreter. If the script directory is not
available (e.g. if the interpreter is invoked interactively or if the
script is read from standard input), path[0] is the empty string,
which directs Python to search modules in the current directory
first.

http://docs.python.org/py3k/library/sys.html#sys.path

The trick is to place the script in the directory that you want in the path, ie. next to top-level packages. But from your code above this seems to be the case already...

s/trick/hack/ and it will be just what I am saying. Not many Python projects use this structure.
-- 
anatoly t. 
_______________________________________________
Python-ideas mailing list
Python-ideas@...
http://mail.python.org/mailman/listinfo/python-ideas
Nick Coghlan | 20 Feb 11:38 2012
Picon

Re: sys.path is a hack - bringing it back under control

On Mon, Feb 20, 2012 at 7:47 PM, anatoly techtonik
<techtonik@...> wrote:
> Hi,
>
> I often find this in my scripts/projects, that I run directly from checkout:
>
> DEVPATH = os.path.dirname(os.path.abspath(__file__))
> sys.path.insert(0, DEVPATH)

PEP 395 describes my current plan to fix sys.path initialisation
(however, I can't yet promise that it will make it into 3.3, since it
doesn't even have a reference implementation yet, and I have several
other things I want to get done first).

Cheers,
Nick.

--

-- 
Nick Coghlan   |   ncoghlan@...   |   Brisbane, Australia

Gmane