Valentin V. Bartenev | 28 Sep 19:42 2011
Picon

Reading from a regular file in async mode

Hallo everybody.

In my asynchronous application (besides other tasks) I have to read in memory 
and process a ~3-7 Mb files from disk. After some research I haven't found any 
proper way to do it without blocking.

I can set O_NONBLOCK flag and use pooling, but it seems that I can't 
async_sleep less then a second, which is too long delay for my task.

Looks like, there remains one thing to do: detach a file reading task from 
async application into a special separate uwsgi service, where the read will 
be blocking and call it through async_send_message.

Am I right or there is a more convenient way for such tasks?

p.s. I use Linux system.

wbr, Valentin V. Bartenev
Roberto De Ioris | 28 Sep 19:43 2011
Picon

Re: Reading from a regular file in async mode


> Hallo everybody.
>
> In my asynchronous application (besides other tasks) I have to read in
> memory
> and process a ~3-7 Mb files from disk. After some research I haven't found
> any
> proper way to do it without blocking.
>
> I can set O_NONBLOCK flag and use pooling, but it seems that I can't
> async_sleep less then a second, which is too long delay for my task.
>
> Looks like, there remains one thing to do: detach a file reading task from
> async application into a special separate uwsgi service, where the read
> will
> be blocking and call it through async_send_message.
>
> Am I right or there is a more convenient way for such tasks?
>
> p.s. I use Linux system.
>
> wbr, Valentin V. Bartenev
> _______________________________________________
>

Hi, nginx moved to async_io some time ago as reading big files could hurt
too much. I will probably add something very similar in the near future.

In the mean time, while not simply call uwsgi.suspend() continuosly until
the file is fully read ?
(Continue reading)

Roberto De Ioris | 28 Sep 19:44 2011
Picon

Re: Reading from a regular file in async mode


> Hallo everybody.
>
> In my asynchronous application (besides other tasks) I have to read in
> memory
> and process a ~3-7 Mb files from disk. After some research I haven't found
> any
> proper way to do it without blocking.
>
> I can set O_NONBLOCK flag and use pooling, but it seems that I can't
> async_sleep less then a second, which is too long delay for my task.
>
> Looks like, there remains one thing to do: detach a file reading task from
> async application into a special separate uwsgi service, where the read
> will
> be blocking and call it through async_send_message.
>
> Am I right or there is a more convenient way for such tasks?
>
> p.s. I use Linux system.
>
> wbr, Valentin V. Bartenev
> _______________________________________________
>

Hi, nginx moved to async_io some time ago as reading big files could hurt
too much. I will probably add something very similar in the near future.

In the mean time, why not simply call uwsgi.suspend() continuosly until
the file is fully read ?
(Continue reading)

Valentin V. Bartenev | 28 Sep 20:10 2011
Picon

Re: Reading from a regular file in async mode

On Wednesday 28 September 2011 21:44:02 Roberto De Ioris wrote:
> Hi, nginx moved to async_io some time ago as reading big files could hurt
> too much. I will probably add something very similar in the near future.

It's definitely a great news. Thanks.

> In the mean time, why not simply call uwsgi.suspend() continuosly until
> the file is fully read ?

Is it not too cpu-expensive? It looks like "while True" loop, especially when 
my async app concurrent load is low, and there is no other task to switch.

wbr, Valentin V. Bartenev
Valentin V. Bartenev | 28 Sep 22:53 2011
Picon

Re: Reading from a regular file in async mode

On Wednesday 28 September 2011 22:10:16 Valentin V. Bartenev wrote:
> > In the mean time, why not simply call uwsgi.suspend() continuosly until
> > the file is fully read ?
> 
> Is it not too cpu-expensive? It looks like "while True" loop, especially
> when my async app concurrent load is low, and there is no other task to
> switch.

Whoops, never mind. I was under the delusion that I can set O_NONBLOCK flag on 
a regular file descriptor in Python and it will not block. Actually, it 
doesn't have any effect.

wbr, Valentin V. Bartenev

Gmane