Re: symlinks in chroot

> Dnia poniedziałek, 30 kwietnia 2012 14:19:46 Roberto De Ioris pisze:
>
>> I have not tested your setup, but you are using namespaces, so all of
>> the
>> mounted fs will disapeear in the new namespace.
>
> That seems logical (now), uWSGI namespace is not a simple chroot(), it
> uses
> linux namespace support developed mostly for paravirtualization (AFAIK) so
> I
> shouldn't expect that if chroot works than my app will also work.
>
>> I do not know if it is a viable solution, but can't you mount the
>> moosefs
>> from uWSGI itself using
>>
>> --exec-in-jail "fusemount ...."
>>
>> ?
>
> I tried that but I don't get any information in the logs that uWSGI
> executed
> my command and the filesystem is not mounted. strace doesn't show anything
> and
> I tried adding --exec-in-jail="date > /tmp/xxx" but nothing happens.
> Does --exec-in-jail works for anyone?
>
>

Re: Can uWSGI support streaming service?

> It works using uWSGI http router. I'm using apache2.2.21, it looks like
> apache buffers uWSGI's output but not mod_wsgi's... Is there any way to
> disable apache's buffering?

You have to use the mod_uwsgi from the tip:

http://projects.unbit.it/uwsgi/browser/apache2/mod_uwsgi.c

It will stream content only when in no-cgi mode

If you plan big deployment on apache i suggest you to give a look at the
upcoming mod_proxy_uwsgi that will be fully integrated with apache2.x and
its "new" (>=2.2) load balancing platform.

--

-- 
Roberto De Ioris
http://unbit.it

Re: uWSGI AMQP messaging system

_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi

Re: symlinks in chroot

Dnia wtorek, 1 maja 2012 08:14:05 Roberto De Ioris pisze:
> Sorry, exec-in-jail run before fs-rebuilt so whatever you do will be
> overwritten.
> Use --exec-as-root, it will be triggered before privileges drop in the jail.
> 
> Remember that you have no stdin, and stdout/stderr are mapped to uWSGI log.

It works with exec-in-jail = mfsmount <mount options>.
The only issue remaining is that mfsmount is a client daemon for MooseFS mount 
and it runs all the time in the background. During vassal shutdown/reload 
uWSGI hangs waiting for mfsmount to die (if I kill mfsmount than it reloads, 
but only then). uWSGI doesn't seem to send any signal to it, I guess that it 
doesn't expect exec-as-root command to be long living process and all it does 
is to run it and wait untill it is terminated.
Is there a way to execute something also at exit time so that I can umount my 
MooseFs (mfsmount would die)?

Maybe instead of (possibly) growing number if exec-* options You could add 
exec slots  into code (moments in app lifetime when one can run some command, 
like at startup, or during reload), so there would be only one options:

exec = <slot>:<command>

for example:

exec = post-jail:<something>
exec = as-root-startup:<at startup command>
exec = as-root-shutdown:<at shutdown command>

Instead of long list of --exec-* commands there would be one, but with a list 
of possible slots to use.

Łukasz Mierzwa
_______________________________________________
uWSGI mailing list
uWSGI <at> lists.unbit.it
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi

Spooler stuck on single job

Hello,

I've noticed that with recent version of uWSGI, the spooler does not
treat jobs as it used to. Before, if 10 jobs were in the queue and job
1 failed, the spooler would proceed to try job 2 and then job 3, etc.
But recently, I've noticed that if job1 fails, it will continue to try
and execute job1 completely ignoring jobs 2-10 until job1 is
successfully processed.  Here is my config:

uwsgi:
  auto-procname: true
  chdir: /opt/webapps/example.com
  cpu-affinity: 1
  enable-threads: true
  env: DJANGO_SETTINGS_MODULE=example.settings
  harakiri: 7200
  master: true
  max-requests: 500
  memory-report: true
  module: django.core.handlers.wsgi:WSGIHandler()
  pidfile: /opt/webapps/example.com/pid/uwsgi.pid
  processes: 16
  pythonpath: example
  socket: /opt/webapps/example.com/sock/uwsgi.sock
  spooler: /opt/webapps/example.com/spooler
  spooler-chdir: /opt/webapps/example.com
  spooler-import: custom.uwsgi_utils
  spooler-max-tasks: 20
  vacuum: true
  virtualenv: /opt/webapps/example.com

Also, on a slightly related note.. I've noticed that the spooler
workers will occasionally take up VAST amounts of memory (upwards of
256MB) with no jobs in the queue.  This is why I've set the
spooler-max-tasks option, but it's even happened to me once with that
option set.  I don't currently have debug logs from when this has
happened but I'll post them when it happens to me again.

BTW, uWSGI is still the most amazing piece of software EVER.  Thank
you for all the hard work :)

Thanks,
Ryan-

Re: Spooler stuck on single job

Sorry, I should have noted that I'm using uWSGI version 1.2

Ryan-

On Tue, May 1, 2012 at 11:55 PM, Ryan Showalter <ryanshow <at> gmail.com> wrote:
> Hello,
>
> I've noticed that with recent version of uWSGI, the spooler does not
> treat jobs as it used to. Before, if 10 jobs were in the queue and job
> 1 failed, the spooler would proceed to try job 2 and then job 3, etc.
> But recently, I've noticed that if job1 fails, it will continue to try
> and execute job1 completely ignoring jobs 2-10 until job1 is
> successfully processed.  Here is my config:
>
> uwsgi:
>  auto-procname: true
>  chdir: /opt/webapps/example.com
>  cpu-affinity: 1
>  enable-threads: true
>  env: DJANGO_SETTINGS_MODULE=example.settings
>  harakiri: 7200
>  master: true
>  max-requests: 500
>  memory-report: true
>  module: django.core.handlers.wsgi:WSGIHandler()
>  pidfile: /opt/webapps/example.com/pid/uwsgi.pid
>  processes: 16
>  pythonpath: example
>  socket: /opt/webapps/example.com/sock/uwsgi.sock
>  spooler: /opt/webapps/example.com/spooler
>  spooler-chdir: /opt/webapps/example.com
>  spooler-import: custom.uwsgi_utils
>  spooler-max-tasks: 20
>  vacuum: true
>  virtualenv: /opt/webapps/example.com
>
>
> Also, on a slightly related note.. I've noticed that the spooler
> workers will occasionally take up VAST amounts of memory (upwards of
> 256MB) with no jobs in the queue.  This is why I've set the
> spooler-max-tasks option, but it's even happened to me once with that
> option set.  I don't currently have debug logs from when this has
> happened but I'll post them when it happens to me again.
>
> BTW, uWSGI is still the most amazing piece of software EVER.  Thank
> you for all the hard work :)
>
> Thanks,
> Ryan-
_______________________________________________
uWSGI mailing list
uWSGI <at> lists.unbit.it
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi

Re: symlinks in chroot

Dnia środa, 2 maja 2012 00:43:06 Łukasz Mierzwa pisze:
> Dnia wtorek, 1 maja 2012 08:14:05 Roberto De Ioris pisze:
> > Sorry, exec-in-jail run before fs-rebuilt so whatever you do will be
> > overwritten.
> > Use --exec-as-root, it will be triggered before privileges drop in the
> > jail.
> > 
> > Remember that you have no stdin, and stdout/stderr are mapped to uWSGI
> > log.
> 
> It works with exec-in-jail = mfsmount <mount options>.
> The only issue remaining is that mfsmount is a client daemon for MooseFS
> mount and it runs all the time in the background. During vassal
> shutdown/reload uWSGI hangs waiting for mfsmount to die (if I kill mfsmount
> than it reloads, but only then). uWSGI doesn't seem to send any signal to
> it, I guess that it doesn't expect exec-as-root command to be long living
> process and all it does is to run it and wait untill it is terminated.
> Is there a way to execute something also at exit time so that I can umount
> my MooseFs (mfsmount would die)?

I've managed to get around that by:

# add fuse perms for www-data user
chgrp www-data /chroot/dev/fuse

# allow non-root users to mount fuse filesystems
echo user_allow_other > /chroot/etc/fuse.conf

# create MooseFS mount dir
mkdir /chroot/mfs
chown www-data:www-data /chroot/mfs

in my vassal config I have now:

exec-as-user = mfsmount /mfs -o <mfs options>
exec-as-user-atexit = fusermount -u /mfs

Since there is now uWSGI option to run any command at worker exit, I had to 
write very simple patch that adds this. There is no way to run that command as 
root since this is done after privileges drop so the command is being run as 
user (unless someone runs workers as root user).

Does this patch qualifies for merging into uWSGI?

Łukasz Mierzwa

_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi

Re: Spooler stuck on single job

Il giorno 02/mag/2012, alle ore 08:55, Ryan Showalter ha scritto:

> Hello,
> 
> I've noticed that with recent version of uWSGI, the spooler does not
> treat jobs as it used to. Before, if 10 jobs were in the queue and job
> 1 failed, the spooler would proceed to try job 2 and then job 3, etc.
> But recently, I've noticed that if job1 fails, it will continue to try
> and execute job1 completely ignoring jobs 2-10 until job1 is
> successfully processed.  Here is my config:

This is pretty strange, as the spooler is not supposed to work in that way.

How the job is failing ? I mean, it is returning a SPOOL_RETRY or an Exception ?

> 
> 
> Also, on a slightly related note.. I've noticed that the spooler
> workers will occasionally take up VAST amounts of memory (upwards of
> 256MB) with no jobs in the queue.  This is why I've set the
> spooler-max-tasks option, but it's even happened to me once with that
> option set.  I don't currently have debug logs from when this has
> happened but I'll post them when it happens to me again.
> 

The spooler can only destroy objects created by him, if the code you are running cannot be freed it will leak.
I suggest you to use 

sys.gettotalrefcount()

before exiting the function to see if your callable is leaking

--
Roberto De Ioris
http://unbit.it
JID: roberto@...

Re: symlinks in chroot

Dnia środa, 2 maja 2012 11:41:04 Łukasz Mierzwa pisze:
> Since there is now uWSGI option to run any command at worker exit, I had to
> write very simple patch that adds this. There is no way to run that command
> as root since this is done after privileges drop so the command is being
> run as user (unless someone runs workers as root user).
> 
> Does this patch qualifies for merging into uWSGI?

It turns out that I need to make another patch, with this one exec-as-user-
atexit would be executed at each worker shutdown, and I need to execute it 
only at master exit.

Łukasz Mierzwa

_______________________________________________
uWSGI mailing list
uWSGI <at> lists.unbit.it
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi

Re: Spooler stuck on single job

>> I've noticed that with recent version of uWSGI, the spooler does not
>> treat jobs as it used to. Before, if 10 jobs were in the queue and job
>> 1 failed, the spooler would proceed to try job 2 and then job 3, etc.
>> But recently, I've noticed that if job1 fails, it will continue to try
>> and execute job1 completely ignoring jobs 2-10 until job1 is
>> successfully processed.  Here is my config:
>
>
> This is pretty strange, as the spooler is not supposed to work in that way.
>
> How the job is failing ? I mean, it is returning a SPOOL_RETRY or an Exception ?

I'm noticing this issue when the spooler is failing with an Exception.
 I have not yet tried a test case with SPOOL_RETRY.

> The spooler can only destroy objects created by him, if the code you are running cannot be freed it will leak.
> I suggest you to use
>
> sys.gettotalrefcount()
>
> before exiting the function to see if your callable is leaking

Sounds good, i'll implement this and get back to you with the results.
_______________________________________________
uWSGI mailing list
uWSGI <at> lists.unbit.it
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi