xin | 2 Feb 04:06 2016

zerg mode and emperor problem

hello:

         recently  I try to use uwsgi zerg mode to deploy my project with real zero-downtime reload, but it was not progressing smoothly
         
         (1)
         firstly with no emperor,  start the main instance with zerg-server, and then attach a zerg instance,  random pause any instance, another instance can work 
well

        main instance config
       [uwsgi]
      socket = 127.0.0.1:9998
      master = true
      processes = 1
      gevent = 100
      enable-threads = true
      lazy-apps = true
      zerg-server = /tmp/zergtest
     wsgi-file = app.py
    
     zerg instance config
     [uwsgi]
      master = true
      processes = 1
      gevent = 100
      enable-threads = true
      lazy-apps = true
      zerg = /tmp/zergtest
     wsgi-file = app.py
      
      I list all options which may effect the uwsgi action, for example gevent & lazy-apps.  zerg-server is ok, but change it to zergpool, project can not work.  when I pause the main instance,  request was also be suspended until I resume the main instance,  but  as I expected,  zerg instance should handle the request. so which mistake I made ?  zerg-server do not equal to zergpool ?

     (2)  I ignore the zergpool problem and try to use zerg-server to finish zero-downtime reload.

        main instance config

     zerg-server = /tmp/zergtest
     wsgi-file = app.py
     master-fifo = /tmp/new.fifo
     master-fifo = /tmp/running.fifo
     hook-accepting1-once = writefifo:/tmp/new.fifo 1

      and zerg instance config

     master-fifo = /tmp/new.fifo
     master-fifo = /tmp/running.fifo
     if-exists = /tmp/running.fifo
        hook-accepting1-once = writefifo:/tmp/running.fifo q
     endif =
    hook-accepting1-once = writefifo:/tmp/new.fifo 1

     my purpose is simple.  when I reload project,  spawn a zerg instance and graceful shutdown the first instance.  all  reload repeat this procedure.  but zerg-server also shutdown obviously.
     I try to start a pure zerg-server without any app load,  and attach zerg-instance to it. when I reload,  use a new zerg-instance and shutdown an old zerg-instance. but uwsgi report no app load error when handle request.  main instance still spawn a worker and handle request. 
     Maybe I can pause the main instance worker, and attach zerg-instance continuous,  one zerg replace another,  but I suspect it is the best solution. so 
any best practice for zerg zero-downtime reload ?

      In doc https://uwsgi-docs.readthedocs.org/en/latest/articles/TheArtOfGracefulReloading.html,  exists the same problem I think. If I shutdown the sleeping instance ( hook-accepting1-once = writefifo:/var/run/sleeping.fifo Q), the instance with zerg server will quit. BTW,  I want only one instance exists at the same time.

     (3)  

       If use emperor to manage instance,  if I spawn a zerg instance and graceful shutdown the current instance,  emperor always try to respawn the main instance,  although the zerg instance indeed exists. so how to use emperor to finish zero-downtime reload with zerg mode ?


best regargs
        



 

_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Daniel Nicoletti | 1 Feb 14:03 2016
Picon
Gravatar

Data in post-buffering mode

Hi,

I'm getting a crash on post buffering mode, this used to work, so I'm
not sure it's
related to some version o uwsgi, or if I am doing something wrong.

basically my code calls uwsgi_request_body_read()
to do the multipart parsing, after I know the body part offsets I call
uwsgi_request_body_seek(0)
after that the user will likely want to save one or all the parts to a file
so another seek is set to point to the beggining of the
part and
uwsgi_request_body_read() is called.
Now the function returns fine, as it has read data, but
accessing the buffer for debugging or for calling
memcpy crashes as it had already freed that buffer.

I know this is the behavior of when not using post-buffering
so I just fill a buffer to enable the same behavior.

What could I be doing wrong?

the code:
https://github.com/cutelyst/cutelyst/blob/master/uwsgiEngine/bodyuwsgi.cpp#L58

thanks

--

-- 
Daniel Nicoletti

KDE Developer - http://dantti.wordpress.com
Bastian Venthur | 29 Jan 11:32 2016
Picon
Gravatar

python, websocket_recv with timeout using select

Hello,

I'm trying to emulate a websocket_recv with a timeout, and I wonder if
it is safe to use select. I don't use any asyncio stuff. The example
below seems to work, but are there any side effects I should consider?
Could I also use uwsgi.websocket_recv (i.e. w/o _nb) I this case and
have a guarantee that it will not block?

Thanks,

Bastian

#!/usr/bin/envpython

import logging
import select

import uwsgi

logger = logging.getLogger()
logging.basicConfig(level=logging.NOTSET)

def app(env, sr):

    uwsgi.websocket_handshake()
    logger.debug('Initialized WebSocket connection.')

    websocket_fd = uwsgi.connection_fd()

    while True:

        readables, _, _ = select.select([websocket_fd], [], [], 3)
        if not readables:
            # timeout
            pass
        for fd in readables:
            if fd == websocket_fd:
                try:
                    msg = uwsgi.websocket_recv_nb()
                except IOError:
                    # connection was closed
                    logger.debug('Connection was closed by client')
                    return []
                logger.debug('Got message: {0}'.format(msg))

    return []

--

-- 
Dr. Bastian Venthur                                  http://venthur.de
Debian Developer                                 venthur at debian org
Braňo Žarnovičan | 28 Jan 20:54 2016
Picon
Gravatar

reload master by forking, without Zerg

Hi,

I'm attempting to gracefully deploy code update on Django app by
forking vassal's master process.

We have long-running http sessions, so "r" reload was not graceful.
Due to memory consumption, we avoid lazy-apps. So, we use pre-forking
mode, which rules out pure worker reload ("w" and "c"). What I'm
trying next is fork reload "f" without Zerg. Fortunately, we don't
have any issues with conflicting pid/log files. Thou, I'm hitting
something else..

Here is my Vassal config (inspired by "The Art of Graceful Reloading"):
[uwsgi]
master-fifo     = /home/foo/var/uwsgi/%n.new.fifo
master-fifo     = /home/foo/var/uwsgi/%n.master.fifo
master          = true
lazy-apps       = false
enable-threads  = true
thunder-lock    = true
processes       = 4
socket          = /home/foo/var/uwsgi/%n.sock
chmod-socket    = 666
vacuum          = true
worker-reload-mercy = 600
if-exists = /home/foo/var/uwsgi/%n.master.fifo
  hook-accepting1-once = writefifo:/home/foo/var/uwsgi/%n.master.fifo q
endif =
hook-accepting1-once = writefifo:/home/foo/var/uwsgi/%n.new.fifo 1

Emperor (running from systemd):
uwsgi --emperor %h/.config/uwsgi/vassal

After triggering master fork..
echo f > /home/foo/var/uwsgi/helloworld.master.fifo

..this is what happens:
* new master is forked and app loaded
* new workers are forked
* old master is triggered to shutdown
* then this appears in log - PROBLEM
uwsgi_master_manage_emperor()/read(): Resource temporarily unavailable
[core/emperor.c line 2424]
lost connection with my emperor !!!
SIGINT/SIGQUIT received...killing workers...
* new master will kill its workers and shutdown itself
* emperor forks another master (3rd one)

If in the hook, I "pause" the old master instead of shutdown "q", it
works. It appears as if new master won't survive the death of his
parent. I have also tested it with three states new/running/sleeping.
New master will survive first iteration (parent is sleeping), but not
the second (parent is shut down).

I did test it also with "vacuum = false" with no noticeable effect on the issue.

Q: is this scenario supported ? Forking master and shutting down his
parent.. In Emperor setup.

Thanks,

BranoZ

Full log from forking reload:

Thu Jan 28 18:35:00 2016 - new master copy spawned with pid 5311
fork()'ing uWSGI...
chdir() to /home/foo/.config/uwsgi/vassal
closing all non-uwsgi socket fds > 2 (max_fd = 1024)...
found fd 3 mapped to socket 0 (/home/foo/var/uwsgi/helloworld.sock)
running /home/foo/venv/uwsgi/bin/uwsgi
*** has_emperor mode detected (fd: 7) ***
[uWSGI] getting INI configuration from helloworld.ini
*** Starting uWSGI 2.0.12 (64bit) on [Thu Jan 28 18:35:00 2016] ***
compiled with version: 4.9.2 on 28 January 2016 15:37:16
os: Linux-3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04)
nodename: debian.example.com
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/foo/.config/uwsgi/vassal
detected binary path: /home/foo/venv/uwsgi/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
chdir() to /home/foo/helloworld
your processes number limit is 3934
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: enabled
uwsgi socket 0 inherited UNIX address /home/foo/var/uwsgi/helloworld.sock fd 3
Python version: 2.7.9 (default, Mar  1 2015, 13:01:26)  [GCC 4.9.2]
Set PythonHome to /home/foo/venv/helloworld
Python main interpreter initialized at 0x1a4ffc0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 600 seconds
mapped 363840 bytes (355 KB) for 4 cores
*** Operational MODE: preforking ***
Loading helloworld/settings.py
Importing helloapp/__init__.py
pid 5311: Slowly starting app.. 1/10
pid 5311: Slowly starting app.. 2/10
pid 5311: Slowly starting app.. 3/10
pid 5311: Slowly starting app.. 4/10
pid 5311: Slowly starting app.. 5/10
pid 5311: Slowly starting app.. 6/10
pid 5311: Slowly starting app.. 7/10
pid 5311: Slowly starting app.. 8/10
pid 5311: Slowly starting app.. 9/10
pid 5311: Slowly starting app.. 10/10
Importing helloapp/apps.py
Importing helloapp/models.py
Running ready() on helloapp
WSGI app 0 (mountpoint='') ready in 11 seconds on interpreter
0x1a4ffc0 pid: 5311 (default app)
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 5311)
spawned uWSGI worker 1 (pid: 5453, cores: 1)
spawned uWSGI worker 2 (pid: 5454, cores: 1)
spawned uWSGI worker 3 (pid: 5455, cores: 1)
spawned uWSGI worker 4 (pid: 5456, cores: 1)
Thu Jan 28 18:35:11 2016 - [emperor] vassal helloworld.ini is ready to
accept requests
running "writefifo:/home/foo/var/uwsgi/helloworld.master.fifo q"
(accepting1-once)...
Thu Jan 28 18:35:11 2016 - graceful shutdown triggered...
Gracefully killing worker 1 (pid: 5088)...
Gracefully killing worker 2 (pid: 5089)...
Gracefully killing worker 3 (pid: 5090)...
Gracefully killing worker 4 (pid: 5091)...
running "writefifo:/home/foo/var/uwsgi/helloworld.new.fifo 1"
(accepting1-once)...
Thu Jan 28 18:35:11 2016 - active master fifo is now
/home/foo/var/uwsgi/helloworld.master.fifo
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
goodbye to uWSGI.
VACUUM: unix socket /home/foo/var/uwsgi/helloworld.sock removed.
Thu Jan 28 18:35:14 2016 - [emperor] removed uwsgi instance helloworld.ini
uwsgi_master_manage_emperor()/read(): Resource temporarily unavailable
[core/emperor.c line 2424]
lost connection with my emperor !!!
SIGINT/SIGQUIT received...killing workers...
*** has_emperor mode detected (fd: 7) ***
[uWSGI] getting INI configuration from helloworld.ini
*** Starting uWSGI 2.0.12 (64bit) on [Thu Jan 28 18:35:14 2016] ***
compiled with version: 4.9.2 on 28 January 2016 15:37:16
os: Linux-3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04)
nodename: debian.example.com
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/foo/.config/uwsgi/vassal
detected binary path: /home/foo/venv/uwsgi/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
chdir() to /home/foo/helloworld
your processes number limit is 3934
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: enabled
uwsgi socket 0 bound to UNIX address /home/foo/var/uwsgi/helloworld.sock fd 3
Python version: 2.7.9 (default, Mar  1 2015, 13:01:26)  [GCC 4.9.2]
Set PythonHome to /home/foo/venv/helloworld
Python main interpreter initialized at 0x1d16ff0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 600 seconds
mapped 363840 bytes (355 KB) for 4 cores
*** Operational MODE: preforking ***
Loading helloworld/settings.py
Importing helloapp/__init__.py
pid 5504: Slowly starting app.. 1/10
pid 5504: Slowly starting app.. 2/10
VACUUM WARNING: unix socket /home/foo/var/uwsgi/helloworld.sock
changed inode. Skip removal
pid 5504: Slowly starting app.. 3/10
pid 5504: Slowly starting app.. 4/10
pid 5504: Slowly starting app.. 5/10
pid 5504: Slowly starting app.. 6/10
pid 5504: Slowly starting app.. 7/10
pid 5504: Slowly starting app.. 8/10
pid 5504: Slowly starting app.. 9/10
pid 5504: Slowly starting app.. 10/10
Importing helloapp/apps.py
Importing helloapp/models.py
Running ready() on helloapp
WSGI app 0 (mountpoint='') ready in 10 seconds on interpreter
0x1d16ff0 pid: 5504 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 5504)
Thu Jan 28 18:35:24 2016 - [emperor] vassal helloworld.ini has been spawned
spawned uWSGI worker 1 (pid: 5627, cores: 1)
spawned uWSGI worker 2 (pid: 5628, cores: 1)
spawned uWSGI worker 3 (pid: 5629, cores: 1)
spawned uWSGI worker 4 (pid: 5630, cores: 1)
Thu Jan 28 18:35:24 2016 - [emperor] vassal helloworld.ini is ready to
accept requests
running "writefifo:/home/foo/var/uwsgi/helloworld.master.fifo q"
(accepting1-once)...
open("/home/foo/var/uwsgi/helloworld.master.fifo"): No such device or
address [core/hooks.c line 134]
running "writefifo:/home/foo/var/uwsgi/helloworld.new.fifo 1"
(accepting1-once)...
Thu Jan 28 18:35:24 2016 - active master fifo is now
/home/foo/var/uwsgi/helloworld.master.fifo
Srikanth Bemineni | 24 Jan 19:55 2016
Picon

UWSGI + redis session Workers is taking too much time to die...NO MERCY !!!


Hi.

My application is built using pyramid framework

I started to use UWSGI + redis(pyramid_redis_sessions) as external session application. After starting to use this, UWSGI is not able to kill the workers on exit.

I always get an error killing the workers

"Fri Jan 22 12:49:31 2016 - worker 1 (pid: 4121) is taking too much time to die...NO MERCY !!!"

It took me a lot of time disabling different parts of the code to root cause redis session management as prime suspect.

Srikanth
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Johann Spies | 20 Jan 14:02 2016
Picon

Strange permission-denied (No Gateway) problem

After an upgrade from wheezy to jessy the nginx, uswgi-combination does not play nicely together.  We have also tried to move the socket to tmp, but that did not change anything in as far as the refused connection is concerned.

Any idea why this problem is there? And how to solve it?

The socket:
ls -la /var/uwsgi/app/web2py/socket
srwxrwxrwx 1 www-data www-data 0 Jan 20 14:40 /var/uwsgi/app/web2py/socket

The logs:

$ tail -n 1 /var/log/nginx/error.log
2016/01/20 14:33:27 [error] 4103#0: *8 connect() to unix:///var/uwsgi/app/web2py/socket failed (111: Connection refused) while connecting to upstream, client: 146.232.117.197, server: crest2, request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///var/uwsgi/app/web2py/socket:", host: "crest2.sun.ac.za"


$ tail emperor.log -n 5
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /var/uwsgi/app/web2py/socket fd 3
bind(): Permission denied [core/socket.c line 227]
Wed Jan 20 14:40:45 2016 - [emperor] curse the uwsgi instance web2py.xml (pid: 4957)
Wed Jan 20 14:40:45 2016 - [emperor] removed uwsgi instance web2py.xml

The configurations:

server {
        listen          80;
        server_name     $hostname;
        uwsgi_read_timeout 2400;
        location ~* /(\w+)/static(?:/_[\d]+\.[\d]+\.[\d]+)?/(.*)$ {
            alias /home/www-data/web2py/applications/$1/static/$2;
            expires max;
        }
 
        location / {
            uwsgi_pass      unix:///var/uwsgi/app/web2py/socket;
            include         uwsgi_params;
            uwsgi_param     UWSGI_SCHEME $scheme;
            uwsgi_param     SERVER_SOFTWARE    nginx/$nginx_version;

            }
}


server {
        listen 443 default_server ssl;
        server_name     $hostname;
        ssl_certificate         /etc/nginx/ssl/web2py.crt;
        ssl_certificate_key     /etc/nginx/ssl/web2py.key;
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 10m;
        ssl_ciphers ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA;
        ssl_protocols TLSv1.2;
        keepalive_timeout    70;
        location ~* /(\w+)/static(?:/_[\d]+.[\d]+.[\d]+)?/(.*)$ {
            alias /home/www-data/web2py/applications/$1/static/$2;
            expires max;
        }
        location / {
            uwsgi_pass   unix:///var/uwsgi/app/web2py/socket;                                                             
            uwsgi_buffer_size   128k;
            uwsgi_buffers   8 256k;
            uwsgi_busy_buffers_size   256k;
            include         uwsgi_params;                                                                                    
            uwsgi_param     UWSGI_SCHEME $scheme;
            uwsgi_param     SERVER_SOFTWARE    nginx/$nginx_version;
 
        }
}

--
Because experiencing your loyal love is better than life itself,
my lips will praise you.  (Psalm 63:3)
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
xin | 19 Jan 08:53 2016

emperor heartbeat can't work with gevent

hello:

       It seems that uwsgi worker don't send heartbeat to emperor when uwsgi in gevent mode, if I use prefork mode, everything is ok
       And I didn't locate related code about send heartbeat in gevent plugin. Is it not implemented?



 

_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
gordon | 16 Jan 18:11 2016
Picon

How to use "http-to-https" option on command line?

Hello,

I am trying to get uwsgi to redirect http traffic on port 80 to https on port 443.  It seems like the "http-to-https" option is designed to make this happen without any additional configuration which is ideal.  However, I cannot seem to get it working... I have tried the following:

uwsgi --strict --http-to-https=:80 --https=:443,foo.crt,foo.key --master --enable-threads --processes=4, --threads=2 --module project.wsgi:application

and

uwsgi --strict --http=:80 --http-to-https --https=:443,foo.crt,foo.key --master --enable-threads --processes=4, --threads=2 --module project.wsgi:application

Can anyone please tell me how to use --http-to-https with an example or let me know it isn't meant for what I am attempting to do.

Just for completeness, my app works fine with the following (minus redirecting http to https):

uwsgi --strict --http=:80 --https=:443,foo.crt,foo.key --master --enable-threads --processes=4, --threads=2 --module project.wsgi:application

Thanks,
Gordon
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Riccardo Magliocchetti | 14 Jan 10:07 2016
Picon

debugging workers memory usage in a python application

Hello,

i have a python app that is consuming a lot of resident memory, 4 workers are 
consuming 600MB of resident memory each and growing so some leaks or very bad 
fragmentation.

In order to get more clue about what i going on i'd like to try something like this:
http://smira.ru/wp-content/uploads/2011/08/heapy.html

So setting the baseline memory on postfork and dump the heap every now and then. 
Before digging in the source code was wondering if we have some hook before 
max-requests work reload is called. Do you think this approach makes sense?

thanks

--

-- 
Riccardo Magliocchetti
 <at> rmistaken

http://menodizero.it
immerrr again | 13 Jan 19:27 2016
Picon
Gravatar

Sending output to stdout AND stderr

Hi everyone

I'm running a uwsgi server process in an environment where only stdout &
stderr are readily available, and would like to separate request and
non-request logs.  I think I have run into an incompatibility between Py2
and Py3.

In Py2 I just configured python root logging handler to send app-specific
logs to stdout and uwsgi logs along with access log went to stderr. 

In Py3 the situation seems to have changed: now sys.stdout duplicates
sys.stderr and I have hard time figuring out the proper configuration to
achieve the desired effect.  I tried playing with --logger and --req-logger
parameters but of no avail.

Here's a gist that reproduces the incompatibility between Py2 & Py3:
https://gist.github.com/immerrr/fcba019e21dd823ee486 (observe lines starting
with "###OUT:" closer to the end of py27 log.

Please, advise, what's the correct configuration parameter combination to
achieve my goal.

Cheers,
immerrr
xin | 12 Jan 09:43 2016

error when use rrd plugin

hello:

        when I config uwsgi like below:

    enable-metrics = true
    stats-push = rrdtool:/home/project/project/data/rrd
    rrdtool-freq = 30
    rrdtool-lib = librrd.so
  
    uwsgi report all metric like this :  ERROR: rrd_update("/home/project/user/data/rrd/core.avg_response_time.rrd", "N:2270")

    I use apt-get intall librrd-dev for the librrd.so

    I think I had made some mistakes,  and how to deal with the problem ? thanks very much


 

_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi

Gmane