Adrien Cossa | 22 Jul 13:06 2016
Picon

how to prevent uwsgi to shot celery own sub-processes?

Hi,

I have an issue using uwsgi and celery. When celery is stopped by uwsgi, it tries to kill one of its pool subprocess but it fails because this subprocess doesn't exist anymore (as if uwsgi would have done the job already). I posted some more information here:

http://stackoverflow.com/questions/38521586/billiard-writes-errno-3-no-such-proces-when-uwsgi-shuts-celery-down

Thanks for any help :)
Adrien

body, td { text-align: left; color: #000000; font-family: Arial, Helvetica, sans-serif; font-size: 9pt; font-style: normal; font-weight: normal; } a, a:link, a:visited, a:hover, a:active { color: #93A71B; text-decoration: none; }

Adrien Cossa
Entwicklung




init.at informationstechnologie GmbH
Fockygasse 29-31 | 4.OG | 1120 Wien
Mail: cossa-0SGIby7uL1k@public.gmane.org
Tel: +43 1 522 53 77 39
Fax: +43 1 522 53 77 55
Web: http://www.init.at

website | vCard | map | email


Diese Nachricht ist vertraulich und nur für den Adressaten bestimmt. Sollten Sie kein autorisierter Empfänger sein, ist jede Offenlegung, Weiterleitung oder sonstige Verwendung dieser Nachricht nicht gestattet. Bitte informieren Sie in diesem Fall den Absender und löschen Sie alle Kopien - vielen Dank.

This message is confidential and only intended for its recipient. If you are not the intended recipient, any disclosure, distribution or any other use of this mail is prohibited. In this case, please notify the sender and delete all copies of the message - thank you.


_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Joshua Lang | 22 Jul 02:44 2016
Picon

Graceful shutdown/cleanup of mule processes

I have a simple uWSGI setup with a mule process for background processing. I've set up my own signal handlers in mule-brain script to handle SIGINT in the mule process. I was hoping this would allow my script to finish its current task and exit after that, but it seems as though the uWSGI master process sends SIGINT to the mule process, but then just goes ahead and SIGKILLs the process right before the master process exits.

Is it possible to prevent this behavior so my mule process can get a chance to gracefully exit when the whole uWSGI "service" is exiting?

Thanks,

-j
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Damjan Georgievski | 19 Jul 21:30 2016
Picon
Gravatar

writing to stderr from uwsgi-php

I need to debug a php application running in uwsgi,
what's the best way to write to stderr from php itself?

ps.
for some reason I don't get any output from the error_log() function

--

-- 
damjan
Dustin C. Hatch | 19 Jul 03:26 2016
Gravatar

systemd socket activation and --http

Hello, I am trying to set up uWSGI to use systemd socket activation,
along with the http plugin. I am able to get each aspect working
correctly on its own, but whenever I combine them, client requests
simply hang.

Here is my configuration:

[uwsgi]
master = true
plugin = python,http
optimize = 1
http = fd://0
wsgi = fmosadmin.web.app

My socket unit:

[Unit]
Description=socket description

[Socket]
ListenStream=9090

[Install]
WantedBy=multi-user.target

And my service unit:

[Unit]
Description=service description

[Service]
Type=simple
EnvironmentFile=-/etc/sysconfig/fmos-admin
ExecStart=/usr/sbin/uwsgi --die-on-term /etc/firemon/fmos-admin.ini
StandardInput=socket
StandardOutput=journal
StandardError=journal

The only thing that even suggests a problem on the server side is this
message in the log (repeated twice at startup):

uwsgi[9493]: epoll_ctl(): Invalid argument [core/event.c line 521]

Socket inheritance seems to be working:

uwsgi[9493]: uwsgi socket 1 inherited INET address :9090 fd 0

But, `curl http://localhost:9090/` just hangs forever.

Everything works correctly if I change `http` to `http-socket` in the
configuration, but since there will be no front-end proxy in this
situation, I understood the former was the correct choice.

Any suggestions are appreciated.

--

-- 
♫Dustin
_______________________________________________
uWSGI mailing list
uWSGI <at> lists.unbit.it
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Roberto De Ioris | 12 Jul 10:21 2016
Picon

Re: Flat file caching in subdirectories


> On Tue, 12 Jul 2016 09:22:01 +0200
> "Roberto De Ioris" <roberto@...> wrote:
>
> Hi Roberto
>
>> Hi, i think some file is missing from the uwsgi-router-isfreshfile
>> repository
>
> Ah, I did not expect a reply that quickly ;)
> Ok, just pushed the code for "uwsgi-router-isfreshfile".
>
>> > - The intended modifier/parameter to set the file's age is still a bit
>> of
>> > a mystery
>>
>>
>> what about using xattr ?
>>
>> https://github.com/unbit/uwsgi-docs/blob/master/Changelog-1.9.9.rst#the-xattr-plugin
>>
>> https://github.com/unbit/uwsgi/blob/master/plugins/xattr/xattr.c
>
> Forgive my thick head at the moment ... but it seems with "xattr" I can
> access a file's attributes and use it as a variable inside the routing
> system ... but what I am searching for is a way to pass in a parameter to
>

oh, ok, check how the 'ishigher' conditional is implemented:

https://github.com/unbit/uwsgi/blob/master/core/routing.c#L1559

basically the code checks for ';' and split it in two buffers. The second
one will be your age.

--

-- 
Roberto De Ioris
http://unbit.com
Tim van der Linden | 10 Jul 10:12 2016
Picon
Gravatar

Flat file caching in subdirectories

Hi folks

I have a caching/router based question.

I need to cache a site with a large number of pages (think 20 million+). These pages are generated by a Python
application with a fair amount of database calls, data crunching and template rendering, hence the need
to cache the responses.

First I looked into uWSGI's Cache2 setup, which is awesome but "only" memory based. With a potential of 20
million pages at ~30 Kb each ... well I do not have that kind of memory :)

File based caching it is. As far as I could find in the docs Cache2 does not support dumping responses into
files, only into memory (tell me if I'm wrong here...). The only way to accomplish file based caching is
through the router and the "transformation_tofile" plugin as demonstrated in the caching cookbook.

I have tested this and it works very well. Not as fast as memory based caching, but still a considerable speed boost.

Here comes the crux: If more than 20 million files could potentially be dumped into a single cache directory
on disk this could wreak havoc on filesystem read times in that specific directory.

So here is what I try to do: I would like to create subdirectories inside the main cache directory based on the
URL path. Each path in the application points to a different type of page so I'm looking at the following
cache setup:

- If the path is: "/path-a/some-title/" the cache on disk "/main-cache-dir/path-a/some-title/file.html

In essence this is no problem and can be achieved by the following router rule:

route-if = isfile:/disk1/uwsgi_cache/files${PATH_INFO}file.html static:/disk1/uwsgi_cache/files${PATH_INFO}file.html

However, uWSGI needs to create this subdirectory structure (and the arbitrary named "file.html" file)
first. I know uWSGI can do this with the "exec-asap" directive:

exec-asap = mkdir -p <some-path>

Yet "exec-asap" is not part of the router setup and does not fire during routing nor does it carry the needed
routing variables (such as ${PATH_INFO}).

Does this sound like a scenario for uWSGI "flat file" caching ... can it cache to subdirectories instead of
flat all in one?
Is it possible to create directories (and their parents) from within the router?

Cheers,
Tim
Jose Jachuf | 10 Jul 03:42 2016
Picon

python plugin

Enviroment:
FreeBSD 10.3
Python27 (default pyhon)
uwsgi 2.0.13 installed from ports [1]

I want to deploy a web application that runs on a virtualenv with python34.

My configuration file is:

[uwsgi]
socket = 127.0.0.1:9002
protocol = http
chdir = /home/jose/trabajos/farmaenv/farma

virtualenv = /home/jose/trabajos/farmaenv/venv
##pythonpath = /home/jose/trabajos/farmaenv/venv
pythonpath = /home/jose/trabajos/farmaenv/venv/lib64/python3.4
pythonpath = /home/jose/trabajos/farmaenv/venv/lib64/python3.4/site-packages

module = wsgi
callable = application

no-site = true
master = true
processes = 2
uid = www
gid = www
chmod-socket = 664
vacuum = true


if I install uwsgi in virtualenv it works well. However, and I want to use the system uwsgi.


For that and based on the documentation [2]

# cd /usr/ports/www/uwsgi
# make extract
# cd work/uwsgi-2.0.13
# python3.4 uwsgiconfig.py --plugin plugins/python core python34

With this got python34_plugin.so file

# cp python34_plugin.so /home/jose/trabajos/farmaenv/plugins/

Add the following to the configuration file:

plugins-dir = /home/jose/trabajos/farmaenv/plugins
plugin = python34

running wsgi:

# uwsgi -T --ini /home/jose/trabajos/farmaenv/farma/farma.ini
[uWSGI] getting INI configuration from /home/jose/trabajos/farmaenv/farma/farma.ini
!!! UNABLE to load uWSGI plugin: ./python34_plugin.so: Undefined symbol "uwsgi_route_api_func" !!!

Best Regards
José
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
John Burk | 9 Jul 03:52 2016

Yet another thread about logging with uWSGI and Pyramid

Been paying the rent with python for about 8 years, but new to uWSGI and web dev't in general, so be kind please.  I'll promptly respond to "you need to post x,y,z for us to help you", just let me know what you need.

I've inherited an app that was built on Pyramid, was only ever using pserve and waitress, and never had a working production setup.

After some researching various WSGI options, I switched everything over to uWSGI and nginx.  Everything's working for me both in development and production, with the exception of logging in my development environment.  I'm not getting anything out of Pyramid, just the uWSGI stderr/out on the console, and the usual in nginx's access.log, and nothing in nginx/error.log

CentOS-6.6

Starting up uswgi with:  

../bin/uwsgi --ini-paste-logged ./development.ini --http :9090

​yields:

*** Starting uWSGI 2.0.13.1 (64bit) on [Fri Jul  8 17:59:10 2016] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-4) on 08 July 2016 17:37:58
os: Linux-3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015
nodename: VM-CentOS-72-x64
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/jburk/dev/main/meteredLicensing/server/metered
detected binary path: /home/jburk/dev/main/meteredLicensing/server/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 4096
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :9090 fd 3
uwsgi socket 0 bound to UNIX address /var/www/uwsgi/mls.sock fd 6
Python version: 2.7.5 (default, Nov 20 2015, 02:00:19)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x1f3a880
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145520 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
Loading paste environment: config:/home/jburk/dev/main/meteredLicensing/server/metered/./development.ini
ImportError: No module named script.util.logging_config
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x1f3a880 pid: 21967 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 21967)
spawned uWSGI worker 1 (pid: 21974, cores: 1)
spawned uWSGI http 1 (pid: 21975)

​the interesting error being ​"
ImportError: No module named script.util.logging_config
​".  Googling that shows that I should only have to install pastescript to get this all working

(server) [jburk <at> VM-CentOS-72-x64 metered]$ pip install pastescript
Collecting pastescript
  Using cached PasteScript-2.0.2-py2.py3-none-any.whl
Collecting Paste>=1.3 (from pastescript)
  Using cached Paste-2.0.3-py2-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): six in /home/jburk/dev/main/meteredLicensing/server/lib/python2.7/site-packages/six-1.10.0-py2.7.egg (from pastescript)
Requirement already satisfied (use --upgrade to upgrade): PasteDeploy in /home/jburk/dev/main/meteredLicensing/server/lib/python2.7/site-packages/PasteDeploy-1.5.2-py2.7.egg (from pastescript)
Installing collected packages: Paste, pastescript
Successfully installed Paste-2.0.3 pastescript-2.0.2

​but as soon as I do, I get:

*** Operational MODE: single process ***
Loading paste environment: config:/home/jburk/dev/main/meteredLicensing/server/metered/./development.ini
ImportError: No module named deploy

​installing 'deploy' seems to have no effect:

(server) [jburk <at> VM-CentOS-72-x64 metered]$ pip install deploy
Collecting deploy
  Downloading deploy-1.9.1.tar.gz
Building wheels for collected packages: deploy
  Running setup.py bdist_wheel for deploy ... done
  Stored in directory: /home/jburk/.cache/pip/wheels/1a/3f/ba/03dba49461ef7b067914a6f2208d1fb3eba3121f50e3788258
Successfully built deploy
Installing collected packages: deploy
Successfully installed deploy-1.9.1

(server) [jburk <at> VM-CentOS-72-x64 metered]$ ../bin/uwsgi --ini-paste-logged ./development.ini --http :9090
[uWSGI] getting INI configuration from ./development.ini
*** Starting uWSGI 2.0.13.1 (64bit) on [Fri Jul  8 18:21:55 2016] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-4) on 08 July 2016 17:37:58
os: Linux-3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015
nodename: VM-CentOS-72-x64
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/jburk/dev/main/meteredLicensing/server/metered
detected binary path: /home/jburk/dev/main/meteredLicensing/server/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 4096
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :9090 fd 3
uwsgi socket 0 bound to UNIX address /var/www/uwsgi/mls.sock fd 6
Python version: 2.7.5 (default, Nov 20 2015, 02:00:19)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x13de920
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145520 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
Loading paste environment: config:/home/jburk/dev/main/meteredLicensing/server/metered/./development.ini
ImportError: No module named deploy

​at this point my virtualenv is dead.  Uninstalling deploy, Paste, and pastescript doesn't get me back to a working state, I keep getting "no module named deploy", and have to to delete and re-make my virtualenv (which only takes about 2 minutes, so it's not painful but it sure is getting stale...)
​I've attached my development.ini, what else do you need to help me sort this out?​

running CentOS-6.6

(server) [jburk <at> VM-CentOS-72-x64 metered]$ pip list
alembic (0.8.6)
bcrypt (3.1.0)
cffi (1.7.0)
colander (1.3.1)
docutils (0.12)
iso8601 (0.1.11)
Jinja2 (2.8)
Mako (1.0.4)
MarkupSafe (0.23)
marshmallow (2.9.0)
marshmallow-sqlalchemy (0.8.1)
metered (0.0, /home/jburk/dev/main/meteredLicensing/server/metered)
MySQL-python (1.2.5)
PasteDeploy (1.5.2)
pip (8.1.2)
pycparser (2.14)
Pygments (2.1.3)
PyJWT (1.4.0)
pyramid (1.7)
pyramid-debugtoolbar (3.0.2)
pyramid-jinja2 (2.6.2)
pyramid-mako (1.0.2)
pyramid-tm (0.12.1)
python-editor (1.0.1)
repoze.lru (0.6)
requests (2.10.0)
setuptools (24.0.2)
six (1.10.0)
SQLAlchemy (1.0.13)
transaction (1.6.1)
translationstring (1.3)
uWSGI (2.0.13.1)
venusian (1.0)
WebOb (1.6.1)
wheel (0.29.0)
zope.deprecation (4.1.2)
zope.interface (4.2.0)
zope.sqlalchemy (0.7.6)
Attachment (development.ini): application/octet-stream, 2459 bytes
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Attila-Mihaly Balazs | 7 Jul 15:39 2016
Picon

Leaving uWSGI behind

Hello all,

Unfortunately I wasn't able to "bend uwsgi to my will" so I choose to
use mod_wsgi for this latest project. A couple of observations which
people might find helpful (ie. pointers as to what could be improved):

- the documentation could use a lot of improvement. There are a lot of
options and most of them are documented in a most unhelpful way. Ie:
"socket-timeout - set internal sockets timeout". Great, but what's an
internal socket? What is the default value? In what case would I want
to change this and how?

- using apache2 mod-proxy-uwsgi with unix sockets still doesn't work.
I remember having issues more than two years ago and now it's still
happening. The issue is the same one as documented here:

- http://lists.unbit.it/pipermail/uwsgi/2015-May/007992.html
- and here https://github.com/unbit/uwsgi/issues/973

Ie. I get "AH00898: DNS lookup failure for: path returned by /path". I
also tried adding something at the end (ie. "...|uwsgi://localhost/"
instead of "...|uwsgi://") but then it just tries to connec through
TCP to 127.0.0.1:3031 instead of using the unix socket

This is on ubuntu 16.04 with Apache 2.4 libapache2-mod-proxy-uwsgi
2.0.12-5ubuntu3 - this one:
http://packages.ubuntu.com/xenial/libapache2-mod-proxy-uwsgi

Cheers,
Attila
Attila-Mihaly Balazs | 6 Jul 16:53 2016
Picon

Enable uwsgi requests to run for a long time?

Hello,

I'm setting up uWSGI as part of a batch processing system which has
the following flow:

(work distributor) --> (http) -> [ apache -> uwsgi -> python3 flask
application ] x N

That is: I have N nodes with apache + uwsgi + a python3 flask
application to do the processing. The "work distributor" submits a
task through HTTP to such a machine, which runs it synchronously and
then replies on the same HTTP connection. Such a HTTP request can take
5-10 minutes. What do I need to set in my uwsgi ini file so that it
doesn't kill the request? I'm concerned because I saw this in the log:

"your mercy for graceful operations on workers is 60 seconds"

I tried to search around a little but couldn't find the setting which
changes this (tried "harakiri" but that wasn't it).

Thank you,
Attila
David Montgomery | 20 Jun 22:46 2016
Picon

uwsgi python module - now to get fixed id for logging per proccess



Hi,

I am using both python logging and set_logvar to log extra vars.

So  for the filename, I would one process per file to be consistent on restarts

Kinda like this -> upload-8030-stdout---supervisor-H0_GkI.log

In python..how so I get "H0_GkI" portion so I can use it too?

then handler = RotatingFileHandler(filename='/tmp/upload_server_bottle-H0_GkI.log', maxBytes=1024 * 1024 * 10,backupCount=1)

Thanks
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi

Gmane