Harry Percival | 25 May 14:09 2016

vassal config: set per-worker cgroups?

Hi there,

We're trying to apply some memory limitations to apps that may contain a variable number of workers.

We know we can set

  cgroup = /mnt/cgroups/memory/domains/vassal-domain-name
  cgroup-opt = memory.limit_in_bytes=1800M
  cgroup-opt = memory.memsw.limit_in_bytes=1800M

And that's fine for limiting a user with one worker to 1.8GB of RAM, but the problem is that if a user has 10 workers, the cgroup limits the *total* memory usage to 1.8GB, which is now down to less than 200Mb per worker, and that's too little.   We also don't want to just bump that user up to an 18GB limit, since that's potentially higher than the system ram, and besides, this is mainly about catching accidental memory leaks in individual processes rather than limiting total consumption.  So the question is:

* Is there any way to put each worker in its own cgroup? *

I could imagine it working if uwsgi had some magical template variable syntax, a bit like nginx?  eg:

  cgroup = /mnt/cgroups/memory/domains/vassal-domain-name/worker-$WORKER_ID

Is there any such thing?

Or perhaps there would be a way of running a script, pre-jail put post-fork, that ran as root/privileged user, that would be able to identify the spawned worker processes and manually put them into their own cgroups?   Could exec-post-jail or exec-pre-app work for this?  Would there be any way for such a job to identify the worker process ids?

thanks,
Harry

-- Harry Percival Developer harry-7DvliJ76plxWUWZDIUQIQ9BPR1lH4CV8@public.gmane.org PythonAnywhere - a fully browser-based Python development and hosting environment <http://www.pythonanywhere.com/> PythonAnywhere LLP 17a Clerkenwell Road, London EC1M 5RD, UK VAT No.: GB 893 5643 79 Registered in England and Wales as company number OC378414. Registered address: 28 Ely Place, 3rd Floor, London EC1N 6TD, UK
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Giles Thomas | 23 May 19:12 2016

Order in which cgroup-opt options are applied

Hi all,

Short question: it looks like cgroup-opt options, when specified in a 
uWSGI .ini file, are being applied in the reverse of the order in which 
they appear in the .ini file.  Is this correct?  If so, is it expected 
defined behaviour, or is it likely to change?

Background:

We're trying to set the memory.memsw.limit_in_bytes and 
memory.limit_in_bytes cgroup options.  When we set them from the command 
line for non-uWSGI processes, we have to set the memory.limit_in_bytes 
option first and then the memory.memsw.limit_in_bytes option second.   
Trying to set them the other way around gives an OS error.

When we set them using cgroup-opt in a uWSGI .ini config file, we have 
to specify memory.memsw.limit_in_bytes first and memory.limit_in_bytes 
second.  If we try to set them the other way around, the vassal (we're 
using emperor mode) never starts -- it keeps getting cursed.

 From a quick look at the source code, it looks like the cgroup-opt 
options might be being applied in the reverse order to the way they're 
specified -- they're being put into a linked list, which is naturally 
easier to prepend to than append to, and then applied in the order they 
appear in in the linked list.

This is all fine -- we can just specify them in the order that works.  
But it would be good to be sure that we're interpreting what's happening 
correctly, and to know if we can rely on this staying the same in the 
future.

All the best,

Giles

--

-- 
Giles Thomas <giles@...>

PythonAnywhere: Develop and host Python from your browser
<https://www.pythonanywhere.com/>

A product from PythonAnywhere LLP
17a Clerkenwell Road, London EC1M 5RD, UK
VAT No.: GB 893 5643 79
Registered in England and Wales as company number OC378414.
Registered address: 28 Ely Place, 3rd Floor, London EC1N 6TD, UK
David Montgomery | 20 May 21:05 2016
Picon

gevent loop does not work with mysql.connector

Hi,

uWSGI with gevent loop appears to not be working with flask and mysql.connection sad to say.


/usr/local/bin/uwsgi --chdir /var/zenapi --pidfile /var/run/api.pid --loop gevent --socket 127.0.0.1:8030 --pythonpath /var/zenapi/app --pythonpath /var/zenapi  --
processes 1 --file /var/zenapi/wsgi.py --callable app -b 32000 --master --gevent 30 --enable-threads --listen 2048  --logto2 /tmp/api.log


INSERT INTO api.loadtest (loadtest) VALUES ('d2a2c9fd-24ed-4bee-87ec-d21fef78fd02')
register error: (<class 'gevent.hub.ConcurrentObjectUseError'>, ConcurrentObjectUseError('This socket is already used by another greenlet: <bound method Waiter.switch of <gevent.hub.Waiter object at 0x7fe487cc1f00>>',), <traceback object at 0x7fe487d06710>)
Traceback (most recent call last):
  File "/var/zenapi/resources/load.py", line 58, in post
    results = db.execute_query(query,select=False,table=table,key=load)
  File "/var/zenapi/mysql_tools.py", line 117, in execute_query
    self.conn.set_property(tables=["%s.%s" % (self.database,table)], scope=fabric.SCOPE_GLOBAL, key=key, mode=fabric.MODE_READWRITE)
  File "/usr/lib/python2.7/dist-packages/mysql/connector/fabric/connection.py", line 1321, in set_property
    self.close()
  File "/usr/lib/python2.7/dist-packages/mysql/connector/fabric/connection.py", line 1503, in disconnect
    self.rollback()
  File "/usr/lib/python2.7/dist-packages/mysql/connector/fabric/connection.py", line 1591, in rollback
    self._mysql_cnx.rollback()
  File "/usr/lib/python2.7/dist-packages/mysql/connector/connection.py", line 857, in rollback
    self._execute_query("ROLLBACK")
  File "/usr/lib/python2.7/dist-packages/mysql/connector/connection.py", line 869, in _execute_query
    self.cmd_query(query)
  File "/usr/lib/python2.7/dist-packages/mysql/connector/connection.py", line 488, in cmd_query
    result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
  File "/usr/lib/python2.7/dist-packages/mysql/connector/connection.py", line 267, in _send_cmd
    return self._socket.recv()
  File "/usr/lib/python2.7/dist-packages/mysql/connector/network.py", line 226, in recv_plain
    chunk = self.sock.recv(4 - packet_len)
  File "/usr/local/lib/python2.7/dist-packages/gevent-1.1.0-py2.7-linux-x86_64.egg/gevent/_socket2.py", line 280, in recv
    self._wait(self._read_event)
  File "/usr/local/lib/python2.7/dist-packages/gevent-1.1.0-py2.7-linux-x86_64.egg/gevent/_socket2.py", line 173, in _wait
    raise _socketcommon.ConcurrentObjectUseError('This socket is already used by another greenlet: %r' % (watcher.callback, ))
ConcurrentObjectUseError: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent.hub.Waiter object at 0x7fe487cc1f00>>

_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Marco De Paoli | 20 May 17:53 2016
Picon
Gravatar

post-buffering and harakiri not working together

Hi all,
I noticed that harakiri is not triggered
But if I comment out "post-buffering=8192" then harakiri works again

At the moment removing post-buffering is not an option because I am not sure if any part of the django project depends on it

Any idea?

tia,
Marco

My box:

# uwsgi --version
2.0.13.1
# nginx -v
nginx version: nginx/1.4.6
# uname -a
Linux server-name 2.6.32-431.29.2.el6.centos.plus.i686 #1 SMP Tue Sep 9 22:29:30 UTC 2014 i686 i686 i386 GNU/Linux


_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Alex Hall | 19 May 18:13 2016

Python application suddenly missing?

Hello all,
I made some changes to some HTML in my site, and was having a bit of trouble with getting them to render. I restarted UWSGI, and suddenly I'm seeing in the log for my app that the Python application cannot be found. I didn't move anything, change any configuration files, install anything new in my virtual environment, move any directories, or in any other way change anything I can think of. I just added a new variable to a WTForm class, and put in a new function in a Python file and in part of some Javascript. My templates (I'm using Flask) failed to render correctly, so as part of trying to fix that I figured I'd restart Nginx and UWSGI. Now it can't locate Python, which makes no sense.

This was all working perfectly yesterday, as it has been doing for over a week. I'm still looking around to see what I might have missed, but is there anything obvious I could try that is known to cause this problem? Anything in particular I should be watching for? I'm on Debian 8; let me know if you need more details about my setup. Thanks.

--
Alex Hall
Automatic Distributors, IT department
ahall <at> autodist.com
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Davide Setti | 13 May 16:15 2016
Picon
Gravatar

how to know what's killed by oom-killer

Hi,
due to changes in an application we have memory issues and therefore workers killed by the oom-killer. What uWSGI logs is:

DAMN ! worker 1 (pid: 2428) died, killed by signal 9 :( trying respawn ...
Is there a way to log the request path that the worker was processing when killed? It would be very helpful to easily spot problems without a profiler.

Thanks
--

Davide Setti
code: http://github.com/vad
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Roberto De Ioris | 13 May 01:24 2016
Picon

[ANNOUNCE] uWSGI 2.0.13.1

Hi everyone, a minor release fixing a couple of regressions is available:

https://uwsgi-docs.readthedocs.io/en/latest/Changelog-2.0.13.1.html

users of python2.5 and python2.6 can now build uWSGI again, as well of
users of older glibc versions.

Unfortunately, the EPOLLEXCLUSIVE patch has been reverted to allow us to
better design it. Riccardo and INADA have already found a solution, so
expect improvements in this area pretty soon.

--

-- 
Roberto De Ioris
http://unbit.com
Alex Hall | 12 May 23:09 2016

Always an extra UWSGI on Debian 8?

Hi list,
I seem to be unable to close UWSGI completely. Each time I make a change to my site, I do:

/etc/init.d/uwsgi stop
/etc/init.d/nginx restart
uwsgi --ini-paste /etc/uwsgi/sites-available/myapp.ini

But it never works correctly. If I then do

fuser -k 9876/tcp

I see two processes stopped, or at least, I assume the two numbers are PIDs. Needless to say, 9876 is the TCP port in use by my app. Anyway, after I do that and then start UWSGI again, everything is fine, but until then, I can't seem to get changes to take. I'm probably doing something wrong, as usual, but what this time? Debian 8, hence the use of /etc/init.d. Thanks.

--
Alex Hall
Automatic Distributors, IT department
ahall-vRNBg8LEEi9Wk0Htik3J/w@public.gmane.org
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Alex Hall | 11 May 20:07 2016

Unable to open log file despite chmod and chown

Hi all,
Sometimes, when I start my uwsgi from an ini file in which a log file is specified, I get an error 288. This says that permission to write to the log file was denied. The file is
/var/log/uwsgi/myapp.log
and I've done
chown www-data /var/log/uwsgi
and
chmod 755 /var/log/uwsgi
then
chmod 777 /var/log/uwsgi
(www-data is the user used by Nginx, so I'm using that one).

Yet, this error persists. I'm not well versed in managing permissions or owners, having not dealt much with a Linux shell (this is a Debian 8 system). Am I doing something wrong in uwsgi, or is this a problem with my chown or chmod commands that has nothing to do with anything? The confusing thing is that logging sometimes works, and sometimes doesn't. My best guess is that it fails when uwsgi wants to create a new log file and archive the previous one--it seems to like to make gz versions of logs on its own. I can't confirm that, though.

--
Alex Hall
Automatic Distributors, IT department
ahall-vRNBg8LEEi9Wk0Htik3J/w@public.gmane.org
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
xnrl | 11 May 11:30 2016
Picon
Gravatar

Logging to anywhere in a programmed mule not working

Hello, I've got a bit of a strange problem while trying to utilize a mule. I start uwsgi from shell like this: "uwsgi --protocol http --plugins python3 --socket :8080 --mule=script /srv/www/app/main.py --uid user" At one point I noticed that logging doesn't work - neither to a file nor to stdout; so I created a very simple script to test it: ------------------ #!/usr/bin/python3 import logging logging.basicConfig(filename = '/var/log/app/app.log', level = logging.DEBUG) log = logging.getLogger('log') log.debug('--> testscript working') ------------------ and it sort of worked - only the output appeared not in the file but on stdout. ------------------ # uwsgi --protocol http --plugins python3 --socket :8080 --mule=testscript /srv/www/app/main.py --uid user [...] spawned uWSGI master process (pid: 25224) spawned uWSGI worker 1 (pid: 25238, cores: 1) spawned uWSGI mule 1 (pid: 25239) DEBUG:log:--> testscript working ------------------ I then proceeded to comment everything in the 'script1' except for the same line as in test - and the result is nothing at all, I don't see the log line anywhere. Permissions on the logfile are 777, owned by the user under whom uwsgi is run (by the --uid option). Both scripts have the same permissions/ownership, reside in the same directory and work as intended when started from shell. I did a diff on both files to determine if the uncommented lines are the same (grepped them with "grep -vE '(^#)|(^\s*$)' script > /tmp/diffX") - they are. Also tried writing something to a file, with and without "if __name__ == '__main__'" clause - test script works, original doesn't. I can't understand why basically identically scripts work differently, or why the log doesn't go where it should. Do I completely misunderstand how mules are supposed to operate, or is it a simpler problem I'm facing? Using uwsgi 1.2.3+dfsg-5+deb7u1 from repos on debian jessie with python 3.2.3-6 Thanks in advance.
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
Alex Hall | 10 May 17:22 2016

Stupid question: where'd the main ini file go?

Hi all,
I feel pretty bad that I have to ask this at all, but where's my UWSGI main ini file get to? I thought I had it in /etc/uwsgi, but it's not there. Starting UWSGI fails, and I don't know why as the log is empty (as usual). Some forum suggested using pidfile2 instead of pidfile, so I wanted to try that, but I just can't find the ini file no matter where I look. I see my apps-available, but not the main ini file for UWSGI itself. I'm sure I didn't move or delete it. Debian 8, if it matters.

--
Alex Hall
Automatic Distributors, IT department
ahall-vRNBg8LEEi9Wk0Htik3J/w@public.gmane.org
_______________________________________________
uWSGI mailing list
uWSGI@...
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi

Gmane