I sat down this week to start writing a small Python application to aggregate my data and generate a status page like the AWS Service Health Dashboard but with my goal completion instead. This is largely supposed to motivate me to find ways to spend more time doing deep work, but more automatically planned than my past efforts. Well, as I feared, it has taken a few days to lay the groundwork. The last piece involved more trial and error than I would have liked due to lacking documentation, but perhaps someone can make use of this configuration.

Hosting multiple Python versions

I have been using Braden MacDonald’s nicely laid out strategy for running Python apps on my server, which supports both Python 2.7 and 3.4. The problem is uWSGI launches many processes for each app, most of which are rarely used in my case. This was never a problem back when I made one off apps in PHP (eeks!), but running Python is a bit more involved. Still, I knew there must be a better way.

On demand vassals

uWSGI supports socket activated vassals since 1.9.1:

When on demand is active for particular vassal, emperor won’t spawn it on start (or when it’s config changes), but it will rather create socket for that vassal and monitor if anything connects to it.

I tried following the documentation, but was stuck on a bind(): Address already in use error. I also got it working with my sockets in /tmp, but the purist in me wanted them in /run, which is a tmpfs on Ubuntu. The docs contain no complete examples of a configuration that uses on demand vassals, so I read what information there was over and over hoping to be divined some understanding. The options documents were marginally helpful. Dozens of options have identical descriptions to other options, and there is no clear indication of which options can go in the master config versus the vassal configs. I was uncertain where my problem lay.

A StackOverflow answer seemed to address my problem, but it contradicted the docs by using both of the following two options:

--emperor-on-demand-directory /run/uwsgi --emperor-on-demand-extension .sock

Whereas the docs said, “defining on demand vassals involves defining one of three additional settings for your emperor”, before detailing the options: –emperor-on-demand-extension, –emperor-on-demand-directory and –emperor-on-demand-exec. (Note also the quoted official documentation erroneously renders all the double leading hyphens as em dashes.)

The GitHub repository @smn/uwsgi-emperor-example showed up in my searching for mentions of emperor-on-demand-directory, but it only uses sockets for activating the vassals and not for serving the apps as I aimed to do.

Eventually I got it working, and the relevant configuration is below. My server is still running Ubuntu 14.04, so I am running uWSGI via Upstart instead of Systemd. (Ubuntu switched to Systemd in 15.04 apparently due to Debian pressure.) I am also using both Nginx and Apache depending on the app because I am weird.

/etc/init/uwsgi.conf

I diverge from all the tutorials by putting my config in config.ini instead of cramming all the options on the command line. Thus my call to launch uWSGI is simple. The pre-start and post-start sections are specifically to support my putting the sockets in /run, a tmpfs, which is cleared on reboot. I put them in a uwsgi subfolder both for organization and for permissions isolation.

# Emperor uWSGI script

description "uWSGI Emperor"
start on runlevel [2345]
stop on runlevel [06]

respawn

pre-start script
    mkdir -p /var/run/uwsgi
end script

post-stop script
    rm -rf /var/run/uwsgi
end script

exec uwsgi /etc/uwsgi/config.ini

/etc/uwsgi/config.ini

This is the top level uWSGI config.

I use some placeholders (%(var_name)) and some magic variables (%n) in the individual vassal configs. These are explained in How uWSGI parses config files and The uWSGI Emperor – multi-app deployment: Special configuration variables.

The final line causes each vassal to inherit vassals-default.ini. This felt slightly cleaner than having all the config in a .skel file symlinked to each vassal config (and maybe there’s hope uWSGI can optimize if the settings are closer to core?), but it did mean I had to lean on more placeholders. At least I have the option to put custom config (not great for consistency, but nice for troubleshooting) in a vassal now that they are not symlinks.

[uwsgi]
master = true
thunder-lock
emperor = %dvassals
emperor-tyrant
emperor-tyrant-nofollow
emperor-on-demand-directory = /var/run/uwsgi
logto = /var/log/uwsgi.log
vassals-include = %dvassals-default.%e

Also, I copied master and thunder-lock here from the vassal config based on the startup log messages. I am unclear on where they belong.

/etc/uwsgi/vassals-default.ini

I am still working out my Python app structure, so I needed a modification that I handled by defining a django variable in the end vassal config solely so I can test for it here to determine the root directory and module name.

The section at the bottom with cheaper* is apparently how to controll the behavior when using socket activation. I expected having cheaper = 0 and cheaper-initial = 0 would prevent spawning excess workers. No workers were spawned on startup, but upon the first request, 10 workers were spawned. I fixed this by commenting out workers = 10. I am not sure what my worker maximum is now, but my goal of not wasting resources is at least being met. This would obviously require more research or testing with apps that actually get traffic, a problem I don’t have. In that case, I probably wouldn’t care about spawning extra workers on first request unless I had a ton of apps. Really, it doesn’t much matter here either since I believe they were all killed off after a minute of no requests.

[uwsgi]
apps_path = /var/www/votecharlie.com/www/projects/
log_path = /var/www/votecharlie.com/log/
post-buffering = 8192

if-not-opt = plugin
plugin = python34
endif =

if-not-opt = django
chdir = %(apps_path)%(app_name)/app/%(app_name)
module = main
endif =

if-opt = django
chdir = %(apps_path)%(app_name)/app
module = %(app_name).main
endif =

# home|virtualenv|venv|pyhome: set PYTHONHOME/virtualenv
virtualenv = %(apps_path)%(app_name)/venv
# pythonpath|python-path|pp: add directory (or glob) to pythonpath
# pythonpath = %(apps_path)%(app_name)/app/%(app_name)

# process-related settings
procname = %(app_name)
thunder-lock
# master
master = true
# clear environment on exit
vacuum = true
daemonize = %(log_path)uwsgi_%(app_name).log
max-requests = 500
harakiri = 6000

# set cheaper algorithm to use, if not set default will be used
cheaper-algo = spare
# minimum number of workers to keep at all times
cheaper = 0
# number of workers to spawn at startup
cheaper-initial = 0
# maximum number of workers that can be spawned
#workers = 10
# how many workers should be spawned at a time
cheaper-step = 1

idle = 20
die-on-idle

/etc/uwsgi/vassals/*

Here are examples of some configs for individual vassals.

/etc/uwsgi/vassals/project1

[uwsgi]
app_name = %n
django

/etc/uwsgi/vassals/project2

[uwsgi]
app_name = %n
plugin = python27

/etc/uwsgi/vassals/project3

[uwsgi]
app_name = %n
python-autoreload = 1

/etc/nginx/sites-enabled/project1.conf

This is Nginx config for the apps I serve with Nginx.

 upstream django {
     server unix:///run/uwsgi/project1.socket;   }

 server {
     # the port your site will be served on
     listen      4533;
     # the domain name it will serve for
     server_name .project1.com; # substitute your machine's IP address or FQDN
     charset     utf-8;

     # max upload size
     client_max_body_size 75M;   # adjust to taste

     # Django media
     location /media  {
         alias /var/www/votecharlie.com/www/projects/project1/app/media;
     }
     location /static {
         alias /var/www/votecharlie.com/www/projects/project1/app/static;
     }

     # Send all non-media requests to the Django server.
     location / {
         uwsgi_pass  django;
         include     /var/www/votecharlie.com/www/projects/project1/app/uwsgi_params;
     }
 }

/etc/apache2/sites-available/votecharlie.com.conf

Within my main Apache config, I have an IncludeOptional line pointing to the conf-enabled directory. This allows enabling apps with the command sudo a2enconf votecharlie-project2 and disabling with a2disconf, and no errors are thrown if no apps are enabled (unlike with straight Include).

IncludeOptional conf-enabled/${site}-*.conf

/etc/apache2/conf-available/votecharlie.com-project2.conf

Here is an example of the config I would add to enable an app under Apache before running sudo a2enconf votecharlie-project2 and reloading Apache.

Alias /projects/project2/static ${path}www/projects/project2/project2/static
<Location /projects/project2>
  Options FollowSymLinks Indexes
  SetHandler uwsgi-handler
  uWSGISocket /var/run/uwsgi/project2.socket
</Location>
<Location /projects/project2/static>
  SetHandler default-handler
</Location>