ionel's codelog

Proxy issues with Apache

2018-03-25T00:00:00+02:00

So few days ago I had a very weird bug with an Apache httpd SSL terminator setup. So basically just a proxy to Nginx+uWSGI serving plain http. Do not ask why such a contrived setup, it's beyond reason.

Sometimes httpd would give this sort of error and big file uploads:

[proxy:error] [pid 18:tid 140393219815168] (32)Broken pipe: [client 10.0.2.2:55239] AH01084: pass request body failed to 172.21.0.9:80 (web), referer: https://localhost/upload/
[proxy_http:error] [pid 18:tid 140393219815168] [client 10.0.2.2:55239] AH01097: pass request body failed to 172.21.0.9:80 (web) from 10.0.2.2 (), referer: https://localhost/upload/

Trying few requests locally didn't reproduce the problem.

Not considering it only happened with large files I suspected there's some sort of boundary or timeout that is being hit. Firefox has a slow connection simulator in its Web Developer tools. Open up Responsive Design Mode and there are the throttling settings:

Note the funny "Good 3G" choice. As if someone would say "good devils" or "nice jerks".

Fortunately for me this reproduced the problem, and after few tries it turned out it reproduced with small files. So not a boundary, but a timeout problem.

Now I've tried to no avail making various Nginx settings like:

proxy_read_timeout 600s;
client_body_timeout 600s;
client_header_timeout 600s;
send_timeout 600s;

And in the Apache conf:

ProxyTimeout 600

So basically playing with this settings was a bit of shotgun debugging. Time to get serious and break out sysdig:

sysdig container.name=app_ssl_1 or container.name=app_web_1 -A -s 1000 > log

I had two containers, thus the container.name filters. If you don't have containers then probably you'll want to filter by process name, as the output is already very verbose.
The -A (or --print-ascii) is to strip non-ascii stuff from output.
The -s 1000 is to see 1000 bytes from the various string being passed around.

So at this point I have a 30mb log file. Grepping [1] backwards through the output for close or connections seems like a good start. It turned out:

21:48:04.525468894 2 httpd (17039) < close res=0
21:48:04.525577726 2 httpd (17039) > close fd=10(<f>/tmp/modproxy.tmp.pmo53y)
21:48:04.525578150 2 httpd (17039) < close res=0
21:48:09.498690963 1 httpd (17064) > close fd=11(<4t>10.0.2.2:58997->172.21.0.8:443)
21:48:09.498691856 1 httpd (17064) < close res=0
21:48:46.935441278 1 nginx (25919) > close fd=18(<4t>172.21.0.8:35834->172.21.0.9:80)
21:48:46.935442465 1 nginx (25919) < close res=0

So basically Apache closing some weird file and then the ssl socket. Then Nginx closes connection from Apache. Not very helpful. At least I know from what port Apache connects to Nginx so I'll grep for :35834? Turns out this is a bogus connection, not my POST file upload. When looking at syscalls it's very important to not get bogged down in irrelevant details.

So now I search backwards for POST /upload. This has turned out something very promising:

21:48:04.525366305 2 httpd (17039) > writev fd=13(<4t>172.21.0.8:35836->172.21.0.9:80) size=882
21:48:04.525401643 2 httpd (17039) < writev res=882 data=
POST /upload/ HTTP/1.1
Host: web
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://localhost/upload/
Content-Type: multipart/form-data; boundary=---------------------------156582199825391
Cookie: ...
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache
X-Forwarded-For: 10.0.2.2
X-Forwarded-Host: localhost
X-Forwarded-Server: localhost

21:48:04.525407958 2 httpd (17039) > mmap addr=0 length=4194304 prot=1(PROT_READ) flags=1(MAP_SHARED) fd=10(<f>/tmp/modproxy.tmp.pmo53y) offset=0
21:48:04.525417756 2 httpd (17039) < mmap res=7FE14C0AF000 vm_size=770340 vm_rss=14720 vm_swap=0
21:48:04.525420158 2 httpd (17039) > mmap addr=0 length=3093534 prot=1(PROT_READ) flags=1(MAP_SHARED) fd=10(<f>/tmp/modproxy.tmp.pmo53y) offset=4194304
21:48:04.525422012 2 httpd (17039) < mmap res=7FE13E4F9000 vm_size=773364 vm_rss=14720 vm_swap=0
21:48:04.525422762 2 httpd (17039) > writev fd=13(<4t>172.21.0.8:35836->172.21.0.9:80) size=7304273
21:48:04.525424849 2 httpd (17039) < writev res=-32(EPIPE) data=
Connection: Keep-Alive
Content-Length: 7304222

-----------------------------156582199825391
... tons of form data ...

Very interesting, the writev syscall error matches the initial error I've seen in the Apache logs (Broken pipe: [client 10.0.2.2:55239] AH01084: pass request body failed to 172.21.0.9:80). Can't be a simple coincidence.

So at this point it makes sense to look at the whole lifecycle of fd=13(<4t>172.21.0.8:35836->172.21.0.9:80). Now because FDs can easily have collisions (they get recycled, other processes can have a FD with same number) it's easier to just grep for the host and port (172.21.0.8:35836):

21:46:45.783868277 0 nginx (25919) > sendfile out_fd=19(<4t>172.21.0.8:35836->172.21.0.9:80) in_fd=20(<f>/var/app/static/bower_components/jquery-ui/themes/smoothness/images/ui-icons_cd0a0a_256x240.png) offset=0 size=4599
21:46:45.783927701 0 nginx (25919) > recvfrom fd=19(<4t>172.21.0.8:35836->172.21.0.9:80) size=1024
21:46:45.783936625 0 httpd (17039) > read fd=13(<4t>172.21.0.8:35836->172.21.0.9:80) size=8000
21:46:45.790029511 0 httpd (17039) > writev fd=13(<4t>172.21.0.8:35836->172.21.0.9:80) size=843
21:46:45.790072392 0 httpd (17039) > read fd=13(<4t>172.21.0.8:35836->172.21.0.9:80) size=8000
21:46:45.790090824 0 nginx (25919) > recvfrom fd=19(<4t>172.21.0.8:35836->172.21.0.9:80) size=1024
21:46:45.790214052 0 nginx (25919) > writev fd=19(<4t>172.21.0.8:35836->172.21.0.9:80) size=323
21:46:45.790219039 0 nginx (25919) > sendfile out_fd=19(<4t>172.21.0.8:35836->172.21.0.9:80) in_fd=20(<f>/var/app/static/bower_components/jquery-ui/themes/smoothness/images/ui-icons_888888_256x240.png) offset=0 size=7092
21:46:45.790298595 0 nginx (25919) > recvfrom fd=19(<4t>172.21.0.8:35836->172.21.0.9:80) size=1024
21:46:45.790309355 0 httpd (17039) > read fd=13(<4t>172.21.0.8:35836->172.21.0.9:80) size=8000
21:47:50.791539134 0 nginx (25919) > close fd=19(<4t>172.21.0.8:35836->172.21.0.9:80)
21:48:04.525366305 2 httpd (17039) > writev fd=13(<4t>172.21.0.8:35836->172.21.0.9:80) size=882
21:48:04.525422762 2 httpd (17039) > writev fd=13(<4t>172.21.0.8:35836->172.21.0.9:80) size=7304273
21:48:04.525467926 2 httpd (17039) > close fd=13(<4t>172.21.0.8:35836->172.21.0.9:80)

So two things to learn from this:

Nginx suddenly decides to close connection after 65 seconds. Note that it closes connection before Apache tries to send the request.

21:46:45.790298595 0 nginx (25919) > recvfrom fd=19(<4t>172.21.0.8:35836->172.21.0.9:80) size=1024
21:47:50.791539134 0 nginx (25919) > close fd=19(<4t>172.21.0.8:35836->172.21.0.9:80)
21:48:04.525366305 2 httpd (17039) > writev fd=13(<4t>172.21.0.8:35836->172.21.0.9:80) size=882
21:48:04.525422762 2 httpd (17039) > writev fd=13(<4t>172.21.0.8:35836->172.21.0.9:80) size=7304273
21:48:04.525467926 2 httpd (17039) > close fd=13(<4t>172.21.0.8:35836->172.21.0.9:80)

Apache will do two writes before realising connection is busted, something is funny there (looks like a bug, or some unrelenting optimization).

Apache is pipelining the proxied requests (note the sendfile for static files). Basically multiple HTTP requests and responses are sent/received on the same connection - an often tricky feature of HTTP 1.1.

So now, a wtf-moment.

In a situation like this the only way forward is to peel layers from the onion: remove code, disable components etc.

Since mod_ssl is the most annoying component around (we don't see any meaningful data in the sysdig logs because it's encrypted) we could try to disable that first. Easy enough, just comment out the SSL stuff and leave the port:

<VirtualHost *:443>
     # SSLEngine On
     # SSLCertificateFile ...
     # SSLCertificateKeyFile ...
     ...

Now we can send plain HTTP test requests to http://localhost:443/. It works why wouldn't it, and it also reproduces the problem.

Digging again in the logs (grep for POST /upload) yields:

09:22:24.700375109 0 httpd (653) > read fd=10(<4t>10.0.2.2:57454->172.20.0.6:443) size=8000
09:22:24.700381289 0 httpd (653) < read res=1460 data=
POST /upload/ HTTP/1.1
Host: localhost:443
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://localhost:443/upload/
Content-Type: multipart/form-data; boundary=---------------------------8182195208398
Content-Length: 8243581
Cookie: csrftoken=...
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache

-----------------------------8182195208398
...
150926 09:22:24.700635979 0 httpd (653) > read fd=10(<4t>10.0.2.2:57454->172.20.0.6:443) size=8000
150927 09:22:24.700638437 0 httpd (653) < read res=-11(EAGAIN) data=
150928 09:22:24.700642986 0 httpd (653) > read fd=10(<4t>10.0.2.2:57454->172.20.0.6:443) size=8000
150929 09:22:24.700644098 0 httpd (653) < read res=-11(EAGAIN) data=
150930 09:22:24.700645959 0 httpd (653) > poll fds=10:41 timeout=20875

Note the funny two subsequent reads failing with EAGAIN - another unrelenting optimization?

And some time later (depending on throttling settings) Apache decides it's time to send the headers:

09:23:49.386532541 3 httpd (653) > writev fd=11(<4t>172.20.0.6:56980->172.20.0.5:80) size=1132
09:23:49.386624313 3 httpd (653) < writev res=1132 data=
POST /upload/ HTTP/1.1
Host: web
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://localhost:443/upload/
Content-Type: multipart/form-data; boundary=---------------------------8182195208398
Cookie: csrftoken=...
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache
Proxy-Secret: 17610e1815bed35364ea65a8c723e10101b64e0c2e824fca6f67eb120b7b7fc6
X-Forwarded-For: 10.0.2.2
X-Forwarded-Host: localhost:443
X-Forwarded-Server: localhost

09:23:49.386660855 3 httpd (653) > writev fd=11(<4t>172.20.0.6:56980->172.20.0.5:80) size=8243632
09:23:49.386664671 3 httpd (653) < writev res=-32(EPIPE) data=

So basically what happens here is that out keepalive connection times out. Turns out there was a Nginx setting for this, the default being:

keepalive_timeout 75s;

Now you don't really want to increase that, you'd end up with lots of tied up sockets and fds in dead connections.

So the mystery remains: why does Apache only send out the request after buffering the whole request body?

Unfortunately this is only clarified by looking at the not exactly short mod_proxy documentation. The fix ...

SetEnv proxy-sendchunked

And now I wasted your time too 👿

[1]

Pro tip: use rg instead of grep. You probably know the impressive silversearcher (ag) but https://github.com/BurntSushi/ripgrep buries everything else.

However, having the output already in a smallish file I've just used less. Just press / (search) or > (jump to end) and ? (search backwards) and fire way your regex.

Rehashing the src layout

2017-09-25T00:00:00+03:00

Looking back at my Python packaging post, now it got more popular. Even Twisted is using it now but there's still confusion about it.

I mean, places that you'd expect would use a src-layout don't. Take flit for instance, I wrote a rant on packaging almost 1 year later, and that prompted Thomas Kluyver to make flit but strangely enough it uses the ad-hoc layout [1].

And there are plenty of guides on the Internet with the ol' ad-hoc layout.

So people are wondering: "I don't understand this packaging mumbo-jumbo, what should I use?"

It's not that complicated, just have these thoughts in mind when choosing:

If you use a project template (e.g.: cookiecutter) the choice of layout is not so important. The template should solve all the annoying packaging problems.

You should use a template anyway.
If you already have a project using the ad-hoc layout, and you don't have problems with testing or packaging then there's little value in switching over to a src-layout.
The ad-hoc layout is more convenient and that's why there's high resistance to src-layouts.

You need tooling to use the src-layout effectively. If you don't use a tool to manage virtualenvs and install your project like Tox then it's going to be painful.
Any layout will work. Mistakes can be corrected.

If this don't make sense (I agree it's a bit heavy on jargon and technicalities) and you don't want to use a template then perhaps you're looking to learn packaging the hard way. Just flip a coin and start with something.

[1]	Now people have already asked for flit to support the `src`-layout. I have even added my thoughts on the issue. Don't go over there and spam the issue with comments, I don't think there's anything new to add.

Dealing with Docker

2016-05-07T00:00:00+03:00

I'm giving Docker a shot again, after couple years, and this time I'm "dockerizing" an old Python app that don't even have configuration management or orchestration. Docker is easy to install now. No more bullshit about kernel features and filesystem support. They even claim it works seamlessly on OS X and Windows [*].

The good parts *

Docker has several things going for it: a good name, huge community, huge amount of images you can start from. Sure, most images are not great, even the official images can have issues but you can always take out the Dockerfile and customize things.

Right now it's hard to find an alternative. CoreOS is trying to make an alternative called "rkt". With a flikr-esque name at that! How do you even read it? ercati? erkit? rocket? rockit? rackit? reckit? You know what, wreckit. If you think I'm lambasting the name unfairly then consider that most of the world don't speak very good English and interpretable spelling is a problem.

Even if rkt didn't have such a terrible name, it still has a long way to reaching Docker's convenience.

What irks me *

Note

If it matters, I'm using Docker version 1.11.0. They call it the "Engine" now.

Docker has some strange oversights and quirks. There's a very strong focus on not breaking any interfaces, it's a bit depressing to look at the bug tracker if you're the impatient type.

These things annoy me:

Documentation has no search. Seriously? I understand that there's Google but come on, I don't want to look at docs for old Docker versions and other junk Google gives me.
Command line seem clumsy:
- Inflexible parsing, eg: docker build . --help ain't valid. That makes no sense, there's only a single non-option argument, why can't I have options after that single possible argument?
- Most arguments have short form (eg: -i) but not --help. Nooo, not that. No one needs a short form for that, God forbid!
- Bad help for most options. Is this supposed to help?
```
-v, --volume=[]                 Bind mount a volume
```
  Cause that sure as hell don't tell me anything about what I can pass in there. Now I have to go into the docs with no search. After rummaging trough 10 useless pages of documentation I eventually I get to this:
```
-v, --volume=[host-src:]container-dest[:<options>]
                      Bind mount a volume. The comma-delimited
                      `options` are [rw|ro], [z|Z],
                      [[r]shared|[r]slave|[r]private], and
                      [nocopy]. The 'host-src' is an absolute path
                      or a name value.
```
  What. Why can't that be in the command line?
If you got a run or build error in docker-compose you're usually left with half running containers and you have to manually cleanup the mess. If you don't pay attention you're left wondering why docker-compose build doesn't do anything. I did some changes, ran docker-compose build, why don't docker-compose up don't run new images? Cause some stuff is already running, that's why!

If you stop a docker-compose up all services stop. If you stop a docker-compose up myservice it leaves all the dependencies running around. Argh! Why the inconsistency?
There is no garbage collection. Leaves around huge piles of useless containers and images. Sure, there's docker rmi $(docker images -q) and docker rm $(docker ps -a -q) but that's like cleaning your computer dust with a water hose. It sure does the job but your computer probably don't work after that.

Dockerfiles are quirky and non-orthogonal. Similarly to command line interface, the Dockerfile syntax is a culmination of ill-advised choices.

Support for multiline values is an afterthought. Dockerfiles are littered with "\" and "&&" all over the place. Lots of noise, lots of mistakes.

There two ways to do the same thing. The parser takes JSON for most commands but if parse fails then it takes whatever gunk is in there as a string.

For example the CMD and ENTRYPOINT take this to the extreme: they have two exec mode. Take a look at this crazy table:

	No `ENTRYPOINT`	ENTRYPOINT exec_entry p1_entry	ENTRYPOINT ["exec_entry", "p1_entry"]
No `CMD`	error, not allowed	/bin/sh -c exec_entry p1_entry	exec_entry p1_entry
CMD ["exec_cmd", "p1_cmd"]	exec_cmd p1_cmd	/bin/sh -c exec_entry p1_entry exec_cmd p1_cmd	exec_entry p1_entry exec_cmd p1_cmd
CMD ["p1_cmd", "p2_cmd"]	p1_cmd p2_cmd	/bin/sh -c exec_entry p1_entry p1_cmd p2_cmd	exec_entry p1_entry p1_cmd p2_cmd
CMD exec_cmd p1_cmd	/bin/sh -c exec_cmd p1_cmd	/bin/sh -c exec_entry p1_entry /bin/sh -c exec_cmd p1_cmd	exec_entry p1_entry /bin/sh -c exec_cmd p1_cmd

There is clear overlap there. Maybe there are historical reasons for it but that's why we have version numbers. Why do we need to keep all that cruft?

No volumes/mounts during docker build. I have found ways to deal with this (read on) but still, it's an irk.

If you look in the bug tracker you'll notice that lots of people have come up with ideas to improve Dockerfiles, but these issues just don't seem important enough.

There's something rotten about how Docker run containers as root by default. Docker seems to mitigate this by disabling some capabilities like ptrace by default. But this ain't purely a security concern, it affects usability as well.

For example, once you install docker on your Ubuntu you're faced with a choice: use sudo all over the place or give your user full access to the daemon:
```
sudo usermod -aG docker $USER

# then relogin, for the change to take effect - obvious right?
```
Still not sure if that's better than a constant reminder that you're doing lots of stuff as root.

I know these might look like superficial complaints but they rile me up regardless. At least it's not this bad. </rant>

The ecosystem *

There are a lot of images, and you'll probably find anything you'll ever want. And there's even a set of specially branded images that pop up almost everywhere: the "official" images. They are pretty good: up to date, they verify signatures for whatever they download, consistent presentation etc.

Though I have a problem with the Python images. For some reason they decided to just compile Python themselves. Sure, it's the latest point-release but they don't include any of the patches the Debian maintainers made. I don't like those patches either (they customize package install paths and the import system) but it's worse without them:

Suddenly gdb stops working. Just try a docker run --rm -it python:2.7 sh:

# apt-get update
# apt-get install gdb
# gdb python
Traceback (most recent call last):
  File "/usr/lib/python2.7/site.py", line 563, in <module>
    main()
  File "/usr/lib/python2.7/site.py", line 545, in main
    known_paths = addusersitepackages(known_paths)
  File "/usr/lib/python2.7/site.py", line 272, in addusersitepackages
    user_site = getusersitepackages()
  File "/usr/lib/python2.7/site.py", line 247, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/usr/lib/python2.7/site.py", line 237, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/usr/lib/python2.7/sysconfig.py", line 582, in get_config_var
    return get_config_vars().get(name)
  File "/usr/lib/python2.7/sysconfig.py", line 528, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/usr/lib/python2.7/sysconfig.py", line 412, in _init_posix
    from _sysconfigdata import build_time_vars
  File "/usr/lib/python2.7/_sysconfigdata.py", line 6, in <module>
    from _sysconfigdata_nd import *
ImportError: No module named _sysconfigdata_nd

Looks like there's some import path trampling going on. I don't want broken debug tools when my app is broken.

You can't use any Python package from the APT repos. Sure, most of them are old and easy to install with pip, but there are exceptions [2].
Strange C.UTF-8 locale. I can understand they don't want to put a specific language in there but if you run any locale using applications you'll run into issues.

What I ended up was using the ubuntu:xenial image (Xenial being the new LTS). It ships latest point release for 2.7 so why compile it again? I took the good parts from python:2.7-slim and I got this Dockerfile:

FROM ubuntu:xenial

RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

RUN apt-get update \
 && apt-get install -y --no-install-recommends \
                    ca-certificates curl \
                    strace gdb lsof locate net-tools htop \
                    python2.7-dbg python2.7 libpython2.7 \
 && rm -rf /var/lib/apt/lists/*

ENV PYTHON_PIP_VERSION 8.1.1
RUN set -eEx \
 && curl -fSL 'https://bootstrap.pypa.io/get-pip.py' | python2.7 - \
         --no-cache-dir --upgrade pip==$PYTHON_PIP_VERSION

COPY .wheels /wheels

Everything works great, and there's some magic in those debug packages that make gdb give me some real nice commands like py-bt [3]. Note that I snuck in some other tools to help debugging.

All I'm saying there is that even official images need scrutiny. Check their Dockerfile and decide if it really fits your needs.

The challenges *

Docker has some interesting challenges, or limitations. The build system has something called "layers" and there's a hard limit of how many layers you can have. Each command in your Dockerfile makes a new layer. If you look at the "official" best practices guide you'll see most of the stuff there revolves around this limitation. You inevitably end up with some damn ugly RUN commands.

There's a good thing about these layers - they are cached. However, the context is not. Never one layer needs all the context, or the same part of the context. Layers should be able to have individual context, but alas, docker build wasn't designed with that in mind.

Another limitation by design is that docker build doesn't allow any mounts or volumes during build. The only way to get stuff in the container that eventually becomes the image is by network or by the "context".

What's this context? *

When you run docker build foobar Docker will make an archive of foobar/* (minus what you have in .dockerignore) and build an image according to what you have in foobar/Dockerfile. You can specify the context path and Dockerfile path individually but, never too odd, that Dockerfile must be in the context. You can't get creative here.

Optimizing the build process *

You can parametrize this build process but the lack of mounts or volumes exposes you to some pretty annoying slowness if you have to build external packages for example. This problem is still pervasive in Python, most of the stuff in PyPI is just source packages. Even if now you can publish Linux binaries on PyPI it's still years till most packages will publish those manylinux1 wheels [1]. Even if we'd have wheels for everything there's still the question of network slowness. Setting up caching proxies is inconvenient.

Most Dockerfiles I've seen have something like this:

RUN mkdir -p /app

COPY requirements.txt /
# slow as hell ...
RUN pip install -r /requirements.txt

COPY app /app

Now for simple projects this is fine, because you only have a handful of dependencies. But for a larger projects, hundreds of dependencies is order of the day. Changing them or upgrading versions (as you should always pin versions [4]) will introduce serious delays in build times. Because the container running the build process is pretty insulated (no volumes or mount remember?) pip can't really cache anything.

Staging the build process *

A way to solve this is having a "builder image" that you run to build wheels for all your dependencies. When you run an image you can use volumes and mounts.

Before jumping in, lets look briefly at file layout. I like to have a docker directory and then another level for each kind of image. Quite similar to the layout the builder for official images have. And no weird filenames, just Dockerfile everywhere:

docker
├── base
│   └── Dockerfile
├── builder
│   └── Dockerfile
├── deps
│   └── Dockerfile
├── web
│   └── Dockerfile
├── worker
│   └── Dockerfile
└── ...

In this scenario we'd deploy two images: web and worker. The inheritance chain would look like this:

buildpack-deps:xenial ➜ builder
ubuntu:xenial ➜ deps ➜ base ➜ web
ubuntu:xenial ➜ deps ➜ base ➜ worker

In which:

builder has development libraries, compilers and other stuff we don't want in production.
deps only has python and the dependencies installed.
base has the source code installed.
web and worker have specific customizations (like installing Nginx or different settings).

And in .dockerignore we'd have:

# just ignore that directory, we don't need that stuff when we have "." as the context
docker

This layout might seem artificial but not quite:

Both the worker and web need the same source code.
The deps and base are not in the same image because their contexts are distinct: one needs a bunch of wheels and the other one only needs the sources. This setup allows us to skip building the deps image if the requirement files did not change.
The web and worker images do not need to have the source code in the context. This allows faster build times. For development purposes we can just mount the sourcecode. More about that later.

In builder/Dockerfile there would be:

# we start from an image with build deps preinstalled
FROM buildpack-deps:xenial

# seems acceptable for building, note the notes above
# about C.UTF-8 - it's not really good for running apps
ENV LANG C.UTF-8

# we'd add all the "-dev" packages we need here
RUN apt-get update \
 && apt-get install -y --no-install-recommends \
                    python2.7 python2.7-dbg python2.7-dev libpython2.7 \
                    strace gdb lsof locate \
 && rm -rf /var/lib/apt/lists/*

ENV PYTHON_PIP_VERSION 8.1.1
RUN set -eEx \
 && curl -fSL 'https://bootstrap.pypa.io/get-pip.py' | python2.7 - \
         --no-cache-dir --upgrade pip==$PYTHON_PIP_VERSION

ARG USER
ARG UID
ARG GID

# we set some default options for pip here
# so we don't have to specify them all the time

# this will make pip additionally look for available wheels here
ENV PIP_FIND_LINKS=/home/$USER/wheelcache
# and this is the default output dir when we run `pip wheel`
ENV PIP_WHEEL_DIR=/home/$USER/wheelcache
ENV PIP_TIMEOUT=60
# one network request less, we don't care about latest version
ENV PIP_DISABLE_PIP_VERSION_CHECK=true

RUN echo "Creating user: $USER ($UID:$GID)" \
 && groupadd --system --gid=$GID $USER \
 && useradd --system --create-home --gid=$GID --uid=$UID $USER \
 && mkdir /home/$USER/wheelcache

WORKDIR /home/$USER

The interesting part here is the USER, UID and GID build arguments. Unless you do something special the processes inside the container runs with root user. This is fine, right? That's the whole point of using a container, processes in the container actually have all sort of limitations - so it don't matter what user runs inside. However, if you mount something from the host inside the container then the owner any new file inside that mount is going to be the same user that the container runs with. The result is that you're going to get a bunch of files owned by root in the host. Not nice.

Because I don't do development with a root account and because user namespaces are surprisingly inconvenient to use [6] I have resorted to recreating my user inside the container. It needs to have the exact uid and git, otherwise I get files owned by an account that don't exist.

Similarly to what was shown before, deps/Dockerfile would have:

FROM ubuntu:xenial

RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

RUN apt-get update \
 && apt-get install -y --no-install-recommends \
                    ca-certificates curl \
                    strace gdb lsof locate net-tools htop \
                    python2.7-dbg python2.7 libpython2.7 \
 && rm -rf /var/lib/apt/lists/*

ENV PYTHON_PIP_VERSION 8.1.1
RUN set -eEx \
 && curl -fSL 'https://bootstrap.pypa.io/get-pip.py' | python2.7 - \
         --no-cache-dir --upgrade pip==$PYTHON_PIP_VERSION

COPY .wheels /wheels
RUN pip install --force-reinstall --ignore-installed --upgrade \
                --no-index --use-wheel --no-deps /wheels/* \
 && rm -rf /wheels

And base/Dockerfile:

FROM app-deps

# copy the application files and add them on the import path
RUN mkdir /app
WORKDIR /app
COPY setup.py /app/
COPY src /app/src
# there are other ways (like .pth file) but this one allows
# for easy setup of various bin scripts app might need
RUN python2.7 setup.py develop

# create an user for the application and install basic tools
# to change user (pysu) and wait for services (holdup)
ARG USER=app
ENV USER=$USER
RUN echo "Creating user: $USER" \
 && groupadd --system  $USER \
 && useradd --system --create-home --gid=$USER --base-dir=/var $USER \
 && pip install pysu==0.1.0 holdup==1.0.0 \
 && pysu $USER id
# this last one just tests that pysu works

For web/Dockerfile we can have something like:

FROM app-base

RUN apt-get update \
 && apt-get install -yq --no-install-recommends nginx-core supervisor \
 && rm -rf /var/lib/apt/lists/*

COPY site.conf /etc/nginx/sites-enabled/default
COPY supervisor.conf /etc/supervisor/conf.d/
COPY entrypoint.sh /

RUN echo "daemon off;" >> /etc/nginx/nginx.conf

EXPOSE 80
CMD ["/entrypoint.sh"]

To build the images we can run this:

#!/bin/sh
set -eux

docker build --tag=app-builder \
             --build-arg USER=$USER \
             --build-arg UID=$(id --user $USER) \
             --build-arg GID=$(id --group $USER) \
             docker/builder
# we run this image two times, once to prime a wheel cache
mkdir -p .dockercache
docker run --rm \
           --user=$USER \
           --volume="$PWD/requirements.txt":/requirements.txt:ro \
           --volume="$PWD/.dockercache":/home/$USER \
           app-builder \
           pip wheel --requirement=/requirements.txt
# and the second time to create the final wheel set
rm -rf "$PWD/docker/deps/.wheels"
mkdir  "$PWD/docker/deps/.wheels"
docker run --rm \
           --user=$USER \
           --volume="$PWD/requirements.txt":/requirements.txt:ro \
           --volume="$PWD/.dockercache":/home/$USER \
           --volume="$PWD/docker/deps/.wheels":/home/$USER/wheels \
           app-builder \
           pip wheel --wheel-dir=wheels \
                     --requirement=/requirements.txt
# and now there are going to be tons of wheels in "docker/deps/.wheels/"
docker build --tag=app-deps docker/deps

docker build --tag=app-base --file=docker/base/Dockerfile .
# this is why we simply ignore "docker/" in .dockerignore -- it
# would inflate the context a lot and we don't need the wheels in this step

docker build --tag=app-web docker/web

If you look close enough you'll notice a small flaw here: rebuilding the base image will invalidate everything that depends on it. Because the app-base image has the code in the context it will get invalidated often enough to be annoying. However, there's a solution for that, and lets not forget the whole point of this - we can produce a large fleet of images cheaply (context only includes docker/<kind>). This is quite good if you have a huge codebase and a large number of image types (in addition to app-web and app-worker you could have various other services).

Doing development *

For development we don't really need to rebuild the app-base image (which includes the sources) - we can just mount our project checkout in the right place. This could work for development:

# no need to rebuild the base image (that has
# the /app/src) if we just mount it anyway
docker run --volume=$PWD/src:/app/src:ro app-web

Enter `docker-compose` *

A nicer way to do it, especially if you depend on services like a database, is with docker-compose - all you need is a docker-compose.yml file like this:

version: '2'
services:
  web:
    build: 'docker/web'
    image: 'app-web'
    ports:
    - '8080:80'
    volumes:
    - '${PWD}/src:/app/src:ro'
    links:
    - 'pg'
    environment:
      DATABASE_HOST: 'pg'
  pg:
    image: 'postgres:9.5'
    environment:
      POSTGRES_DB: 'app'
      POSTGRES_USER: 'app'
      POSTGRES_PASSWORD: 'app'

With this configuration we solve the problem with the app-base that was outlined earlier: note the '${PWD}/src:/app/src:ro' mount.

Living inside the container *

Working inside the container is not easy. Take debugging for example - normally you'd expect strace and gdb to just work but they require some privileges that are disabled by default. Thus this pattern appears:

docker run --privileged --rm -it myimage bash
# or, if you want to debug a running container
docker exec --privileged -it mycontainer bash

While strace can work from the outside gdb is not feasible to use from the host due to the library paths being mismatched. It's unlikely that you have exact OS on the host, so you will lack the sorely needed debug symbols.

Avoiding `bash` *

Turns out this is a thing, for example Alpine don't have bash by default. You can still install it but if you want the smallest possible images then you should avoid it.

My rule of thumb here is: if at any point you'll want to have an interactive session in a container just install it. Space can't be be more important that tooling.

Container dependencies *

A frequent problem with dependencies is that startup is slow. Take for instance the pg service shown in the docker-compose.yml example: it can take few seconds for it to start the first time due to setup. Then the web service will fail to start if it needs to connect to the database at startup (e.g.: migrations).

People definitely want a solution for this. Currently there are 3 ways to handle this issue:

Use one of Docker's restart policies. This might be a bit too coarse.
Use your orchestration tool's healthcheck features to handle this. This might be incovenient, for example docker-compose don't have this feature. Kubernetes has it but it's a bit more involved.
Make your container wait a bit for services by doing basic port or health checks. There are some recommendations here, however, none of them support unix domain sockets so I made my own tool: holdup.

Process handling *

An easy mistake is not using exec in the shell scripts you'll inevitably need. If you forget to do this then your container will fail to stop correctly and Docker will be forced to send a SIGKILL. This can lead to data loss. If your PID 1 is something like /bin/bash or /bin/sh, or you don't have any process with PID 1 then you have this problem.

Suppose you have a CMD ["script.sh"] or ENTRYPOINT ["script.sh"]. Then:

#!/bin/sh

# general good practice (stop on error, missing variables, verbose mode):
set -eux

# avoid this:
pure-ftpd

# do this instead:
exec pure-ftpd

To make it worse, the su bin doesn't support an "exec"-mode. Thus tools like gosu [5] were created.

Another problem are sloppy applications that don't cleanup after child processes. Containers don't have a "init" process by default - so there's no "catch-all" for zombie processes. If you have that sort of application then your container will inevitably exhaust all available PIDs. You can work around this with minimal init systems like pidunu but a better solution is to fix your application's code - it only needs a os.waitpid(-1, WNOHANG) [7] in the right place.

Lack of standard services *

This is a surprise problem, especially if you grew on well established services like syslog or cron.

For example pure-ftpd has an unhealthy marriage to syslog. You can either:

Get the source package, patch/configure it to log to stdout and recompile. Unfortunatelly this results in a bloated image, unless you inconvenience yourself with layer merging tools or using a builder image.

There may be a distro having a package with all the right compile options but I haven't found it.
Mount a syslog socket in the container (example: /var/log). But this goes against the grain of Docker managing the logs. Good luck finding what container emitted the messages.

Run pure-ftpd and syslogd with supervisord or similar. You'd need a Dockerfile like:

FROM ubuntu:xenial
RUN apt-get update \
 && apt-get install -y --no-install-recommends supervisor inetutils-syslogd \
 && rm -rf /var/lib/apt/lists/* \
 && echo '*.* /dev/stdout' > /etc/syslog.conf
COPY supervisor.conf /etc/supervisor/conf.d/

And a supervisor.conf like:

[program:syslog]
command=syslogd -n --no-klog
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
redirect_stderr=true
priority=1

[program:mystuff]
command=mystuff

But this creates some friction with the way things should be run in Docker: should mystuff fail to run, Docker won't know.

Run something like:

# just forward everything to stdout
socat -u unix-recv:/dev/log - &

exec pure-ftpdexec

And hope socat never dies ...

Cron is another kind of I don't need this issue. In addition to Cron requiring a /dev/log (syslog), your cron jobs are ran with a clean environment. All those environment variables are gone unless you have a contraption like this:

Dump all the env vars somewhere: gosu myuser env > /saved-environ
And then in the cron job, make sure to load that: env - `cat /saved-environ` python ... >/dev/null 2>&1

Alternatively you could (depending on what base image you use) dump all the env vars in this location: gosu myuser env > /etc/environment

Output redirection is another issue with Cron, you can't simply log to stdout, so your only resort is, guess what, syslog: 2>&1 | logger -t myjob.

Is that all you say? Of course not. Cron also wants a mailer. You probably don't want that so you add a MAILTO="" in your cron config.

Managing configuration *

Turns out Docker doesn't have anything builtin for this. You either have to use external services to store configuration and orchestrate restarting the containers, or rely on volumes. Using volumes has some limitations but it seems way more approachable.

The gist of this is having an image just for configuration, a docker/conf/Dockerfile like this:

FROM alpine:3.3

RUN mkdir -p /etc/app/conf /etc/app/conf-update

# this will be the default data, if volume wasn't created
COPY settings.conf /etc/app/conf/
VOLUME /etc/app/conf

# this second copy is used for updating
COPY settings.conf /etc/app/conf-update

COPY update-conf.sh /
# similarly to a cp command
ENTRYPOINT ["/update-conf.sh", "/etc/app/conf-update", "/etc/app/conf"]

This update-conf.sh script is the difference to Jeff Nickoloff's example - it logs what gets changed:

#!/bin/sh
set -eu
echo "Updating configuration in $2 with files from $1"

for to_update in $1/*; do
    name="${to_update##*/}"
    echo "Updating: $name"

    echo "* diff from: $to_update to: $2/$name"
    diff -U 5 "$to_update" "$2/$name"
    echo "* copying: $to_update to: $2/$name"
    cp "$to_update" "$2/$name"
done

echo Success. Configuration updated!

Then to use this you just link the volumes (assuming docker-compose):

conf:
  image: 'app-conf'
  build: 'docker/conf'
web:
  ...
  volumes_from:
  - 'conf:ro'

This is all I got so far, more in a future article!

[*]	Haven't tried it on Windows but it does use VirtualBox so I expect a painfully slow experience.

[1]	Wheels are binaries for Python packages.

[2]	Try building GUI library bindings or pysvn. Enter a world of pain.

[3]	`py-bt` prints the callstack of Python code, while `bt` prints the C call stack which don't show python line numbers or function names.

[4]

Pinning the versions is a best practice because:

Even minor/patch releases can introduce regressions.
They allow faster installs: pip doesn't need to find all the available versions every time, there are less network requests and generally shorter path to a cache hit.

You can read more about it here, here or here.

[5]	If you already use Python and don't want an extra 1.7Mb binary use pysu instead.

[6]	You need to configure your Docker daemon and `subuid`/`subgid` (another system-wide thing). User namespace is a security feature for deployment, not development. Imagine having to configure that on every developer's machine, good grief!

[7]	See: https://docs.python.org/3/library/os.html#os.waitpid

Notes on debugging

2016-02-18T00:00:00+02:00

Debugging is a favourite subject of mine. It's an incredibly interesting area of programming but unfortunately not always a trivial one. It's also very important - it plays a significant part in programmer's productivity. No matter how many QA tools you have at your disposal, and no matter how hard you try to create bug-free code you will need to do debugging. It's just one of the intractable facets of programming.

Here's my attempt to define some concepts and guidelines. What are the tenets of debugging?

Find the cause first *

This is the most important, for many reasons. Yes, some issues can be quite hard to understand but with time, as you build up knowledge, it gets easier. If you just try random changes (shotgun debugging) in hopes that it will fix the problem then you're in fact depriving yourself of knowledge and assurance that you actually fixed the problem. It may seem easier, especially if you get lucky but overall you'll waste inordinate amounts of time with that "technique". Focus on finding the cause first.

Down the rabbit hole *

Unfortunately you have to accept the harsh reality that there are no shortcuts:

You'll have to read lots of code to understand what's going on. Yes, that includes other peoples code. You're using a framework? You'll have to read the code.

Lots of people avoid this because it's hard and many frameworks are complicated. But understanding your libraries will pay off.
You won't see the problem by merely looking at the code [1] so you'll have to inspect the execution of the code and reason about it.
Not understanding the cause can be grueling and frustrating; it can take a while. But once you figure it out it's pleasant. It's totally worth the pain, plus next time it get easier - the accumulated knowledge helps. Don't give up!

Rooting it out *

Assuming you got the right tools, there are various "techniques" you can use.

Simpler programs are easier to understand than complicated programs with lots of moving parts.

One of the best approaches is to isolate the issue. Some call it divide & conquer. The idea is to reduce the amount of code you need to comprehend. It's similar to making a very good bug report:

Remove code or don't run code that's unnecessary or unrelated. If you're not sure, take it out. If the issue doesn't reproduce anymore then you know what to focus on. Just remove half of the code till you see where the problem is.

For example, you'd do it when execution stops at some unknown point and you need to figure out where it is. It may indicate you don't have good tooling (like an adequate tracer).
If you're having a regression you can use git bisect (or hg bisect) to find the offending commit. That tool will run a command of your choosing till the first commit that makes it fail is found.
Isolate any environment issues. Try a different OS, library or Python version. Note that this can be misleading - if the problem doesn't reproduce with a different Python it doesn't mean that the issue is in the Python interpreter - it may very well be an issue with your code (example: misusing some API).

A more risky way to approach things is "proving the hypothesis": you make some theory about what might be wrong and then test it. There are some pitfalls:

It's only good if you got some intuition in the problem space. Otherwise you'd practically be doing shotgun debugging.
If you find a way to make the bug disappear but don't understand why do not stop there. Understanding the cause is paramount.

Another way is the "secret police" method: assume everything is bad, vet any component, including external ones. Usually this is a "last resort" thing for crazy issues.

Plan for disaster *

Everything is easier when you're prepared. When you write code, or choose what code to reuse have in mind that one fateful day you will have to deal with the choices you've made. There are two aspects you should consider.

Tooling *

Think about what tooling and information you'll have available when you need to:

Do post-mortem analysis. Things like logs, core dumps, stacktraces and other details you might not have access to otherwise.
Inspect process state (memory, object, files, sockets and other resources).
Trace or step through execution.
Quickly change code and run it again.
Inspect code (jump to definition, usage search)
Look at logs. Reconfigure logging easily to get the firehose [2] when you need it.

Some design choices make some tools harder to use. Examples:

Highly dynamic code make inspecting code harder. Not to mention reasoning about it ...
Some tools don't work with threads or other concurrency implementations like gevent or eventlet.
Some tools may depend on threads, which may interfere with other libraries (like gevent or eventlet).
Hardcoded configuration for logging.
Availability of debug builds for various things. Example: can you deploy your Python app with a debug build of the interpreter?
Can you install tools or dependencies easily?
Can you elevate permissions for debugging purposes? Various tools may need root access or special permissions.

Code *

Most certainly you'll have to read code, possibly not your own code. Factor that in when writing code or when choosing a framework or library, along other factors like documentation quality, number of users and how well it is maintained.

Code you don't understand might as well be technical debt!

Have these ideas in mind when writing/reviewing code, or when choosing what library to use:

If it's big, complicated and hard to understand it will cost you. Plan the time you'll need to spend understanding it. It's surprising how much time you'll waste on a library that's advertised to save time.

It's not just the effort to understand the internals. Inordinate amounts of time are often wasted on shoehorning slightly different (but unplanned for) use cases. Big fancy frameworks optimize for typical use-cases and make everything else hard to accomplish. There are exceptions of course, but rare.
A community can help a lot. Having a place to ask for help means that you don't need that much upfront knowledge.
Code quality. Less bugs means less debugging right?

There are many practices but here are few ones:
- Code reviews.
- Good exception handling. Don't discard or cripple exceptions. Give descriptive errors that indicates what needs to be fixed.
  
  Good error handling makes debugging more predictable, and less of a bottomless hole to sink your time in.

Plan for disaster, but live dangerously *

Safety don't present the same opportunities for acquiring knowledge. It's often necessary to do more tricky things, at least for the sake of getting the information you need while debugging, if not for figuring why that metaclass-using ORM don't work as you expect.

For example, you wouldn't normally do monkey-patching in your code, but maybe it's an option when otherwise you'd have to patch up lots of files.

On the other hand, experimentation is prone to failure so there's the open question of where's a good place to experiment ...

Ask for help but understand *

You got a bug, you ask your colleague and 5 minutes later it's sorted out. Don't leave it at that. Ask how and why. Understand the process used to find the case. Identify best practices if possible.

Integration tests pay off *

The interesting thing about tests it's not that they make debugging easier, it's that they force you to resolve bugs early. They give you a warning early so you can fix it at a convenient time. Compare that to a production issue that you gotta fix on a Friday night.

To make an analogy: it's like owning a car - you'd want to get warning indicators and service it at an opportune time rather than having it break down in the middle of nowhere.

Integration tests, and not unit tests because they reach out those areas between various components (interactions) where the nasty bugs are in. Simply put, production won't run just a function, it will a combination thereof. Debate about what's the right balance of unit tests vs functional tests still rages on, but you need to be aware that you can't simply go for a pure unit test suite and hope for the best.

Other peoples misery is gold *

Reading what issues other people had [3] is pure gold. There is simply no way you can accumulate all that knowledge only from the software you develop yourself. It's a huge boost to your intuition as well. Plus you won't feel that bad when you have a problem - there's always someone you had it worse.

If you have the time you can also watch videos or read other peoples blogs. Every so often there something describing an interesting problem or various obscure but useful details.

[1]	If you do code reviews that is ...

[2]	firehose noun Something that carries a crushing stream of `DEBUG` level logging messages.

[3]	Make sure you look over the other lists too: https://github.com/danluu/post-mortems#other-lists-of-postmortems

Memory use and speed of JSON parsers

2015-11-22T00:00:00+02:00

Note

TL;DR: In decode oriented use-case with big payloads JSON decoders often use disproportionate amounts of memory. I gave up on JSON and switched to Msgpack.

You should draw your own conclusions by running the test code yourself.

⸻

Based on various feedback [*] I've did the benchmarks again, using ru_maxrss instead of Valgrind and with few more implementations.

Updated results: *

I have a peculiar use-case where I need to move around big chunks of data (some client connects to some HTTP API and gets some data). For whatever reason [1], JSON was chosen as the transport format. And one day that big chunk became very big - around few hundred megabytes. And it turned out that processes doing the JSON decoding were using lots of RAM - 4.4GB for a mere 240MB JSON payload? Insane. [2]

I was using the builtin json library, and the first thing I thought - "there must be a better JSON parser". So I've started measuring ...

Now measuring memory usage is a tricky thing, you can look at ps or look around in /proc/<pid> but you'd get very coarse snapshots and would be very hard to find out the real peak usage. Luckily enough Valgrind can instrument any program to track allocations (as opposed to recompiling everything to use a custom memory allocator) and it has a really nice tool called massif.

So I've started building a little benchmark using Valgrind. My input looks like this:

{
    "foo": [{
        "bar": [
            'A"\\ :,;\n1' * 20000000,
        ],
        "b": [
            1, 0.333, True,
        ],
        "c": None,
    }]
}

That generates a 240MB JSON with a structure pretty close to my app's problematic data.

Running valgrind --tool=massif --pages-as-heap=yes --heap=yes --threshold=0 --peak-inaccuracy=0 --max-snapshots=1000 ... for each parser gets me something like this on Python 2.7 (scroll down for results on Python 3.5):

Peak memory usage (Python 2.7):

           cjson:   485.4 Mb
       rapidjson:   670.5 Mb
            yajl: 1,199.2 Mb
           ujson: 1,862.0 Mb
        jsonlib2: 2,882.7 Mb
         jsonlib: 2,884.2 Mb
      simplejson: 2,953.6 Mb
            json: 4,397.9 Mb

Would you look at that. Now you can argue that my sample data is crazy but sadly, but that's just how my data looks sometimes. Few of the strings blow up to horrid proportions once in a while.

json has a severe weakness here, it needs a dozen more times memory than the input. WAT.

cjson is right there in my face, begging me to use it. There are some rumours that it has VeryBadBugs™ [6] but I think the lack of a bug tracker is what makes that project ultimately unappealing.

rapidjson seems to be a new player [3], however the Python 2 binding seems to have some gaps in essential parts. Still, it's interesting to at least get an idea of how it performs. The Python 3-only binding looks more mature, but sadly this app only run on Python 2 right now.

yajl and ujson appear to be mature enough but they simply still use lots of memory. There must be a better way ...

It looks like whatever I choose it's bad. There's a very good proverb [†] that applies here:

Best solution to problem is not having the problem in the first place.

Remember that time a customer asked for a thing but in fact he only needed something simpler and less costly. Talking through requirements and refining them solves lots of problems right there. This is that kind of situation. I wish I had realized I don't really need JSON at all sooner ...

I have to do more changes to switch the format of the HTTP API but that can't be worse than maintaining/fixing the cjson or rapidjson bindings myself.

If we try msgpack (and some old friends [‡], just for kicks) we get this:

Peak memory usage (Python 2):

          pickle:   368.9 Mb
         marshal:   368.9 Mb
         msgpack:   373.2 Mb
           cjson:   485.4 Mb
       rapidjson:   670.4 Mb
            yajl: 1,199.2 Mb
           ujson: 1,862.0 Mb
        jsonlib2: 2,882.7 Mb
         jsonlib: 2,884.2 Mb
      simplejson: 2,953.6 Mb
            json: 4,397.9 Mb

If you look at the test code you'll notice that I use msgpack with very specific options. Because the initial version of Msgpack wasn't very smart about strings (it had a single string type [5]) some specific options are needed:

msgpack.dumps(obj, use_bin_type=True) - use a different type for byte-strings. By default Msgpack will lump all kinds of strings into same type and you can't tell what the original type was.

On Python 2:
- str goes into the bin type
- unicode goes into the string type
On Python 3:
- bytes goes into the bin type
- str goes into the string type
msgpack.loads(payload, encoding='utf8') - decode the strings (so you get unicode back).

What about speed? *

Using pytest-benchmark we get this [5]:

Speed (Python 2.7):

    -----------------------------------------------
    Name (time in ms)              Min
    -----------------------------------------------
    test_speed[marshal]           59.2630 (1.0)    
    test_speed[pickle]            59.4530 (1.00)   
    test_speed[msgpack]           59.7100 (1.01)   
    test_speed[rapidjson]        443.0561 (7.48)   
    test_speed[cjson]            676.6071 (11.42)  
    test_speed[ujson]            681.8101 (11.50)  
    test_speed[yajl]           1,590.4601 (26.84)  
    test_speed[jsonlib]        1,873.3799 (31.61)  
    test_speed[jsonlib2]       2,006.7949 (33.86)  
    test_speed[simplejson]     3,592.2401 (60.62)  
    test_speed[json]           5,193.2762 (87.63)  
    -----------------------------------------------

Only the minimum time is shown. This is intentional - run the test code on your own hardware if you care about anything else.

Python 3 *

This app where I had the issue runs only on Python 2 for a very good (and also sad) reason. But no reason to dig myself further into a hole - gotta see how this performs on the latest and greatest. It will get ported one day ...

Peak memory usage (Python 3.5):

         marshal:   372.1 Mb
          pickle:   372.9 Mb
         msgpack:   376.6 Mb
       rapidjson:   668.6 Mb
            yajl:   687.3 Mb
           ujson: 1,578.9 Mb
            json: 3,422.3 Mb
      simplejson: 6,681.4 Mb

Speed (Python 3.5)

    -----------------------------------------------
    Name (time in ms)              Min
    -----------------------------------------------
    test_speed[msgpack]           69.0613 (1.0)    
    test_speed[pickle]            69.9465 (1.01)   
    test_speed[marshal]           74.9914 (1.09)   
    test_speed[rapidjson]        337.5243 (4.89)   
    test_speed[ujson]            902.8647 (13.07)  
    test_speed[yajl]           1,195.4298 (17.31)  
    test_speed[json]           4,404.9523 (63.78)  
    test_speed[simplejson]     6,524.9919 (94.48)  
    -----------------------------------------------

No cjson or jsonlib on Python 3. I don't even know what's the story behind jsonlib2. Looks like Msgpack is a safe bet here.

Different kind of data *

Now this is highly skewed towards some might call a completely atypical data shape. So I advise you take the test code and run the benchmarks with your own data.

But if you're lazy here are some results with different kinds of data, just to get an idea of how much the input can change memory use and speed.

Lots off small objects *

The 189MB citylots.json gets us wildly different results.

It appears simplejson works way better on small objects, and json is quite improved on Python 3:

Peak memory usage (Python 2.7):

      simplejson: 1,171.7 Mb
           cjson: 1,304.2 Mb
         msgpack: 1,357.2 Mb
         marshal: 1,385.2 Mb
            yajl: 1,457.1 Mb
            json: 1,468.0 Mb
       rapidjson: 1,561.6 Mb
          pickle: 1,854.1 Mb
        jsonlib2: 2,134.9 Mb
         jsonlib: 2,137.0 Mb
           ujson: 2,149.9 Mb

Peak memory usage (Python 3.5):

         marshal:   951.0 Mb
            json: 1,059.8 Mb
      simplejson: 1,063.6 Mb
          pickle: 1,098.4 Mb
         msgpack: 1,115.9 Mb
            yajl: 1,226.6 Mb
       rapidjson: 1,404.9 Mb
           ujson: 2,077.6 Mb

Speed:

Speed (Python 2.7):

    -----------------------------------------------
    Name (time in ms)              Min
    -----------------------------------------------
    test_speed[marshal]         3.9999 (1.0)    
    test_speed[ujson]           4.2569 (1.06)   
    test_speed[simplejson]      5.1105 (1.28)   
    test_speed[cjson]           5.2355 (1.31)   
    test_speed[msgpack]         5.9742 (1.49)   
    test_speed[yajl]            6.1059 (1.53)   
    test_speed[json]            6.3822 (1.60)   
    test_speed[jsonlib2]        6.7880 (1.70)   
    test_speed[jsonlib]         6.9587 (1.74)   
    test_speed[rapidjson]       7.4734 (1.87)   
    test_speed[pickle]         18.8649 (4.72)   
    -----------------------------------------------

Speed (Python 3.5):

    -----------------------------------------------
    Name (time in ms)              Min
    -----------------------------------------------
    test_speed[marshal]        1.1784 (1.0)    
    test_speed[ujson]          3.6378 (3.09)   
    test_speed[msgpack]        3.7226 (3.16)   
    test_speed[pickle]         3.7739 (3.20)   
    test_speed[rapidjson]      4.1379 (3.51)   
    test_speed[json]           5.1150 (4.34)   
    test_speed[simplejson]     5.1530 (4.37)   
    test_speed[yajl]           5.9426 (5.04)   
    -----------------------------------------------

Smaller data *

The tiny 2.2MB canada.json, again, gives us very different results. Memory use becomes irrelevant:

Peak memory usage (Python 2.7):

         marshal:    35.2 Mb
           cjson:    38.9 Mb
            yajl:    39.0 Mb
            json:    39.3 Mb
         msgpack:    39.5 Mb
      simplejson:    40.5 Mb
          pickle:    42.1 Mb
        jsonlib2:    47.4 Mb
       rapidjson:    48.5 Mb
         jsonlib:    48.8 Mb
           ujson:    50.9 Mb

Peak memory usage (Python 3.5):

         marshal:    38.3 Mb
          pickle:    40.4 Mb
            yajl:    42.1 Mb
            json:    42.2 Mb
         msgpack:    42.7 Mb
      simplejson:    45.3 Mb
       rapidjson:    52.3 Mb
           ujson:    55.5 Mb

And speed is again different:

Speed (Python 2.7):

    -----------------------------------------------
    Name (time in ms)              Min
    -----------------------------------------------
    test_speed[msgpack]         12.3210 (1.0)    
    test_speed[marshal]         15.1060 (1.23)   
    test_speed[ujson]           19.8410 (1.61)   
    test_speed[json]            48.0320 (3.90)   
    test_speed[cjson]           48.6560 (3.95)   
    test_speed[simplejson]      52.0709 (4.23)   
    test_speed[yajl]            62.1090 (5.04)   
    test_speed[jsonlib2]        81.6209 (6.62)   
    test_speed[jsonlib]         83.2670 (6.76)   
    test_speed[rapidjson]      102.3500 (8.31)   
    test_speed[pickle]         258.6429 (20.99)  
    -----------------------------------------------

Speed (Python 3.5):

    -----------------------------------------------
    Name (time in ms)              Min
    -----------------------------------------------
    test_speed[marshal]        10.0271 (1.0)    
    test_speed[msgpack]        10.2731 (1.02)   
    test_speed[pickle]         17.2853 (1.72)   
    test_speed[ujson]          17.7634 (1.77)   
    test_speed[rapidjson]      25.6136 (2.55)   
    test_speed[json]           54.8634 (5.47)   
    test_speed[yajl]           58.3519 (5.82)   
    test_speed[simplejson]     65.0913 (6.49)   
    -----------------------------------------------

I suppose better use of freelists happens here?

Bottom line *

Both speed and memory are affected by data shape. Speed is not always proportional to memory use.

Again, don't trust the numbers, run the benchmarks yourself, with your own data. Even if your data is identical in shape your hardware might behave differently than mine. Even the memory use can be different on your machine (example: different architecture, different shared libraries). And what's the chance your data has the exact shape as whatever was used in the benchmark?

Test setup:

Ubuntu 14.04 (a VM on an unoccupied host, at least in theory)
Sandy Bridge i7 (TurboBoost off, but frequency scaling was on)
Python 2.7.6
Python 3.5.0
Valgrind 3.11.0

As you can see the setup is less then perfect. If you really care you'd run the test code yourself.

[1]	I believe I've fallen victim to cargo-culting ...

[2]	But not more insane than Linux doing it's memory overcommit thing and swapping everything. You can dismiss this as an ignorant remark about the kernel but bottom line is the same: trashing the swap is very bad.

[3]	I've noticed rapidjson in this C++ JSON library benchmark. It seems it was the only library with a working Python binding.

[4]	The fact that there are lots of languages you can use Msgpack on and they don't have an Unicode type doesn't help either.

[5]	(1, 2) Generated with something like `script -c tox \| ansi2html`.

[6]

It seems that no one can really point out what's wrong with cjson. The author of jsonlib even did this FUD post and it seems people have been buying that bullshit. StackOverflow is practically plastered with transcriptions of that.

He may be right, and there are actual issues with encoding and conformance in cjson, but that ain't the right way to explain it.

Also, kinda ironic that jsonlib was abandoned one year later after that post ...

[*]	Feedback from commenters here, Reddit, HackerNews and Google+.

[†]	Where did this idea originate from? If you got a source please comment.

[‡]	Not something that you would use in a HTTP API at all. Put security concerns aside, the problem is mixing different versions of python. But they are still interesting to look at, for ideas of what's possible performance-wise.

Publishing to GitHub Pages from Travis CI

2015-07-11T00:00:00+03:00

One neat trick you can do is publish to GitHub Pages is publish to GitHub Pages from Travis.

I needed to do this for a static blog. However, the repository is under an organization and multiple people have commit access there. Using my Personal access token [1] is not the right way to do the authentication because:

Personal access tokens can only restrict kinds of operations, restricting to specific repository is not possible.
If something bad happens all my public repositories are at risk, as the Personal access token is the equivalent of a password.

There's a better way to solve the authentication: repositories can have deploy keys. The idea is that we'll use a brand new ssh key to perform the authentication. We do not use that key for anything else.

First we need to make a new ssh key:

ionel@localhost$ ssh-keygen -f publish-key
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in publish-key.
Your public key has been saved in publish-key.pub.

Then use use the Travis CLI (use gem install travis to install it) to encrypt it:

Note

I didn't use --add cause it reformats my .travis.yml, and it needs editing anyway.

ionel@localhost$ travis encrypt-file publish-key
Detected repository as ionelmc/sphinx-py3doc-enhanced-theme, is this correct? |yes|
encrypting publish-key for ionelmc/sphinx-py3doc-enhanced-theme
storing result as publish-key.enc
storing secure env variables for decryption

Please add the following to your build script (before_install stage in your .travis.yml, for instance):

    openssl aes-256-cbc -K ionel@localhost$encrypted_87d9ccdaebfd_key -iv ionel@localhost$encrypted_87d9ccdaebfd_iv -in publish-key.enc -out publish-key -d

Pro Tip: You can add it automatically by running with --add.

Make sure to add publish-key.enc to the git repository.
Make sure not to add publish-key to the git repository.
Commit all changes to your .travis.yml.

The unencrypted secret key should be removed (we don't want to add it by accident in a commit 😅):

rm publish-key

Then we add the encrypted key:

git add publish-key.enc

We need to change the decode command as the decoded file will go into a different path and some other fixups for the ssh authentication. In .travis.yml we need to have:

before_install:
- openssl aes-256-cbc -K ionel@localhost$encrypted_87d9ccdaebfd_key -iv ionel@localhost$encrypted_87d9ccdaebfd_iv -in publish-key.enc -out ~/.ssh/publish-key -d
- chmod u=rw,og= ~/.ssh/publish-key
- echo "Host github.com" >> ~/.ssh/config
- echo "  IdentityFile ~/.ssh/publish-key" >> ~/.ssh/config
- git --version
- git remote set-url origin git@github.com:ionelmc/sphinx-py3doc-enhanced-theme.git
- git fetch origin -f gh-pages:gh-pages

Now we can use a tool like ghp-import to simply publish (somewhere in .travis.yml after you generate the HTML in output):

ghp-import -n -p -m "Update gh-pages." output

All is left is to add the public key in your repo's deploy keys (make sure to tick the write access checkbox):

In some cases you only want to publish for the changes made on the master branch. Make sure you have this in your .travis.yml:

branches:
  only:
  - master

You can see the working examples of this in ropython-site or sphinx-py3doc-enhanced-theme.

[1]	Sadly, there are several guides out there that recommend using a Personal access token.

ionel's codelog

Proxy issues with Apache

Rehashing the src layout

Dealing with Docker

The good parts *

What irks me *

The ecosystem *

The challenges *

What's this context? *

Optimizing the build process *

Staging the build process *

Doing development *

Enter docker-compose *

Living inside the container *

Avoiding bash *

Container dependencies *

Process handling *

Lack of standard services *

Managing configuration *

Notes on debugging

Find the cause first *

Down the rabbit hole *

Rooting it out *

Plan for disaster *

Tooling *

Code *

Plan for disaster, but live dangerously *

Ask for help but understand *

Integration tests pay off *

Other peoples misery is gold *

Memory use and speed of JSON parsers

What about speed? *

Python 3 *

Different kind of data *

Lots off small objects *

Smaller data *

Bottom line *

Publishing to GitHub Pages from Travis CI

Enter `docker-compose` *

Avoiding `bash` *