Dealing with Docker

The good parts *

Docker has several things going for it: a good name, huge community, huge amount of images you can start from. Sure, most images are not great, even the official images can have issues but you can always take out the Dockerfile and customize things.

Right now it's hard to find an alternative. CoreOS is trying to make an alternative called "rkt". With a flikr-esque name at that! How do you even read it? ercati? erkit? rocket? rockit? rackit? reckit? You know what, wreckit. If you think I'm lambasting the name unfairly then consider that most of the world don't speak very good English and interpretable spelling is a problem.

Even if rkt didn't have such a terrible name, it still has a long way to reaching Docker's convenience.

What irks me *

Docker has some strange oversights and quirks. There's a very strong focus on not breaking any interfaces, it's a bit depressing to look at the bug tracker if you're the impatient type.

These things annoy me:

• Documentation has no search. Seriously? I understand that there's Google but come on, I don't want to look at docs for old Docker versions and other junk Google gives me.

• Command line seem clumsy:

  • Inflexible parsing, eg: docker build . --help ain't valid. That makes no sense, there's only a single non-option argument, why can't I have options after that single possible argument?

  • Most arguments have short form (eg: -i) but not --help. Nooo, not that. No one needs a short form for that, God forbid!

  • Bad help for most options. Is this supposed to help?

    -v, --volume=[]                 Bind mount a volume
    

    Cause that sure as hell don't tell me anything about what I can pass in there. Now I have to go into the docs with no search. After rummaging trough 10 useless pages of documentation I eventually I get to this:

    -v, --volume=[host-src:]container-dest[:<options>]
                          Bind mount a volume. The comma-delimited
                          `options` are [rw|ro], [z|Z],
                          [[r]shared|[r]slave|[r]private], and
                          [nocopy]. The 'host-src' is an absolute path
                          or a name value.
    

    What. Why can't that be in the command line?

• If you got a run or build error in docker-compose you're usually left with half running containers and you have to manually cleanup the mess. If you don't pay attention you're left wondering why docker-compose build doesn't do anything. I did some changes, ran docker-compose build, why don't docker-compose up don't run new images? Cause some stuff is already running, that's why!

  If you stop a docker-compose up all services stop. If you stop a docker-compose up myservice it leaves all the dependencies running around. Argh! Why the inconsistency?

• There is no garbage collection. Leaves around huge piles of useless containers and images. Sure, there's docker rmi $(docker images -q) and docker rm $(docker ps -a -q) but that's like cleaning your computer dust with a water hose. It sure does the job but your computer probably don't work after that.

• Dockerfiles are quirky and non-orthogonal. Similarly to command line interface, the Dockerfile syntax is a culmination of ill-advised choices.

  • Support for multiline values is an afterthought. Dockerfiles are littered with "\" and "&&" all over the place. Lots of noise, lots of mistakes.

  • There two ways to do the same thing. The parser takes JSON for most commands but if parse fails then it takes whatever gunk is in there as a string.

    For example the CMD and ENTRYPOINT take this to the extreme: they have two exec mode. Take a look at this crazy table:

    	No ENTRYPOINT	ENTRYPOINT exec_entry p1_entry	ENTRYPOINT ["exec_entry", "p1_entry"]
    No CMD	error, not allowed	/bin/sh -c exec_entry p1_entry	exec_entry p1_entry
    CMD ["exec_cmd", "p1_cmd"]	exec_cmd p1_cmd	/bin/sh -c exec_entry p1_entry exec_cmd p1_cmd	exec_entry p1_entry exec_cmd p1_cmd
    CMD ["p1_cmd", "p2_cmd"]	p1_cmd p2_cmd	/bin/sh -c exec_entry p1_entry p1_cmd p2_cmd	exec_entry p1_entry p1_cmd p2_cmd
    CMD exec_cmd p1_cmd	/bin/sh -c exec_cmd p1_cmd	/bin/sh -c exec_entry p1_entry /bin/sh -c exec_cmd p1_cmd	exec_entry p1_entry /bin/sh -c exec_cmd p1_cmd

    There is clear overlap there. Maybe there are historical reasons for it but that's why we have version numbers. Why do we need to keep all that cruft?

  • No volumes/mounts during docker build. I have found ways to deal with this (read on) but still, it's an irk.

  If you look in the bug tracker you'll notice that lots of people have come up with ideas to improve Dockerfiles, but these issues just don't seem important enough.

• There's something rotten about how Docker run containers as root by default. Docker seems to mitigate this by disabling some capabilities like ptrace by default. But this ain't purely a security concern, it affects usability as well.

  For example, once you install docker on your Ubuntu you're faced with a choice: use sudo all over the place or give your user full access to the daemon:

  sudo usermod -aG docker $USER
  
  # then relogin, for the change to take effect - obvious right?
  

  Still not sure if that's better than a constant reminder that you're doing lots of stuff as root.

I know these might look like superficial complaints but they rile me up regardless. At least it's not this bad. </rant>

The ecosystem *

There are a lot of images, and you'll probably find anything you'll ever want. And there's even a set of specially branded images that pop up almost everywhere: the "official" images. They are pretty good: up to date, they verify signatures for whatever they download, consistent presentation etc.

Though I have a problem with the Python images. For some reason they decided to just compile Python themselves. Sure, it's the latest point-release but they don't include any of the patches the Debian maintainers made. I don't like those patches either (they customize package install paths and the import system) but it's worse without them:

• Suddenly gdb stops working. Just try a docker run --rm -it python:2.7 sh:

  # apt-get update
  # apt-get install gdb
  # gdb python
  Traceback (most recent call last):
    File "/usr/lib/python2.7/site.py", line 563, in <module>
      main()
    File "/usr/lib/python2.7/site.py", line 545, in main
      known_paths = addusersitepackages(known_paths)
    File "/usr/lib/python2.7/site.py", line 272, in addusersitepackages
      user_site = getusersitepackages()
    File "/usr/lib/python2.7/site.py", line 247, in getusersitepackages
      user_base = getuserbase() # this will also set USER_BASE
    File "/usr/lib/python2.7/site.py", line 237, in getuserbase
      USER_BASE = get_config_var('userbase')
    File "/usr/lib/python2.7/sysconfig.py", line 582, in get_config_var
      return get_config_vars().get(name)
    File "/usr/lib/python2.7/sysconfig.py", line 528, in get_config_vars
      _init_posix(_CONFIG_VARS)
    File "/usr/lib/python2.7/sysconfig.py", line 412, in _init_posix
      from _sysconfigdata import build_time_vars
    File "/usr/lib/python2.7/_sysconfigdata.py", line 6, in <module>
      from _sysconfigdata_nd import *
  ImportError: No module named _sysconfigdata_nd
  

  Looks like there's some import path trampling going on. I don't want broken debug tools when my app is broken.

• You can't use any Python package from the APT repos. Sure, most of them are old and easy to install with pip, but there are exceptions [2].

• Strange C.UTF-8 locale. I can understand they don't want to put a specific language in there but if you run any locale using applications you'll run into issues.

What I ended up was using the ubuntu:xenial image (Xenial being the new LTS). It ships latest point release for 2.7 so why compile it again? I took the good parts from python:2.7-slim and I got this Dockerfile:

FROM ubuntu:xenial

RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

RUN apt-get update \
 && apt-get install -y --no-install-recommends \
                    ca-certificates curl \
                    strace gdb lsof locate net-tools htop \
                    python2.7-dbg python2.7 libpython2.7 \
 && rm -rf /var/lib/apt/lists/*

ENV PYTHON_PIP_VERSION 8.1.1
RUN set -eEx \
 && curl -fSL 'https://bootstrap.pypa.io/get-pip.py' | python2.7 - \
         --no-cache-dir --upgrade pip==$PYTHON_PIP_VERSION

COPY .wheels /wheels

Everything works great, and there's some magic in those debug packages that make gdb give me some real nice commands like py-bt [3]. Note that I snuck in some other tools to help debugging.

All I'm saying there is that even official images need scrutiny. Check their Dockerfile and decide if it really fits your needs.

The challenges *

Docker has some interesting challenges, or limitations. The build system has something called "layers" and there's a hard limit of how many layers you can have. Each command in your Dockerfile makes a new layer. If you look at the "official" best practices guide you'll see most of the stuff there revolves around this limitation. You inevitably end up with some damn ugly RUN commands.

There's a good thing about these layers - they are cached. However, the context is not. Never one layer needs all the context, or the same part of the context. Layers should be able to have individual context, but alas, docker build wasn't designed with that in mind.

Another limitation by design is that docker build doesn't allow any mounts or volumes during build. The only way to get stuff in the container that eventually becomes the image is by network or by the "context".

RUN mkdir -p /app

COPY requirements.txt /
# slow as hell ...
RUN pip install -r /requirements.txt

COPY app /app

docker
├── base
│   └── Dockerfile
├── builder
│   └── Dockerfile
├── deps
│   └── Dockerfile
├── web
│   └── Dockerfile
├── worker
│   └── Dockerfile
└── ...

# just ignore that directory, we don't need that stuff when we have "." as the context
docker

# we start from an image with build deps preinstalled
FROM buildpack-deps:xenial

# seems acceptable for building, note the notes above
# about C.UTF-8 - it's not really good for running apps
ENV LANG C.UTF-8

# we'd add all the "-dev" packages we need here
RUN apt-get update \
 && apt-get install -y --no-install-recommends \
                    python2.7 python2.7-dbg python2.7-dev libpython2.7 \
                    strace gdb lsof locate \
 && rm -rf /var/lib/apt/lists/*

ENV PYTHON_PIP_VERSION 8.1.1
RUN set -eEx \
 && curl -fSL 'https://bootstrap.pypa.io/get-pip.py' | python2.7 - \
         --no-cache-dir --upgrade pip==$PYTHON_PIP_VERSION

ARG USER
ARG UID
ARG GID

# we set some default options for pip here
# so we don't have to specify them all the time

# this will make pip additionally look for available wheels here
ENV PIP_FIND_LINKS=/home/$USER/wheelcache
# and this is the default output dir when we run `pip wheel`
ENV PIP_WHEEL_DIR=/home/$USER/wheelcache
ENV PIP_TIMEOUT=60
# one network request less, we don't care about latest version
ENV PIP_DISABLE_PIP_VERSION_CHECK=true

RUN echo "Creating user: $USER ($UID:$GID)" \
 && groupadd --system --gid=$GID $USER \
 && useradd --system --create-home --gid=$GID --uid=$UID $USER \
 && mkdir /home/$USER/wheelcache

WORKDIR /home/$USER

FROM ubuntu:xenial

RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

RUN apt-get update \
 && apt-get install -y --no-install-recommends \
                    ca-certificates curl \
                    strace gdb lsof locate net-tools htop \
                    python2.7-dbg python2.7 libpython2.7 \
 && rm -rf /var/lib/apt/lists/*

ENV PYTHON_PIP_VERSION 8.1.1
RUN set -eEx \
 && curl -fSL 'https://bootstrap.pypa.io/get-pip.py' | python2.7 - \
         --no-cache-dir --upgrade pip==$PYTHON_PIP_VERSION

COPY .wheels /wheels
RUN pip install --force-reinstall --ignore-installed --upgrade \
                --no-index --use-wheel --no-deps /wheels/* \
 && rm -rf /wheels

FROM app-deps

# copy the application files and add them on the import path
RUN mkdir /app
WORKDIR /app
COPY setup.py /app/
COPY src /app/src
# there are other ways (like .pth file) but this one allows
# for easy setup of various bin scripts app might need
RUN python2.7 setup.py develop

# create an user for the application and install basic tools
# to change user (pysu) and wait for services (holdup)
ARG USER=app
ENV USER=$USER
RUN echo "Creating user: $USER" \
 && groupadd --system  $USER \
 && useradd --system --create-home --gid=$USER --base-dir=/var $USER \
 && pip install pysu==0.1.0 holdup==1.0.0 \
 && pysu $USER id
# this last one just tests that pysu works

FROM app-base

RUN apt-get update \
 && apt-get install -yq --no-install-recommends nginx-core supervisor \
 && rm -rf /var/lib/apt/lists/*

COPY site.conf /etc/nginx/sites-enabled/default
COPY supervisor.conf /etc/supervisor/conf.d/
COPY entrypoint.sh /

RUN echo "daemon off;" >> /etc/nginx/nginx.conf

EXPOSE 80
CMD ["/entrypoint.sh"]

#!/bin/sh
set -eux

docker build --tag=app-builder \
             --build-arg USER=$USER \
             --build-arg UID=$(id --user $USER) \
             --build-arg GID=$(id --group $USER) \
             docker/builder
# we run this image two times, once to prime a wheel cache
mkdir -p .dockercache
docker run --rm \
           --user=$USER \
           --volume="$PWD/requirements.txt":/requirements.txt:ro \
           --volume="$PWD/.dockercache":/home/$USER \
           app-builder \
           pip wheel --requirement=/requirements.txt
# and the second time to create the final wheel set
rm -rf "$PWD/docker/deps/.wheels"
mkdir  "$PWD/docker/deps/.wheels"
docker run --rm \
           --user=$USER \
           --volume="$PWD/requirements.txt":/requirements.txt:ro \
           --volume="$PWD/.dockercache":/home/$USER \
           --volume="$PWD/docker/deps/.wheels":/home/$USER/wheels \
           app-builder \
           pip wheel --wheel-dir=wheels \
                     --requirement=/requirements.txt
# and now there are going to be tons of wheels in "docker/deps/.wheels/"
docker build --tag=app-deps docker/deps

docker build --tag=app-base --file=docker/base/Dockerfile .
# this is why we simply ignore "docker/" in .dockerignore -- it
# would inflate the context a lot and we don't need the wheels in this step

docker build --tag=app-web docker/web

# no need to rebuild the base image (that has
# the /app/src) if we just mount it anyway
docker run --volume=$PWD/src:/app/src:ro app-web

version: '2'
services:
  web:
    build: 'docker/web'
    image: 'app-web'
    ports:
    - '8080:80'
    volumes:
    - '${PWD}/src:/app/src:ro'
    links:
    - 'pg'
    environment:
      DATABASE_HOST: 'pg'
  pg:
    image: 'postgres:9.5'
    environment:
      POSTGRES_DB: 'app'
      POSTGRES_USER: 'app'
      POSTGRES_PASSWORD: 'app'

docker run --privileged --rm -it myimage bash
# or, if you want to debug a running container
docker exec --privileged -it mycontainer bash

#!/bin/sh

# general good practice (stop on error, missing variables, verbose mode):
set -eux

# avoid this:
pure-ftpd

# do this instead:
exec pure-ftpd

FROM alpine:3.3

RUN mkdir -p /etc/app/conf /etc/app/conf-update

# this will be the default data, if volume wasn't created
COPY settings.conf /etc/app/conf/
VOLUME /etc/app/conf

# this second copy is used for updating
COPY settings.conf /etc/app/conf-update

COPY update-conf.sh /
# similarly to a cp command
ENTRYPOINT ["/update-conf.sh", "/etc/app/conf-update", "/etc/app/conf"]

#!/bin/sh
set -eu
echo "Updating configuration in $2 with files from $1"

for to_update in $1/*; do
    name="${to_update##*/}"
    echo "Updating: $name"

    echo "* diff from: $to_update to: $2/$name"
    diff -U 5 "$to_update" "$2/$name"
    echo "* copying: $to_update to: $2/$name"
    cp "$to_update" "$2/$name"
done

echo Success. Configuration updated!

conf:
  image: 'app-conf'
  build: 'docker/conf'
web:
  ...
  volumes_from:
  - 'conf:ro'