Tox tricks and patterns

14 April 2015 (updated 17 February 2016)

Tox is a relatively new tool [1] that does one thing very well: test a python package in a set of virtual environments. This roughly translates to something like:

  • python sdist
  • virtualenv .tox/py27
  • .tox/py27/pip install some dependencies
  • .tox/py27/pip install (where is whatever sdist produced)
  • Test commands of your choosing. Some typical examples:
    • py.test
    • nosetests
    • test - people insist using the setuptools test command even if it's unnecessary complication [3].

A minimal tox.ini looks like this:

envlist = py27,py34

deps = pytest
commands = py.test

That will run py.test with both Python 2.7 and 3.4.

Why should you use tox? *

There are several advantages of using tox:

  • It simplifies contributor ramp-up. Just run tox and you're set - no more reading dozen install, setup and testing instructions just to run the tests.
  • It's portable on Linux, OS X and Windows.
  • Reduces that ugly shell script boilerplate.

There are some alternatives to tox, however, they have some disadvantages:

  • Make: tricky to install on Windows and it's not really designed for testing python packages. You'd still need to do lots of shell scripting to achieve what tox does out of the box.
  • Invoke (or its alternatives): again, this is not designed for testing python packages - it's something way more generic and you'd have to either write everything yourself or use a 3rd party task libraries. [4]
  • Shell scripts: this is the worst of the lot. Besides them being unportable and hard to read, I'm pretty tired of seeing people doing accidental rm -rf / (because shell scripts don't error or uninitialized variables by default) or having some other obscure bugs. Read this terrifying list of Bash pitfalls.

There are probably other tools I haven't considered, feel free to comment, if you still think there's something better than tox. :-)

Effective use *

Because tox was designed for the aforementioned workflow some things are hard to do, or the best way is unclear. I'm going to go over few less known tricks and patterns.

Command overriding *

I usually allow overriding the test command [2]. This allows me to use tox for ad-hoc development.

deps = pytest
commands = {posargs:py.test}

With that you can conveniently do any of:

  • tox -- py.test -k test_something to run tests matching test_something in all the environments.
  • tox -e py34 -- python to get the python REPL in that environment.
  • tox -e py27 -- django-admin runserver to run the Django development server in that environment.

And that would be run over all the available environments.

Lack of packaging *

Sometimes you lack a and it doesn't make sense to have one. But you still want to use tox because you want to use virtualenvs and run something in them. To make that work have this in your tox.ini:

skipsdist = true

Note that this applies to all environments you use.

Compact configuration *

Using conditional settings you can make tox.ini a lot more brief.

Compare this:

envlist = py2.7-A,py3.4-A,py2.7-B,py3.4-B

commands = <do some stuff with A>

commands = <do some stuff with B>

commands = <do some stuff with A>

commands = <do some stuff with B>
envlist = py{2.7,3.4}-{A,B}

commands =
    A: <do some stuff with A>
    B: <do some stuff with A>

Way more brief :-)

Environment reuse *

Sometimes you want to run several commands. Now you have two choices:

  • Have one environment with all the commands. This makes it hard to select an individual command.
  • Have a environment for each command. This makes command line selection very neat. However, now you have a virtualenv for each environment, which makes everything very slow unless ...

... you have all the environments use the same virtualenv directory! As seen here, you can use a specific directory that doesn't need to be unique:

skipsdist = true
envlist = build

envdir = {toxinidir}/.env

commands =
    build: pelican --output output --settings --delete-output-directory []
    watch: pelican --output output --settings --delete-output-directory --autoreload []
    run: twistd -n web --path=.
    publish: python -m "Update gh-pages." output
    publish: git push origin master
    publish: git push origin gh-pages
deps =

With such an configuration you can do any of:

  • tox -e build,publish to build and deploy
  • tox -e watch to build and rebuild on file changes
  • tox -e run to just run a webserver

Note the [] in the build command allows adding extra arguments. Example:

  • tox -e build -- --debug to get some verbose logging.

When it inevitably leads to shell scripts *

This is one thing that's annoying to do in Tox - sometimes you do want to run some shell scripts, as bad as they are. [6]

Suppose you have a in your project's root:

commands =

This will work fine, however, on Windows you need a different shell script. Currently tox doesn't have conditional settings for the platform so you need to have a script that can run both as Shell and Batch (for Windows). A way to solve this is to have something like this in tox.ini:

commands = test.cmd

Then you make test.cmd executable and have this peculiar mash-up of heredocs (<<END ... END), labels (:end) and goto:

#!/bin/bash -eE
@echo off
powershell -ExecutionPolicy ByPass -File test.ps1 %*
goto :end
::batch $*
exit $?

And then you write regular Bash code in and PowerShell (on Windows) in test.ps1. The ExecutionPolicy ByPass is required because PowerShell doesn't allow external scripts to run by default (the default is a sort of "interactive only" mode) [5].

Partial environment reuse *

Suppose you want to test a set of configurations. Using a conditional settings you can have something like this:

envlist = {2.7,3.4}-{A,B,C}

basepython =
    2.7: python2.7
    3.4: python3.4

# {toxworkdir} defaults to .tox
envdir =
    2.7: {toxworkdir}/2.7
    3.4: {toxworkdir}/3.4

# I'm building here on the previous example but we could have any other command here ...
commands =
    {2.7,3.4}-A: {toxinidir}/test.cmd A
    {2.7,3.4}-B: {toxinidir}/test.cmd B
    {2.7,3.4}-C: {toxinidir}/test.cmd C

With that you'll have only 2 environments and that will significantly reduce environment rebuild time.

Dependency caching *

This is not necessarily tox specific but I'm mentioning it because it can make installing some dependencies so much faster: use wheelhouses preloaded with the deps.

Know other tricks? Comment below!

[1]0.5 was released almost 5 years ago.
[2]See: substitutions for positional arguments in commands.
[3]Writing a well behaved is not a walk in the park and a custom test command doesn't bring any advantages.
[4]I haven't looked much into this, tho I have came across rituals. Ironically it still uses tox for testing :-)

It could be more strict than ByPass but I haven't bothered. Read any of these if you care:


However, you shouldn't do this, unless you have strong shell scripting skills.

I'm just putting it here because I got it to work, just like I've got MySQL to work (somewhat), even if it made me angry.

This entry was tagged as python testing