Packaging a python library

Sun 25 May 2014

Note

This is about packaging libraries, not applications.

For now there are no special considerations for C extensions here.

All the advice here is implemented in a project template: cookiecutter-pylibrary (introduction).

I think the packaging best practices should be revisited, there are lots of good tools now-days that are either unused or underused. It's generally a good thing to re-evaluate best practices all the time.

I assume here that your package is to be tested on multiple Python versions, with different combinations of dependency versions, settings etc.

And few principles that I like to follow when packaging:

  • If there's a tool that can help with testing use it. Don't waste time building a custom test runner if you can just use nose or py.test. They come with a large ecosystem of plugins that can improve your testing.
  • When possible, prevent issues early. This is mostly a matter of strictness and exhaustive testing. Design things to prevent common mistakes.
  • Collect all the coverage data. Record it. Identify regressions.
  • Test all the possible configurations.

The structure*

This is fairly important, everything revolves around this. I prefer this sort of layout:

├─ src
│  └─ packagename
│     ├─ __init__.py
│     └─ ...
├─ tests
│  └─ ...
└─ setup.py

The src directory is a better approach because:

  • You get import parity. The current directory is implicitly included in sys.path; but not so when installing & importing from site-packages. Users will never have the same current working directory as you do.

    This constraint has beneficial implications in both testing and packaging:

    • You will be forced to test the installed code (e.g.: by installing in a virtualenv). This will ensure that the deployed code works (it's packaged correctly) - otherwise your tests will fail. Early. Before you can publish a broken distribution.
    • You will be forced to install the distribution. If you ever uploaded a distribution on PyPI with missing modules or broken dependencies it's because you didn't test the installation. Just beeing able to successfuly build the sdist doesn't guarantee it will actually install !
  • It prevents you from readily importing your code in the setup.py script. This is a bad practice because it will always blow up if importing the main package or module triggers additional imports for dependencies (which may not be available [5]). Best to not make it possible in the first place.

  • Simpler packaging code and manifest. It makes manifests very simple to write (e.g.: you package a Django app that has templates or static files). Also, zero fuss for large libraries that have multiple packages. Clear separation of code being packaged and code doing the packaging.

    Without src writting a MANIFEST.in is tricky [6]. If your manifest is broken your tests will fail. It's much easier with a src directory: just add graft src in MANIFEST.in.

    Publishing a broken package to PyPI is not fun.

  • Without src you get messy editable installs ("setup.py develop" or "pip install -e"). Having no separation (no src dir) will force setuptools to put your project's root on sys.path - with all the junk in it (e.g.: setup.py and other test or configuration scripts will unwittingly become importable).

  • There are better tools. You don't need to deal with installing packages just to run the tests anymore. Just use tox - it will install the package for you [2] automatically, zero fuss, zero friction.

  • Less chance for user mistakes - they will happen - assume nothing !

  • Less chance for tools to mixup code with non-code.

Another way to put it, flat is better than nested [*] - but not for data. A file-system is just data after all - and cohesive, well normalized data structures are desirable.

You'll notice that I don't include the tests in the installed packages. Because:

  • Module discovery tools will trip over your test modules. Strange things usually happen in test module. The help builtin does module discovery. E.g.:

    >>> help('modules')
    Please wait a moment while I gather a list of all available modules...
    
    __future__          antigravity         html                select
    ...
    
  • Tests usually require additional dependencies to run, so they aren't useful by their own - you can't run them directly.

  • Tests are concerned with development, not usage.

  • It's extremely unlikely that the user of the library will run the tests instead of the library's developer. E.g.: you don't run the tests for Django while testing your apps - Django is already tested.

Alternatives*

You could use src-less layouts, few examples:

Tests in package Tests outside package
├─ packagename
│  ├─ __init__.py
│  ├─ ...
│  └─ tests
│     └─ ...
└─ setup.py
├─ packagename
│  ├─ __init__.py
│  └─ ...
├─ tests
│  └─ ...
└─ setup.py

These two layouts became popular because packaging had many problems few years ago, so it wasn't feasible to install the package just to test it. People still recommend them [4] even if it based on old and oudated assumptions.

Most projects use them incorectly, as all the test runners except Twisted's trial have incorrect defaults for the current working directory - you're going to test the wrong code if you don't test the installed code. trial does the right thing by changing the working directory to something temporary, but most projects don't use trial.

The setup script*

Unfortunately with the current packaging tools, there are many pitfalls. The setup.py script should be as simple as possible:

# -*- encoding: utf8 -*-
import glob
import io
from os.path import basename
from os.path import dirname
from os.path import join
from os.path import splitext

from setuptools import find_packages
from setuptools import setup


def read(*names, **kwargs):
    return io.open(
        join(dirname(__file__), *names),
        encoding=kwargs.get("encoding", "utf8")
    ).read()

setup(
    name="nameless",
    version="0.1.0",
    license="BSD",
    description="An example package. Replace this with a proper project description. Generated with https://github.com/ionelmc/cookiecutter-pylibrary",
    long_description="%s\n%s" % (read("README.rst"), read("CHANGELOG.rst")),
    author="Ionel Cristian Mărieș",
    author_email="contact@ionelmc.ro",
    url="https://github.com/ionelmc/python-nameless",
    packages=find_packages("src"),
    package_dir={"": "src"},
    py_modules=[splitext(basename(i))[0] for i in glob.glob("src/*.py")],
    include_package_data=True,
    zip_safe=False,
    classifiers=[
        # complete classifier list: http://pypi.python.org/pypi?%3Aaction=list_classifiers
        "Development Status :: 5 - Production/Stable",
        "Intended Audience :: Developers",
        "License :: OSI Approved :: BSD License",
        "Operating System :: Unix",
        "Operating System :: POSIX",
        "Operating System :: Microsoft :: Windows",
        "Programming Language :: Python",
        "Programming Language :: Python :: 2.6",
        "Programming Language :: Python :: 2.7",
        "Programming Language :: Python :: 3",
        "Programming Language :: Python :: 3.3",
        "Programming Language :: Python :: 3.4",
        "Programming Language :: Python :: Implementation :: CPython",
        "Programming Language :: Python :: Implementation :: PyPy",
        "Topic :: Utilities",
    ],
    keywords=[
        # eg: "keyword1", "keyword2", "keyword3",
    ],
    install_requires=[
        # eg: "aspectlib==1.1.1", "six>=1.7",
    ],
    extras_require={
        # eg: 'rst': ["docutils>=0.11"],
    },
    entry_points={
        "console_scripts": [
            "nameless = nameless.__main__:main"
        ]
    }

)

What's special about this:

  • No exec or import trickery.
  • Includes everything from src: packages or root-level modules.
  • Explicit encodings.

Running the tests*

Again, it seems people fancy the idea of running python setup.py test to run the package's tests. I think that's not worth doing - setup.py test is a failed experiment to replicate some of CPAN's test system. Python doesn't have a common test result protocol so it serves no purpose to have a common test command [1]. At least not for now - we'd need someone to build specifications and services that make this worthwhile, and champion them. I think it's important in general to recognize failure where there is and go back to the drawing board when that's necessary - there are absolutely no services or tools that use setup.py test command in a way that brings added value. Something is definitely wrong here.

I believe it's too late now for PyPI to do anything about it, Travis is already a solid, reliable, extremely flexible and free alternative. It integrates very well with Github - builds will be run automatically for each Pull Request.

To test locally tox is a very good way to run all the possible testing configurations (each configuration will be a tox environment). I like to organize the tests into a matrix with these additional environments:

  • check - check package metadata (e.g.: if the restructured text in your long description is valid)
  • clean - clean coverage
  • report - make coverage report for all the accumulated data
  • docs - build sphinx docs

I also like to have environments with and without coverage measurement and run them all the time. Race conditions are usually performance sensitive and you're unlikely to catch them if you're everything with coverage measurements.

The test matrix*

Depending on dependencies you'll usually end up with a huge number of combinations of python versions, dependency versions and different settings. Generally people just hard-code everything in tox.ini or only in .travis.yml. They end up with incomplete local tests, or test configurations that run serially in Travis. I've tried that, didn't like it. I've tried duplicating the environments in both tox.ini and .travis.yml. Still didn't like it.

As there were no readily usable alternatives to generate the configuration, I've implemented a generator script that uses templates to generate tox.ini and .travis.yml. This is way better, it's DRY, you can easily skip running tests on specific configurations (e.g.: skip Django 1.4 on Python 3) and there's less work to change things.

The essentials:

setup.cfg*

The generator script uses a configuration file (setup.cfg for convenience):

# Just append this to your setup.cfg

[matrix]
# This is the configuration for the `./bootstrap.py` script.
# It generates `.travis.yml`, `tox.ini` and `appveyor.yml`.
#
# Syntax: [alias:] value [!variable[glob]] [&variable[glob]]
#
# alias:
#  - is used to generate the tox environment
#  - it's optional
#  - if not present the alias will be computed from the `value`
# value:
#  - a value of "-" means empty
# !variable[glob]:
#  - exclude the combination of the current `value` with 
#    any value matching the `glob` in `variable`
#  - can use as many you want
# &variable[glob]:
#  - only include the combination of the current `value` 
#    when there's a value matching `glob` in `variable`
#  - can use as many you want

python_versions =
    2.6
    2.7
    3.3
    3.4
    pypy

dependencies =
    1.4: Django==1.4.13 !python_versions[3.*]
    1.5: Django==1.5.8
    1.6: Django==1.6.5
    1.7: https://www.djangoproject.com/download/1.7.b4/tarball/ !python_versions[2.6]

coverage_flags =
    : true
    nocover: false

environment_variables =
    -

bootstrap.py*

This is the generator script. You run this whenever you want to regenerae the configuration:

#!/usr/bin/env python
import os
import sys
if not os.path.exists('.tox/configure'):
    import virtualenv
    import subprocess
    print("Bootstrapping ...")
    virtualenv.create_environment('.tox/configure')
    print("Installing `jinja2` and `matrix` into bootstrap environment ...")
    if sys.platform == 'win32':
        subprocess.check_call(['.tox/configure/Scripts/pip', 'install', 'jinja2', 'matrix'])
    else:
        subprocess.check_call(['.tox/configure/bin/pip', 'install', 'jinja2', 'matrix'])
if sys.platform == 'win32':
    execfile(
        '.tox/configure/Scripts/activate_this.py', 
        dict(__file__='.tox/configure/Scripts/activate_this.py')
    )
else:
    execfile(
        '.tox/configure/bin/activate_this.py', 
        dict(__file__='.tox/configure/bin/activate_this.py')
    )
import jinja2
import matrix

jinja = jinja2.Environment(
    loader=jinja2.FileSystemLoader('.'),
    trim_blocks=True,
    lstrip_blocks=True,
    keep_trailing_newline=True
)
tox_environments = {}
for alias, conf in matrix.from_file('setup.cfg').items():
    python = conf['python_versions']
    deps = conf['dependencies']
    cover = {'false': False, 'true': True}[conf['coverage_flags'].lower()]
    env_vars = conf['environment_variables']

    tox_environments[alias] = {
        'python': 'python' + python if 'py' not in python else python,
        'deps': deps.split(),
        'cover': cover,
        'env_vars': env_vars.split(),
    }

with open('tox.ini', 'w') as fh:
    fh.write(jinja.get_template('tox.tmpl.ini').render(
        tox_environments=tox_environments
    ))

with open('.travis.yml', 'w') as fh:
    fh.write(jinja.get_template('.travis.tmpl.yml').render(
        tox_environments=tox_environments
    ))

with open('appveyor.yml', 'w') as fh:
    fh.write(jinja.get_template('appveyor.tmpl.yml').render(
        tox_environments=tox_environments
    ))

print("DONE.")

.travis.tmpl.yml*

This has some goodies in it: the very useful libSegFault.so trick.

It basically just runs tox.

language: python
python: 2.7
env:
  global:
    LD_PRELOAD=/lib/x86_64-linux-gnu/libSegFault.so
  matrix:
    - TOXENV=check
{% for env, config in tox_environments|dictsort %}
    - TOXENV={{ env }}{% if config.cover %},coveralls{% endif %}

{% endfor %}
before_install:
  - python --version
  - virtualenv --version
  - pip --version
  - uname -a
  - lsb_release -a
install:
  - pip install tox
script:
  - tox -v
notifications:
  email:
    on_success: never
    on_failure: always

tox.tmpl.ini*

[tox]
envlist =
    clean,
    check,
{% for env in tox_environments|sort %}
    {{ env }},
{% endfor %}
    report,
    docs

[testenv]
setenv =
    PYTHONPATH={toxinidir}/tests
    PYTHONUNBUFFERED=yes
deps =
    pytest
    pytest-capturelog
commands =
    {posargs:py.test -vv}

[testenv:spell]
setenv =
    SPELLCHECK = 1
commands =
    sphinx-build -b spelling docs dist/docs
usedevelop = true
deps =
    -r{toxinidir}/docs/requirements.txt
    sphinxcontrib-spelling
    pyenchant

[testenv:docs]
whitelist_externals =
    rm
commands =
    rm -rf dist/docs || rmdir /S /Q dist\docs
    sphinx-build -b html docs dist/docs
    sphinx-build -b linkcheck docs dist/docs
usedevelop = true
deps =
    -r{toxinidir}/docs/requirements.txt

[testenv:configure]
deps =
    jinja2
    matrix
usedevelop = true
commands =
    python configure

[testenv:check]
basepython = python3.4
deps =
    docutils
    check-manifest
    flake8
    collective.checkdocs
usedevelop = true
commands =
    python setup.py checkdocs
    python setup.py check --restructuredtext --strict --metadata
    check-manifest {toxinidir}
    flake8 src

[testenv:coveralls]
deps =
    coveralls
usedevelop = true
commands =
    coverage combine
    coverage report
    coveralls

[testenv:report]
basepython = python3.4
commands =
    coverage combine
    coverage report
usedevelop = true
deps = coverage

[testenv:clean]
commands = coverage erase
usedevelop = true
deps = coverage

{% for env, config in tox_environments|dictsort %}
[testenv:{{ env }}]
basepython = {{ config.python }}
{% if config.cover or config.env_vars %}
setenv =
    {[testenv]setenv}
{% endif %}
{% for var in config.env_vars %}
    {{ var }}
{% endfor %}
{% if config.cover %}
    WITH_COVERAGE=yes
usedevelop = true
commands =
    {posargs:py.test --cov src -vv}
{% endif %}
deps =
    {[testenv]deps}
    coverage
    pytest-cov
{% for dep in config.deps %}
    {{ dep }}
{% endfor %}

{% endfor %}

appveyor.tmpl.ini*

For Windows-friendly projects:

build: false
environment:
  matrix:
    - TOXENV: check
{% for env, config in tox_environments|dictsort %}{% if env.startswith('2.7') or env.startswith('3.4') %}
    - TOXENV: "{{ env }}"
{% endif %}{% endfor %}
init:
  - "ECHO %TOXENV%"
  - ps: "ls C:\\Python*"
install:
  - ps: Invoke-WebRequest "https://bootstrap.pypa.io/ez_setup.py" -OutFile "c:\\ez_setup.py"
  - ps: Invoke-WebRequest "https://bootstrap.pypa.io/get-pip.py" -OutFile "c:\\get-pip.py"
  - "c:\\python27\\python c:\\ez_setup.py"
  - "c:\\python27\\python c:\\get-pip.py"
  - "c:\\python27\\Scripts\\pip install tox virtualenv"
test_script:
  - "c:\\python27\\Scripts\\tox --version"
  - "c:\\python27\\Scripts\\virtualenv --version"
  - "c:\\python27\\Scripts\\pip --version"
  - "c:\\python27\\Scripts\\tox"

If you've been patient enough to read through that you'll notice:

  • The Travis configuration uses tox for each item in the matrix. This makes testing in Travis consistent with testing locally.
  • The environment order for tox is clean, check, 2.6-1.3, 2.6-1.4, ..., report.
  • The environments with coverage measurement run the code without installing (usedevelop = true) so that coverage can combine all the measurements at the end.
  • The environments without coverage will sdist and install into virtualenv (tox's default behavior [2]) so that packaging issues are caught early.
  • The report environment combines all the runs at the end into a single report.

Having the complete list of environments in tox.ini is a huge advantage:

  • You run everything in parallel locally (if your tests don't need strict isolation) with detox. And you can still run everything in parallel if you want to use drone.io instead of Travis.
  • You can measure cummulated coverage for everything (merge the coverage measurements for all the environments into a single one) locally.

Test coverage*

There's Coveralls - a nice way to track coverage over time and over multiple builds. It will automatically add comments on Github Pull Request about changes in coverage.

TL;DR*

  • Put code in src.
  • Use tox and detox.
  • Test both with coverage measurements and without.
  • Use a generator script for tox.ini and .travis.ini.
  • Run the tests in Travis with tox to keep things consistent with local testing.
[1]There's subunit and probably others but they are widely used.
[2](1, 2) See example.
[3]There is a feature specification/proposal in tox for multi-dimensional configuration but it still doesn't solve the problem of generating the .travis.yml file. There's also tox-matrix but it's not flexibile enough.
[4]cookiecutter-pypackage is acceptable at the surface level (tests outside, correct MANIFEST) but still has the core problem (lack of src separation) and gives the wrong idea to glancing users.
[5]

It's a chicken-and-egg problem: how can pip know what dependencies to install if running the setup.py script requires unknownable dependencies ?

There are so many weird corners you can get into by having the power to run arbitrary code in the setup.py script. This why people tried to change setup.py to pure metadata.

[6]Did you know the order of the rules in MANIFEST.in matters ?
[*]PEP-20's 5th aphorism: Flat is better than nested.

This entry was tagged as python packaging django

blog comments powered by Disqus