Speed up pip install

Fri 02 January 2015

Note

The technique presented here has been obsoleted in pip-7.0. See the changelog and PR#2618.

On my Windows machine numpy takes 8 minutes to build and install [*]. And no, that doesn't include the time to install the compiler tools, it wouldn't even be close. I'm not sure what's going on, in an Ubuntu VM it builds and installs in under 3 minutes. It's annoying, but fortunately enough there's a nice trick to speed up installation: store all your wheels somewhere and then use that directory as a cache. This makes installing large packages that don't have wheels on PyPI faster even if they don't have C extensions as the build step is skipped.

In case you were wondering, pip doesn't cache the wheels built locally [1], so this is worth doing when there are no wheels for your platform on PyPI.

On Windows just make a file c:\Users\username\pip\pip.ini and add this in:

[global]
find-links = file://c:/Users/username/.pip-wheelhouse

[wheel]
wheel-dir = c:/Users/username/.pip-wheelhouse

For Linux it's /home/username/.pip/pip.conf:

[global]
find-links = file:///home/username/.pip-wheelhouse

[wheel]
wheel-dir = /home/username/.pip-wheelhouse

Alternatively, you can use environment variables for this. In your .bashrc or .zshrc add:

export STANDARD_CACHE_DIR="${XDG_CACHE_HOME:-${HOME}/.cache}/pip"
export WHEELHOUSE="${STANDARD_CACHE_DIR}/wheelhouse"
export PIP_FIND_LINKS="file://${WHEELHOUSE}"
export PIP_WHEEL_DIR="${WHEELHOUSE}"

Then just run:

pip wheel numpy

That will store the wheel in the default wheel directory, which you've set in the configuration.

After that pip wheel numpy will be very fast. This is highly desirable especially if you use tools that frequently rebuild the virtualenv, like Tox.

Wishful thinking *

Ideally package maintainers would upload Windows wheels to PyPI if they have C extensions, as it's fairly easy [3] with the (free for open-source) AppVeyor CI Service.

However, for Linux it's not possible to upload wheels [3] so this trick is still worth doing even when package maintainers notice how easy is to build wheels for Windows :-)

[*]numpy has wheels for OSX, but not for Windows. For shame!
[1]pip doesn't cache wheels (as of the 6.0 release). There's discussion about the improvement in issues #878 and #2140. There's also a proof of concept in PR#1572.
[2]See an example. The build jobs have have wheels as artifacts.
[3](1, 2)

See: https://packaging.python.org/en/latest/extensions.html#publishing-binary-extensions

PyPI disallows upload of Linux wheels because of binary compatibility issues - what's built on Ubuntu won't really run on RedHat because the two distros have different versions of libraries.

This entry was tagged as linux pip python windows