Overview of cibuildwheel 🎡

This is the first of two posts on cibuildwheel, a fantastic project I joined after switching to it from my own azure-wheel-helpers, which I’ve blogged about here before. It is the best wheelbuilding system available for Python today, and can make something that is normally a pain to setup and a headache to maintain a breeze (by forcing all the headaches on us, of course, as maintainers, but it’s better to solve issues centrally! Obviously we rather like solving these problems. Or we are just crazy, which is also possible ;) ).

Be sure to checkout the followup post over new features in 1.8.0 and 1.9.0, too! Also, cibuildwheel was recently accepted into the PyPA!

Background

cibuildwheel is a tool to build Python wheels on all supported platforms. To answer one of the most common questions: no, you don’t need it if you don’t have any compiled extensions. In that case, just run:

pip install build
python -m build
# or
pipx run build

And you’ll get a wheel and an SDist. You can put this in CI; GitHub Actions has pipx installed by default, and it doesn’t even need setup-python to run.

Building without cibuildwheel

If you have any binary components, suddenly the above process becomes much harder. You now need to build on every Python version¹ and every OS you support. Each platform has unique hurdles you have to work with.

On Linux, you need to use the ManyLinux docker images, which are carefully stripped down and prepared to ensure you only interact with the allowed subset of features to make a manylinux compliant wheel, and you need to run auditwheel on the resulting wheel to verify it is a manylinux wheel. And, of course, you’ll need emulation most of the time if you want to build for ARM, PowerPC, or other special architectures.

On macOS, unlike Linux, you can set a target version (say, 10.9), and then the compiler will avoid using anything binary from the system that was introduced after that point. Since most of the changes in C++ are in headers, you actually can use most of the new features from new C++ versions even if you target an older version of macOS! You never need an older version of macOS to target an older macOS (which is important, because Apple doesn’t sign older versions). But, in order to work with CPython itself, it’s best to download and use the “official” python.org CPythons, which are compiled with 10.9+ compatibility; you don’t want to use homebrew Python, which is compiled to be optimized to your OS. After running, you need to run deallocate to ensure all requirements are bundled. And to build Universal2 or Arm64 wheels on Intel, especially on 10.15, is really a challenge!

Windows, ironically, is the easiest to get right as far as Python version; you just have to deal with selecting the compiler (we’ll leave 2.7 out of this discussion, it is much harder and project dependent). And there isn’t a tool (yet) to bundle dependencies (though delvewheel may be solving this issue).

On top of this, there are lots of issues and caveats. For example, PyPy is rather slow to release new versions, so often you need patches to build correctly (that is true at the time of writing with PyPy 7.3.3). You also really should test your wheels, and the testing shouldn’t happen in the build environment, to ensure the wheel works as intended.

And once you’ve solved the above issues, then you will have at least 100 lines of mostly CI specific code that can’t be easily moved to a different CI if your provider suddenly stops supporting you (cough, Travis CI, cough).

I’ve manually solved most of these issues in the past in azure-wheel-helpers, a set of templates from before Azure supported proper remote templates with substitution. In fact, I wrote a detailed writeup here some time ago; if you want to dig into specifics, it’s still a good read. But this was hacky, had to be maintained for each new Python release, and was specific to one CI system (Azure). It also had to be updated in each project via git subtree, but a modern version wouldn’t need to be. But there is a better way.

Enter cibuildwheel

Instead of a CI-specific solution, cibuildwheel is a Python package from PyPI. It is a bit like tox or Nox, but designed specifically for building wheels. It handles all the above complications for you, making wheel building beautiful, simple, and very powerful. The simplest project in GitHub Action looks like this:

on: [push, pull_request]

jobs:
  build_wheels:
    name: Build wheels on ${{ matrix.os }}
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-20.04, windows-2019, macOS-10.15]

    steps:
      - uses: actions/checkout@v2

      - uses: pypa/cibuildwheel@v1.12.0

      - uses: actions/upload-artifact@v2
        with:
          path: ./wheelhouse/*.whl

Think I’m cheating by using an action? Replace that line with run: pipx run cibuildwheel==1.12.0 and it still works. It’s just a simple Python package; it should run on any CI (and is tested against half a dozen CI systems).

There’s a lot going on here - cibuildwheel is running on each OS; it detects it is in GitHub Actions and enables smart folding of output and a few other optional things.² It reads the PEP 621 value for Python-Requires (it falls back to setup.cfg or setup.py simple AST parsing if it can’t find it there) and limits the versions of Pythons it tries to build based on that setting. From the list of possible “auto” architectures (which include 32 bit architectures on Windows and Linux), and PyPy, it builds one wheel for each. If you tell cibuildwheel how to run your tests, it will test those too, in a separate environment.

To further configure cibuildwheel, environment variables are used, of the form CIBW_* or CIBW_*_<OS>, where <OS> is one of the three major OS’s. You can use this to further split up the builds, install extra dependencies, run tests, and more. cibuildwheel’s variable system was carefully designed to ensure you almost never have to split up your matrix into separate jobs on cibuildwheel’s behalf, unless you have special workarounds such as Windows + Python 2.7 or some other horrendous concoction.

Adding Apple Silicon

This is a “default” build, but extending it to all known architectures and universal2 wheels for Apple Silicon is actually quite trivial! To add Apple Silicon:

env:
  CIBW_ARCHS_MACOS: auto universal2
  CIBW_TEST_SKIP: "*universal2:arm64"

This just adds the universal2 arch (for Python 3.9, the only Python to support Apple Silicon currently), and it skips testing the arm part of Universal2 (really just skips a warning, because you can’t emulate Arm64 on Intel). Yes, if you ran cibuildhweel from a hypothetical (at this point) macOS Apple Silicon runner, it would actually test Universal2 wheels twice, once with emulation for Intel!

Adding Linux alternative architectures

For Linux, you can build natively for Arm, PowerPC, and IBM Z, but that is only available on Travis CI at the moment(unless you use a self-hosted runner), and the recent reduction and removal of open-source support has really make running on Travis difficult, so cibuildwheel can now also run using emulation on other platforms, like GitHub Actions. This is what that would look like:

strategy:
  matrix:
    arch: [auto32, auto64, aarch64, ppc64le, s390x]
---
- name: docker/setup-qemu-action@v1
  if: runner.os == 'Linux'
  with:
    platforms: all

- uses: pypa/cibuildwheel@v1.12.0
  env:
    CIBW_ARCHS: ${{ matrix.arch }}

Distributing

You can upload artifacts on most CI systems (expect Travis CI), and then manually download them and then upload with twine. Or, you can add the following job:

  upload_all:
    needs: [build_wheels, build_win27_wheels, make_sdist] # List all jobs needed
    runs-on: ubuntu-latest
    if: github.event_name == 'release' && github.event.action == 'published'

    - uses: actions/download-artifact@v2
      with:
        name: artifact
        path: dist

    - uses: pypa/gh-action-pypi-publish@v1.4.2
      with:
        user: __token__
        password: ${{ secrets.pypi_password }}

This will trigger only when you make a release in the GitHub UI or command line client, and will upload all wheels (and your SDist) to PyPI for you. See Scikit-HEP: GHA Wheels for the full example and more info.

Other details

The Tips and tricks page has points for common issues you may face building wheels, such as how to avoid an extra dependency when compiling with MSVC 2019. Also see the modern C++ standards page, especially if you need to support Python 2.7 and Windows.

Best practices: package design

Beyond just cibuildwheel, there are several practices you should follow that will both help cibuildwheel produce your package, and help your users. I would highly recommend you visit Scikit-HEP: packaging, which is a great resource for the finer points of making a conformant, easy to install and build project (and not HEP-specific). Then visit Scikit-HEP: GHA Wheels, which as details on setting up a GitHub Actions build for a cibuildwheel project.

In short:

• Always have a pyproject.toml. This will ensure reproducible builds, which is useful in cibuildwheel and important if a user doesn’t manage to get a wheel (say, a Alpine Linux or ClearLinux user).
  • See the NumPy tip if you use Cython.
• Use setup.cfg as much as possible. This keeps your setup.py simple and build-focused, and makes metadata (python_requires) easy to parse by cibuildwheel. It also avoids common mistakes that tend to pop up on Windows and PyPy.
• Use a test extra and tell cibuildwheel to install it for testing via CIBW_TEST_EXTRAS. Also set CIBW_TEST_COMMAND to pytest {project}/tests.

Other resources

If you’d like to learn more about building wheels, here are some of the other sources you can consult:

• My followup post over new features in 1.8.0 and 1.9.0
• azure-wheel-helpers, with detailed writeup here, where this is done from scratch.
• multibuild, the original tool from Matthew Brett, used by NumPy and more. Lots of shell scripts.
• azure-pipeline-templates, a tool used by Anthony Sottile. Doesn’t do old versions of macOS.