🎡 cibuildwheel 1.8.0 and 1.9.0

cibuildwheel has just had two back-to-back releases, two weeks apart, representing several months of hard work and some exciting few features! I will be covering both releases at once, so we will discuss Apple Silicon support, architecture emulation on Linux, integrated PEP 621 Requires-Python support, the native GitHub Action, extended build and test controls, and more!

If you are following the releases, 1.7.0 came out last November (2020), and included the fantastic output folding feature, which makes logs much easier to read on CI systems that support folding, and makes it much easier to see how long each step takes. The 1.7.x series also included the addition of the working examples section of the documentation, which tracks some known projects using cibuildwheel, such as scikit-learn, Matlotlib, and MyPy; it is a great place to go to look into how other projects have integrated cibuildwheel into their workflow.

I have an general overview post as well. Now let’s look at what’s new! Update: cibuildwheel is now an official package of the PyPA!

Architectures: Emulation

The main new feature of both 1.8 and 1.9 is the architecture option. In 1.7, it enabled Linux emulation, and in 1.8, Apple Silicon support and a shortcut for splitting based on 32/64 bit.

For Linux, you could build natively for ARM, PowerPC, and IBM Z, but that was only available on Travis CI (unless you use a self-hosted runner), and the recent reduction and removal of open-source support has really make running on Travis difficult, so cibuildwheel can now also run using emulation on other platforms, like GitHub Actions. This is what that would look like:

strategy:
  matrix:
    arch: [auto32, auto64, aarch64, ppc64le, s390x]
---
- uses: docker/setup-qemu-action@v1
  if: runner.os == 'Linux'
  with:
    platforms: all

- uses: pypa/cibuildwheel@v1.12.0
  env:
    CIBW_ARCHS: ${{ matrix.arch }}

You an also use all as the architecture, and cibuildwheel will include all architectures, and you can use build-selectors to pick what you want to build; some may find that more natural. But keep in mind that we may add architectures in the future if the PPA adds them to the manylinux spec. You can also use auto32 and auto64, as well as native. The default is auto.

Read the docs.

Architectures: Apple Silicon Support

The headline feature of 1.8.0 was the support for Apple Silicon (AS) cross-compilation, along with preliminary support for running on an Apple Silicon runner (no commercially available systems have AS runners yet, so this can only be prototyped locally currently). This feature is closely tied to the Packaging 20.9 and Pip 21.0.1 releases, which make it possible to load universal2 wheels on AS. As a reminder, there are three kinds of wheels for macOS now (starting with Python 3.9):

• x86_64: Classic wheel, works on Intel. As a reminder, Pip 20.3+ is required on Big Sur to load wheels, even for Intel, due to the numbering change from 10.x to 11.
• arm64: Runs on Apple Silicon only.
• universal2: Runs on Intel with Pip 20.3+ or AS with Pip 21.0.1+. Can be up to 2x larger, since it contains binaries for both architectures; data files are not duplicated, though.

For now, most projects should probably ship x86_64 + universal2 wheels; or possibly arm64 instead for large projects or ones that do not have universal2 libraries available in their dependencies. For Python 3.10, there will be no reason to ship x86_64 also, since universal2 will always work on whatever version of Pip supports Python 3.10.

cibuildwheel builds the auto architecture by default, which is every natively runnable and testable architecture; Intel runners cannot run AS portions of wheels, so it does not include universal2; you have to ask for it. This is what it might look like:

env:
  CIBW_ARCHS_MACOS: auto universal2
  CIBW_TEST_SKIP: "*universal2:arm64"

This will add the universal2 architecture, and will explicitly indicate you are okay to skip testing on it (otherwise you’ll see a warning).

On an AS runner (not currently available in CI), cibuildwheel installs and runs your tests twice; once on AS, and once emulating Intel!

If you use a setuptools based build, it should work out-of-the-box, though you might have to work to ensure any dependencies are of the correct architecture. If you are wrapping CMake, you will likely need to add MACOSX_DEPLOYMENT_TARGET to CIBW_ENVIRONMENT (may be fixed in a future release), add CMAKE_OSX_ARCHITECTURES to CIBW_ENVIRONMENT, such as CMAKE_OSX_ARCHITECTURES=x86_64;arm64. Scikit-Build will need to be updated to work correctly.

Read the docs.

Native GitHub Action

cibuildwheel now has a native GitHub Action, which can be use like this:

- uses: actions/checkout@v2
- uses: pypa/cibuildwheel@v1.12.0
- uses: actions/upload-artifact@v2
  with:
    path: ./wheelhouse/*.whl

With the action, you don’t need to set up a host Python (internally, it uses pipx run, since pipx is supported by GitHub Actions; in fact, they use it to install some of their Python apps). It avoids separate install/run steps. Most importantly, it is easy to schedule weekly updates via GitHub’s Dependabot; just add .github/dependabot.yml:

version: 2
updates:
  # Maintain dependencies for GitHub Actions
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"
    ignore:
      # Official actions have moving tags like v1
      # that are used, so they don't need updates here
      - dependency-name: "actions/checkout"
      - dependency-name: "actions/upload-artifact"
      - dependency-name: "actions/download-artifact"

Now you’ll get a PR, at most once per week, with updates to cibuildwheel. You now should be able to stay up to date with cibuildwheel, but still avoid any last minute surprises when releasing wheels.

Note that the action is really equivalent to the following line:

- run: pipx run cibuildwheel==1.12.0

but with support for Dependabot.

Read the docs.

Test Skipping

With the expanded architecture builds, there is an increasing amount of strain on testing in a matrix. For example, since we support more wheels than NumPy provides, a source build could be triggered by the test phase.¹ Or you might have slow tests that are even slower when running on emulation. You can now list test identifiers to skip, so activating tests is no longer all or nothing! It looks like this:

CIBW_TEST_SKIP: "*-universal2:arm64"

The syntax is identical to CIBW_SKIP, except for universal2, which has two test identifiers, cp39-macosx_universal2:arm64 and cp39-macosx_universal2:x86_64; on an Apple Silicon runner (which does not exist yet), this really will test universal2 wheel twice, once through Intel emulation. You should manually skip the macOS arm64 tests explicitly, as otherwise you’ll get a warning that cibuildwheel can’t emulate arm64 on Intel and a test is being implicitly skipped.

Read the docs.

Brace Expansion

A smaller feature is shell-style brace expansion for identifiers is now supported. If you want to build for CPython 3.7 and 3.8 only, you could use:

CIBW_BUILD: cp{37,38}-*

which is arguably more elegant than cp3[78], and will correctly generalize to cp310 when support is enabled later this year. This also can be combined with the new test skipping feature above.

Read the docs.

Requires-Python

One of the most common issues for users just starting out with cibuildwheel is the (fairly reasonable, in 2021) consternation that they are getting errors coming from Python 2.7 or maybe even Python 3.5. While we continue to match manylinux in support, we have solved this by respecting the Requires-Python setting in your project, as long as you set it using the PEP 621 location in project.requires-python in your pyproject.toml, or the setuptools specific location options.requires_python in setup.cfg, or even the requires_python keyword argument in setup.py if the AST is simple enough.

FYI, manylinux 2010 is joining 2014 in dropping support for Python 2 soon. Manylinux1 will be retired this Summer (2021). And Python 3.5 will likely be retired after the Ubuntu 16.04 EOL in April.

Read the docs.

Final Wheel Printout

At the end of the beautifully folded printout from cibuildwheel’s logs, you now get a printout of exactly what wheels were produced on unsuccessful runs, eliminating the need for an ls step after cibuildwheel runs. It also prints the total time.

Error on Empty

If cibuildwheel is run and no builds are selected, you now get an error (which you may have expected, but before it did not). This is more likely to catch errors than cause problems, but if it does, or if you like setting CIBW_SKIP: "*" to disable builds for testing², then you can pass --allow-empty on the command line.

Internal Improvements

There are a host of internal improvements behind the scenes to make sure cibuildwheel is stable, reliable, and up to date:

• The codebase is now fully statically typed, passing the equivalent of the --strict setting on MyPy.
• An error is now thrown if the image option is empty, which can happen if there is a mistake in your configuration.
• The Python identifiers are now stored in TOML format, instead of hard-coded in Python.
• Weekly pinned dependency updates are now automated by a GitHub Actions bot.
• Python version automation is included in the weekly update, updating the TOML file.
• The codebase is fully pre-commit style checked.
• We use setup.cfg instead of setup.py for most settings, and now have installable extras for simpler docs and testing.
• Big docs updates (and still ongoing).

The Future

Work is being done to provide a bit more support for the Limited API mode of CPython 3+; soon cibuildwheel may just build the first Limited API wheel, and then test on each remaining version of CPython. We will be ready to enable CPython 3.10 when it gets close enough to release.