🎡 cibuildwheel 2.2

Another great release from cibuildwheel, 2.2.0, is out! There are a few important additions in this release that you should be aware of, so I will outline the major changes here. We will cover the new musllinux wheels, overload configuration, and incoming changes to pip and PyPy expected in the next release. As always, it is recommended that you pin your cibuildwheel version and then provide some automated way to keep the pin up-to-date, such as GitHub’s dependabot. You should be updating just before you make a release, as well, but you probably don’t want to be surprised by new wheels during your release process!

If you are building Python 3.10 wheels, you probably are also interested in overrides, since it’s quite likely you will want a newer manylinux image for 3.10 than for older versions of Python (see NumPy, for example, which only provides manylinux2014 for Python 3.10).

I also have several other posts on cibuildwheel, including my general overview post. See them all here.

musllinux

A major, concerted effort across all of Python packaging, from standards writing PEP 656, pip 21.2, packaging 21.0, auditwheel 4.0, manylinux, and of course cibuildwheel has added a new platform: musllinux! This is a distribution that covers the largest missing component of manylinux: MUSL based distributions of Linux (like Alpine). Manylinux, as you might recall, is based on GLIBC. Alpine users are starting to get binaries when pip installing for the first time. Alpine is a very popular docker image that is just under 5 MB in size.

To support this, new identifiers have been added, using musllinux instead of manylinux. There is currently one image for musllinux, musllinux_1_1, but as always you can select or pin musllinux images. If you want to skip musllinux wheels, you will need to add *musllinux* to your skip list, or explicitly request *manylinux* for linux builds.

This also means that installing things inside the images can very even further; manylinux1 (CentOS 5), manylinux2010 (CentOS 6), and manylinux2014 (CentOS 7) all use yum; manylinux_2_24 (Debian 8) uses apt (and is stuck on GCC 6), while musllinux_1_1 uses apk. To help manage this, a new cibuildwheel feature was added, and we will cover that next.

Overrides

A new override system was added to cibuildwheel to make manipulating complex builds much easier. This is very useful for both handling differences in docker images, as well as for selecting a different base manylinux image for different Python versions, which is becoming much more common now that Python 3.10 is out.

The system looks quite a bit like MyPy’s TOML overrides array, this was intentional. Let’s say you want to support manylinux 2010 for older Python’s, but for Python 3.10, you only care about 2014, just like NumPy:

[tool.cibuildwheel]
manylinux-x86_64-image = "manylinux2014"
manylinux-i686-image = "manylinux2014"

# Before Python 3.10, manylinux2010 is the most compatible
[[tool.cibuildwheel.overrides]]
select = "cp3?-*"
manylinux-x86_64-image = "manylinux2010"
manylinux-i686-image = "manylinux2010"

You will now launch manylinux2010 images for Python 3.x, and manylinux2014 for Python 3.xx - anything that matches the select statement will override. You can also expand numbers in brackets, like cp3{6,7,8,9}.

The docker image launcher is smart enough to split launches based on the image as well as before-all, as well, so it just works as expected. In fact, now that PyPy is covered by the official images, CPython and PyPy now can share before-all.

Another common example, if you have a dependency (and the reason we delayed musllinux to get this feature in):

[tool.cibuildwheel.linux]
before-all = [
  "apt-get install libboost-dev",
]

[[tool.cibuildwheel.overrides]]
select = "*musllinux*"
before-all = [
  "apk add boost-dev",
]

If select matches multiple times, they are overridden in order, the last one “winning”, per option. Remember that environment variable specifications still replace older ones; you can’t “add” one environment variable and keep the others. The double brackets on the header are important - this is a TOML list item.

Keep in mind, if you don’t want to put your cibuildwheel configuration in your pyproject.toml, you can specify your own config file for cibuildwheel. I would recommend keeping the pyproject.toml file as a generic recipe for building anywhere, and leaving only CI specific parts like your matrix controls. pipx run cibuildwheel --platform linux should work out of the box on any machine, ideally. Most settings, like test-command and test-extras/test-requirements are general, and are a property of your project, not your CI runner.

If your CI breaks while you are trying to release and you have to build manually locally, you’ll thank me for this advice one day. Using the config mode is great if you ever need to change CIs, or if you split your builds across CI systems. And there are fewer potential collisions in syntax in TOML than YAML, due to most CI’s substitution systems.

Other fixes

An empty manylinux image environment variable will no longer break CI, but will just fall though (regression in 2.0). This is useful in making simpler matrices.

Joining the most of the PyPA packages, we now support TOML 1.0 (by moving to the faster tomli from the mostly unmaintained toml package).

PyPy is now supported on macOS 11 runners; this has been supported for the last PyPy release or two, but we were still filtering on macOS 11. PyPy still does not support Apple Silicon, sadly. You can run cibuildwheel from Python 3.10 now if you’d like (we’ve been building them since the betas, and they have been on by default since RC 1). GitHub Actions is currently moving macos-latest to macos-11, so this is quite timely.

Coming soon

Since I’d rather not write a new post here for our next release, here’s what we have nearly ready:

Pip 21.3 (soon)

As soon as TravisCI fixes their broken Sao Paulo workers (or we switch to emulation on GHA), manylinux should be able to release new images with updated Pip. This version of pip no longer copies the build directory by default, so there will be less copying then before, hopefully making some users with very large packages quite happy. Until that happens, we’ll be pinned to Pip 21.2 across all platforms for consistency (you can always unpin or change pins).

PyPy 7.3.6 (soon)

TravisCI for manylinux is also holding up the update to PyPy 7.3.6, which will add a new Python version - pypy3.8 will join pypy3.7. Expect a new PyPy target when that happens for 3.8.

Other things (soon)

We have a potential fix for using single curly brackets in commands (currently interferes with our command substitution). Specifying a different config-file should be possible using the action soon.