Another great release from cibuildwheel, 2.2.0, is out! There are a few important additions in this release that you should be aware of, so I will outline the major changes here. We will cover the new musllinux wheels, overload configuration, and incoming changes to pip and PyPy expected in the next release. As always, it is recommended that you pin your cibuildwheel version and then provide some automated way to keep the pin up-to-date, such as GitHub’s dependabot. You should be updating just before you make a release, as well, but you probably don’t want to be surprised by new wheels during your release process!
If you are building Python 3.10 wheels, you probably are also interested in overrides, since it’s quite likely you will want a newer manylinux image for 3.10 than for older versions of Python (see NumPy, for example, which only provides manylinux2014 for Python 3.10).
A major, concerted effort across all of Python packaging, from standards writing PEP 656, pip 21.2, packaging 21.0, auditwheel 4.0, manylinux, and of course cibuildwheel has added a new platform: musllinux! This is a distribution that covers the largest missing component of manylinux: MUSL based distributions of Linux (like Alpine). Manylinux, as you might recall, is based on GLIBC. Alpine users are starting to get binaries when pip installing for the first time. Alpine is a very popular docker image that is just under 5 MB in size.
To support this, new identifiers have been added, using
musllinux instead of
manylinux. There is currently one image for musllinux,
musllinux_1_1, but as
always you can select or pin musllinux images. If you want to skip musllinux
wheels, you will need to add
*musllinux* to your skip list, or explicitly
*manylinux* for linux builds.
This also means that installing things inside the images can very even further;
manylinux1 (CentOS 5),
manylinux2010 (CentOS 6), and
(CentOS 7) all use
manylinux_2_24 (Debian 8) uses
apt (and is stuck
on GCC 6), while
apk. To help manage this, a new
cibuildwheel feature was added, and we will cover that next.
Remember, the manylinux1 image has been in maintenance mode, has no support for Python 3.10, and will be fully retired at the end of 2021. The main reason for manylinux1 is actually pip support, not GLIBC support; pip 9 ships on RHEL/CentOS 7/8 and Ubuntu 18.04 by default, even though those systems easily could support manylinux2014 with a pip upgrade. Also, upgrading pip provided by a system package manager is unsupported, do it at your own risk. The correct solution? Use venvs, and upgrade your pip right after creating each venv. Package managers that force old versions can be a pain!
See pypa/manylinux#994 for the deprecation schedule and further discussion about pip versions.
A new override system was added to
cibuildwheel to make manipulating complex
builds much easier. This is very useful for both handling differences in docker
images, as well as for selecting a different base manylinux image for different
Python versions, which is becoming much more common now that Python 3.10 is out.
The system looks quite a bit like MyPy’s TOML
array, this was intentional. Let’s say you want to support manylinux 2010 for
older Python’s, but for Python 3.10, you only care about 2014, just like NumPy:
[tool.cibuildwheel] manylinux-x86_64-image = "manylinux2014" manylinux-i686-image = "manylinux2014" # Before Python 3.10, manylinux2010 is the most compatible [[tool.cibuildwheel.overrides]] select = "cp3?-*" manylinux-x86_64-image = "manylinux2010" manylinux-i686-image = "manylinux2010"
You will now launch manylinux2010 images for Python 3.x, and manylinux2014 for
Python 3.xx - anything that matches the
select statement will override. You
can also expand numbers in brackets, like
The docker image launcher is smart enough to split launches based on the image
as well as
before-all, as well, so it just works as expected. In fact, now
that PyPy is covered by the official images, CPython and PyPy now can share
Another common example, if you have a dependency (and the reason we delayed musllinux to get this feature in):
[tool.cibuildwheel.linux] before-all = [ "apt-get install libboost-dev", ] [[tool.cibuildwheel.overrides]] select = "*musllinux*" before-all = [ "apk add boost-dev", ]
select matches multiple times, they are overridden in order, the last one
“winning”, per option. Remember that environment variable specifications still
replace older ones; you can’t “add” one environment variable and keep the
others. The double brackets on the header are important - this is a TOML list
Keep in mind, if you don’t want to put your cibuildwheel configuration in your
pyproject.toml, you can specify your own config file for cibuildwheel. I would
recommend keeping the
pyproject.toml file as a generic recipe for building
anywhere, and leaving only CI specific parts like your matrix controls.
pipx run cibuildwheel --platform linux should work out of the box on any
machine, ideally. Most settings, like
test-requirements are general, and are a property of your
project, not your CI runner.
If your CI breaks while you are trying to release and you have to build manually locally, you’ll thank me for this advice one day. Using the config mode is great if you ever need to change CIs, or if you split your builds across CI systems. And there are fewer potential collisions in syntax in TOML than YAML, due to most CI’s substitution systems.
An empty manylinux image environment variable will no longer break CI, but will just fall though (regression in 2.0). This is useful in making simpler matrices.
Joining the most of the PyPA packages, we now support TOML 1.0 (by moving to the
tomli from the mostly unmaintained
PyPy is now supported on macOS 11 runners; this has been supported for the last
PyPy release or two, but we were still filtering on macOS 11. PyPy still does
not support Apple Silicon, sadly. You can run cibuildwheel from Python 3.10
now if you’d like (we’ve been building them since the betas, and they have been
on by default since RC 1). GitHub Actions
is currently moving
macos-11, so this is quite timely.
Since I’d rather not write a new post here for our next release, here’s what we have nearly ready:
Pip 21.3 (soon)
As soon as TravisCI fixes their broken Sao Paulo workers (or we switch to emulation on GHA), manylinux should be able to release new images with updated Pip. This version of pip no longer copies the build directory by default, so there will be less copying then before, hopefully making some users with very large packages quite happy. Until that happens, we’ll be pinned to Pip 21.2 across all platforms for consistency (you can always unpin or change pins).
PyPy 7.3.6 (soon)
TravisCI for manylinux is also holding up the update to PyPy 7.3.6, which will add a new Python version - pypy3.8 will join pypy3.7. Expect a new PyPy target when that happens for 3.8.
Other things (soon)
We have a potential fix for using single curly brackets in commands (currently interferes with our command substitution). Specifying a different config-file should be possible using the action soon.