🎡 cibuildwheel 2.10

 Posted on September 13, 2022  (Last modified on October 14, 2022)  |  Henry Schreiner

cibuildwheel 2.10 is out, with some important additions. PEP 517 config settings added, --only (which has an interesting use in GHA), and Cirrus CI support (including our first Apple Silicon native runner!) are highlights. We also support Python 3.11 now (as of 2.11.2, RC’s in older releases).

We’ve had some fantastic releases of cibuildwheel this year, including some very powerful features you might be interested in using, and I haven’t covered releases since 2.2, so let’s take an in-depth look at what’s new for this and the last few releases!

PEP 517 config settings (2.10)

As we start getting more PEP 517 builders (I’m working on a Scikit-build rewrite, for example), and as setuptools tries to move to using the newer mechanisms, being able to pass configuration in to the backend. There’s a mechanism for doing this specified in PEP 517 and supported by pip and build, and we provide support for this now, too.

Here’s an example passing --build-option through:

[tool.cibuildwheel.config-settings]
--build-option = "--use-mypyc"

Note that pip and build differ in support for arrays - build supports arrays of options, while pip only supports single options, so using an array for an option requires the build backend.

Only (2.10)

There’s a new way to run cibuildwheel; this was designed for easy local running of a specific wheel (supported since 2.4, and linux from windows since 2.8). You can pass --only <identifier> to build just one identifier. You don’t need to specify --platform or build/skip/arch settings. This is really handy if you need to test a build locally.

It turns out this is also very useful for GitHub Actions, as well; this enables an easy way to dynamically generate a build matrix for a wheel per job! You start by producing a JSON build matrix:

jobs:
  generate-wheels-matrix:
    name: Generate wheels matrix
    runs-on: ubuntu-latest
    outputs:
      include: ${{ steps.set-matrix.outputs.include }}
    steps:
      - uses: actions/checkout@v3
      - name: Install cibuildwheel
        run: pipx install cibuildwheel==2.11.2
      - id: set-matrix
        run: |
          MATRIX=$(
            {
              cibuildwheel --print-build-identifiers --platform linux \
              | jq -nRc '{"only": inputs, "os": "ubuntu-latest"}' \
              && cibuildwheel --print-build-identifiers --platform macos \
              | jq -nRc '{"only": inputs, "os": "macos-latest"}' \
              && cibuildwheel --print-build-identifiers --platform windows \
              | jq -nRc '{"only": inputs, "os": "windows-latest"}'
            } | jq -sc
          )
          echo "include=$MATRIX" >> $GITHUB_OUTPUT          
    env:
      CIBW_ARCHS_LINUX: x86_64
      CIBW_ARCHS_MACOS: universal2
      CIBW_ARCHS_WINDOWS: x86 AMD64

Now you can use this dynamically generated matrix to create one job per wheel:

build-wheels:
  name: Build ${{ matrix.only }}
  needs: generate-wheels-matrix
  strategy:
    matrix:
      include: ${{ fromJson(needs.generate-wheels-matrix.outputs.include) }}
  runs-on: ${{ matrix.os }}
  steps:
    - uses: actions/checkout@v3

    - uses: pypa/cibuildwheel@v2.11.2
      with:
        only: ${{ matrix.only }}

A few things to keep in mind currently:

  • auto/auto64/auto32 is guessed from your running arch; you can specify the architectures explicitly.
  • You need to keep the cibuildwheel version numbers in sync manually (Dependabot won’t update both).
  • Remember that before-all runs on every job, so if it does something expensive, you might want to group wheels.

You should be able to combine classic matrices and generated matrices, as well; just generate the “missing” only’s. --only does nothing if it is empty. One wheel per job is probably better for emulated architectures, for example. Remember to activate emulation in the build matrix if you do that!

- name: Set up QEMU
  if: runner.os == 'Linux'
  uses: docker/setup-qemu-action@v1
  with:
    platforms: all

SDist support (2.5)

You can now build directly from an SDist by pointing cibuildwheel to an SDist instead of a directory. You can setup a step to produce or even download an SDist, then you can run cibuildwheel on that SDist. You can get your cibuildwheel configuration file from the SDist (defaults to {package}/pyproject.toml), or provide a local file to --config-file.

This has been used to build forges, as well, like galaxyproject/wheelforge.

Environment pass-in (2.3)

On linux, cibuildwheel uses a container, so the environment is isolated from the host. You’ve always been able to set environment variables manually, but sometimes (especially in static config), you really just want to pass some variables through. Now you can:

[tool.cibuildwheel.linux]
environment-pass = ["BUILD_TIME", "SAMPLE_TEXT"]

Not all variables should be passed in - specifically, things with host paths likely are not valid in the container.

Platform info (2.3-2.10)

Let’s combine the various platforms changed and added into one list:

  • You can make builds from Apple Silicon too now (Cirrus CI is the first runner). (2.10)
  • Only native wheels are built on AS by default, like Intel (earlier experimental support produced Universal2 also). (2.9)
  • Added support for manylinux_2_28, which is the “next” RHEL-based runner after manylinux2014. The Debian-based manylinux_2_24 has been deprecated by ManyLinux. (2.7)
  • Default updated to manylinux2014 now, consistent across Python versions and implementations. Older images have been deprecated by ManyLinux. (2.3)
  • Python 3.7+ required to run (you can still target 3.6).
  • CPython 3.11 wheels built by default using the RC (ABI stable); you should be building wheels in preparation for the 3.11 final release. Many of our users, including NumPy, have already released 3.11 wheels. (2.6 prerelease, 2.9 RC and default)
  • Podman supported in addition to Docker for building Linux images, use CIBW_CONTAINER_ENGINE.
  • Linux wheels can be built locally from Windows via Docker (2.8)
  • Integrated support for building ABI3 wheels; if multiple Python versions are present, the wheel will only be tested on the newer Python versions. (2.5)
  • You can run cibuildwheel locally on Windows or macOS, in addition to Linux. (2.4)
  • PyPy 3.9 support added (2.4)
  • Experimental Windows ARM support from (hypothetical still) Windows ARM runners (2.3)

As a quick note, even if you support older versions of manylinux, it is highly recommended you avoid using them for newer Pythons. There’s generally no need to produce a manylinux1 wheel after 3.8, since the last version of pip that can’t use manylinux2010 officially didn’t support 3.7 and technically doesn’t support 3.9. manylinux2010 was dropped before 3.11 and is a bad idea for 3.10 (NumPy for example only provides manylinux2014 for 3.10+).

My personal recommendation is to only support Python 3.7+ and use manylinux2014, unless you need manylinux_2_28. There are enough copies of pip 9 around on Python 3.6 that it’s best just to remove support (by increasing your requires-python lower limit) and let old software users get your old software releases. Trying to support Python 3.6 can actually hurt your Python 3.6 users.

Improved Action with compatible Python version (2.8.1)

We moved to a non-leaky mechanism of ensuring an available Python version that is compatible with cibuildwheel is provided. This required adding a new feature to actions/setup-python (in version 4.1) to disable the environment change. Combined with composite actions, this is a useful way for actions to ensure they have a working Python version but avoid being leaky (wntrblm/nox does this now too).

This is basically how cibuildwheel and nox work; you could use this to easily provide an action for your own package:

name: mypkg
description: "Runs a package"

runs:
  using: composite
  steps:
    - uses: actions/setup-python@v4
      id: python
      with:
        python-version: "3.7 - 3.10"
        update-environment: false

    - run: >
        pipx run --python '${{ steps.python.outputs.python-path }}' --spec '${{
        github.action_path }}' mypkg        
      shell: bash

Other things

We now suggest possible corrections if you misspell something in the TOML config (3.9).

There have been some docs updates too, including this great interactive diagram.

We will continue to track new releases of Python, and look for ways to improve. There’s a project working on Windows ARM cross-compiling. cibuildwheel is part of the PyPA Discord server now.

Bonus: Simpler Dependabot

While not explicitly a change in cibuildwheel, GitHub’s Dependabot now avoids changing v3 into v3.0.1, so a simpler dependabot config now works to maintain your actions:

version: 2
updates:
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "daily"
programming  python  cibuildwheel 
comments powered by Disqus