Python

This is the first of two posts on cibuildwheel, a fantastic project I joined after switching to it from my own azure-wheel-helpers, which I’ve blogged about here before. It is the best wheelbuilding system available for Python today, and can make something that is normally a pain to setup and a headache to maintain a breeze (by forcing all the headaches on us, of course, as maintainers, but it’s better to solve issues centrally! Obviously we rather like solving these problems. Or we are just crazy, which is also possible ;) ).

Be sure to checkout the followup post over new features in 1.8.0 and 1.9.0, too! Also, cibuildwheel was recently accepted into the PyPA!

I am pleased to announce the release of pybind11 2.6.0! This is the largest release since 2.2 (released over three years ago). I would like to highlight some of the key changes below; be sure to check out the changelog and upgrade guide for more information! The focus of this release was stability, packaging, and supporting more platforms, though there are a lot of small features and useful additions, covered by newly expanded docs.

COVID-19 is ravaging the globe. Let’s look at the excellent Johns Hopkins dataset using Pandas. This will serve both as a guideline for getting the data and exploring on your own, as well as an example of Pandas multi-indexing in an easy to understand situation. I am currently involved in science-responds.

The foundational histogramming package for Python, boost-histogram, hit beta status with version 0.6! This is a major update to the new Boost.Histogram bindings. Since I have not written about boost-histogram yet here, I will introduce the library in its current state. Version 0.6.2 was based on the recently released Boost C++ Libraries 1.72 Histogram package. Feel free to visit the docs, or keep reading this post.

This Python library is part of a larger picture in the Scikit-HEP ecosystem of tools for Particle Physics and is funded by DIANA/HEP and IRIS-HEP. It is the core library for making and manipulating histograms. Other packages are under development to provide a complete set of tools to work with and visualize histograms. The Aghast package is designed to convert between popular histogram formats, and the Hist package will be designed to make common analysis tasks simple, like plotting via tools such as the mplhep package. Hist and Aghast will be initially driven by HEP (High Energy Physics and Particle Physics) needs, but outside issues and contributions are welcome and encouraged.

Python 3.8 is out, with new features and changes. The themes for this release have been performance, ABI/internals, and static typing, along with a smattering of new syntax. Given the recent community statement on Python support, we should be staying up to date with the current changes in Python. As Python 2 sunsets, we are finally in an era where we can hope to someday use the features we see coming out of Python release again!

Note: I now highly recommend cibuildwheel instead of custom binary wheels. See GHA Pure Python Wheels and GHA Binary Wheels for modern methods to produce wheels on GitHub Actions (directly applicable to Azure, as well, with minor changes; cibuildwheel works on almost all major CI providers). See my new posts on cibuildwheel!

This is the third post in a series about Azure DevOps. This one is about making Python wheels. If you want to play nice with Python users, or you have a complex build, this will make your package far more accessible to users. They are faster to install and to use and more secure. We will quickly cover making universal wheels, then we will move on to fully compiled binaries, including C++14, manylinux2010, and other hot topics. This series was developed to update the testing and releasing of Python packages for Scikit-HEP. The results of this tutorial can be seen in the boost-histogram repository, under the .ci folder.

This is the second post in a series about Azure DevOps. This one is about release pipelines; if you use Azure to build packages (like binaries, etc.), how do you push them to a final endpoint? In this example, we will be building a simple pure Python package, and pushing the result to Test-PyPI. You can adapt it to your situation, however. The third post will cover building Python binaries. This series was developed to update the testing and releasing of Python packages for Scikit-HEP. Several of the projects in Scikit-HEP are using release pipelines, including boost-histogram and Particle.

Note: I now highly recommend GitHub Actions, which is almost “Azure 2.0”. You can read my tutorials on GitHub Actions on the Scikit-HEP developer pages. The release process, in particular, is simpler, and you still get the benefit of artifacts.

Continuous Integration (CI) is fantastic for software development and deployment. One of the newest entries into the CI market¹ is Microsoft’s Azure DevOps. Their Open Source support is impressive; it is likely part of the recent push² by Microsoft to be more Open Source friendly. Open Source projects get 10 parallel builds, unlimited build minutes, 6 hour job timeouts, and incredibly fast jobs on macOS, Linux, and Windows, all via a single platform. Quite a few major projects³ have been moving to Azure since the initial release in December 2018. The configuration of DevOps is second only to GitLab CI in ease of use and possibly the most expressive system available. The multiple pipeline support also scales well to complicated procedures.

This is the first in a series of posts covering an introduction to setting up projects in Azure DevOps, developed to update the testing and releasing of Python packages for Scikit-HEP, a project for a coherent High Energy Physics Python analysis toolset. The second post covers release pipelines, and the third covers building binary Python packages using DevOps.

Note: I now highly recommend GitHub Actions, which is almost “Azure 2.0”, if you are interested in setting up CI. The language is very similar, although simplified, with some non-backward compatible bugfixes (such as multiline expressions will error if any line files, instead of just on the last line in Azure). You can read my tutorials on GitHub Actions on the Scikit-HEP developer pages.