Programming on ISciNumPy.dev

Repo Review 1.0

henryfs@princeton.edu (Henry Schreiner) — Sat, 04 Apr 2026 10:52:51 -0500

I’ve just released repo-review 1.0 with a huge update to the WebApp, along with some other solid improvements in speed, simplicity, and stability. Repo-review’s two most popular plugins, sp-repo-review and validate-pyproject, can take full advantage of the new asynchronous fetching mechanism. And if you use the upcoming Python 3.15, the CLI is more responsive than ever thanks to lazy loading!

Working to make Python lazy

henryfs@princeton.edu (Henry Schreiner) — Tue, 24 Mar 2026 09:00:00 -0400

Python 3.15a7, which is now just a uv python install 3.15 away on all major platforms, has lazy imports! This exciting feature, proposed in PEP 810, promises to make CLI applications faster (especially when using flags like --help), and could make a lot of large code with lots of imports that don’t always get used faster too. Unlike the earlier, failed attempt, this requires libraries to put in some work. I’ve developed a helper tool to make it easy; I’d like to cover what lazy imports are and how to use my tool. Since this is the first library that I used AI heavily in developing, the second half of the post will cover how my experience with AI for a task like this went.

TL;DR: run uvx flake8-lazy --apply to make your code magically faster on Python 3.15!

New in boost-histogram 1.7 / Hist 2.10

henryfs@princeton.edu (Henry Schreiner) — Fri, 06 Feb 2026 17:16:40 -0500

I’ve just released boost-histogram 1.7 and hist 2.10, with a major new feature and a big typing improvement! I’d like to quickly outline what’s new here. Python 3.10+ is now required.

How we made Python's packaging library 3x faster

henryfs@princeton.edu (Henry Schreiner) — Fri, 09 Jan 2026 13:30:00 -0500

Along with a pip (and now packaging) maintainer, Damian Shaw, I have been working on making packaging, the library behind almost all packaging related tools, faster at reading versions and specifiers, something tools like pip have to do thousands of times during resolution. Using Python 3.15’s new statistical profiler and metadata from every package ever uploaded to PyPI, I measured and improved core Packaging constructs while keeping the code readable and simple. Reading in Versions can be up to 2x faster and SpecifierSets can be up to 3x faster in packaging 26.0, now released! Other operations have been optimized, as well, up to 5x in some cases. See the announcement and release notes too; this post will focus on the performance work only.

Advent of Code 2025 in Typescript

henryfs@princeton.edu (Henry Schreiner) — Wed, 24 Dec 2025 00:09:00 -0500

After two years of Advent of Code in Rust, I thought I’d try TypeScript. I’ve always wanted to improve repo-review’s webapp, and that requires knowledge of the packaging systems for JavaScript, so I thought I’d try TypeScript this year. I also used this as an opportunity to learn more AI tooling too, mostly CoPilot in VSCode & ChatGPT. I’d like to share my experience and thoughts! My code is at aoc2025 (and aoc2024, aoc2023).

UHI 1.0: histogram serialization

henryfs@princeton.edu (Henry Schreiner) — Wed, 20 Aug 2025 16:00:00 -0400

UHI 1.0 is out, with a major new feature: a new histogram serialization spec! This spec supports multiple formats (HDF5, zip, and JSON initially), and can be supported by multiple libraries (Boost-histogram/hist initially). There’s also a new test suite helper for libraries targeting the UHI indexing spec.

pybind11 3.0.0

henryfs@princeton.edu (Henry Schreiner) — Thu, 10 Jul 2025 00:13:00 -0400

Pybind11 3.0 has been released! I would like to highlight some of the key changes below; be sure to check out the changelog and upgrade guide for more information! This release includes an ABI bump, the first required bump in many years on Unix (Windows has had required bumps more often). This release contains the multi-year smart-holder branch, multi-phase init and subinterpreter support, py::native_enum, an interface to warnings, typing improvements, and more. CMake now defaults to FindPython mode.

Support for Python 3.14, 3.14t, GraalPy, PyPy 3.11, and iOS have been added, while legacy support for Python 3.7, PyPy 3.8/3.9, and CMake <3.15 has been removed. Most deprecated features have been kept for this release, but anything producing a warning in 3.0 may be removed in a future 3.x version.

🎡 cibuildwheel 3.0

henryfs@princeton.edu (Henry Schreiner) — Thu, 05 Jun 2025 01:54:45 -0400

cibuildwheel 3.0.0 is out, with some very big additions. We’ve added GraalPy, Python 3.14 (and 3.14t) betas, and iOS support! We’ve got several new options: test-sources, test-environment, and (experimental) pyodide-version. We now fully use enable (and PyPy requires using it), and we no longer inject setuptools and wheel in build environments. Defaults have changed, too: build is now the default frontend, manylinux_2_28 is the default manylinux image, with 32-bit linux now being opt-in. We’ve removed support for Python 3.6 and 3.7, we now require 3.11+ to run cibuildwheel itself, and EoL manylinux/musllinux images now need to be fully specified.

We’ve had some fantastic releases of cibuildwheel since my last post over 2.19, so I’ll include a few of the new features from those releases, too. I’ll also note a few of the features being worked on for future releases.

Python π

henryfs@princeton.edu (Henry Schreiner) — Thu, 08 May 2025 09:05:54 -0400

Python π (3.14) is out! The big feature this time around is template strings. Free-threading is no longer experimental. There’s also lots more color (including syntax highlighting in the REPL!), remote debugging, deferred evaluation of annotations, and the usual error message and performance improvements. Subintepreters are now accessible without the C-API (finally!), and free-threaded Python is no longer experimental.

scikit-build-core 0.10

henryfs@princeton.edu (Henry Schreiner) — Mon, 05 Aug 2024 23:09:13 -0400

Scikit-build-core 0.10 is out, and it is one of the largest releases we’ve produced in terms of new configuration options. It automatically reads your minimum CMake version, you can tell it to read your scikit-build-core minimum-version, and there are settings for many of the advanced things users used to do with scikit-build (classic), like rebuilds or pure Python fallbacks on build failures.

🎡 cibuildwheel 2.19

henryfs@princeton.edu (Henry Schreiner) — Mon, 10 Jun 2024 12:05:51 -0400

cibuildwheel 2.19 is out, with some very big additions. A new platform, Pyodide, has been added for building WebAssembly wheels. We’ve added CPython 3.13 free-threaded builds, now on all OS’s. And we have an opt-in speed improvement with the build[uv] build-frontend option.

We’ve had some fantastic releases of cibuildwheel since my last post over 2.10, so I’ll include a few of the new features from those releases, too, with a highlight on a larger feature that can use more explaining: inherit for overrides.

Python 3.13

henryfs@princeton.edu (Henry Schreiner) — Thu, 09 May 2024 15:33:00 -0400

Python 3.13 betas are out (Edit: 3.13.0 is out!), which means the features are locked in. For the first time in thirty years, Python has a new, more colorful REPL! There’s also a no-GIL compile-time option (free-threaded), an optional JIT, some new typing features, and better error messages (again).

Inline run dependencies in pipx 1.4.2

henryfs@princeton.edu (Henry Schreiner) — Mon, 04 Dec 2023 15:00:06 -0500

While it can also do much more, Python is a fantastic language for writing small scripts and utilities with it’s expressive syntax and batteries-included standard library. But what if you need just a bit more? PyPI is one of the best package repositories for any language, and being able to access it without having to write a multi-file library and setting up virtual environments would be a dream - one that is becoming reality. Pipx 1.4.2 has an experimental implementation of the ~~provisionally~~ accepted PEP 723, and I’d like to show it off here, as it’s tremendously useful for simple scripts & utilities. Support is also available in Nox 2024.04.15 and Hatch 1.10.

C++23

henryfs@princeton.edu (Henry Schreiner) — Mon, 23 Oct 2023 11:00:00 -0400

C++23 is complete and pretty much finalized, so let’s explore the new features in C++, from a data science point of view. This is not as large of a release as C++20 or C++11, but it’s still a hefty release with lots of nice new features - larger than C++17.

Some of the major new features are building on themes from C++20, such as vastly expanded ranges and views, more constexpr/consteval, module support for the standard library, std::print to supplement std::format, and a std::generator to support coroutines. There are also several general features, lambdas and classes get along much better now, there are new preprocessor additions (!), a stacktrace library, flat maps/sets, and new std::optional monadics, deducing this, and many various additions.

The Scientific Python Development Guide

henryfs@princeton.edu (Henry Schreiner) — Fri, 14 Jul 2023 00:45:32 -0500

One outcome of the 2023 Scientific Python Developer Summit was the Scientific Python Development Guide, a comprehensive guide to modern Python package development, complete with a new project template supporting 12 build backends and a WebAssembly-powered checker with checks linked to the guide. The guide covers topics like modern, compiled, and classic packaging, style checks, type checking, docs, task runners, CI, tests, and much more! There also are sections of tutorials, principles, and some common patterns.

This guide (along with cookie & repo-review) started in Scikit-HEP in 2020. During the summit, it was merged with the NSLS-II guidelines, which provided the basis for the principles section. I’d like to thank and acknowledge Dan Allan and Gregory Lee for working tirelessly during the summit to rework, rewrite, merge, and fix the guide, including writing most of the tutorials pages and first patterns page, and rewriting the environment page as a tutorial.

Introducing repo-review

henryfs@princeton.edu (Henry Schreiner) — Thu, 13 Jul 2023 22:00:00 -0400

I’ve released a new¹ toolkit for running checks, similar to Ruff and Flake8 but designed to check configuration, called repo-review. It requires Python 3.10+² to run and has no built-in checks, but is easy to write plugins for. A set of checks based on the Scientific Python Development Guide (which I also have a post about!) are available as a plugin, sp-repo-review. You can run repo-review in WebAssembly (via Pyodide), or in pre-commit, or as a GitHub Action. It supports multiple output formats, including Rich, HTML, and JSON. The system is based on fixtures (like pytest) and topologically sorts requirements. You don’t need to depend on repo-review to add a repo-review plugin. You can see a live version using sp-repo-review in-place here or standalone here.

Python 3.12

henryfs@princeton.edu (Henry Schreiner) — Mon, 19 Jun 2023 17:00:00 -0400

Python 3.12’s beta’s are out, which means the features are locked in. The theme this year has been cleanup and typing. distutils has been removed, and setuptools is no longer present in default environments.

🎡 cibuildwheel 2.10

henryfs@princeton.edu (Henry Schreiner) — Tue, 13 Sep 2022 08:42:26 -0400

cibuildwheel 2.10 is out, with some important additions. PEP 517 config settings added, --only (which has an interesting use in GHA), and Cirrus CI support (including our first Apple Silicon native runner!) are highlights. We also support Python 3.11 now (as of 2.11.2, RC’s in older releases).

We’ve had some fantastic releases of cibuildwheel this year, including some very powerful features you might be interested in using, and I haven’t covered releases since 2.2, so let’s take an in-depth look at what’s new for this and the last few releases!

Python 3.10

henryfs@princeton.edu (Henry Schreiner) — Sat, 23 Jul 2022 10:00:00 -0400

Python 3.10 is out (and has been for a while, I’m late posting this), with new features and changes. The big new language feature this update is pattern matching. We get much better errors, the always-present typing improvements, and finally some real usage of the new PEG parser from 3.9.

Python 3.11

henryfs@princeton.edu (Henry Schreiner) — Wed, 11 May 2022 16:11:00 -0400

Python 3.11 has hit the beta (now released!) stage, which means no more new features. It’s a perfect time to play with it! The themes in this update are the standard ones: The faster CPython project is now fully going (3.11 is 25% faster on average), along with improved error messages, typing, and asyncio. Beyond this, the only major new feature is a library for reading TOML files; this probably only exciting if you are involved in Python packaging (but I am, so I’m excited!).

Python 3.7

henryfs@princeton.edu (Henry Schreiner) — Sat, 15 Jan 2022 22:41:05 -0500

Python 3.7 has been out for a while. In fact, it’s the oldest version of Python still receiving support when this was written. I’d still like to write a “what’s new”, targeting users who are upgrading to a Python 3.7+ only codebase, and want to know what to take advantage of!

Poetry Versions

henryfs@princeton.edu (Henry Schreiner) — Thu, 09 Dec 2021 11:00:00 -0500

Poetry provides a all in one solution to Python packaging. I want to focus on why I was quite hard on Poetry in my last post, specifically on its default version capping and solver quirks, and also a few other negative things. This is a followup to Should you have upper version bounds, which you should read before this post.

Should You Use Upper Bound Version Constraints?

henryfs@princeton.edu (Henry Schreiner) — Thu, 09 Dec 2021 10:00:00 -0500

Bound version constraints (upper caps) are starting to show up in the Python ecosystem. This is causing real world problems with libraries following this recommendation, and is likely to continue to get worse; this practice does not scale to large numbers of libraries or large numbers of users. In this discussion I would like to explain why always providing an upper limit causes far more harm than good even for true SemVer libraries, why libraries that pin upper limits require more frequent updates rather than less, and why it is not scalable. After reading this, hopefully you will always consider every cap you add, you will know the (few) places where pinning an upper limit is reasonable, and will possibly even avoid using libraries that pin upper limits needlessly until the author updates them to remove these pins.

If this 10,000 word behemoth is a bit long for you, then skip around using the table of contents, or see the TL;DR section at the end, or read version numbers by Bernát Gábor, which is shorter but is a fantastic read with good examples and cute dog pictures. Or Hynek’s Semantic Versioning Will Not Save You Be sure to check at least the JavaScript project analysis before you leave!

Also be warned, I pick on Poetry quite a bit. The rising popularity of Poetry is likely due to the simplicity of having one tool vs. many for packaging, but it happens to also have a special dependency solver, a new upper bound syntax, and a strong recommendation to always limit upper versions - in direct opposition to members of the Python core developer team and PyPA developers. Not all libraries with excessive version capping are Poetry projects (like TensorFlow), but many, many of them are. To be clear, Poetry doesn’t force version pinning on you, but it does push you really, really hard to always version cap, and it’s targeting new Python users that don’t know any better yet than to accept bad recommendations. And these affect the whole ecosystem, including users who do not use poetry, but want to depend on libraries that do! I do really like other aspects of Poetry, and would like to eventually help it build binary packages with Scikit-build (CMake) via a plugin, and it has some great developers. If I don’t pick on Poetry enough for you, don’t worry, I have a follow-up post that picks on it in much more detail. Also, check out pdm, which gives many of the benefits of Poetry while following PEP standards. Also pixi, which works with the Conda ecosystem.

If you come across something that can’t be solved, try using --exclude-newer <DATE> in uv and pdm. This limits the solve by ignoring packages newer than some date.

App vs Library

henryfs@princeton.edu (Henry Schreiner) — Thu, 09 Dec 2021 09:00:00 -0500

What is the difference between an app and a library? This seemingly simple question confuses some, and it turns out to be a harder question to answer than you might expect. While the actual distinction between these common terms will always be muddled in practice, I propose a specific definition to be used when considering dependencies. This distinction is important when discussing bound version constraints in the next post.

Scikit Build Proposal

henryfs@princeton.edu (Henry Schreiner) — Wed, 27 Oct 2021 08:00:00 -0400

I’ve spent the last few years trying to make it easy for anyone to extend Python with compiled languages. I’ve worked on pybind11, a powerful C++ library that allows users to write advanced Python extensions using just C++11, used by some of the largest projects, SciPy, PyTorch, Google, LLVM, and tens of thousands of other libraries, down to very small extensions. I also work on cibuildwheel, which makes building binaries (called wheels) on continuous integration (CI) simple. It is again powerful enough to used by huge projects, like Scikit-learn, matplotlib, mypy; and is simple enough to be used by hundreds of other packages. Recently it was accepted into the Python Packaging Authority (PyPA). There is one missing piece, though, to complete this picture of compiled extensions that easy to use for small projects, and powerful enough for large projects: the build system. I believe the solution to that is scikit-build, and I’d like to work on it over the next three years.

Scikit-build is a tool for integrating a package with a CMake build system into Python. You can utilize the vast collection of packages and projects using CMake already, and you have access to modern building features, like multithreaded builds, library discovery, superb compiler and IDE support, and all sorts of extended tooling. Modern CMake is quite pleasant to write compared to times past; I have written a book and training course on it. We ship up-to-date cmake and ninja wheels for all binary platforms.

Update: Funded! I’ll be working on this starting August 1, 2022!

I wrote a proposal for an NSF CSSI Elements project containing three parts. The first part will cover core development on Scikit-build to address the current shortcomings and to prepare it for a post-distutils (Python 3.12+) world. The second part would cover assisting libraries with a science use case in either transitioning to scikit-build (ideally from an existing CMake build system with Python bindings, but I can help mentor developers in writing bindings (ideally pybind11), setting up CI, and writing CMake code as well (see my book or workshop on Modern CMake, and I’m happy to help old scikit-build projects transition to better practices). As part of this, I would be building up the examples and documentation, leading into the third part of the proposal: A series of training events and training material, including plans for something alongside SciPy.

You can also see an outline at scikit-build/scikit-build/wiki or at the end of this post.

Thank you for all the projects! The proposal was submitted Dec 8, 2021; mid year we should find out if it was accepted!

🎡 cibuildwheel 2.2

henryfs@princeton.edu (Henry Schreiner) — Wed, 20 Oct 2021 11:27:29 -0400

Another great release from cibuildwheel, 2.2.0, is out! There are a few important additions in this release that you should be aware of, so I will outline the major changes here. We will cover the new musllinux wheels, overload configuration, and incoming changes to pip and PyPy expected in the next release. As always, it is recommended that you pin your cibuildwheel version and then provide some automated way to keep the pin up-to-date, such as GitHub’s dependabot. You should be updating just before you make a release, as well, but you probably don’t want to be surprised by new wheels during your release process!

🎡 cibuildwheel 2.0

henryfs@princeton.edu (Henry Schreiner) — Fri, 16 Jul 2021 12:23:20 -0400

The cibuildwheel package has just had a major release with some fantastic features. Python 2.7 and 3.5 support has been removed (and PyPy3.6), allowing us to update to the latest manylinux and auditwheel versions, and support the newly unified manylinux PyPy3.7 images. We now allow users to select pypa/build as a build frontend. We now have a custom option to enable pre-release Pythons (3.10 currently) for testing before they are ABI stable (please don’t release wheels until that happens). Maybe most exciting, cibuildwheel now supports configuration in pyproject.toml, allowing you to be even further isolated from dependence on your CI system; you can easily produce Linux and Windows wheels locally (macOS still installs to system locations). And, since my last post and introduction post, cibuildwheel is now part of the PyPA!

Announcing CLI11 2.0

henryfs@princeton.edu (Henry Schreiner) — Wed, 14 Jul 2021 14:34:02 -0400

CLI11, a powerful library for writing beautiful command line interfaces in C++11, has been updated to 2.0. A lot of deprecated things have been removed, and there was a lot of cleanup under-the-hood; making CLI11 simpler. A few defaults have changed slightly, like better TOML support by default.

CLI11 does a better job than ever understanding any sort of container you provide - complex numbers are natively supported, along with atomic types. A long requested feature, simple version flags, has been added. Subcommands are more customizable. And there have been quite a few bugfixes for rare issues.

Python 3.9

henryfs@princeton.edu (Henry Schreiner) — Fri, 12 Feb 2021 12:00:00 -0400

Python 3.9 is out (and has been for a while, I’m late posting this), with new features and changes. The themes for this release have been heavily internal improvements, such as a new more powerful parser, and the usual static typing improvements, along with a several bits new-user facing new syntax. This makes 3.9 a smaller release, but still it has some nice features of note.

🎡 cibuildwheel 1.8.0 and 1.9.0

henryfs@princeton.edu (Henry Schreiner) — Sun, 07 Feb 2021 20:13:41 -0500

cibuildwheel has just had two back-to-back releases, two weeks apart, representing several months of hard work and some exciting few features! I will be covering both releases at once, so we will discuss Apple Silicon support, architecture emulation on Linux, integrated PEP 621 Requires-Python support, the native GitHub Action, extended build and test controls, and more!

If you are following the releases, 1.7.0 came out last November (2020), and included the fantastic output folding feature, which makes logs much easier to read on CI systems that support folding, and makes it much easier to see how long each step takes. The 1.7.x series also included the addition of the working examples section of the documentation, which tracks some known projects using cibuildwheel, such as scikit-learn, Matlotlib, and MyPy; it is a great place to go to look into how other projects have integrated cibuildwheel into their workflow.

I have an general overview post as well. Now let’s look at what’s new! Update: cibuildwheel is now an official package of the PyPA!

Overview of cibuildwheel 🎡

henryfs@princeton.edu (Henry Schreiner) — Sat, 06 Feb 2021 20:58:16 -0500

This is the first of two posts on cibuildwheel, a fantastic project I joined after switching to it from my own azure-wheel-helpers, which I’ve blogged about here before. It is the best wheelbuilding system available for Python today, and can make something that is normally a pain to setup and a headache to maintain a breeze (by forcing all the headaches on us, of course, as maintainers, but it’s better to solve issues centrally! Obviously we rather like solving these problems. Or we are just crazy, which is also possible ;) ).

Be sure to checkout the followup post over new features in 1.8.0 and 1.9.0, too! Also, cibuildwheel was recently accepted into the PyPA!

pybind11 2.6.0

henryfs@princeton.edu (Henry Schreiner) — Wed, 21 Oct 2020 15:00:00 -0400

I am pleased to announce the release of pybind11 2.6.0! This is the largest release since 2.2 (released over three years ago). I would like to highlight some of the key changes below; be sure to check out the changelog and upgrade guide for more information! The focus of this release was stability, packaging, and supporting more platforms, though there are a lot of small features and useful additions, covered by newly expanded docs.

Johns Hopkins COVID-19 Dataset in Pandas

henryfs@princeton.edu (Henry Schreiner) — Mon, 30 Mar 2020 15:30:00 -0400

COVID-19 is ravaging the globe. Let’s look at the excellent Johns Hopkins dataset using Pandas. This will serve both as a guideline for getting the data and exploring on your own, as well as an example of Pandas multi-indexing in an easy to understand situation. I am currently involved in science-responds.

The boost-histogram beta release

henryfs@princeton.edu (Henry Schreiner) — Fri, 13 Dec 2019 11:00:00 -0500

The foundational histogramming package for Python, boost-histogram, hit beta status with version 0.6! This is a major update to the new Boost.Histogram bindings. Since I have not written about boost-histogram yet here, I will introduce the library in its current state. Version 0.6.2 was based on the recently released Boost C++ Libraries 1.72 Histogram package. Feel free to visit the docs, or keep reading this post.

This Python library is part of a larger picture in the Scikit-HEP ecosystem of tools for Particle Physics and is funded by DIANA/HEP and IRIS-HEP. It is the core library for making and manipulating histograms. Other packages are under development to provide a complete set of tools to work with and visualize histograms. The Aghast package is designed to convert between popular histogram formats, and the Hist package will be designed to make common analysis tasks simple, like plotting via tools such as the mplhep package. Hist and Aghast will be initially driven by HEP (High Energy Physics and Particle Physics) needs, but outside issues and contributions are welcome and encouraged.

Python 3.8

henryfs@princeton.edu (Henry Schreiner) — Fri, 25 Oct 2019 14:11:20 -0400

Python 3.8 is out, with new features and changes. The themes for this release have been performance, ABI/internals, and static typing, along with a smattering of new syntax. Given the recent community statement on Python support, we should be staying up to date with the current changes in Python. As Python 2 sunsets, we are finally in an era where we can hope to someday use the features we see coming out of Python release again!

C++20

henryfs@princeton.edu (Henry Schreiner) — Sat, 20 Jul 2019 22:42:21 -0400

The final meeting for new features in C++ is over, so let’s explore the new features in C++, from a data science point of view. This is the largest release of C++ since C++11, and when you consider C++14 and C++17 to be interim releases, the entire 9 year cycle is possibly the largest yet! It may not feel quite as massive as C++11, since we didn’t have interim releases for C++11 and because C++11 is a much more complete, useful language than C++03, but this is still a really impactful release! This is also the first version to include every major feature from Bjarne Stroustrup’s “The design and Evolution of C++”, also known as D&E, from 1994!

Let’s look at the major new features, as well as collections of smaller ones.

Azure DevOps: Python Wheels

henryfs@princeton.edu (Henry Schreiner) — Tue, 07 May 2019 22:00:00 +0200

Note: I now highly recommend cibuildwheel instead of custom binary wheels. See GHA Pure Python Wheels and GHA Binary Wheels for modern methods to produce wheels on GitHub Actions (directly applicable to Azure, as well, with minor changes; cibuildwheel works on all most major CI providers). See my new posts on cibuildwheel!

This is the third post in a series about Azure DevOps. This one is about making Python wheels. If you want to play nice with Python users, or you have a complex build, this will make your package far more accessible to users. They are faster to install and to use and more secure. We will quickly cover making universal wheels, then we will move on to fully compiled binaries, including C++14, manylinux2010, and other hot topics. This series was developed to update the testing and releasing of Python packages for Scikit-HEP. The results of this tutorial can be seen in the boost-histogram repository, under the .ci folder.

Azure DevOps: Releases

henryfs@princeton.edu (Henry Schreiner) — Tue, 07 May 2019 21:00:00 +0200

This is the second post in a series about Azure DevOps. This one is about release pipelines; if you use Azure to build packages (like binaries, etc.), how do you push them to a final endpoint? In this example, we will be building a simple pure Python package, and pushing the result to Test-PyPI. You can adapt it to your situation, however. The third post will cover building Python binaries. This series was developed to update the testing and releasing of Python packages for Scikit-HEP. Several of the projects in SciKit-HEP are using release pipelines, include boost-histogram and Particle.

Note: I now highly recommend GitHub Actions, which is almost “Azure 2.0”. You can read my tutorials on GitHub Actions on the Scikit-HEP developer pages. The release process, in particular, is simpler, and you still get the benefit of artifacts.

Azure DevOps: Introduction

henryfs@princeton.edu (Henry Schreiner) — Tue, 07 May 2019 20:00:00 +0200

Continuous Integration (CI) is fantastic for software development and deployment. One of the newest entries into the CI market¹ is Microsoft’s Azure DevOps. Their Open Source support is impressive; it is likely part of the recent push² by Microsoft to be more Open Source friendly. Open Source projects get 10 parallel builds, unlimited build minutes, 6 hour job timeouts, and incredibly fast jobs on macOS, Linux, and Windows, all via a single platform. Quite a few major projects³ have been moving to Azure since the initial release in December 2018. The configuration of DevOps is second only to GitLab CI in ease of use and possibly the most expressive system available. The multiple pipeline support also scales well to complicated procedures.

This is the first in a series of posts covering an introduction to setting up projects in Azure DevOps, developed to update the testing and releasing of Python packages for Scikit-HEP, a project for a coherent High Energy Physics Python analysis toolset. The second post covers release pipelines, and the third covers building binary Python packages using DevOps.

Note: I now highly recommend GitHub Actions, which is almost “Azure 2.0”, if you are interested in setting up CI. The language is very similar, although simplified, with some non-backward compatible bugfixes (such as multiline expressions will error if any line files, instead of just on the last line in Azure). You can read my tutorials on GitHub Actions on the Scikit-HEP developer pages.

ROOT on Conda Forge

henryfs@princeton.edu (Henry Schreiner) — Wed, 13 Feb 2019 10:18:37 +0100

For High Energy Physics, the go-to framework for big data analysis has been CERN’s ROOT framework. ROOT is a massive C++ library that even predates the STL in some areas. It is¹ also a JIT C++ interpreter called Cling, probably the best in the business. If you have heard of the Xeus C++ Kernel for Jupyter, that is built on top of Cling. ROOT has everything a HEP physicist could want: math, plotting, histograms, tuple and tree structures, a very powerful file format for IO, machine learning, Python bindings, and more. It also does things like dictionary generation and arbitrary class serialization (other large frameworks like Qt have similar generation tools).

You may already be guessing one of the most common problems for ROOT. It is huge and difficult to install – if you build from source, that’s a several hour task on a single core. It has gotten much better in the last 6 years, and there are several places you can find ROOT, but there are still areas where it is challenging. This is especially true for Python; ROOT is linked to your distro’s Python (both python2 and python3 if your distro supports it, as of ROOT 6.22); but the common rule for using Python is “don’t touch your system Python” - so modern Python users should be in a virtual environment, and for that ROOT requires the system site-packages option be enabled, which is not always ideal. And, if you use the Anaconda Python distribution, which is the most popular scientific distribution of Python and massively successful for ML frameworks, the general rule even for people who build ROOT themselves has been: don’t. But now, you can get a fully featured ROOT binary package for macOS or Linux, Python 2.7, 3.6, 3.7, or 3.8 from Conda-Forge, the most popular Anaconda community channel! Many more HEP recipes have now been added to Conda-Forge, as well! ROOT now also provides a conda docker image, too!

ROOT Install Options

henryfs@princeton.edu (Henry Schreiner) — Mon, 04 Feb 2019 10:44:37 +0100

For particle physicists, ROOT is one of the most important toolkits around. It is a huge suite of tools that predates the C++ standard library, and has almost anything a particle physicist could want. It has driven developments in other areas too. ROOT’s current C++ interpreter, CLING, is the most powerful C++ interpreter available and is used by the Xeus project for Jupyter. The Python work has helped PyPy, with CPPYY also coming from ROOT. However, due to the size, complexity, and age of some parts of ROOT, it can be a bit challenging to install; and it is even more challenging when you want it to talk to Python. I would like to point to the brand-new Conda-Forge ROOT package for Linux and macOS, and point out a few other options for macOS installs. Note for Windows users: Due to the fact that ROOT expects the type long to match the system pointer size, 64-bit Windows cannot be supported for quite some time. While you can use it in 32 bit form, this is generally impossible to connect to Python, which usually will be a 64-bit build.

Illustrations and videos

henryfs@princeton.edu (Henry Schreiner) — Mon, 10 Dec 2018 11:32:00 +0100

These are some illustrations and videos that I have made over the years.

Histogram Speeds in Python

henryfs@princeton.edu (Henry Schreiner) — Thu, 01 Nov 2018 14:58:09 +0100

Let’s compare several ways of making Histograms. I’m going to assume you would like to end up with a nice OO histogram interface, so all the 2D methods will fill a Physt histogram. We will be using a 2 x 1,000,000 element array and filling a 2D histogram, or 10,000,000 elemends in a 1D histogram. Binnings are regular.

1D 10,000,000 item histogram

Example	KNL	MBP	X24
NumPy: histogram	704 ms	147 ms	114 ms
NumPy: bincount	432 ms	110 ms	117 ms
fast-histogram	337 ms	45.9 ms	45.7 ms
Numba	312 ms	58.8 ms	60.7 ms

2D 1,000,000 item histogram

Example	KNL	MBP	X24
Physt	1.21 s	293 ms	246 ms
NumPy: histogram2d	456 ms	114 ms	88.3 ms
NumPy: add.at	247 ms	62.7 ms	49.7 ms
NumPy: bincount	81.7 ms	23.3 ms	20.3 ms
fast-histogram	53.7 ms	10.4 ms	7.31 ms
fast-hist threaded 0.5	(6) 62.5 ms	9.78 ms	(6) 15.4 ms
fast-hist threaded (m)	62.3 ms	4.89 ms	3.71 ms
Numba	41.8 ms	10.2 ms	9.73 ms
Numba threaded	(6) 49.2 ms	4.23 ms	(6) 4.12 ms
Cython	112 ms	12.2 ms	11.2 ms
Cython threaded	(6) 128 ms	5.68 ms	(8) 4.89 ms
pybind11 sequential	93.9 ms	9.20 ms	17.8 ms
pybind11 OpenMP atomic	4.06 ms	6.87 ms	1.91 ms
pybind11 C++11 atomic	(32) 10.7 ms	7.08 ms	(48) 2.65 ms
pybind11 C++11 merge	(32) 23.0 ms	6.03 ms	(48) 4.79 ms
pybind11 OpenMP merge	8.74 ms	5.04 ms	1.79 ms

Binding Minuit2

henryfs@princeton.edu (Henry Schreiner) — Sat, 07 Jul 2018 21:00:00 +0200

Let’s try a non-trivial example of a binding: Minuit2 (6.14.0 standalone edition).

Tools to Bind to Python

henryfs@princeton.edu (Henry Schreiner) — Sat, 07 Jul 2018 21:00:00 +0200

This was originally given as a PyHEP 2018 talk, It is designed to be interactive, and can be run in SWAN if you have a CERN account. If you want to run it manually, just download the repository: github.com/henryiii/pybindings_cc. It is easy to run in Anaconda.

Announcing CLI11 1.6

henryfs@princeton.edu (Henry Schreiner) — Wed, 27 Jun 2018 14:44:40 +0200

CLI11, a powerful library for writing beautiful command line interfaces in C++11, has been updated to 1.6, the largest update ever. CLI11 output is more customizable than ever, and has a better functionality separation under the hood.

CLI11 has had the formatting system completely redesigned, with minor or complete customization of the output possible. Configuration files reading and writing also can be configured; a new example with json instead of ini formatting is included. Validators (finally) have custom help output, as well. Many odd corner cases have been made possible, such as interleaving options.

CMake 3.11

henryfs@princeton.edu (Henry Schreiner) — Mon, 02 Apr 2018 11:16:57 +0200

CMake 3.11 was just released; this is particularly exciting release for CMake. I’d like to give a quick and friendly introduction to the new features that might make the largest difference for CMake users.

Announcing GooFit 2.1

henryfs@princeton.edu (Henry Schreiner) — Thu, 07 Dec 2017 03:20:00 -0500

GooFit 2.1 introduces the full-featured Python bindings to GooFit. These bindings mimic the C++ usage of GooFit, including bindings for all PDFs, and also provide NumPy-centric conversions, live Jupyter notebook printing, pip install, and more. Most of the examples in C++ are provided in Python form, as well.

Several other API changes were made. Observables are now distinguished from Variables and provided as a separate class. Both these classes are now passed around by copy everywhere.¹ The three and four body amplitude classes have been refactored and simplified. OpenMP is now supported via homebrew on macOS; GooFit is one of the only packages that currently can build with OpenMP on the default macOS compiler. Eigen is now available, and CLI11 has been updated to version 1.3.

GooFit 2.1 will receive continuing support while development on GooFit 2.2 presses on with a new indexing scheme for PDFs.

Include What You Use

henryfs@princeton.edu (Henry Schreiner) — Tue, 05 Dec 2017 14:06:35 -0500

Include-what-you-use is a promising little tool for cleaning up a codebase. It didn’t end up working for the use I had for it, but it still could be useful. Here is a quick guideline on installing it on macOS.

Comparing CLI11 and Boost PO

henryfs@princeton.edu (Henry Schreiner) — Sun, 03 Dec 2017 16:31:50 -0500

CLI11 started years ago as a set of tools built on Boost Program Options (PO), and has since matured into the powerful, easy-to-use stand-alone library it is available today. If you would like to see the original inspiration for CLI11, look at Program.hpp in CLI11 0.1. The rest of the post will focus on a comparison between making a CLI app in the two libraries. I am going to assume that you are preparing fairly basic but non-trivial programs in the following comparison.

TL;DR: CLI11 is more concise, and provides more control with better defaults in many cases, but was inspired by Boost PO.

Announcing CLI11 1.3

henryfs@princeton.edu (Henry Schreiner) — Fri, 01 Dec 2017 09:03:50 -0500

CLI11, a powerful library for writing beautiful command line interfaces in C++11, has been updated to 1.3, the largest update ever. CLI11 is more powerful than ever, and has simpler and more consistent parsing under the hood.

This version focused on refactoring several key systems to ensure correct behavior in the interaction of different settings. Most caveats about features only working on the main App have been addressed, and extra arguments have been reworked. Inheritance of defaults makes configuring CLI11 much easier without having to subclass. Policies add new ways to handle multiple arguments to match your favorite CLI programs. Error messages and help messages are better and more flexible. Several bugs and odd behaviors in the parser have been fixed.

Watching GitHub Releases

henryfs@princeton.edu (Henry Schreiner) — Tue, 28 Nov 2017 16:54:14 -0500

If you use an RSS feed reader, you can watch all your favorite GitHub repositories for new releases! Just follow:

https://github.com/USER/REPO/releases.atom

You can see a long discussion with other options here.

OpenMP on High Sierra

henryfs@princeton.edu (Henry Schreiner) — Mon, 13 Nov 2017 15:41:35 -0500

Building OpenMP code is actually possible using Apple Clang that comes default with macOS’s Xcode! Although Apple does not build the OpenMP library, the compiler still supports it. In this post, I demonstrate the procedure necessary to include OpenMP in your build, both with the new support in CMake 3.12, as well as how it would be done without it.

Announcing GooFit 2.0

henryfs@princeton.edu (Henry Schreiner) — Thu, 08 Jun 2017 17:54:00 -0700

The next version of the premier CUDA/OpenMP fitting program for HEP analysis, GooFit 2.0, has been released. GooFit is now easy to build on a wide variety of Unix systems, and supports debuggers and IDEs. GooFit is faster, has unit tests, and working examples. More PDFs and examples have been added, as well as newly released example datasets that are downloaded automatically. GooFit now has built in support for MPI, and can use that to deploy to multiple graphics cards on the same machine. A new command line parser (CLI11) and drastically improved logging and errors have made code easier to write and debug. Usage of GooFit specific terminology is now reduced, using standard Thrust or CUDA terms when possible, lowering the barrier for new developers. A new Python script has been added to assist users converting from pre 2.0 code.

The file structure of GooFit and the build system have been completely revamped. The fake nvcc features have been removed, as have the rootstuff copies of ROOT classes. PDFs are now organized by type and compile and link separately. Multiple PDF caching support has improved. The build system now uses CMake and manages external libraries.

A new feature of the CMake build system is GooFit Packages, which are complete packages that can be added to GooFit and built, allowing analysis code to live in a separate location from GooFit, rather than the old method of simply forking GooFit and adding your analysis manually. A GooFit Package can be made into an example trivially. See this package for an example.

GooFit 2.0 will receive continuing support while development on GooFit 2.1 presses on.

Links:

GooFit on GitHub • GooFit webpage • API documentation

Announcing CLI11 Version 1.0

henryfs@princeton.edu (Henry Schreiner) — Thu, 01 Jun 2017 14:05:00 -0700

CLI11, a powerful library for writing command line interfaces in C++11, has just been released. There are no requirements beyond C++11 support (and even <regex> support not required). It works on Mac, Linux, and Windows, and has 100% test coverage on all three systems. You can simply drop in a single header file (CLI11.hpp available in releases) to use CLI11 in your own application. Other ways to integrate it into a build system are listed in the README.

The library was inspired the Python libraries Plumbum and Click, and incorporates many of their user friendly features. The library is extensively documented, with a friendly introduction, a tutorial filled (in progress) GitBook, and more technical API docs.

Perfect forwarding for methods

henryfs@princeton.edu (Henry Schreiner) — Fri, 17 Mar 2017 10:43:00 -0700

I often see perfect forwarding listed for constructor arguments, but not usually for functions with a return or methods. Here is my solution for an method method of class Cls:

template<typename ...Args>
static auto method(Cls* cls, Args &&  ...args)
  -> typename std::result_of<decltype(&Cls::method)(Cls, Args...)>::type {
    return cls->method(std::forward<Args>(args)...);
}

This is useful if you want to call protected classes from a “helper” friend class, for example, to expose them to tests without having to require GoogleTest/GoogleMock to be available for regular users.

Lua Environment Modules

henryfs@princeton.edu (Henry Schreiner) — Wed, 11 Jan 2017 06:08:00 -0800

This is a guide to setting up Lmod (lua environment modules) on a CentOS system. I’ve used a similar procedure to set them up on a Mac, as well, so this is still a useful guide to the workings of Lmod if you use a different system; mostly paths will change. On a Mac, you’ll want to install Lmod from the science tap in brew. There are several good pages covering environment modules (TCL version), but not many that use the newer Lua syntax. This document aims to fill that roll.

Python 3 upgrade

henryfs@princeton.edu (Henry Schreiner) — Wed, 12 Oct 2016 08:30:00 +0200

About ten years ago, Guido Van Rossum, the Python author and Benevolent Dictator for Life (BDFL), along with the Python community, decided to make several concurrent backward incompatible changes to Python 2.5 and release a new version, Python 3.0.

C++17

henryfs@princeton.edu (Henry Schreiner) — Sat, 10 Sep 2016 08:30:00 +0200

The every-three-year cycle has changed the development of C++; we are now getting consistent releases somewhere in-between the major and minor releases of old. The 2017 release may be called minor by some, with a huge portion of the planned improvements being pushed back another 3-6 years, but there were several substantial changes in useful areas; it is much more impactful than C++14, for example. This almost feels like a lead-in release to C++20.

The std::variant, std::optional, and std::any additions to the standard library are huge, and can restructure the way you program (and are available for older C++ releases through Boost and other libraries).

C++14

henryfs@princeton.edu (Henry Schreiner) — Fri, 09 Sep 2016 08:30:00 +0200

Unlike C++11, this is a minor release, focused mostly on improvements on top of C++11 changes, with very little that one could call “new”. C++14 feels a little more natural than C++11 by expanding the usage of features and implementing common sense additions that were missed in the original C++11 release. There were also quite a few bug fixes; several of these were backported into C++11 mode in compilers.

Also, while C++11 is always available in ROOT 6, C++14 requires a flag and compatible compiler, so C++14 features are often unavailable. The Conda-Forge ROOT package has C++17 enabled.

C++11

henryfs@princeton.edu (Henry Schreiner) — Thu, 08 Sep 2016 08:30:00 +0200

C++11 was the largest change ever made to C++; and due to the changed release schedule, probably will remain the largest single change. It is a well thought out, mostly backward-compatible change that can cause you to completely rethink the way you write code in C++. It is best thought of as almost a new language, a sort of (C++)++ language. There are too many changes to list here, and there are excellent resources available, so this is meant to just give you a taste of some of the most useful changes.

Many of the features work best together, or are related. There already are great resources for learning about C++11 (listed at the bottom of this lesson), and C++11 is already in use in most software. Therefore, the remainder of this lesson will cover a few of the common idioms in C++11 that a programmer experienced with the older C++ might not immediately think of.

GoogleTest and CMake

henryfs@princeton.edu (Henry Schreiner) — Fri, 25 Mar 2016 16:12:00 -0700

This is a quick recipe for setting up CMake to use googletest in your projects. First, make a tests folder in the root of your project. Then, add add_subdirectory(tests) to your CMakeLists.txt, after you’ve finished adding the libraries in your project. Note that the way I’ve written this probably requires CMake 3.4+.

A simple introduction to asyncio

henryfs@princeton.edu (Henry Schreiner) — Thu, 19 Nov 2015 09:46:00 -0800

This is a simple explanation of the asyncio module and new supporting language features in Python 3.5. Even though the new keywords async and await are new language constructs, they are mostly¹ useless without an event loop, and that is supplied in the standard library as asyncio. Also, you need awaitable functions, which are only supplied by asyncio (or in the growing set of async libraries, like asyncssh, quamash etc.).

A little example of how asyncio works

henryfs@princeton.edu (Henry Schreiner) — Thu, 19 Nov 2015 09:43:00 -0800

This is a simple example to show how Asyncio works without using Asyncio itself, instead using a basic and poorly written event loop. This is only meant to give a flavor of what Asyncio does behind the curtains. I’m avoiding most details of the library design, like callbacks, just to keep this simple. Since this is written as an illustration, rather than real code, I’m going to dispense with trying to keep it 2.7 compatible.

Including CRY cosmic ray generator in CMake

henryfs@princeton.edu (Henry Schreiner) — Mon, 19 Oct 2015 09:07:00 -0700

I realized that CRY did not have a CMake based install option, so including it in a GEANT4 cmake project might not be obvious. This is how you would do it in your CMakeLists.txt:

GTest Submodule

henryfs@princeton.edu (Henry Schreiner) — Wed, 07 Oct 2015 06:43:00 -0700

Note: There is a better way to do this described here.

If you’ve ever tried apt-get or brew to try to install gtest, you are probably familiar with the fact that gtest is not “recommend” for global install on your system. As an alternative, the recommendation is that you make it part of your project. The process for making gtest part of your project, however, is not well documented, at least for modern git projects. What follows is the procedure I used to do so.

Slots in Python

henryfs@princeton.edu (Henry Schreiner) — Thu, 06 Aug 2015 16:01:00 -0700

Slots seem to be poorly documented. What they do is simple, but whether they are used is tricky. This is a little mini-post on slots.

Basics of metaclasses

henryfs@princeton.edu (Henry Schreiner) — Thu, 06 Aug 2015 15:59:00 -0700

This is a quick tutorial over the basics of what metaclasses do.

The Metaclass

Metaclasses, while seemingly a complex topic, really just do something very simple. They control what happens when you have code that turns into a class object. The normal place they are executed is right after the class statement. Let’s see that in action by using print as our metaclass.

Factory classmethods in Python

henryfs@princeton.edu (Henry Schreiner) — Wed, 29 Jul 2015 09:13:00 -0700

I haven’t seen a great deal of practical documentation about using classmethods as factories in Python (which is arguably the most important use of a classmethod, IMO). This post hopes to fill in that gap.

Making an autoload extension for IPython

henryfs@princeton.edu (Henry Schreiner) — Fri, 24 Jul 2015 07:06:00 -0700

I recently decided to try my hand at making an auto-load extension for Python and Plumbum. I was planning to suggest it as a new feature, then I thought it might be an experimental feature, and now it’s just a blog post. But it was an interesting idea and didn’t seem to be well documented process on the web. So, here it is.

The plan was to make commands like this:

Uncertainty extension for IPython

henryfs@princeton.edu (Henry Schreiner) — Fri, 24 Jul 2015 06:45:00 -0700

Wouldn’t it be nice if we had uncertainty with a nice notation in IPython? The current method would be to use raw Python,

from uncertainties import ufloat

print(ufloat(12.34, 0.01))

12.340+/-0.010

Let’s use the infix library to make the notation easier. We’ll define |pm| to mean +/-.

Plumbum color

henryfs@princeton.edu (Henry Schreiner) — Wed, 22 Jul 2015 12:25:00 -0700

I’ve been working on a color addition to Plumbum for a little while, and I’d like to share the basics of using it with you now. This library was originally built around a special str subclass, but now is built on the new Styles representation and is far more powerful than the first implementation. It safely does nothing if you do not have a color-compatible systems (posix + tty currently), but can be forced if need be. It is included with Plumbum, so you don’t have to add a requirement for your scripts that is non-essential (as color often is). It is integrated with plumbum.cli, too. Also, I’ve managed to accelerate the color selection algorithms about 8x, allowing near game-like speeds. (see the fullcolor.py example).

Plumbum scripting

henryfs@princeton.edu (Henry Schreiner) — Sun, 12 Jul 2015 14:07:00 -0700

Scripting in Bash is a pain. Bash can do almost anything, and is unbeatable for small scripts, but it struggles when scaling up to doing anything close to a real world scripting problem. Python is a natural choice, especially for the scientist who already is using it for analysis. But, it’s much harder to do basic tasks in Python. So you are left with scripts starting out as Bash scripts, and then becoming a mess, then being (usually poorly) ported to Python, or even worse, being run by a Python script. I’ve seen countless Python scripts that run Bash scripts that run real programs. I’ve even written one or two. It’s not pretty.

I recently came (back) across a really powerful library for doing efficient command line scripts in Python. It contains a set of tools that makes the four (five with color) main tasks of command line scripts simple and powerful. I will also go over the one main drawback of the library (and the possible enhancement!).

Simple Overloading in Python

henryfs@princeton.edu (Henry Schreiner) — Tue, 07 Jul 2015 09:45:00 -0700

This is intended as an example to demonstrate the use of overloading in object oriented programming. This was written as a Jupyter notebook (aka IPython) in Python 3. To run in Python 2, simply rename the variables that have unicode names, and replace truediv with div.

While there are several nice Python libraries that support uncertainty (for example, the powerful uncertainties package and the related units and uncertainties package pint), they usually use standard error combination rules. For a beginning physics class, often ‘maximum error’ combination is used. Here, instead of using a standard deviation based error and using combination rules based on uncorrelated statistical distributions, we assume a simple maximum error and simply add errors.

To implement this, let’s build a Python class and use overloading to implement algebraic operations.