Python on ISciNumPy.dev

scikit-build-core 1.0

henryfs@princeton.edu (Henry Schreiner) — Mon, 06 Jul 2026 12:00:00 -0400

Scikit-build-core 1.0, the culmination of a four year project to build the best compiled code backend for Python, is out. It’s a huge release, with over 90 bugs fixed, over 40 new features, accounting for over a hundred closed issues. This is the completion of my original vision, and I’m really happy with it. I’d like to show off some of the new features, and also discuss some of the last four months of development that made this possible. This is all in time for SIMPLE-Py, the workshop I’m hosting next week over Python packaging alongside SciPy 2026.

Claude Code Reviews with Fable

henryfs@princeton.edu (Henry Schreiner) — Tue, 16 Jun 2026 20:00:00 -0400

Fable came out and then disappeared a couple of days later. In that short time, I managed to run a lot of repository reviews with it, and was really happy with the results. I’ve found and fixed around 400 bugs and 400+ other issues (performance, simplifications, modernizations) across dozens of my projects, making several hundred PRs with a nearly 100% merge rate on decided PRs.

🎡 cibuildwheel 4.0

henryfs@princeton.edu (Henry Schreiner) — Sun, 07 Jun 2026 12:00:00 -0400

cibuildwheel 4.0.0 is out, with some very big additions. We’ve added pyemscripten 3.13 wheels (uploadable to PyPI!), Python 3.15 (and 3.15t) betas, and much more! We’ve got a new stage: audit, which applies abi3audit to stable ABI wheels. We now default to running delvewheel to repair Windows wheels. We now have {project} and {package} placeholders for config-settings, and we set CIBUILDWHEEL_BUILD_IDENTIFIER in the build step. We’ve also made some nice Android improvements. We’ve removed support for Python 3.8, Python 3.13t, GraalPy 3.11, and Cirrus CI.

Python 3.15

henryfs@princeton.edu (Henry Schreiner) — Wed, 06 May 2026 12:00:00 -0500

Python 3.15 beta 1 is out! This is a really impactful release, with some really big additions. A new lazy import system, a powerful sampling profiler, not one but two new builtins, the usual color/types/errors updates, and lots of key changes for developers.

Starting With Agentic AI

henryfs@princeton.edu (Henry Schreiner) — Tue, 05 May 2026 20:00:00 -0400

AI suddenly passed the “more time saved than spent” point around December 2025. A little late, I’ve finally started using agentic AI in various places over the last 2-3 months, and wanted to jot down my thoughts on what works, what doesn’t (yet?), and what the future might look like. If you are curious about AI for programming, or have been skeptical in the past, if “AI slop” is the first thing that comes to mind, this post will show you some ways that AI can be really helpful. Certain tasks have been completely transformed. Notice, I said “some”, not “all”! But it is a lot.

Repo Review 1.0

henryfs@princeton.edu (Henry Schreiner) — Sat, 04 Apr 2026 10:52:51 -0500

I’ve just released repo-review 1.0 with a huge update to the WebApp, along with some other solid improvements in speed, simplicity, and stability. Repo-review’s two most popular plugins, sp-repo-review and validate-pyproject, can take full advantage of the new asynchronous fetching mechanism. And if you use the upcoming Python 3.15, the CLI is more responsive than ever thanks to lazy loading!

Working to make Python lazy

henryfs@princeton.edu (Henry Schreiner) — Tue, 24 Mar 2026 09:00:00 -0400

Python 3.15a7, which is now just a uv python install 3.15 away on all major platforms, has lazy imports! This exciting feature, proposed in PEP 810, promises to make CLI applications faster (especially when using flags like --help), and could make a lot of large code with lots of imports that don’t always get used faster too. Unlike the earlier, failed attempt, this requires libraries to put in some work. I’ve developed a helper tool to make it easy; I’d like to cover what lazy imports are and how to use my tool. Since this is the first library that I used AI heavily in developing, the second half of the post will cover how my experience with AI for a task like this went.

TL;DR: run uvx flake8-lazy --apply=list to make your code magically faster on Python 3.15!

New in boost-histogram 1.7 / Hist 2.10

henryfs@princeton.edu (Henry Schreiner) — Fri, 06 Feb 2026 17:16:40 -0500

I’ve just released boost-histogram 1.7 and hist 2.10, with a major new feature and a big typing improvement! I’d like to quickly outline what’s new here. Python 3.10+ is now required.

How we made Python's packaging library 3x faster

henryfs@princeton.edu (Henry Schreiner) — Fri, 09 Jan 2026 13:30:00 -0500

Along with a pip (and now packaging) maintainer, Damian Shaw, I have been working on making packaging, the library behind almost all packaging related tools, faster at reading versions and specifiers, something tools like pip have to do thousands of times during resolution. Using Python 3.15’s new statistical profiler and metadata from every package ever uploaded to PyPI, I measured and improved core Packaging constructs while keeping the code readable and simple. Reading in Versions can be up to 2x faster and SpecifierSets can be up to 3x faster in packaging 26.0, now released! Other operations have been optimized, as well, up to 5x in some cases. See the announcement and release notes too; this post will focus on the performance work only.

UHI 1.0: histogram serialization

henryfs@princeton.edu (Henry Schreiner) — Wed, 20 Aug 2025 16:00:00 -0400

UHI 1.0 is out, with a major new feature: a new histogram serialization spec! This spec supports multiple formats (HDF5, zip, and JSON initially), and can be supported by multiple libraries (Boost-histogram/hist initially). There’s also a new test suite helper for libraries targeting the UHI indexing spec.

pybind11 3.0.0

henryfs@princeton.edu (Henry Schreiner) — Thu, 10 Jul 2025 00:13:00 -0400

Pybind11 3.0 has been released! I would like to highlight some of the key changes below; be sure to check out the changelog and upgrade guide for more information! This release includes an ABI bump, the first required bump in many years on Unix (Windows has had required bumps more often). This release contains the multi-year smart-holder branch, multi-phase init and subinterpreter support, py::native_enum, an interface to warnings, typing improvements, and more. CMake now defaults to FindPython mode.

Support for Python 3.14, 3.14t, GraalPy, PyPy 3.11, and iOS have been added, while legacy support for Python 3.7, PyPy 3.8/3.9, and CMake <3.15 has been removed. Most deprecated features have been kept for this release, but anything producing a warning in 3.0 may be removed in a future 3.x version.

🎡 cibuildwheel 3.0

henryfs@princeton.edu (Henry Schreiner) — Thu, 05 Jun 2025 01:54:45 -0400

cibuildwheel 3.0.0 is out, with some very big additions. We’ve added GraalPy, Python 3.14 (and 3.14t) betas, and iOS support! We’ve got several new options: test-sources, test-environment, and (experimental) pyodide-version. We now fully use enable (and PyPy requires using it), and we no longer inject setuptools and wheel in build environments. Defaults have changed, too: build is now the default frontend, manylinux_2_28 is the default manylinux image, with 32-bit linux now being opt-in. We’ve removed support for Python 3.6 and 3.7, we now require 3.11+ to run cibuildwheel itself, and EoL manylinux/musllinux images now need to be fully specified.

We’ve had some fantastic releases of cibuildwheel since my last post over 2.19, so I’ll include a few of the new features from those releases, too. I’ll also note a few of the features being worked on for future releases.

Python π

henryfs@princeton.edu (Henry Schreiner) — Thu, 08 May 2025 09:05:54 -0400

Python π (3.14) is out! The big feature this time around is template strings. Free-threading is no longer experimental. There’s also lots more color (including syntax highlighting in the REPL!), remote debugging, deferred evaluation of annotations, and the usual error message and performance improvements. Subinterpreters are now accessible without the C-API (finally!), and free-threaded Python is no longer experimental.

scikit-build-core 0.10

henryfs@princeton.edu (Henry Schreiner) — Mon, 05 Aug 2024 23:09:13 -0400

Scikit-build-core 0.10 is out, and it is one of the largest releases we’ve produced in terms of new configuration options. It automatically reads your minimum CMake version, you can tell it to read your scikit-build-core minimum-version, and there are settings for many of the advanced things users used to do with scikit-build (classic), like rebuilds or pure Python fallbacks on build failures.

🎡 cibuildwheel 2.19

henryfs@princeton.edu (Henry Schreiner) — Mon, 10 Jun 2024 12:05:51 -0400

cibuildwheel 2.19 is out, with some very big additions. A new platform, Pyodide, has been added for building WebAssembly wheels. We’ve added CPython 3.13 free-threaded builds, now on all OS’s. And we have an opt-in speed improvement with the build[uv] build-frontend option.

We’ve had some fantastic releases of cibuildwheel since my last post over 2.10, so I’ll include a few of the new features from those releases, too, with a highlight on a larger feature that can use more explaining: inherit for overrides.

Python 3.13

henryfs@princeton.edu (Henry Schreiner) — Thu, 09 May 2024 15:33:00 -0400

Python 3.13 betas are out (Edit: 3.13.0 is out!), which means the features are locked in. For the first time in thirty years, Python has a new, more colorful REPL! There’s also a no-GIL compile-time option (free-threaded), an optional JIT, some new typing features, and better error messages (again).

Inline run dependencies in pipx 1.4.2

henryfs@princeton.edu (Henry Schreiner) — Mon, 04 Dec 2023 15:00:06 -0500

While it can also do much more, Python is a fantastic language for writing small scripts and utilities with its expressive syntax and batteries-included standard library. But what if you need just a bit more? PyPI is one of the best package repositories for any language, and being able to access it without having to write a multi-file library and setting up virtual environments would be a dream - one that is becoming reality. Pipx 1.4.2 has an experimental implementation of the ~~provisionally~~ accepted PEP 723, and I’d like to show it off here, as it’s tremendously useful for simple scripts & utilities. Support is also available in Nox 2024.04.15 and Hatch 1.10.

The Scientific Python Development Guide

henryfs@princeton.edu (Henry Schreiner) — Fri, 14 Jul 2023 00:45:32 -0500

One outcome of the 2023 Scientific Python Developer Summit was the Scientific Python Development Guide, a comprehensive guide to modern Python package development, complete with a new project template supporting 12 build backends and a WebAssembly-powered checker with checks linked to the guide. The guide covers topics like modern, compiled, and classic packaging, style checks, type checking, docs, task runners, CI, tests, and much more! There also are sections of tutorials, principles, and some common patterns.

This guide (along with cookie & repo-review) started in Scikit-HEP in 2020. During the summit, it was merged with the NSLS-II guidelines, which provided the basis for the principles section. I’d like to thank and acknowledge Dan Allan and Gregory Lee for working tirelessly during the summit to rework, rewrite, merge, and fix the guide, including writing most of the tutorials pages and first patterns page, and rewriting the environment page as a tutorial.

Introducing repo-review

henryfs@princeton.edu (Henry Schreiner) — Thu, 13 Jul 2023 22:00:00 -0400

I’ve released a new¹ toolkit for running checks, similar to Ruff and Flake8 but designed to check configuration, called repo-review. It requires Python 3.10+² to run and has no built-in checks, but is easy to write plugins for. A set of checks based on the Scientific Python Development Guide (which I also have a post about!) are available as a plugin, sp-repo-review. You can run repo-review in WebAssembly (via Pyodide), or in pre-commit, or as a GitHub Action. It supports multiple output formats, including Rich, HTML, and JSON. The system is based on fixtures (like pytest) and topologically sorts requirements. You don’t need to depend on repo-review to add a repo-review plugin. You can see a live version using sp-repo-review in-place here or standalone here.

Python 3.12

henryfs@princeton.edu (Henry Schreiner) — Mon, 19 Jun 2023 17:00:00 -0400

Python 3.12’s beta’s are out, which means the features are locked in. The theme this year has been cleanup and typing. distutils has been removed, and setuptools is no longer present in default environments.

🎡 cibuildwheel 2.10

henryfs@princeton.edu (Henry Schreiner) — Tue, 13 Sep 2022 08:42:26 -0400

cibuildwheel 2.10 is out, with some important additions. PEP 517 config settings added, --only (which has an interesting use in GHA), and Cirrus CI support (including our first Apple Silicon native runner!) are highlights. We also support Python 3.11 now (as of 2.11.2, RC’s in older releases).

We’ve had some fantastic releases of cibuildwheel this year, including some very powerful features you might be interested in using, and I haven’t covered releases since 2.2, so let’s take an in-depth look at what’s new for this and the last few releases!

Python 3.10

henryfs@princeton.edu (Henry Schreiner) — Sat, 23 Jul 2022 10:00:00 -0400

Python 3.10 is out (and has been for a while, I’m late posting this), with new features and changes. The big new language feature this update is pattern matching. We get much better errors, the always-present typing improvements, and finally some real usage of the new PEG parser from 3.9.

Python 3.11

henryfs@princeton.edu (Henry Schreiner) — Wed, 11 May 2022 16:11:00 -0400

Python 3.11 has hit the beta (now released!) stage, which means no more new features. It’s a perfect time to play with it! The themes in this update are the standard ones: The faster CPython project is now fully going (3.11 is 25% faster on average), along with improved error messages, typing, and asyncio. Beyond this, the only major new feature is a library for reading TOML files; this this is probably only exciting if you are involved in Python packaging (but I am, so I’m excited!).

Python 3.7

henryfs@princeton.edu (Henry Schreiner) — Sat, 15 Jan 2022 22:41:05 -0500

Python 3.7 has been out for a while. In fact, it’s the oldest version of Python still receiving support when this was written. I’d still like to write a “what’s new”, targeting users who are upgrading to a Python 3.7+ only codebase, and want to know what to take advantage of!

Poetry Versions

henryfs@princeton.edu (Henry Schreiner) — Thu, 09 Dec 2021 11:00:00 -0500

Poetry provides an all-in-one solution to Python packaging. I want to focus on why I was quite hard on Poetry in my last post, specifically on its default version capping and solver quirks, and also a few other negative things. This is a followup to Should you have upper version bounds, which you should read before this post.

Should You Use Upper Bound Version Constraints?

henryfs@princeton.edu (Henry Schreiner) — Thu, 09 Dec 2021 10:00:00 -0500

Bound version constraints (upper caps) are starting to show up in the Python ecosystem. This is causing real world problems with libraries following this recommendation, and is likely to continue to get worse; this practice does not scale to large numbers of libraries or large numbers of users. In this discussion I would like to explain why always providing an upper limit causes far more harm than good even for true SemVer libraries, why libraries that pin upper limits require more frequent updates rather than less, and why it is not scalable. After reading this, hopefully you will always consider every cap you add, you will know the (few) places where pinning an upper limit is reasonable, and will possibly even avoid using libraries that pin upper limits needlessly until the author updates them to remove these pins.

If this 10,000 word behemoth is a bit long for you, then skip around using the table of contents, or see the TL;DR section at the end, or read version numbers by Bernát Gábor, which is shorter but is a fantastic read with good examples and cute dog pictures. Or Hynek’s Semantic Versioning Will Not Save You Be sure to check at least the JavaScript project analysis before you leave!

Also be warned, I pick on Poetry quite a bit. The rising popularity of Poetry is likely due to the simplicity of having one tool vs. many for packaging, but it happens to also have a special dependency solver, a new upper bound syntax, and a strong recommendation to always limit upper versions - in direct opposition to members of the Python core developer team and PyPA developers. Not all libraries with excessive version capping are Poetry projects (like TensorFlow), but many, many of them are. To be clear, Poetry doesn’t force version pinning on you, but it does push you really, really hard to always version cap, and it’s targeting new Python users that don’t know any better yet than to accept bad recommendations. And these affect the whole ecosystem, including users who do not use poetry, but want to depend on libraries that do! I do really like other aspects of Poetry, and would like to eventually help it build binary packages with Scikit-build (CMake) via a plugin, and it has some great developers. If I don’t pick on Poetry enough for you, don’t worry, I have a follow-up post that picks on it in much more detail. Also, check out pdm, which gives many of the benefits of Poetry while following PEP standards. Also pixi, which works with the Conda ecosystem.

If you come across something that can’t be solved, try using --exclude-newer <DATE> in uv and pdm. This limits the solve by ignoring packages newer than some date.

App vs Library

henryfs@princeton.edu (Henry Schreiner) — Thu, 09 Dec 2021 09:00:00 -0500

What is the difference between an app and a library? This seemingly simple question confuses some, and it turns out to be a harder question to answer than you might expect. While the actual distinction between these common terms will always be muddled in practice, I propose a specific definition to be used when considering dependencies. This distinction is important when discussing bound version constraints in the next post.

Scikit Build Proposal

henryfs@princeton.edu (Henry Schreiner) — Wed, 27 Oct 2021 08:00:00 -0400

I’ve spent the last few years trying to make it easy for anyone to extend Python with compiled languages. I’ve worked on pybind11, a powerful C++ library that allows users to write advanced Python extensions using just C++11, used by some of the largest projects, SciPy, PyTorch, Google, LLVM, and tens of thousands of other libraries, down to very small extensions. I also work on cibuildwheel, which makes building binaries (called wheels) on continuous integration (CI) simple. It is again powerful enough to be used by huge projects, like Scikit-learn, matplotlib, mypy; and is simple enough to be used by hundreds of other packages. Recently it was accepted into the Python Packaging Authority (PyPA). There is one missing piece, though, to complete this picture of compiled extensions that is easy to use for small projects, and powerful enough for large projects: the build system. I believe the solution to that is scikit-build, and I’d like to work on it over the next three years.

Scikit-build is a tool for integrating a package with a CMake build system into Python. You can utilize the vast collection of packages and projects using CMake already, and you have access to modern building features, like multithreaded builds, library discovery, superb compiler and IDE support, and all sorts of extended tooling. Modern CMake is quite pleasant to write compared to times past; I have written a book and training course on it. We ship up-to-date cmake and ninja wheels for all binary platforms.

Update: Funded! I’ll be working on this starting August 1, 2022!

I wrote a proposal for an NSF CSSI Elements project containing three parts. The first part will cover core development on Scikit-build to address the current shortcomings and to prepare it for a post-distutils (Python 3.12+) world. The second part would cover assisting libraries with a science use case in either transitioning to scikit-build (ideally from an existing CMake build system with Python bindings, but I can help mentor developers in writing bindings (ideally pybind11), setting up CI, and writing CMake code as well (see my book or workshop on Modern CMake, and I’m happy to help old scikit-build projects transition to better practices)). As part of this, I would be building up the examples and documentation, leading into the third part of the proposal: A series of training events and training material, including plans for something alongside SciPy.

You can also see an outline at scikit-build/scikit-build/wiki or at the end of this post.

Thank you for all the projects! The proposal was submitted Dec 8, 2021; mid year we should find out if it was accepted!

🎡 cibuildwheel 2.2

henryfs@princeton.edu (Henry Schreiner) — Wed, 20 Oct 2021 11:27:29 -0400

Another great release from cibuildwheel, 2.2.0, is out! There are a few important additions in this release that you should be aware of, so I will outline the major changes here. We will cover the new musllinux wheels, overload configuration, and incoming changes to pip and PyPy expected in the next release. As always, it is recommended that you pin your cibuildwheel version and then provide some automated way to keep the pin up-to-date, such as GitHub’s dependabot. You should be updating just before you make a release, as well, but you probably don’t want to be surprised by new wheels during your release process!

🎡 cibuildwheel 2.0

henryfs@princeton.edu (Henry Schreiner) — Fri, 16 Jul 2021 12:23:20 -0400

The cibuildwheel package has just had a major release with some fantastic features. Python 2.7 and 3.5 support has been removed (and PyPy3.6), allowing us to update to the latest manylinux and auditwheel versions, and support the newly unified manylinux PyPy3.7 images. We now allow users to select pypa/build as a build frontend. We now have a custom option to enable pre-release Pythons (3.10 currently) for testing before they are ABI stable (please don’t release wheels until that happens). Maybe most exciting, cibuildwheel now supports configuration in pyproject.toml, allowing you to be even further isolated from dependence on your CI system; you can easily produce Linux and Windows wheels locally (macOS still installs to system locations). And, since my last post and introduction post, cibuildwheel is now part of the PyPA!

Python 3.9

henryfs@princeton.edu (Henry Schreiner) — Fri, 12 Feb 2021 12:00:00 -0400

Python 3.9 is out (and has been for a while, I’m late posting this), with new features and changes. The themes for this release have been heavily internal improvements, such as a new more powerful parser, and the usual static typing improvements, along with a several bits new-user facing new syntax. This makes 3.9 a smaller release, but still it has some nice features of note.

🎡 cibuildwheel 1.8.0 and 1.9.0

henryfs@princeton.edu (Henry Schreiner) — Sun, 07 Feb 2021 20:13:41 -0500

cibuildwheel has just had two back-to-back releases, two weeks apart, representing several months of hard work and a few exciting features! I will be covering both releases at once, so we will discuss Apple Silicon support, architecture emulation on Linux, integrated PEP 621 Requires-Python support, the native GitHub Action, extended build and test controls, and more!

If you are following the releases, 1.7.0 came out last November (2020), and included the fantastic output folding feature, which makes logs much easier to read on CI systems that support folding, and makes it much easier to see how long each step takes. The 1.7.x series also included the addition of the working examples section of the documentation, which tracks some known projects using cibuildwheel, such as scikit-learn, Matplotlib, and MyPy; it is a great place to go to look into how other projects have integrated cibuildwheel into their workflow.

I have a general overview post as well. Now let’s look at what’s new! Update: cibuildwheel is now an official package of the PyPA!

Overview of cibuildwheel 🎡

henryfs@princeton.edu (Henry Schreiner) — Sat, 06 Feb 2021 20:58:16 -0500

This is the first of two posts on cibuildwheel, a fantastic project I joined after switching to it from my own azure-wheel-helpers, which I’ve blogged about here before. It is the best wheelbuilding system available for Python today, and can make something that is normally a pain to setup and a headache to maintain a breeze (by forcing all the headaches on us, of course, as maintainers, but it’s better to solve issues centrally! Obviously we rather like solving these problems. Or we are just crazy, which is also possible ;) ).

Be sure to checkout the followup post over new features in 1.8.0 and 1.9.0, too! Also, cibuildwheel was recently accepted into the PyPA!

pybind11 2.6.0

henryfs@princeton.edu (Henry Schreiner) — Wed, 21 Oct 2020 15:00:00 -0400

I am pleased to announce the release of pybind11 2.6.0! This is the largest release since 2.2 (released over three years ago). I would like to highlight some of the key changes below; be sure to check out the changelog and upgrade guide for more information! The focus of this release was stability, packaging, and supporting more platforms, though there are a lot of small features and useful additions, covered by newly expanded docs.

Johns Hopkins COVID-19 Dataset in Pandas

henryfs@princeton.edu (Henry Schreiner) — Mon, 30 Mar 2020 15:30:00 -0400

COVID-19 is ravaging the globe. Let’s look at the excellent Johns Hopkins dataset using Pandas. This will serve both as a guideline for getting the data and exploring on your own, as well as an example of Pandas multi-indexing in an easy to understand situation. I am currently involved in science-responds.

The boost-histogram beta release

henryfs@princeton.edu (Henry Schreiner) — Fri, 13 Dec 2019 11:00:00 -0500

The foundational histogramming package for Python, boost-histogram, hit beta status with version 0.6! This is a major update to the new Boost.Histogram bindings. Since I have not written about boost-histogram yet here, I will introduce the library in its current state. Version 0.6.2 was based on the recently released Boost C++ Libraries 1.72 Histogram package. Feel free to visit the docs, or keep reading this post.

This Python library is part of a larger picture in the Scikit-HEP ecosystem of tools for Particle Physics and is funded by DIANA/HEP and IRIS-HEP. It is the core library for making and manipulating histograms. Other packages are under development to provide a complete set of tools to work with and visualize histograms. The Aghast package is designed to convert between popular histogram formats, and the Hist package will be designed to make common analysis tasks simple, like plotting via tools such as the mplhep package. Hist and Aghast will be initially driven by HEP (High Energy Physics and Particle Physics) needs, but outside issues and contributions are welcome and encouraged.

Python 3.8

henryfs@princeton.edu (Henry Schreiner) — Fri, 25 Oct 2019 14:11:20 -0400

Python 3.8 is out, with new features and changes. The themes for this release have been performance, ABI/internals, and static typing, along with a smattering of new syntax. Given the recent community statement on Python support, we should be staying up to date with the current changes in Python. As Python 2 sunsets, we are finally in an era where we can hope to someday use the features we see coming out of Python release again!

ROOT on Conda Forge

henryfs@princeton.edu (Henry Schreiner) — Wed, 13 Feb 2019 10:18:37 +0100

For High Energy Physics, the go-to framework for big data analysis has been CERN’s ROOT framework. ROOT is a massive C++ library that even predates the STL in some areas. It is¹ also a JIT C++ interpreter called Cling, probably the best in the business. If you have heard of the Xeus C++ Kernel for Jupyter, that is built on top of Cling. ROOT has everything a HEP physicist could want: math, plotting, histograms, tuple and tree structures, a very powerful file format for IO, machine learning, Python bindings, and more. It also does things like dictionary generation and arbitrary class serialization (other large frameworks like Qt have similar generation tools).

You may already be guessing one of the most common problems for ROOT. It is huge and difficult to install – if you build from source, that’s a several hour task on a single core. It has gotten much better in the last 6 years, and there are several places you can find ROOT, but there are still areas where it is challenging. This is especially true for Python; ROOT is linked to your distro’s Python (both python2 and python3 if your distro supports it, as of ROOT 6.22); but the common rule for using Python is “don’t touch your system Python” - so modern Python users should be in a virtual environment, and for that ROOT requires the system site-packages option be enabled, which is not always ideal. And, if you use the Anaconda Python distribution, which is the most popular scientific distribution of Python and massively successful for ML frameworks, the general rule even for people who build ROOT themselves has been: don’t. But now, you can get a fully featured ROOT binary package for macOS or Linux, Python 2.7, 3.6, 3.7, or 3.8 from Conda-Forge, the most popular Anaconda community channel! Many more HEP recipes have now been added to Conda-Forge, as well! ROOT now also provides a conda docker image, too!

ROOT Install Options

henryfs@princeton.edu (Henry Schreiner) — Mon, 04 Feb 2019 10:44:37 +0100

For particle physicists, ROOT is one of the most important toolkits around. It is a huge suite of tools that predates the C++ standard library, and has almost anything a particle physicist could want. It has driven developments in other areas too. ROOT’s current C++ interpreter, CLING, is the most powerful C++ interpreter available and is used by the Xeus project for Jupyter. The Python work has helped PyPy, with CPPYY also coming from ROOT. However, due to the size, complexity, and age of some parts of ROOT, it can be a bit challenging to install; and it is even more challenging when you want it to talk to Python. I would like to point to the brand-new Conda-Forge ROOT package for Linux and macOS, and point out a few other options for macOS installs. Note for Windows users: Due to the fact that ROOT expects the type long to match the system pointer size, 64-bit Windows cannot be supported for quite some time. While you can use it in 32 bit form, this is generally impossible to connect to Python, which usually will be a 64-bit build.

Illustrations and videos

henryfs@princeton.edu (Henry Schreiner) — Mon, 10 Dec 2018 11:32:00 +0100

These are some illustrations and videos that I have made over the years.

Histogram Speeds in Python

henryfs@princeton.edu (Henry Schreiner) — Thu, 01 Nov 2018 14:58:09 +0100

Let’s compare several ways of making Histograms. I’m going to assume you would like to end up with a nice OO histogram interface, so all the 2D methods will fill a Physt histogram. We will be using a 2 x 1,000,000 element array and filling a 2D histogram, or 10,000,000 elements in a 1D histogram. Binnings are regular.

1D 10,000,000 item histogram

Example	KNL	MBP	X24
NumPy: histogram	704 ms	147 ms	114 ms
NumPy: bincount	432 ms	110 ms	117 ms
fast-histogram	337 ms	45.9 ms	45.7 ms
Numba	312 ms	58.8 ms	60.7 ms

2D 1,000,000 item histogram

Example	KNL	MBP	X24
Physt	1.21 s	293 ms	246 ms
NumPy: histogram2d	456 ms	114 ms	88.3 ms
NumPy: add.at	247 ms	62.7 ms	49.7 ms
NumPy: bincount	81.7 ms	23.3 ms	20.3 ms
fast-histogram	53.7 ms	10.4 ms	7.31 ms
fast-hist threaded 0.5	(6) 62.5 ms	9.78 ms	(6) 15.4 ms
fast-hist threaded (m)	62.3 ms	4.89 ms	3.71 ms
Numba	41.8 ms	10.2 ms	9.73 ms
Numba threaded	(6) 49.2 ms	4.23 ms	(6) 4.12 ms
Cython	112 ms	12.2 ms	11.2 ms
Cython threaded	(6) 128 ms	5.68 ms	(8) 4.89 ms
pybind11 sequential	93.9 ms	9.20 ms	17.8 ms
pybind11 OpenMP atomic	4.06 ms	6.87 ms	1.91 ms
pybind11 C++11 atomic	(32) 10.7 ms	7.08 ms	(48) 2.65 ms
pybind11 C++11 merge	(32) 23.0 ms	6.03 ms	(48) 4.79 ms
pybind11 OpenMP merge	8.74 ms	5.04 ms	1.79 ms

Binding Minuit2

henryfs@princeton.edu (Henry Schreiner) — Sat, 07 Jul 2018 21:00:00 +0200

Let’s try a non-trivial example of a binding: Minuit2 (6.14.0 standalone edition).

Tools to Bind to Python

henryfs@princeton.edu (Henry Schreiner) — Sat, 07 Jul 2018 21:00:00 +0200

This was originally given as a PyHEP 2018 talk. It is designed to be interactive, and can be run in SWAN if you have a CERN account. If you want to run it manually, just download the repository: github.com/henryiii/pybindings_cc. It is easy to run in Anaconda.

Python 3 upgrade

henryfs@princeton.edu (Henry Schreiner) — Wed, 12 Oct 2016 08:30:00 +0200

About ten years ago, Guido Van Rossum, the Python author and Benevolent Dictator for Life (BDFL), along with the Python community, decided to make several concurrent backward incompatible changes to Python 2.5 and release a new version, Python 3.0.

A simple introduction to asyncio

henryfs@princeton.edu (Henry Schreiner) — Thu, 19 Nov 2015 09:46:00 -0800

This is a simple explanation of the asyncio module and new supporting language features in Python 3.5. Even though the new keywords async and await are new language constructs, they are mostly¹ useless without an event loop, and that is supplied in the standard library as asyncio. Also, you need awaitable functions, which are only supplied by asyncio (or in the growing set of async libraries, like asyncssh, quamash etc.).

A little example of how asyncio works

henryfs@princeton.edu (Henry Schreiner) — Thu, 19 Nov 2015 09:43:00 -0800

This is a simple example to show how Asyncio works without using Asyncio itself, instead using a basic and poorly written event loop. This is only meant to give a flavor of what Asyncio does behind the curtains. I’m avoiding most details of the library design, like callbacks, just to keep this simple. Since this is written as an illustration, rather than real code, I’m going to dispense with trying to keep it 2.7 compatible.

Slots in Python

henryfs@princeton.edu (Henry Schreiner) — Thu, 06 Aug 2015 16:01:00 -0700

Slots seem to be poorly documented. What they do is simple, but whether they are used is tricky. This is a little mini-post on slots.

Basics of metaclasses

henryfs@princeton.edu (Henry Schreiner) — Thu, 06 Aug 2015 15:59:00 -0700

This is a quick tutorial over the basics of what metaclasses do.

The Metaclass

Metaclasses, while seemingly a complex topic, really just do something very simple. They control what happens when you have code that turns into a class object. The normal place they are executed is right after the class statement. Let’s see that in action by using print as our metaclass.

Factory classmethods in Python

henryfs@princeton.edu (Henry Schreiner) — Wed, 29 Jul 2015 09:13:00 -0700

I haven’t seen a great deal of practical documentation about using classmethods as factories in Python (which is arguably the most important use of a classmethod, IMO). This post hopes to fill in that gap.

Making an autoload extension for IPython

henryfs@princeton.edu (Henry Schreiner) — Fri, 24 Jul 2015 07:06:00 -0700

I recently decided to try my hand at making an auto-load extension for Python and Plumbum. I was planning to suggest it as a new feature, then I thought it might be an experimental feature, and now it’s just a blog post. But it was an interesting idea and didn’t seem to be well documented process on the web. So, here it is.

The plan was to make commands like this:

Uncertainty extension for IPython

henryfs@princeton.edu (Henry Schreiner) — Fri, 24 Jul 2015 06:45:00 -0700

Wouldn’t it be nice if we had uncertainty with a nice notation in IPython? The current method would be to use raw Python,

from uncertainties import ufloat

print(ufloat(12.34, 0.01))

12.340+/-0.010

Let’s use the infix library to make the notation easier. We’ll define |pm| to mean +/-.

Plumbum color

henryfs@princeton.edu (Henry Schreiner) — Wed, 22 Jul 2015 12:25:00 -0700

I’ve been working on a color addition to Plumbum for a little while, and I’d like to share the basics of using it with you now. This library was originally built around a special str subclass, but now is built on the new Styles representation and is far more powerful than the first implementation. It safely does nothing if you do not have a color-compatible system (posix + tty currently), but can be forced if need be. It is included with Plumbum, so you don’t have to add a requirement for your scripts that is non-essential (as color often is). It is integrated with plumbum.cli, too. Also, I’ve managed to accelerate the color selection algorithms about 8x, allowing near game-like speeds. (see the fullcolor.py example).

Plumbum scripting

henryfs@princeton.edu (Henry Schreiner) — Sun, 12 Jul 2015 14:07:00 -0700

Scripting in Bash is a pain. Bash can do almost anything, and is unbeatable for small scripts, but it struggles when scaling up to doing anything close to a real world scripting problem. Python is a natural choice, especially for the scientist who already is using it for analysis. But, it’s much harder to do basic tasks in Python. So you are left with scripts starting out as Bash scripts, and then becoming a mess, then being (usually poorly) ported to Python, or even worse, being run by a Python script. I’ve seen countless Python scripts that run Bash scripts that run real programs. I’ve even written one or two. It’s not pretty.

I recently came (back) across a really powerful library for doing efficient command line scripts in Python. It contains a set of tools that makes the four (five with color) main tasks of command line scripts simple and powerful. I will also go over the one main drawback of the library (and the possible enhancement!).