The Scientific Python Development Guide

One outcome of the 2023 Scientific Python Developer Summit was the Scientific Python Development Guide, a comprehensive guide to modern Python package development, complete with a new project template supporting 12 build backends and a WebAssembly-powered checker with checks linked to the guide. The guide covers topics like modern, compiled, and classic packaging, style checks, type checking, docs, task runners, CI, tests, and much more! There also are sections of tutorials, principles, and some common patterns.

This guide (along with cookie & repo-review) started in Scikit-HEP in 2020. During the summit, it was merged with the NSLS-II guidelines, which provided the basis for the principles section. I’d like to thank and acknowledge Dan Allan and Gregory Lee for working tirelessly during the summit to rework, rewrite, merge, and fix the guide, including writing most of the tutorials pages and first patterns page, and rewriting the environment page as a tutorial.

The guide

The core of the project is the guide, which is comprised of four sections:

• Tutorials: How to go from “research” code to a basic package, for beginning readers.
• Topical guides: The core of the guide, for intermediate to advanced readers.
• Principles: Some general principles from the NSLS-II guide.
• Patterns: Recipes for common situations. Three pages are there now; including data, backports, and exports.

From the original Scikit-HEP dev pages, a lot was added:

• Brand new guide page on documentation, along with new sp-repo-review checks to help with readthedocs.
• A compiled projects page! Finally! With scikit-build-core, meson-python, and maturin. The page shows real outputs from the cookiecutter, kept in sync with cog (a huge benefit of using a single repo for all three components!)
• Big update to GHA CI page including a section on Composite Actions given at the Dev summit.
• CLI entry points are now included.
• Python 3.12 support added, Python 3.7 dropped.
• New sp-repo-review badges throughout (more on that later!)
• Updates for Ruff’s move and support for requires-python.
• Lots of additions for GitHub Actions.

The infrastructure was updated too:

• Using latest Jekyll (version 4) and latest Just the Docs theme. More colorful callout boxes. Plugins are used now.
• Live PR preview (provided by probably the world’s first readthedocs Jekyll build!). Developed with the Zarr developers during the summit.
• Better advertising for cookie and sp-repo-review on the index page(s).
• Auto bump and auto-sync CI jobs.

Cookie

We also did something I’ve wanted to do for a long time: the guide, the cookiecutter template, and the checks are all in a single repo! The repo is scientific-python/cookie, which is the moved scikit-hep/cookie (the old URL for cookiecutter still works!). A lot of work went into the cookiecutter, too!

• Generalized defaults. We still have special integration if someone sets the org to scikit-hep; the same integration can be offered to other orgs.
• All custom hooks removed; standard jinja now used throughout templates. Using cookiecutter computed variables idiom too. Windows still fully supported and tested. Adding new choices is much easier now.
• Added cruft (a cookiecutter updater) testing.
• Dual-supporting copier now too, a cookiecutter replacement with a huge CLI improvement (and also supports updating). Might be the first project to support both at the same time. CI (or nox locally) checks to ensure generation is identical. Much better interface with copier, including validation, descriptive text, arrow selection, etc.
• Improved CLI experience even if using cookiecutter (like no longer requesting current year).
• Reworked docs template.
• Support for cookiecutter 2.2 pretty descriptions (added about four hours after cookiecutter 2.2.0 was released).
• GitLab CI support when not targeting github.com URLs (added by Giordon Stark).

Repo-review

I wrote an introduction post about this one!

Along with this was probably the biggest change, one requested by several people at the summit: scientific-python/repo-review (was scikit-hep/repo-review) is now a completely general framework for implementing checks in Python 3.10+. The checks have been moved to sp-repo-review, which is now part of scientific-python/cookie. There are too many changes to list here, so just the key ones in 0.6, 0.7 & 0.8:

• Extensive, beautiful documentation for check authors at (used to help guide the new docs guide page & template update!)
• Support for four output formats, rich (improved), svg, json (new), html (new).
• Support for listing all checks.
• GitHub Actions support with HTML step summary, pre-commit support.
• Generalized topological sorting to fixtures, dynamic fixtures.
• Dynamic check selection (via fixtures! Basically everything is powered by fixtures now.)
• URL support in all output formats (including the terminal and WebApp!)
• Support for package not at root of repo.
• Support for running on a remote repo from the command line.
• Support for select/ignore config in pyproject.toml or command line.
• Pretty printed and controllable sorting for families.
• Supports running from Python, including inside a readme with something like cog.

The full changelog has more - you can even see the 10 beta releases in-between 0.6.x and 0.7.0 where a lot of this refactoring work was happening. If you have configuration you’d like to write check for, feel free to write a plugin!

sp-repo-review

Finally, sp-repo-review contains the previous repo-review plugins with checks:

• Fully cross-linked with the development guide. Every check has a URL that points to a matching badge inside the development guide where the thing the check is looking for is being discussed!
• Full list of checks (including URLs), produced by cog, in readme.
• Also ships with GitHub Action and pre-commit checks
• Released (in sync with cookie & guide, as they are in the same repo) as CalVer, with release notes.
• Split CI that selects just want to run based on changed files, with green checkmark that respects skips (based on the excellent contrition to pypa/build).

Using the guide

If you have a guide, I’d like for you to compare it with the Scientific Python Development Guide, and see if we are missing anything - bring it to our attention, and maybe we can add it. And then you can link to the centerally maintained guide instead of manually maintaining a complete custom guide. See scikit-hep/developer for an example; many pages now point at this guide now. We can also provide org integrations for the cookiecutter, providing some customizations when a user targets your org (targeting scikit-hep will add a badge).