Python 3.11 has hit the beta (now released!) stage, which means no more new features. It’s a perfect time to play with it! The themes in this update are the standard ones: The faster CPython project is now fully going (3.11 is 25% faster on average), along with improved error messages, typing, and asyncio. Beyond this, the only major new feature is a library for reading TOML files; this probably only exciting if you are involved in Python packaging (but I am, so I’m excited!).

Faster CPython

I will predict this is one of the main reasons users will be going through the effort of upgrading. The faster CPython project has some lofty goals, and it’s already showing results: CPython 3.11 is 25% faster on average than CPython 3.10, and 60% faster for some workloads. It will depend on what you are doing; compiled code (like NumPy) will not change, since that’s not dependent on CPython’s performance in the first place.

This also means there are lots of internal API cleanups and changes, and lots of bytecode changes (things like Numba will likely have extra work to do to upgrade). If you use a tool like pybind11 or Cython to write compiled extensions, you should be able to just upgrade to get CPython 3.11 support.

A few specific optimizations:

• Exceptions are zero-cost - the try statement itself has almost no cost associated with it if nothing is thrown.
• C-style formatting in very simple cases is now as fast as f-strings. Not that you should use this, but legacy code will be faster.
• Dicts with all str keys are optimized (smaller memory usage).
• re is up to 10% faster using computed goto’s (so not applicable for WebAssembly).
• Faster startup reading core modules (10-15%).
• Cheaper frames (function calls).
• Inlined (faster) calls for Python calling Python.
• Specializing adaptive interpreter (running the same operation multiple times can be faster).

Error messages

This version has a smaller set of changes for error messages than 3.10, but the main two it has are huge.

Error messages can now show you exactly where in the expression the error occurred. This makes debugging massively easier, since you can tell what variable is broken in a longer expression. Here’s an example:

obj = {"a": {"b": None}}

obj["a"]["b"]["c"]["d"]

Traceback (most recent call last):
  File "tmp.py", line 3, in <module>
    obj["a"]["b"]["c"]["d"]
    ~~~~~~~~~~~~~^^^^^
TypeError: 'NoneType' object is not subscriptable

That ~~~^^^ part is new, and tells you which subscript failed - before 3.11, you couldn’t tell which of the four subscripts there was None. Now you can tell that the third one is the problematic one. Note that this must be in a file; it will not do this if it’s just in the REPL. It also adds a tiny memory cost, so it can be disabled, but please don’t.

The second big feature is all Exceptions now have an .add_note(msg) method, which will inject a note to the exception that will be printed at the bottom. This allows the classic “suggest” pattern to finally be written properly:

try:
    import skbuild
except ModuleNotFoundError as err:
    err.add_note("Please pip or conda install 'scikit-build', or upgrade pip.")
    raise

This produces:

Traceback (most recent call last):
  File "tmp.py", line 2, in <module>
    import skbuild
    ^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'skbuild'
Please pip or conda install 'scikit-build', or upgrade pip.

If you are writing an application, you shouldn’t dump exceptions, but instead print proper error messages, but for libraries, this is fantastic. This can be called multiple times, and each string gets added to a __notes__ tuple on the exception.

A related change is the addition of sys.exception(), which is a nicer way to spell sys.exc_info()[1], along with some related changes making it easier to just use the current exception without the extra type & traceback. Modifications to the traceback are properly propagated.

Typing

This continues to be a place where great strides are made each version, though most of the new features also are available to older versions, either from typing_extensions or by using string annotations via from __future__ import annotations.

Variadic generics

Generics can now be variadic, supporting a variable number of arguments, using TypeVarTuple. For example, NumPy wanted this feature to indicate the sizes and dimensions of an array. Here’s a quick example from PEP 646:

from typing import Generic, TypeVar, TypeVarTuple, NewType

DType = TypeVar("DType")
Shape = TypeVarTuple("Shape")


class Array(Generic[DType, *Shape]):
    def __abs__(self) -> Array[DType, *Shape]: ...

    def __add__(self, other: Array[DType, *Shape]) -> Array[DType, *Shape]: ...


Height = NewType("Height", int)
Width = NewType("Width", int)

x: Array[float, Height, Width] = Array()

Self type

This is a huge one, because it is such a common pattern. There now is a Self type that describes the current class. This is perfect for classes that return self (for easy chaining) or a new instance (including classmethods!):

# Before
Self = TypeVar("Self", bound="Vector")


class Vector:
    def square(self: Self) -> Self:
        return self**2

    @classmethod
    def from_coords(
        cls: Type[Self],
        *,
        x: float,
        y: float,
    ) -> Self:
        return cls(x, y)

# After
class Vector:
    def square(self) -> Self:
        return self**2

    @classmethod
    def from_coords(
        cls,
        *,
        x: float,
        y: float,
    ) -> Self:
        return cls(x, y)

If you are tempted to type the name of class in a return annotation as a string, consider Self - it’s more accurate as a return most of the time.

LiteralString

One common security issue is forgetting to escape strings that get generated from external input. For example, sqlite will correctly escape input as long as you don’t try to build the string input yourself. This now can be expressed with the type system. A LiteralString is a string typed into the code.

def f(other: str) -> None:
    a_literal_str = "I am a literal string"
    also_literal = f"{a_literal_string} too!"
    not_literal = f"Not a literal due to {other}"

    typing.assert_type(also_literal, LiteralString)
    typing.assert_type(not_literal, str)

Notice that a f-string composed of LiteralString’s is also a LiteralString!

Exhaustiveness checking

Another nice addition is the common exhaustiveness pattern is now official and included, and has better spelling.

# Before
def assert_never(v: NoReturn, /) -> NoReturn:
    assert False, f"Unhandled: {v}"


def f(x: Literal["a", "b"]) -> None:
    if x == "a":
        return
    if x == "b":
        return
    assert_never(x)

# After
def f(x: Literal["a", "b"]) -> None:
    if x == "a":
        return
    if x == "b":
        return
    typing.assert_never(x)

This will display a message if you forget to check every possible branch - and unlike the old way to write this, the error message will include typing.Never instead of the seemingly unrelated typing.NoReturn.

Other

• Dataclass transforms - helping libraries that have dataclass-like decorators.
• Required/NotRequired for TypedDict.
• typing.reveal_type is now an official (and available in typing) function. reveal_types() is not (yet), however.
• More support for Generic subclasses (TypedDict, NamedTuple).
• Any subclasses supported (if you are making a fully dynamic class, for example).

Also, typing.assert_type lets you verify a typing construct is true. This is great for Protocol checking:

# Before
if typing.TYPE_CHECKING:
    _: Duck = typing.cast(MyDuck, None)

# After
typing.assert_type(
    typing.cast(Duck, None),
    MyDuck,
)

AsyncIO

The only major new syntax feature in Python 3.11 comes in support for asyncio: ExceptionGroups. You can manually build an ExceptionGroup, or they are produced from the next feature, but the interesting part is handling them. Here’s an example:

try:
    raise ExceptionGroup(
        "multiexcept",
        [TypeError(1), KeyError("two")],
    )
except* TypeError as err:
    print("Got", *(repr(r) for r in err.exceptions))
except* KeyError as err:
    print("Got", *(repr(r) for r in err.exceptions))

This will print out:

Got TypeError(1)
Got KeyError('two')

Notice that multiple exceptions match (unlike normal try/except), and that you get a new ExceptionGroup with all errors that match (since there might be more than one). You can also manually catch and handle ExceptionGroup, it’s just an exception type for multiple exceptions with nice pretty printing - the new syntax just makes handling them much easier.

This has been backported as exceptiongroup (without the new syntax), and is already in use by cattrs to bundle all parsing errors into a single grouped exception. Instead of breaking on the first failure, cattrs will show all failures at once! This is going to be transformational for error reporting for things that are not linear, even if they are not running in parallel, like validating a data model.

Sadly, all libraries that handle tracebacks manually (pytest, IPython, Rich, etc) will have to update to support exceptiongroup (but it’s basically the same work needed to support 3.11’s new formatting too).

This has enabled asyncio TaskGroups, which are similar to Trio nurseries.

import asyncio


async def printer(n):
    await asyncio.sleep(n)
    print("Hi from", n)


async def main():
    async with asyncio.TaskGroup() as g:
        g.create_task(printer(2))
        g.create_task(printer(1))


asyncio.run(main())

Hi from 1
Hi from 2

Here’s a fun example using the Rich library’s scroll bars:

from rich.progress import Progress
import asyncio


async def lots_of_work(n: int, progress: Progress) -> None:
    for i in progress.track(range(n), description=f"[red]Computing {n}..."):
        await asyncio.sleep(0.05)


async def main():
    with Progress() as progress:
        async with asyncio.TaskGroup() as g:
            g.create_task(lots_of_work(120, progress))
            g.create_task(lots_of_work(90, progress))


asyncio.run(main())

Tomllib

A TOML parser (not writer) is now part of the standard lib. It looks just like tomli (because it is basically just a stdlib version of tomli). It’s hard not to be just a little bit sad that YAML wasn’t chosen for packaging configuration, because this likely would have then been a stdlib YAML parser like Ruby has, but still nice to see. This is great for configuration - now libraries can support pyproject.toml (or any other TOML files) without a third-party dependency.

If you have a TOML file:

[tool.mylib]
hello = "world"

Then parsing it is simple:

import tomllib

with open("mylib.toml", "rb") as f:
    config = tomllib.load(f)

assert config["tool"]["mylib"]["hello"] == "world"

If you want to write a TOML file, you can continue to use the tomli-w package. As a quick reminder, the toml package is dead and should not be used, use tomli instead if you need to support Python 3.10 or earlier.

Other features

This is the first version of CPython to directly support WebAssembly (wasm32-emscripten / wasm32-wasi)! You can use Python 3.10 today through Pyodide, but 3.11 should directly support it, making it easier on Pyodide, as well as enabling other distributions for web browsers. Native support should mean good performance and light weight download sizes, too! Over time, the support tier hopefully will improve to tier 2, currently targeting tier 3.

Here are a few other features of note:

• contextlib.chdir provides a thread unsafe way to change directory temporarily.
• You can disable the automatic injection of the current working directory to the path when Python starts with PYTHONSAFEPATH. Check sys.flags.safe_path from code.
• Unions now work in functools.singledispatch.
• operator.call added
• Atomic grouping and possessive qualifiers in re. You can sometimes rewrite regex to be much faster with this. Here’s an article on them.
• PyBuffer was added to the Limited API / Stable ABI.

There are quite a few other minor features that you might like that were not notable enough for this list, like * unpacking directly inside for. Check the release notes!

Other developer changes

Library developers may need to be aware of the following changes:

• venv uses sysconfig installation schemes.
• Lots of bytecode changes.
• Lots of deprecations, like chaining classmethods (which has always been buggy).
• Some removed deprecated features, like asyncio.coroutine and stuff in inspect.
• More legacy stuff for supporting Python 2 is being removed. Supporting Python 2 and 3.11 at the same time is likely much harder, please support 3.7+ or better.
• Lots of build system updates, include C11 required.
• Lots of C API changes - see python/pythoncapi-compat for help in supporting multiple versions if you aren’t using pybind11, Cython, or some other binding tool.

There are also lots of new deprecations, including a bunch of rarely used modules (see PEP 594).

Final Words

This is an exciting release of Python that hits all the right buttons. Faster, better error messages, better typing, and better asyncio. The support for pre-release pythons is fantastic these days, with GitHub Actions supporting new alphas/betas/RCs in after about a day (python-version: 3.11-dev will give you the latest dev release). Please test! If you ship binaries, CPython 3.11 is ABI stable now and cibuildwheel 2.9 includes 3.11 wheels by default. Ship binaries now, before October hits if possible so we can hit the ground running!

Pybind11 currently (2.10) supports 3.11, minus a small dynamic multiple inheritance bug from the new API. Cython and MyPyC are affected by the same issue, as well. The CPython maintainers have rolled back a change for us, so there will likely be a pybind11 2.10.1 soon that requires CPython 3.11rc1 or newer to target 3.11. (Only required if you embed Python 3.11, otherwise 2.10.0 works).

Sources and other links

• Official docs
• Deepsource (lots of examples)
• Anthony Explains (video, top 10 things)
• MCoding (video)