Poetry provides a all in one solution to Python packaging. I want to focus on why I was quite hard on Poetry in my last post, specifically on its default version capping and solver quirks, and also a few other negative things. This is a followup to Should you have upper version bounds, which you should read before this post.
Why so hard on Poetry?
Regardless of the tone of the rest of this post, I do like Poetry, and I provide it as one of eight backends for the Scikit-HEP/cookie project generation cookiecutter and use it for several projects, including at least one library. I have great respect for what they managed to pull off, and they were one of the first alternatives to the standard tools, which was great. It’s wildly popular, though, so I don’t think I need to sing it’s praises, but rather issue warnings about some of the decisions it makes. I’m not going to talk about things like the developers intentionally adding a 5% failure chance to “deprecate” a feature, but instead on the intentionally unpythonic versioning schemes Poetry pushes.
And I should also point out the nearly drop-in replacement pdm which provides the same benefits, but lets you select your version capping strategy, and doesn’t cap as badly, and follows more PEP standards. hatch doesn’t have a lockfile yet (it’s waiting for a standard), but provides an environment per task, which is conceptually better than one massive dev environment.
I believe most users don’t realise it has a unique, slow, and opinionated solver. Also, Poetry users are often intimidated by the plethora of tools it can replace, like setuptools/flit, venv/virtualenv, pip, pip-tools, wheel, twine, bump2version, and nox/tox; and that sort of user is very easily influenced by the defaults and recommendations they are seeing, since they do not have enough experience in the Python ecosystem to know when a recommendation is a bad one. The draw of mix-and-match tends to come later once they start having stronger opinions on the way things should work.
Not only does running
poetry add <package> automatically use
generating a new project adds both a caret cap to pytest and Python itself! And
if you have
python = ^3.6, all Poetry users who depend on your package will
have to have a cap on the Python version. It doesn’t matter if you’ve read the
version capping discussion agree with every
single line; if you depend on just a single package that caps Python and use
Poetry, you must add the cap. And, if that dependency (after reading this
discussion) removes the cap, you will still be capped. Even if they removed the
cap in a patch version so therefore it does not apply to you anymore.
Example (click to expand)
Let’s say I depend on library
A==1.0.0 caps Python to
<3.10. Poetry will force me to also
cap my library to at most
<3.10 in my
Now let’s say
A==1.0.1 is released, and it loosened the cap to
package now is not constrained to
3.11 by my dependencies, since I allow
A==1.0.* , except by Poetry forcing me to write
<3.10 in my
pyproject.toml. Now I have to update, anything that depends on me has to
update, and so on down the chain.
If I dependent on A=1.0.0, then this would be more reasonable. But you can’t predict the future, specifically that your dependencies may loosen or remove upper bounds; in fact, unless they are abandoned, that’s exactly what they will do over time!
I believe a resolver should only force limits on you if you pin a dependency exactly. Any pin that allows a single “newer” version of any form should never force you to duplicate the limits in those unpinned dependencies in your file. However, Poetry developers have said “this behavior by Poetry will never change”. I personally believe caps should only be made for known incompatibilities, but it doesn’t matter, I can’t use Poetry and a single dependency that caps Python version without being forced to do so myself. Even if I’m making a simple library that uses textbook Python with uncapped dependencies that I know will update to the new Python.
I do realise that the reason Poetry is making this constraint on you is due to the lock file (a useful comparison between pipenv, Poetry, and PDM helps illustrate this). When it generates the lock file, then it is selecting an exact, locked version that explicitly states it will not be compatible, which means your lock file will not load on all the versions you specify. The reason Python is “special” is because you can’t lock Python, while all other dependencies are locked. So it’s forcing you to make a truthful lock file (still would rather just a warning here, though).
Besides making it doubly important to never cap the Python version, this is due
to the clash between specifying lock file dependencies and library
dependencies that get propagated to the metadata and therefore PyPI. There
might be a great solution to this: Add PEP 621 support, then if both PEP 621’s
project.requires-python is set and
tool.poetry.dependencies has a
entry, use the former for the project metadata and the latter for the lock file.
A tool that lets you produce PyPI packages should not force you to set a
metadata slot as important as
Requires-Python based on a lock file you are not
even including in the package.
I’ve discussed this in Poetry, and as a result, instead of fixing the resolver, fixing the default add, and/or fixing the default template, they have a page describing why you should always cap your dependencies. As I’ve pointed out, this reasoning is invalid - you can’t ensure your code works forever by adding pins; just the opposite, in fact, you will have reduced future compatibility - especially important for a library. And if you are available to make quick updates, you can quickly update to add a pin if something breaks (and then fix it). You can ask a user to pin in an issue until you fix it if you are not available for a quick release. A user can use a locking system (like Poetry provides). Etc. Anything is better than solver errors when they are invalid.
I also do not like the dependency syntax in Poetry using TOML tables. I have one complaint with the standard dependency syntax; there was no ability for one extra to depend on another, but this was solved in pip 21.2+, and Poetry’s new syntax doesn’t actually solve that. Instead, it seems to be overly complex, depends on long inline TOML tables (which are slightly broken in TOML for users IMO since they arbitrarily don’t support newlines or trailing commas), and require as much or more repetition, and don’t actually support exposing the “dev” portion directly as an extra. If you have an extremely complex set of dependencies, maybe this would be reasonable, but I’ve avoided mixing really complex projects and Poetry.
I have also asked for Poetry to also support PEP 621, and so far they have
held back, saying their system is “better” than supporting a standard they
helped develop, maybe because they are unhappy that no one else liked their
dependency syntax? Now GitHub has
poetry.lock) as a replacement for
setup.py for their dependency scanning, but not the standards-based settings
that would have also benefited flit, pdm, trampolim, and whey (and probably
many more in the future, including setuptools). Also, you have to learn the
standard syntax anyway for PEP 518’s
requires field that Poetry depends on
to work, so you are always going to have to learn the PEP 440 syntax to use
Poetry was also very slow to support installing binary wheels Apple Silicon, or even macOS 11; while most1 of the PyPA tooling supported it quickly. This means that things like NumPy installed from source, which made Poetry basically useless for scientific work for quite a while on macOS, where source installs for NumPy don’t work. I would like to see them prioritize patch releases if there’s an entire OS affected - their own pinning system forces users to make patch releases more often, but they haven’t been doing so themselves.
My recommendation would be to consider it if you are writing an application and
maybe for a library, but just make sure you fix the restrictive limits and
understand the limitations and quirks. As long as you know what you are doing,
it can be a great system. The “all-in-one” structure is really impressive, and
using it is fun. I think the new plugin system will likely make it even more
popular. But using individual tools is more Pythonic, and lets you select
exactly what you like best for each situation. Flit is just as simple or
simpler than Poetry for configuration, and supports PEP 621 (even if rather
secretly at the moment). Setuptools is not that bad as long as you use
setup.cfg instead of
setup.py, and has
setuptools_scm, which is really
nice for some workflows. I would recommend reading either
https://scikit-hep.org/developer to see what the composable, standard tools
What would I recommend Poetry do to improve the situation for libraries?
This is just my personal wishlist, and I don’t think it’s going to happen, but here it is:
- Drop the default capping on
poetry add. This should be a clear and explicit choice by a developer. Since it is already setting the current version as a minimum (which is fine for a first guess), these are very tight requirements, too!
- Drop the default capping in the new project template. There is no reason at
all that pytest should be limited to
^6for developers! Your test suite will not break in pytest 7 unless there’s a bug in pytest 7, and there could be a bug in any minor or patch version (looking at 6.1.0). And it will get fixed if you report it.
- Drop the default
<4requirement on Python as well; this will make the Python 4 transition harder by making it impossible to test or run old code, and will not “fix” a single solve, ever.
- Provide a way around the forced requirement that you cap matching your dependents. Libraries should not be unable to specify what they want to support due to dependents forcing a certain lock file that the library is not going to use. PEP 621 would be a perfect opportunity to provide a way to specify the metadata separately from the solve if needed.
If Poetry does not want to make these changes, they should implement local
npm does. Or there are other possible solutions, too, like
having two sets of requirements, one “recommended” one and one “required” one
(this one would be the PyPI published one). Poetry would try to use the
recommended (capped) requirements but would fall back to the required caps if a
solution could not be found, etc.
Pipenv doesn’t count in “most” here… ↩︎