February 2021. Maintainers of the cryptography package release a new version that depends on Rust to build. Within hours, CI pipelines across the world start burning, and the hallucinogenic smoke awakens internet trolls. What happened, and how does it relate to Poetry, the popular Python dependency manager?
Pin Your Transitive Dependencies, My Friends
I’m going to ignore the Rust part of the cryptography kerfuffle – if these folks thirst for Rust, who am I to complain? To me, the real problem was that many users had failed to pin the cryptography package. Either they didn’t pin it at all because it was a transitive dependency, or they used a version range that assumed cryptography followed semantic versioning, which it didn’t.
I was in the first group – one of my projects had only pinned direct dependencies, so my builds started failing. I often
pip freeze dependencies into requirements files, but for this small app, I hadn’t bothered and instead wrote my direct dependencies by hand in
I disliked running
pip freeze every time I added a dependency, and I also wanted to avoid managing the requirements file by hand. If this also describes you, then you’ll love Poetry.
Poetry brings an experience on par with the dependency management tools in other languages, like Bundler, npm, and crate.
The Goal: Repeatable Builds
Why should we even use Poetry? Because we need repeatable builds. Say it with me: Repeatable Builds!
A transitive dependency should never kill your CI pipeline or screw up an engineer’s development environment.
To get there, we need a great Python dependency manager that makes adding direct dependencies to your project easy and handles resolving and tracking transitive dependencies. That’s what Poetry does.
But What About <tool that does this already>?
What about Pipenv, pip-tools, and Conda? Don’t they do something like this?
Pipenv is a direct competitor to Poetry, but I like Poetry better. Poetry is a more active project and works with the relatively new pyproject.toml file. I’m also a bit of a rebel, so having a group of Python developers who call themselves the “Python Packaging Authority” tell me to use Pipenv is a turn-off.
And Conda? Data science people have recommended Conda to me for years, but the Conda workflow never clicked with me. I like virtual environments and pip!
On the other hand, I have used pip-tools. If you aren’t ready to jump into a dependency manager like Poetry yet, but you would like to automate tracking transitive dependencies, pip-tools is a good start. The post RIP Pipenv has great arguments for using pip-tools. I would still use pip-tools, but I grew tired of writing
pip-compile. Poetry makes everything easier.
The main thing about Poetry I hope you take away from this post is how smooth Poetry’s
add command is. This command is how you add dependencies. For example, you type
poetry add aioredis to add the latest version of the library aioredis-py.
Here’s what running
poetry add aioredis looks like:
❯ poetry add aioredis Using version ^1.3.1 for aioredis Updating dependencies Resolving dependencies... (0.2s) Writing lock file Package operations: 3 installs, 0 updates, 0 removals • Installing async-timeout (3.0.1) • Installing hiredis (2.0.0) • Installing aioredis (1.3.1)
To add a specific version, you can write
poetry add email@example.com. You can also install a package directly from GitHub like this:
poetry add git+https://firstname.lastname@example.org.
When you run
poetry add with or without a version number, Poetry does the following things:
- Installs the package
pyproject.tomlwith the new top-level dependency
poetry.lockto pin both top-level and transitive dependencies
If you’ve tried to use
pip freeze or
pip-tools to keep track of dependencies and found all the steps you had to take annoying, you’ll love how freeing Poetry’s automation is.
Worried about breaking your existing tools with a new workflow while you try out Poetry? You can always use the export command to copy dependencies to a requirements.txt file.
And there is much more to Poetry: aside from adding, removing, and updating dependencies, Poetry also manages virtual environments, gives you an interactive shell, and more.
If you test it out, let me know what you think!