Why Is Poetry Essential to the Modern Python Stack?

A Story

February 2021. Maintainers of the cryptography package release a new version that depends on Rust to build. Within hours, CI pipelines across the world start burning, and the hallucinogenic smoke awakens internet trolls. What happened, and how does it relate to Poetry, the popular Python dependency manager?

Pin Your Transitive Dependencies, My Friends

I’m going to ignore the Rust part of the cryptography kerfuffle – if these folks thirst for Rust, who am I to complain? To me, the real problem was that many users had failed to pin the cryptography package. Either they didn’t pin it at all because it was a transitive dependency, or they used a version range that assumed cryptography followed semantic versioning, which it didn’t.

I was in the first group – one of my projects had only pinned direct dependencies, so my builds started failing. I often pip freeze dependencies into requirements files, but for this small app, I hadn’t bothered and instead wrote my direct dependencies by hand in requirements.txt.

I disliked running pip freeze every time I added a dependency, and I also wanted to avoid managing the requirements file by hand. If this also describes you, then you’ll love Poetry.

Poetry brings an experience on par with the dependency management tools in other languages, like Bundler, npm, and crate.

The Goal: Repeatable Builds

Why should we even use Poetry? Because we need repeatable builds. Say it with me: Repeatable Builds!

A transitive dependency should never kill your CI pipeline or screw up an engineer’s development environment.

To get there, we need a great Python dependency manager that makes adding direct dependencies to your project easy and handles resolving and tracking transitive dependencies. That’s what Poetry does.

But What About <tool that does this already>?

What about Pipenv, pip-tools, and Conda? Don’t they do something like this?

Pipenv is a direct competitor to Poetry, but I like Poetry better. Poetry is a more active project and works with the relatively new pyproject.toml file. I’m also a bit of a rebel, so having a group of Python developers who call themselves the “Python Packaging Authority” tell me to use Pipenv is a turn-off.

And Conda? Data science people have recommended Conda to me for years, but the Conda workflow never clicked with me. I like virtual environments and pip!

On the other hand, I have used pip-tools. If you aren’t ready to jump into a dependency manager like Poetry yet, but you would like to automate tracking transitive dependencies, pip-tools is a good start. The post RIP Pipenv has great arguments for using pip-tools. I would still use pip-tools, but I grew tired of writing pip-compile. Poetry makes everything easier.

A Demo

The main thing about Poetry I hope you take away from this post is how smooth Poetry’s add command is. This command is how you add dependencies. For example, you type poetry add aioredis to add the latest version of the library aioredis-py.

Here’s what running poetry add aioredis looks like:

❯ poetry add aioredis
Using version ^1.3.1 for aioredis

Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 3 installs, 0 updates, 0 removals

  • Installing async-timeout (3.0.1)
  • Installing hiredis (2.0.0)
  • Installing aioredis (1.3.1)

To add a specific version, you can write poetry add aioredis@1.3.1. You can also install a package directly from GitHub like this: poetry add git+https://github.com/karantan/ansible.git@v2.6.0.1.

When you run poetry add with or without a version number, Poetry does the following things:

• Installs the package
• Updates pyproject.toml with the new top-level dependency
• Updates poetry.lock to pin both top-level and transitive dependencies

If you’ve tried to use pip freeze or pip-tools to keep track of dependencies and found all the steps you had to take annoying, you’ll love how freeing Poetry’s automation is.

Worried about breaking your existing tools with a new workflow while you try out Poetry? You can always use the export command to copy dependencies to a requirements.txt file.

And there is much more to Poetry: aside from adding, removing, and updating dependencies, Poetry also manages virtual environments, gives you an interactive shell, and more.

If you test it out, let me know what you think!