Publishing a Package to PyPI with Poetry

I recently published my first ever package to the Python Package Index (PyPI). The process was particularly novel for me as it meant my first ever use of the Poetry package management tool. If you happen to work in a production studio that uses ftrack and is now working remotely, you might even want to take a look at it as it enables sharing of assets using the Simple Storage Service (S3) from Amazon Web Services.

Why not use setuptools?

Typically, if you wanted to distribute a library you’d written you’d likely be pointed in the direction of setuptools/distutils in order to build your module for distribution. It’s part of Pythons standard library and has been since 1998 and was my first choice. The problem I quickly found was a single setup.py would generally be tailored to a particular version of Python and I wanted to publish to both Python 2 (!) / 3 due to ftracks desktop application which drove my package still running on Python 2. This is fairly common within desktop vfx software though there’s efforts underway to standardise to using Python 3.7. This meant potentially maintaining two separate versions of this file and two separate Pipfiles for my virtualenvs. Additionally, I’d need to write a number of utility functions to read in details of the version of the library and README.md to show in the descriptions on PyPI. I’d also need to install a number of other tools in order to publish.

Each of these problems in themselves aren’t that much of an issue, but I decided to investigate using Poetry after being recommended it in the past.

Poetry

Poetry is a somewhat new contender in the world of package management. When first getting started with Python, I used pip for all my package installations. Over the last few years, I’ve settled on using Pipenv in combination with pyenv in order to manage the installs of Python and the virtual environments I use with them. Pipenv is typically recommended as a way of installing packages but it doesn’t however offer any way of publishing your project to PyPI.

Poetry uses a pyproject.toml file in order to describe your project and its dependencies. If you haven’t yet heard of it, there’s a great description here on what pyproject.toml is and why it exists. It also defaults to working with the activated python version if it is compatible with your project. In contrast to setuptools, it doesn’t require utility code to load in markdown or version numbers. It also is able to manage dependencies for both versions of Python alongside one another, without multiple files in play. It also is only a single library, meaning I only have to remember a single tool. This seemed like a pretty good choice to me for my project.

To generate a new pyproject.toml you use:

poetry new my-new-project

My project only has a couple of dependencies, boto3 and the core ftrack python library. Specifying them is a simple add call:

poetry add boto3

This will install the boto3 library to a virtualenv and and generate a poetry.lock detailing the exact versions of the boto3 library that has been installed.

The full pyproject.toml with all dependencies looks as follows:

[tool.poetry]
name = "ftrack-s3-accessor"
version = "0.1.7.post1"
description = "A ftrack s3 accessor updated to work with ftrack-python-api and boto3."
authors = ["Ian Wootten <hi@niftydigits.com>"]
license = "Apache 2"
readme = "README.md"
homepage = "https://github.com/niftydigits/ftrack-s3-accessor"
repository = "https://github.com/niftydigits/ftrack-s3-accessor"

include = [
    "LICENSE.md",
]

[tool.poetry.dependencies]
python = ">=2.7.9,<2.8 || >=3.7,<4.0"
boto3 = "^1.15.10"
ftrack-python-api = "^2.0.0"
ftrack-action-handler = {version = "^0.2.1", optional = true}

[tool.poetry.dev-dependencies]
pytest = {version="^5.2", python = ">=3.7,<4.0"}

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.extras]
action = ["ftrack-action-handler"]

You can see here the versions of Python can be edited to those I want to target. Poetry is pretty clever about the way dependencies are tracked and is able to alert you to whether there may be any complications for particular combinations of Python and dependencies when you try to build your project.

Publishing to PyPI

The Python Package Index has both a test site and the official site, both of which anyone is able to publish to. You’ll need an account with at least the public site, but it’s strongly recommended you go ahead and create an account on test.pypi.org too for testing purposes. I’ve made several mistakes as I was experiementing, so I’d definitely recommend this.

You can configure Poetry to remember the test index as a separate repositority in its configuration:

poetry config repositories.testpypi https://test.pypi.org/legacy/

Poetry has a simple way of building the project for PyPI:

poetry build

You should see output something similar to the following if everything is in order. Note for my project, a universal wheel has been built for both Python 2 and 3:

Building ftrack-s3-accessor (0.1.7.post1)
  - Building sdist
  - Built ftrack-s3-accessor-0.1.7.post1.tar.gz
  - Building wheel
  - Built ftrack_s3_accessor-0.1.7.post1-py2.py3-none-any.whl

You won’t be able to publish the same version number of a file to either repository more than once, so you need to be very certain you’re happy with the configuration you’ve specified before going ahead and publishing.

Once you’re happy, the package is then be able to test and publish to test.pypi.org:

poetry publish -r testpypi

You’ll be asked for your credentials but you can configure an api key in poetry to be used for publishes for each repository if you don’t want to have to remember these each time you publish a new version.

Congratulations, your package should now be available on test.pypi.org! You should inspect how looks there and make any neccessary changes to your pyproject.toml if there are mistakes. Originally, I’d left out the README definition and so it didn’t appear in PyPI. Once you’re happy with it, you can publish your package to the official PyPI site:

poetry publish

Final Thoughts

I definitely appreciated having a single simple tool to create my first publishes to PyPI. Managing the complexity that comes with wanting to publish for multiple versions of Python is not something I would want to have to do myself. If you’re looking into simplifying your package publication, I definitely recommend taking a look. PyPI publication always seemed a bit of a dark art to me, given now I know how simple it can be I hope I’ll be able to make more of my work available.

In the next post I take things a step further and use github actions to automate publishing to PyPI.