poetry local config file

Summary

When working with poetry, you can configure poetry with some specifics using the command line interface e.g.:

poetry config virtualenvs.create true

The recommendation here is to always hard-encode those configurations in a local file. This can be done adding a --local flag in the poetry config command:

poetry config --local virtualenvs.create true

The above command will create the a poetry.toml file at the root of your project, with the following information:

[virtualenvs]
create = true

This ensures that your configuration is always hard-encoded and replicable.

List the current poetry configuration

This can be done via a simple command:

poetry config --list

Use-case: system-git-client true

I rarely have to define poetry configurations, however it helps me to fix the following problem I encounter from time to time:

(1) I have a CI/CD pipeline running on Gitlab;

(2) This pipeline has 3 different stages: quality-checks, build and deploy;

(3) Each of these stages might require python dependencies. These dependencies are managed by poetry. You need the Gitlab CI/CD runner to install them within the virtual environment where the stages are gonna execute their scripts. This means, you need each job to run poetry install. E.g.:

variables:
  GIT_SUBMODULE_STRATEGY: recursive

stages:
  - test
  - build
  - deploy

quality-checks:
  image: "<your-custom-docker-path>/ci-cd-python-test-harness:latest"
  stage: test
  script:
    - <setup-gitlab-ssh-access>
    - <add-safe-git-directories>
    - poetry install
    - make checks

...

Note: here, the last line make checks triggers a Makefile action, running black, mypy, pylint and pytest on the codebase. You can have a look on what this make command looks like in the snippet immediately below:

black:
    poetry run black .

mypy:
    poetry run mypy <your-src-folder>

pylint:
    poetry run pylint <your-src-folder>

test:
    PYTHONWARNINGS=ignore poetry run pytest -vvvs <your-test-folder>

checks: black mypy pylint test

Thus, I have my CI/CD running poetry install before the CI/CD runner to be able to test my code in its virtual environment.

(4) Part of the poetry install will install all the dependencies my project contains, including git submodules. This is exactly where our issue lies!

$ poetry install
Creating virtualenv <your-repository>_-py3.10 in /home/gitlab-runner/.cache/pypoetry/virtualenvs
Installing dependencies from lock file
No git repository was found at ../../<your-submodule-repository>.git
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

In order for your project to be able to use git submodules and the CI/CD to run successfully, you need to run the following command:

poetry config --local experimental.system-git-client true

This creates a poetry.toml file with the following lines:

[experimental]
system-git-client = true

This trick should fix the No git repository was found error occurring in your CI/CD pipeline.

Use Poetry as Python Package Manager

Installing poetry is super easy. On macOS, simply run:

brew install poetry

Now, let’s have a look how to use it.

Poetry Cheat Sheet

I have gathered for you in this section the poetry commands you will always need. You can refer to this section later on and simply save the link for later use.

poetry init
poetry install
poetry update
poetry add <your-python-package>
poetry run

Getting started with an example

  1. Create your python project and move under its repository:

    mkdir playground-poetry && cd playground-poetry
    
  2. Init poetry. You will be prompted to fill-in the following configuration fields:

    > poetry init
    Package name [playground-poetry]:
    Version [0.1.0]:
    Description []:
    Author [None, n to skip]:
    License []:
    Compatible Python versions [^3.10]:
    Would you like to define your main dependencies interactively? (yes/no) [yes]
    Would you like to define your development dependencies interactively? (yes/no) [yes]
    Do you confirm generation? (yes/no) [yes]
    
  3. This will generate the pyproject.toml configuration file:

    [tool.poetry]
    name = "playground-poetry"
    version = "0.1.0"
    description = "A primer on poetry."
    authors = ["John Doe <john.doe@gmail.com>"]
    
    [tool.poetry.dependencies]
    python = "^3.10"
    
    [tool.poetry.dev-dependencies]
    
    [build-system]
    requires = ["poetry-core>=1.0.0"]
    build-backend = "poetry.core.masonry.api"
    

After generation, your project’s architecture should look like the following:

playground-poetry
└── pyproject.toml

Add a package

You can simply use the following command e.g.:

poetry add black

In our case, we wanted to install the python black code formatter.

Note: for more general codebase formatting, I recommend super-linter.

You will see that our pyproject.toml poetry configuration file has been updated as it now contains reference to the black package:

[tool.poetry]
name = "playground-poetry"
version = "0.1.0"
description = "A primer on poetry."
authors = ["John Doe <john.doe@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.10"
black = "^22.10.0"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Note: you are wondering what the weird ^ sign stands for? You well soon find an article about it.

You can check that black is indeed accessible and installed within poetry virtual environment via:

> poetry run black --version
black, 22.10.0 (compiled: yes)
Python (CPython) 3.10.4

The pyproject.toml is not the only thing that has changed. If you have a look on our project’s architecture, you will see that it now contains an additional poetry.lock file:

playground-poetry
├── poetry.lock
└── pyproject.toml

Note: poetry is storing the state the same way terraform is doing. If you are new to poetry this might be too much details right now. If you want to know more about poetry.lock an article will follow soon!

Run python code on a poetry environment

Imagine that you have now a python code that requires a couple of dependencies (e.g. could be black, pandas, logging, etc.) to run. E.g.:

"""
A simple module containing maths methods.
"""


def add(number_1: float, number_2: float) -> float:
    """
    Add the numbers.
    Args:
        number_1 (float): the first number.
        number_2 (float): the second number.
    Returns:
        float: the sum of both numbers.

    >>> add(-2, 1)
    -1
    >>> add(42, 0)
    42
    """
    return number_1 + number_2


def main() -> None:
    """
    Main function.
    """

    res = add(4, 7)
    print(res)


if __name__ == "__main__":
    main()

with the following project’s architecture:

playground-poetry
├── playground_poetry
    ├── __init__.py
    └── main.py
├── poetry.lock
└── pyproject.toml

Note: fair enough, in our project we do not really need those dependencies at this point, but let’s say that this is just an extract and other parts in the code do actually use logging or pandas.

You have those dependencies installed on your poetry environment (you can see them on the pyproject.toml dependencies section).

You then need to execute your python code within the umbrella of this poetry virtual environment.

This is done using poetry run python <your-python-file>.

In our example:

> poetry run python playground_poetry/main.py
11

Note: we recommend you to have a similar architecture on your projects as it makes the development of python’s package easier, using the snake_case.

your-project-name
└── your_project_name
    ├── __init__.py
    └── main.py

Get started on a cloned poetry project

Now let’s say you already inherit from an existing poetry project with an already existing pyproject.toml and poetry.lock files.

The first time you need to instantiate the virtual environment, reading from the pyproject.toml file:

poetry install

This will create the poetry.lock file if not existing or resolves the dependencies if so.

You can also update the poetry.lock file if needed:

poetry update

Note: more info https://python-poetry.org/docs/cli/.

Run the extra mile using poetry run and a Makefile

Let’s improve our example project.

Have you noticed that to run our main.py file, you need to explicitly state the whole path:

poetry run python playground_poetry/main.py

You can make things better, editing the pyproject.toml file for the following:

[tool.poetry]
name = "playground-poetry"
version = "0.1.0"
description = "A primer on poetry."
authors = ["John Doe <john.doe@gmail.com>"]
packages = [{include="playground_poetry"}]

[tool.poetry.dependencies]
python = "^3.10"
black = "^22.10.0"
pandas = "^1.5.1"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.scripts]
main = "playground_poetry.main:main"

and from now on simply do the same thing as before but shorter (and faster) via:

> poetry run main
11

Note: this is thanks to the packages line, the __init__.py file nested under it that makes the main.py file visible and the [tool.poetry.scripts] layer.

But that’s not all: we can do even better. Let’s make this command even shorter, saving it under a Makefile command.

In our example project, let’s add a Makefile with the following lines:

main:
    poetry run main

The final structure should look like the following:

playground-poetry
├── playground_poetry
    ├── __init__.py
    └── main.py
├── Makefile
├── poetry.lock
└── pyproject.toml

Finally, you can run the main function using:

> make main
poetry run main
11

And you thought our main job were to “write” code? The less the more! 😇

What have you learned

  • You can create a poetry environment from scratch to manage your python dependencies.
  • You can reuse an existing one.
  • You can scale and automate using Makefile commands.
  • You got a short primer on Software Development Standardization and Best Principles.