Category: python

Python type annotations are an excellent way to communicate your intentions across your development teams (to enforce your intention, you need to use a type checker like mypy though).

This makes your code base more robust.

Multiple types are accessible in Python. Here we go into the detail of Literal, NewType and Final.

To illustrate the concepts, we will work with the following Cheese object:

from dataclasses import dataclass

@dataclass
class Cheese:
    """
    Class representing a cheese object.
    """

    name: str
    price_per_kilo: float
    aoc: bool


if __name__ == "__main__":

    cheese = Cheese(
        name="roquefort",
        price_per_kilo=25.75,
        aoc=True
    )

Note: AOC stands for Appellation d’Origine Controlée. As a trademark, this label intends to protect French Cheeses from counterfeiting. Some cheese are protected. Some are not. There are over 1,600 varieties of cheese. Each of them pairing well with specific wines.

Literal

The Literal type allows you to restrict the variable to a very specific set of values.

from dataclasses import dataclass
from typing import Literal

@dataclass
class Cheese:

    name: Literal["roquefort", "comté", "brie"]
    price_per_kilo: float
    aoc: bool


if __name__ == "__main__":

    cheese = Cheese(
        name="abondance",
        price_per_kilo=28,
        aoc=True
    )

If you run mypy on the command line against that file, you will get an error:

error: Argument "name" to "Cheese" has incompatible type "Literal['abondance']";
expected "Literal['roquefort', 'comté', 'brie']"  [arg-type]

Tip: In most development environments you can get the typechecker analysis in real time. In Visual Studio Code, you can get notified of errors as you type using the MyPy Type Checking extension: code --install-extension matangover.mypy.

Notes:

• To restrict possible values of a variable, you can also use Python Enumerations. However, Literal is more lightweight.

• You can also use Annotated types to specify more complex constraints (e.g. to constrain a string to a specific size or to match a regular expression) but this type is best served as a communication method.

NewType

A NewType takes an existing type and creates a brand new type that possesses the exact same fields and methods as the existing type.

NewType is useful in a handful of real-world scenario.

For instance, to make sure you only operate upon sanitized strings to prevent SQL injections you could establish a distinction between a str and a SanitizedString.

Same to separate between a User and a LoggedInUser.

Let’s say we are a gastronomic restaurant. We only want to serve Protected Cheese to our customers.

For the service, a dedicated method called dispense_protected_cheese_to_customer makes sure that
only protected cheese can be served. And for sure, as it only takes our new ProtectedCheese type (based on the Cheese type) as argument and refuses everything else.

from dataclasses import dataclass
from typing import NewType

@dataclass
class Cheese:

    name: str
    price_per_kilo: float
    aoc: bool


ProtectedCheese = NewType("ProtectedCheese", Cheese)


def prepare_for_serving(
    cheese: Cheese
) -> ProtectedCheese:
    protected_cheese = ProtectedCheese(cheese)
    # you can image other suitable methods being
    # applied here...
    return protected_cheese


def dispense_protected_cheese_to_customer(
    protected_cheese: ProtectedCheese
) -> None:
    # ...
    return None


if __name__ == "__main__":

    cheese = Cheese(
        name="charolais",
        price_per_kilo=47.60,
        aoc=True
    )

    protected_cheese = prepare_for_serving(cheese)
    dispense_protected_cheese_to_customer(protected_cheese)

You can try twisting around the above snippet yourself, by trying to serve a non-protected cheese to a customer:

protected_cheese = prepare_for_serving(cheese)
dispense_protected_cheese_to_customer(cheese)

Mypy will complain:

error: Argument 1 to "dispense_protected_cheese_to_customer" has incompatible type "Cheese";
expected "ProtectedCheese"  [arg-type]

Note: The prepare_for_serving method acts as a blessed function, creating our protected cheese from our original cheese blueprint type. It is important to note that the only way to create new types is through a set of blessed functions.

One more thing; NewType is a useful pattern to be aware of. However, classes and invariants provide a similar but much stronger guarantees to avoid illegal states.

Final Types

Final types allow you to prevent a type from changing its value over time (e.g. if you do not want the name of a variable to be changed by accident).

from typing import Final

RESTAURANT_NAME: Final = "Le Central"

Should you try set the variable to a new value, mypy will complain:

from typing import Final

RESTAURANT_NAME: Final = "Le Central"

if __name__ == "__main__":

    RESTAURANT_NAME = "Le Bois sans feuilles"

error: Cannot assign to final name "RESTAURANT_NAME"  [misc]

Final is hence useful to prevent a variable from being rebound.

Note: Both restaurants belong to the Troisgros family. Both are based in Roanne. Le Bois sans feuilles is a prestigious three-stars restaurant. A documentary about it was released in 2023: Menus Plaisirs – Les Troisgros.

Type Aliases and Variables Annotations

Even though it is cumbersome, you can add type annotations not only to methods but to variables:

def reverse_string(string_candidate: str) -> str:
    reversed_string: str = string_candidate[::-1]
    return reversed_string

You can also alias a particularly long type:

from typing import Dict, List

DictOfLists = Dict[str, List[int]]

def element_in_lists(
    dictionary: DictOfLists,
    element: int
) -> list[bool]:
    return [element in l for _, l in dictionary.items()]

if __name__ == "__main__":

    my_dict = {
        "a": [1,2],
        "b": [],
        "c": [42, 2, -1]
    }

    result = element_in_lists(
        dictionary=my_dict,
        element=42
    )
    print(result)

Acknowledgment

This post is based on the excellent book: Robust Python: Write Clean and Maintainable Code, Patrick Viafore, O’REILLY.

SimleNamespace is a Python utility that allow you to create simple Python objects.

With it, you can turn a dictionary into an object where the keys are accessible using the “dot notation”.

python> from types import SimpleNamespace
python> person_dict = {
    "firstname": "John",
    "lastname": "Doe",
    "age": 29
}
python> person = SimpleNamespace(**person_dict)
python> person.firstname
'John'

python> person_dict.firstname
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'firstname'

You can see it as a simple data structure. You can use it instead of a Class when you need an object that do need to implement any kind of behavior.

SimpleNamespace vs. Dataclass vs. namedtuple

Here is a recap of the main strength for each types:

Use-Case: Mocking & Prototyping

You have an API endpoint you want to query.

This endpoint returns JSON data.

You have a python method taking this JSON data as an input, transforms it and returns the transformed JSON as output.

Note: this is typically the case when you have e.g. a Google Cloud Function, listening to the outbound webhook where data are delivered by one external service and passing on the data to another external service after having performed some transformation over it. Usually, extracting data from the received HTTP payload and preparing a JSON card to be send to another endpoint via an HTTP POST request. Could be alerts you want to send to Microsoft Teams.

Here is the snippet doing the aforementioned actions:

from typing import Any
import requests
from pprint import pprint
from types import SimpleNamespace


def request_random_user() -> requests.Response:
    """
    Method fetching one random user from the API.
    """

    response = requests.get(
        url="https://random-data-api.com/api/v2/users?size=1",
        timeout=300
    )
    response.raise_for_status()
    return response


def reduce_json_content(
    response: requests.Response
) -> dict[str, Any]:
    """
    Method returning a dict with selected
    subfields from the json content in the
    HTTP response.
    """

    content = response.json()
    selected_fields = ("first_name", "last_name", "username")
    return {field: content[field] for field in selected_fields}


def main() -> None:

    response = request_random_user()
    content = reduce_json_content(response)
    pprint(content)


if __name__ == "__main__":
    main()

Let’s give it a shot:

zsh> poetry run python main.py
{
    'first_name': 'Eula',
    'last_name': 'Morar',
    'username': 'eula.morar'
}

Now, let’s say you want to test the reduce_json_content method without calling the API.

You can simply pass on to the method a mock json content.

SimpleNamespace is the perfect too for that.

Let’s edit the above snippet, replacing the last lines by the following:

if __name__ == "__main__":

    response_mock = SimpleNamespace()
    response_mock.json = lambda: {
        "first_name": "John",
        "last_name": "Doe",
        "username": "johndoe",
    }

    content = reduce_json_content(response_mock)
    pprint(content)

Note: response is of type requests.Response. It possess a .json() method. Therefore, we need to mock this method using a lambda function. This lambda function does not require any parameters.

zsh> poetry run python main.py
{'first_name': 'John', 'last_name': 'Doe', 'username': 'johndoe'}

Voilà! You can then continue the development without calling the API every time.

This article explains why fstring interpolation is bad in logging functions and why you should rather use:

logging.info("Your string %s", my_var)

instead of:

logging.info(f"Your string {my_var}")

or:

logging.info("Your string %s" % my_var)

Context

In python you can have code using logging functions for better observability. For instance:

"""
Demonstration module for logging fstring
interpolation pylint error.
"""

import logging

logging.basicConfig(level="INFO")
logger = logging.getLogger(__name__)


def main() -> None:
    """
    Main method. Nothing fancy about it.
    """
    try:
        print(8 / 0)
    except ZeroDivisionError as exc:
        logger.info(f"The division failed: {exc}")


if __name__ == "__main__":
    main()

Running the above code will raise the following error:

zsh> poetry run python main.py
INFO:__main__:The division failed: division by zero

So far so good.

Error: Use lazy % formatting in logging functions

The next step is to push our code into production, hence, applying black, mypy and pylint formatting over your code.

Note: you can encompass all of the above within a similar Makefile, see hereafter.

black:
    poetry run black main.py

mypy:
    poetry run mypy main.py

pylint:
    poetry run pylint main.py

checks: black mypy pylint

zsh> make checks
main.py:19:8: W1203: Use lazy % formatting in logging functions (logging-fstring-interpolation)

------------------------------------------------------------------
Your code has been rated at 9.00/10

So what’s happening here?

You can see that pylint complains about us using a Python fstring in the logging function.

The error message hints us to use the lazy % formatting instead.

Because we want to comply with its instruction, we edit our code, changing the line by the following:

logger.info("The division failed: %s" % exc)

Confident, we run our code again, switching for a lazy % formatting and expecting a 10/10 rating:

zsh> make checks
main.py:19:8: W1201: Use lazy % formatting in logging functions (logging-not-lazy)
main.py:19:20: C0209: Formatting a regular string which could be an f-string (consider-using-f-string)

------------------------------------------------------------------
Your code has been rated at 8.00/10 (previous run: 9.00/10, -1.00)

Surprise! Pylint seems not to like either fstring or lazy % formatting…

It even comes up with a new error into place!

We could obviously silent the error, adding the following inline comment:

logger.info(  # pylint: disable=logging-fstring-interpolation
    f"The division failed: {exc}"
)

Which would gives us a nice and neat 10/10 rating:

zsh> make checks
-------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 9.00/10, +1.00)

However, better than hiding the dirt under the rug, it is better to understand why pylint is raising this error.

So, why fstring interpolation is bad in logging functions?

Hidden Motivation: Performances!

The problem is not so much about us using fstrings in place of lazy % formatting.

The problem lies about performances.

Jumping back into our code example, what fixes it – without us having to cheat around by muting the error – is a lazy % formatting with a comma:

logger.info("The division failed: %s", exc)

The Devil is in the details!

The motivation behind the warning is around performance.

With this final version of our code, if a log statement is not emitted, then the interpolation cost is saved.

The solution was to shift from:

logger.info("The division failed: %s" % exc)

to:

logger.info("The division failed: %s", exc)

so that the string will only be interpolated if the message is actually emitted.

Working with datetime and timezones in python can generate lot of confusions and errors.

It is therefore a best practice to always specify the timezone you are working with (e.g. UTC, CEST…) explicitly as you can be sure the bread will always fall on the marmalade side.

For instance, the system time in your CICD pipeline runners might be different from your local system time, causing your unittests to fail on your runner but to succeed in local.

As a rule of thumb, it is always best to compare Coordinated Universal Time (UTC) with UTC.

Let’s see how to do this in practice with a code we will also unittest. Hence, we will catch up all the necessary lingo and discover the necessary tools of the trade.

Usage

Let’s say you have created a custom type Message:

from dataclasses import dataclass

@dataclass
class Message:
    content: str
    created_at: str

You want to convert the Message data class into a JSON-like dictionary – e.g. for better human readability:

from pprint import pprint
from dateutil.parser import parse
from datetime import datetime, timezone
from typing import Dict


def convert_message_to_dict(
    message: Message
) -> Dict[str, str]:
    """
    Method returning a dictionary populated with the
    Message's attributes.
    """
    return {
        "content": message.content,
        "created_at": parse(message.created_at)
        .astimezone(timezone.utc).isoformat(),
        "converted_at": datetime.now(timezone.utc)
        .isoformat(),
    }


if __name__ == "__main__":

    message = Message(
        content="foo",
        created_at="2024-02-14T01:32:00Z"
    )

    converted_message = convert_message_to_dict(message)

    pprint(converted_message)

Notes:

1. I want my dictionary to only contain str so I need to convert datetimes to isoformat – which is the ISO_8601 standard.

2. I explicitly want to define the timezone as UTC, using timezone.utc, to avoid any confusion.

3. In isoformat you can explicitly define the timezone information at the end of the “YYYY-MM-DDTHH:mm:ss” string. To declare an UTC, either “Z” or the offset “+00:00”, “+0000” or “+00” can be used as postfix interchangeably.

4. “Z” stands for “Zulu”, the military designation for UTC. It is the same as the UTC time. Read more.

5. As for the “+” or “-” prefixing the offset, you add “+” moving East from the London meridian and subtract “-” moving West. Read more.

6. pprint stands for pretty print and displays the results in a more human readable manner.

Here we go, running the script at “2024-02-13T13:52:09.265050+00:0” gives me the following output:

{
    "content": "foo",
    "created_at": "2024-02-14T01:32:00+00:00",
    "converted_at": "2024-02-13T13:52:09.265050+00:00"
}

Let’s unittest the above code.

Unittest datetime

In order to unittest the aforementioned code, we gonna use freezegun.

freeze_time is a nice python module that allow you to mock datetime.

import pytest
from freezegun import freeze_time
from datetime import datetime, timezone


@pytest.fixture()
def message() -> Message:
    """
    Fixture returning a minimal Message
    with 'created_at' expressed in UTC.
    """
    return Message(
        content="foo",
        created_at="2023-09-01T07:00:00Z"
    )

@freeze_time("2024-02-24T06:00:00", tz_offset=0)
def test_datetime():
    dt = datetime.now().isoformat()
    assert dt == "2024-02-24T06:00:00"


@freeze_time("2024-02-24T06:00:00", tz_offset=0)
def test_feed_to_dict(message):

    outcome = message_to_dict(message)

    expected_outcome = {
        "content": "foo",
        "created_at": "2023-09-01T07:00:00+00:00",
        "converted_at": "2024-02-24T06:00:00"
    }

    assert outcome == expected_outcome

Notes:

• the trick is to compare UTC dates to UTC dates;

• thanks to the fixture, the above unittest code respects the Arrange-Act-Assert pattern.