Concurrency

How to use concurrency

In Python, there are multiple ways of handling concurrency. Without taking into account specific technical differences between them, there are two main approaches.

Using threads

from concurrent.futures import ThreadPoolExecutor

def print_letter(letter: str) -> None:
  print(f"Letter: {letter}")

letters = ['A', 'B', 'C']

with ThreadPoolExecutor() as executor:
  executor.map(print_letter, letters)

Using asyncio

import asyncio

async def a_print_letter(letter: str) -> None:
  print(f"Letter: {letter}")

letters = ['A', 'B', 'C']

async def main():
  await asyncio.gather(\*(a_print_letter(letter) for letter in letters))

asyncio.run(main())

Essentially, ThreadPoolExecutor is used in synchronous code, while asyncio is used in asynchronous code.

The Concurrent construct

To harmonize the way we write (and most importantly, read) concurrent code, we have the Concurrent construct. It is a wrapper around ThreadPoolExecutor and asyncio, abstracting away the differences between them.

It exposes the functions map and a_map, which offer a homogeneous interface for dealing with synchronous and asynchronous code respectively.

from modules.concurrency.core import Concurrent

def print_letter(letter: str) -> None:
  print(f"Letter: {letter}")

async def a_print_letter(letter: str) -> None:
  print(f"Letter: {letter}")

letters = ['A', 'B', 'C']

Concurrent.map(print_letter, letters)
await Concurrent.a_map(a_print_letter, letters)

Internal concurrent functions

It is very common to have a function that is only used within a Concurrent construct. These functions should always be declared as top level private functions of the module. For example:

def _print_letter(letter: str) -> None:
  print(f"Letter: {letter}")

letters = ['A', 'B', 'C']

def print_letters(letters: list[str]) -> None:
  Concurrent.map(_print_letter, letters)

Note that these functions follow the private function naming convention.