TIL - asyncio.Queue Is the Backbone of Async Producer-Consumer Pipelines

asyncio.Queue is a coroutine-safe FIFO that decouples producers from consumers inside a single event loop — no threads, no locks, no race conditions.

import asyncio

async def producer(queue: asyncio.Queue, items: list):
    for item in items:
        await asyncio.sleep(0.05)   # simulate I/O
        await queue.put(item)
        print(f"Produced: {item}")
    await queue.put(None)           # sentinel to signal done

async def consumer(queue: asyncio.Queue):
    while True:
        item = await queue.get()
        if item is None:
            break
        print(f"Consumed: {item}")
        queue.task_done()

async def main():
    q = asyncio.Queue(maxsize=3)    # back-pressure: blocks producer if full
    await asyncio.gather(
        producer(q, list(range(6))),
        consumer(q),
    )

asyncio.run(main())

maxsize=N applies natural back-pressure — the producer awaits when the queue is full, preventing unbounded memory growth. queue.join() blocks until all task_done() calls have been made, useful for batch processing.