FastAPI Event Driven Development With Kafka, Zookeeper and Docker Compose: A Step-by-Step Guide Part -1

Setting up Kafka and Docker compose

Full Code on github here

Introduction

In part 1, we will set up a single-node Kafka and Zookeeper environment using Docker Compose. We will then produce and consume test messages using the Kafka console producer and consumer. In part 2, we will add a FastAPI endpoint that handles requests and produces messages.

Kafka

Kafka is a distributed data streaming platform that allows you to publish, store, and process streams of records (messages) in a fault-tolerant and scalable manner. In simple terms, Kafka is like a messaging system that enables applications to send and receive data in real-time.

Kafka has three main components:

1. Producers: Applications that send (publish) messages (also called records) to Kafka.
2. Brokers: The Kafka servers that store and manage the messages. They work together to form a Kafka cluster, ensuring fault tolerance and scalability.