Apache Kafka

Apache Kafka is a distributed event streaming platform used to publish, store, and process continuous streams of events at scale. Unlike a traditional message broker, Kafka retains events in an ordered, append-only log for a configurable retention period, so multiple independent consumers can read - and re-read - the same stream of events at their own pace without competing for the same message.

This log-based model makes Kafka well suited to event sourcing, real-time analytics pipelines, and connecting many producers and consumers around a shared, replayable source of truth, with Kafka Streams and ksqlDB available for processing data directly within the pipeline. Kafka is designed to run as a distributed cluster and is built to sustain very high write and read throughput.

Kafka is a fit for teams building event-driven architectures or real-time data pipelines that need durability and replay, complementing rather than duplicating RabbitMQ (also in this catalog), whose queue-and-routing model is better suited to traditional background-job and task-queue workloads.