Overview#
So what is Kafka? Kafka is an immutable & distributed log of events.
Cluster#
Kafka’s functionality is orchestrated via a horizontally scalable cluster of nodes.
Brokers#
Each node in the Kafka cluster is called a broker. Kafka broker, server, and node are used synonymously - however I’ll be using the term node as it’s used as a common term for other data-systems which are distributed.
Topics#
A topic is a collection of partitions. Producers write data to partitions which are append-only logs of data. Consumers read from these partitions composing a topic.
Partitions#
A partition is what it sounds like - the data written to a topic partitioned into individual sets.
Each partition is an append-only log of events.
An example of events a through f written to Topic 1 which has two partitions.
Note that events can be out of order - if events do need to be in order, a partition key can be used to send events to an individual partition to allow consumers to read them in order, sequentially (FIFO: first in, first out).
Partition Key:
An example of two different types of events events X and Y with a through c events written to each topic which has two partitions.
Replication Factor#
These partitions can live on multiple nodes for a topic. How we accomplish this is by setting a replication factor.
For the example below, this topic’s partitions are distributed across three nodes with a replication factor of 2.
Producers#
As mentioned earlier, producers are what write to the append only partitions for a topic.
Producers interact with the topic’s partitions via the Producer API via any of the Kafka Clients.
An example of events a through f written by two separate producers (pr1 & pr2) to Topic 1 which has three partitions.
Consumers#
As mentioned earlier, consumers are what read from the append only partitions for a topic.
Consumers are fully independent of each other. Consumers interact with the topic’s partitions via the Consumer API.
Consumer Groups#
Rebalancing#
What happens if a consumer in a consumer group runs into an error and stops consuming?
If this occurs - Kafka performs a rebalancing of the read partitions across the consumer group.
For example, let’s say we have 5 partitions and 3 consumers in the consumer group:
Consumer C dies, and leaves the consumer group
Offsets#
Offsets are the index of where a consumers has read up to in the log of events.
Overview Example#
- n:3 Nodes
- t:2 Topics
- rf:2 Replication Factor
- p:4 Partitions per topic
Producers Perspective#
via any of the Kafka Clients.
References#
Clients
API’s
Source Code