[Easy to understand] What is Kafka Consumer, Consumer Group, Topic & Partition?

Kafka:

Kafka is a distributed streaming platform that allows you to publish and subscribe to streams of records, similar to a message queue or enterprise messaging system.

Consume:

Consuming in Kafka means reading data from a topic. Consumers read messages from topics and process them.

Consumer Group:

A consumer group is a set of consumers that cooperate to consume data from Kafka brokers. Each message within a topic is delivered to one consumer instance within each subscribing consumer group. This allows you to scale processing by adding more consumers to a group.

Topic:

A topic is a category/feed name to which records are sent by producers. Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it.

Partition:

Topics in Kafka are divided into partitions. Each partition is an ordered, immutable sequence of records that are continually appended to. Each message within a partition is assigned a unique offset. Partitions allow you to parallelize a topic by splitting the data across multiple brokers.

Example Story:

Imagine you have a bookstore where books are continuously being published. Each book is like a message in Kafka. The bookstore has different sections (topics) like Fiction, Non-Fiction, and Sci-Fi. Each section has multiple shelves (partitions) where books are placed. Now, readers (consumers) are part of different book clubs (consumer groups) where each club reads books from a specific section. By having multiple shelves (partitions) in each section, the bookstore can serve more readers efficiently.