Explaining Kafka Load Balancing and usage of key based hashing in a Pictorial way

Hi, Lets try to find out how messages generated from a Producer is distributed across the partitions of a particular Topic inside a Kafka Cluster.

We shall also see how the message can be controlled and streamlined to a specific Partition with the help of a ‘Key’.

A Topic is configured with several Partitions, generally the number of nodes in the Kafka cluster decides the number of Partitions in a Topic.

The purpose of having different Partitions is to share the load and improve the performance of the Kafka cluster.

Let’s now go through the images below and understand how messages are handled and shared across the Partitions.

1st — Create a Topic

2nd- Zooming into the topic

3rd- Producers and Partitions

4th — Send message ‘A’

5th — Send message ‘B’

6th- Send message ‘D’ from 2nd Producer

7th- Send message ‘K’ from Producer-2

8th — Send message ‘S’ from Producer-2

9th — Message distribution across Partitions

10th — Send message ‘Z’ with key ‘IND’

11th — message ‘Z’ with key ‘IND’

12th — send message ‘Z’ with key ‘USA’

13th — message ‘Z’ with key ‘USA’

14th — Send message ‘12’ with key ‘IND’

15th — message ‘12’ with key ‘IND’

16th — message ‘12’ with key ‘IND’

Now comes the Consumers. Lets see how consumer/group(s) consume the messages from the kafka message bus.

17th — Adding Consumers to the Cluster

18th-Consumer-P pulls message from Topic

19th-Consumer-Q pulls message from Topic

20th-Consumer-Q pulls next message from Topic

21st-Consumer-R of Consumer-Group-2 pulls message from Topic

22nd-Consumer-P of Consumer-Group-1 pulls message from Topic

23rd-Consumer-Q of Consumer-Group-1 pulls message from Topic

24th-Consumer-R of Consumer-Group-2 pulls message from Topic

25th-Consumer-P of Consumer-Group-1 pulls message from Topic

26th-Consumer-R of Consumer-Group-2 pulls message from Topic

27th-Consumer-R of Consumer-Group-2 pulls next message from Topic

28th-Consumer-R of Consumer-Group-2 pulls another message from Topic

29th-Consumer-P of Consumer-Group-1 pulls next message from Topic

30th-Consumer-R of Consumer-Group-2 pulls message from Topic

31st-Consumer-R of Consumer-Group-2 pulls next message from Topic

Well, this is how a message is sent across partitions of a Topic and a Consumer from a Consumer-Group consumes the messages.

Thank you for the read. If you like it, then please click the Clap button :)