Apache Kafka Buzz Words

Introduction

Hello everyone, in our academics we have read a lot about distributed system taking over the future.

Apache kafka is one such distributed system. Few buzz words associated with apache kafka are topics, records, partitions, replications, cluster, broker, logs and not to metion the most widely used terms producer and consumer.

A kafka cluster consist of multiple brokers. A kafka topic is split into multiple partitions. By default the partition is num.partitions=1 defined in $KAFKA_ROOT/config/server.properties. Each partition is ordered and each message in each partition get an incremental id called offset. This offset must be remembered for consumer to work properly.

If kafka is properly setup on your system, then running kafka-console-consumer.sh --help should give few argument to be passed in, notice --offset argument earliest/latest which are somehow tied to offset and reading. This is not a complete insight of commands and its usage i would link some useful blogs below.

Say If we have a kafka cluster of 3 brokers and created a topic named order with 3 partitions then the sytem would create topic[0, 1, 2] on all the brokers[0, 1, 2] with each partition present on each server. Means brokers would have

• Broker 9092 => topic-0[order]{partition(0)}
• Broker 9093 => topic-1[order]{partition(1)}
• Broker 9094 => topic-2[order]{partition(2)}

Also there is concept of leader when dealing with replication factor. Say you have decided to add replication factor then, topic-0[order] can be present in either broker 9093 or 9094 based on replication factor value. Buy default a master leader is selected to which all slaves sync this is called in-sync-replicas(ISR).

If one of the broker goes down then new leader is selected. This all need not required to be done by developer but it is internally taken care by Apache Kafka :smile:

By default most of us while studying create only 1 broker by launching

kafka-server-start.sh $KAFKA_ROOT/config/server.properties from terminal

To boot new server we just need to create server-{number}.properties and pass it to our program kafka-server-start.

NOTE

Change broker.id in $KAFKA_ROOT/config/server.properties.

Default it starts with 0, you can give it as incremental like 1, 2, 3... and so on for servers.

Reference

Official Apache Kafka Documentation