Kafka

If you have any doubts in the below, contact us by dropping a mail to the Kung Fu Panda. We will get back to you very soon.

Basics

Kafka Use Cases

Entities in Kafka

Note

How producers produce data

Kafka Flow

Kafka Producer Requirements

Creating a producer needs the following.

Configuring producers

Note

Messages can be sent to the broker in the following ways

Retention

Kafka Broker

Consumers

Consumer groups.

how are partitions assigned to consumers in a group.

Creating a Kafka Consumer

How consumer works

Consumer additional configuration params

Consumer commits and offsets

Automatic Commits

Commit current offset

Rebalance Listeners

Replication

each topic can have multiple replicas.

Note

Kafka has a binary protocol (over TCP) that specifies the format of the requests and how brokers respond to them— both when the request is processed successfully or when the broker encounters errors while processing the request.

How kafka works

If a request is made to a different broker than who is the leader of the partition, it returns with the error "Not the leader for this partition". Kafka clients are responsible for directing the data to appropriate brokers.

Types of requests from kafka clients to brokers

metadata requests

how does a broker handle produce requests.

How does a broker handle get requests

Note

Partition Allocation

Data Retention

Segment

Message

How does kafka quickly get the data for an offset within a topic partition

Compaction

Note

Imp Config params

Imp things to do at producer level

Reliable Consumers

should commit offsets appropriately.

Imp consumer config

Building data pipelines

MirrorMaker

Cross Cluster Replicator

Use cases of mirrormaker

Issues in cross DC communication

Failover to other DC

Mirrormaker

consists of consumers which are part of the same consumer group and read data from the topics which are replicated.

how does mirrormaker work

things to monitor in mirrormaker.

Administering Kafka

Commands

Stream Processing

Property of Data Streams

Existing programming paradigms

Stream Processing Concepts

Stream Table duality

Joins in Streaming

Streaming Use Cases