Kafka The Definitive Guide Notes - Part 2

Zookeeper - metadata about cluster(brokers, topic)
kraft - replaces Z
Z cluster - ensemble-odd num of servers - 5
dont go beyond 7
common config
initLimit, syncLimit - followers-leader
leader election
controller
---
broker.id - unique in cluster
port - 9092 
z.conn - broker meta
log.dirs
partitions ~ brokers
h/w, rack
---
producer record - topic, value - optional: partition, key
serializer
partitioner
topic
partition
batch
broker
--
prodr - bootstrap.servers, key.serial, val.serial
fire & forget, sync send-p.send(pr).get(), async send - p.send(pr, new CBF())
--
acks - 0,1, all - jhow many replicas must receive b4 prodr can consider it successful
retries
batch size
client.id
--
custom serial, avro
--
PR - t, k, v
msg - (k, v)--null k : round robin, same key -same partition
num of partitions consistent - mapping consistent
custom partitiong
--
consumer group - receives all msgs
consumers - part of them from partitions
rebalance - ownership
group coord - heartbeats to broker
polls n commits consumed recs
JoinGroup req to GC
1st cons to join - Group Leader - list of cons from GC- assigns subset of partns to cons
--
cons - boot, deserial, group id
c.subscrbe(t)
loop - c.poll
1 cons/thread
--
update current position in partn - commit
reads latest committed offset
auto commit, sync, async
--
rebalance code hooks - clean up
--
broker start - reg id Z - ephemereal mode
conn lost - auto node removed, watch list of brokers notified, id still exists in other DS
--
controller - 1st ephe node - broker fnct+ partn leader election-Z watch
leader - serves reqs, follower - replicates
new broker - controller checks for replicas - notifies
--
replication - distri, partnd, replicated commit log -availability,durability
partn - scale, parallel, order
out of sync followers cant become leaders
preferred leader-t creation with balanced load
--