Kafka The Definitive Guide Notes - Part 3

reqs - standard header - req type, version, correl id, client id
port - broker listens - acceptor thread - creates conn - processor thread -handles: req in req Q, Q to resp to clients
should be sent to proper partn which is leader
where to send - req type - metadata - list of topics, pns, replicas, leader replica
all brokers - metadata cache
--
PR
validations - priviliges to send, acks -all- enough in sync replicas?
req buffer  - purgatory
leader - responds after replicas
msgs - local disk, linux - FS cache
--
FR
offset, P, T, limit - min, max
zerocopy - dir file to n/w
consumer can see only if replicated everywhere
--
storage unit - partn replica
partn - cant be split btn multi brokers or disks
log.dirs - partn storage
--
T create - partn alloc
round robin, inc offset from leader, alternating racks
add new partn to least amount of partn dir
--
partn - split to segs-retention size/duration
current/active seg never deleted
--
format - seg: file - msg, offset, k, v, size, checksum, timestamp etc
dumlogseg tool
--
index: partn, offset:seg mapping, position
--
compaction--store only latest value
--
reliable data delivery-order within partn, commit - all in sync replicas, commited wont be lost as long as at least one in sync is alive, consumer can read only committed
follower - active z sessions, fetch latest from leader
--
replcn factor-3
unclean leader elec
min in syncs
--
commit configs, freq
offset reset - earliest/latest
auto, explicit 
--
exactly once - UK
--
pipeline - kafka -endpoint/mid
timeline - bulk/ stream
reliability, thruput, in mem, coupling, 
--