Kafka

Topic Management

Create a topic

kafka-topics --create --bootstrap-server localhost:9092 --topic blah --replication-factor 1 --partitions 3

Describe a topic

kafka-topics --bootstrap-server localhost:9092 --topic blah --describe

Show dynamic config on a topic

kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name blah --describe

Alter a dynamic config on a topic

kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name blah --alter  --add-config min.insync.replicas=2

Deleting a dynamic config on a topic

kafka-configs --bootstrap-server localhost:9092 --entity-type topics --entity-name blah --alter  --delete-config min.insync.replicas

Any consumer that is reading from the tail of a log (most current data) will still see all the messages sent to the topic.
Ordering of messages is kept, log compaction only removes some messages, but does not reorder them
The offset of a message is immutable (it never changes). Offsets are just skipped if a message is missing
Deleted records can still be seen by consumers for a period of delete.retention.ms (default is 24 hours)

segment.ms (default 7 days): Max amount of time to wait to close active segment
segment.bytes (default 1G): Max size of segment
min.compaction.lag (default 0): how long to wait before a message can be compacted
delete.retention.ms (default 24 hours): wait before deleting data marked for compaction
min.cleanable.dirty.ratio (default 0.5): higher ⇒ less, more efficient cleaning power. Lower ⇒ opposite

Checking Kakfa Connect status:

curl connect:8083

should show version number

Checking Kafka from Python services, use kafkacat

kafkacat -b kafka:29092 -L