現(xiàn)象：

搭建了一個(gè) kafka 服務(wù), 使用 kafka-python 包正常生產(chǎn)數(shù)據(jù), 但是 kafka 過(guò)五分鐘就把我的 topic 刪除掉. 但是配置 log 的已經(jīng)配置了, 我認(rèn)為 kafka 不應(yīng)該刪除我的 topic 歷史數(shù)據(jù).

關(guān)于 log 清理的配置文件:

############################# Log Flush Policy #############################

log.flush.interval.messages=10000

############################# Log Retention Policy #############################

log.retention.hours=168

log.retention.bytes=-1

log.segment.bytes=1073741824

打印的 log:

[2019-11-26 10:19:11,478] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Found deletable segments with base offsets [0] due to retention time 604800000ms breach (kafka.log.Log)

[2019-11-26 10:19:11,485] INFO [ProducerStateManager partition=falcon_source_created-0] Writing producer snapshot at offset 16 (kafka.log.ProducerStateManager)

[2019-11-26 10:19:11,487] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Rolled new log segment at offset 16 in 8 ms. (kafka.log.Log)

[2019-11-26 10:19:11,488] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Scheduling log segment [baseOffset 0, size 5861] for deletion. (kafka.log.Log)

[2019-11-26 10:19:11,491] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Incrementing log start offset to 16 (kafka.log.Log)

[2019-11-26 10:19:11,497] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Found deletable segments with base offsets [0] due to retention time 604800000ms breach (kafka.log.Log)

[2019-11-26 10:19:11,499] INFO [ProducerStateManager partition=event_mapped-0] Writing producer snapshot at offset 16 (kafka.log.ProducerStateManager)

[2019-11-26 10:19:11,500] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Rolled new log segment at offset 16 in 2 ms. (kafka.log.Log)

[2019-11-26 10:19:11,500] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Scheduling log segment [baseOffset 0, size 6452] for deletion. (kafka.log.Log)

[2019-11-26 10:19:11,500] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Incrementing log start offset to 16 (kafka.log.Log)

[2019-11-26 10:20:11,489] INFO [Log partition=falcon_source_created-0, dir=/kafka/kafka-logs-0d2f113835a6] Deleting segment 0 (kafka.log.Log)

[2019-11-26 10:20:11,493] INFO Deleted log /kafka/kafka-logs-0d2f113835a6/falcon_source_created-0/00000000000000000000.log.deleted. (kafka.log.LogSegment)

[2019-11-26 10:20:11,500] INFO [Log partition=event_mapped-0, dir=/kafka/kafka-logs-0d2f113835a6] Deleting segment 0 (kafka.log.Log)

[2019-11-26 10:20:11,500] INFO Deleted log /kafka/kafka-logs-0d2f113835a6/event_mapped-0/00000000000000000000.log.deleted. (kafka.log.LogSegment)

[2019-11-26 10:20:11,503] INFO Deleted offset index /kafka/kafka-logs-0d2f113835a6/falcon_source_created-0/00000000000000000000.index.deleted. (kafka.log.LogSegment)

[2019-11-26 10:20:11,503] INFO Deleted offset index /kafka/kafka-logs-0d2f113835a6/event_mapped-0/00000000000000000000.index.deleted. (kafka.log.LogSegment)

[2019-11-26 10:20:11,504] INFO Deleted time index /kafka/kafka-logs-0d2f113835a6/falcon_source_created-0/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)

[2019-11-26 10:20:11,504] INFO Deleted time index /kafka/kafka-logs-0d2f113835a6/event_mapped-0/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)

查看一些資料發(fā)現(xiàn)應(yīng)該跟他的偏移量的刪除策略有關(guān)~ 但是我設(shè)置了 LOG_CLEANER_ENABLE 關(guān)閉刪除策略, kafka 依然會(huì)刪除我的 topic

總結(jié)

以上是生活随笔為你收集整理的清理offset_关于 kafka 日志清理策略的问题的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。