什么是 MirrorMaker？

MirrorMaker是Kafka附帶的一個用于在Kafka集群之間制作鏡像數(shù)據(jù)的工具。該工具主要動作就是從源集群中消費并生產(chǎn)到目標群集。

一個集群可以啟動多個MirrorMaker配置到多個集群

?

運行 MirrorMaker方法

kafka-mirror-maker.sh --consumer.config ./consumer.properties --producer.config ./producer.properties --num.streams 8 --whitelist ".*"

實際操作一遍

consumer.properties文件

root@DESKTOP-I0EG1MJ:~/kafka-client/mirror-maker# cat consumer.properties bootstrap.servers=192.168.3.194:9092 group.id=mirrormaker auto.offset.reset=earliest

producer.properties文件

root@DESKTOP-I0EG1MJ:~/kafka-client/mirror-maker# cat producer.properties bootstrap.servers=192.168.2.123:9092

?運行mirror-maker

root@DESKTOP-I0EG1MJ:~/kafka-client/mirror-maker# cat run-mirrormaker.sh kafka-mirror-maker.sh --consumer.config ./consumer.properties --producer.config ./producer.properties --offset.commit.interval.ms 5000 --num.streams 2 --whitelist "test0428"

參數(shù)說明：

--consumer.config 消費者的配置文件（要消費的集群）

--producer.config 指定生產(chǎn)配置文件（要發(fā)送到的目標集群）

--whitelist 要同步的topic白名單，可以匹配正則，也可以指定具體topic

--offset.commit.interval.ms 消費端提交offset時間間隔

--num.streams ?MirrorMaker 要創(chuàng)建多少個 KafkaConsumer 實例

更多可以使用 --help查看

root@DESKTOP-I0EG1MJ:~/kafka-client/mirror-maker# kafka-mirror-maker.sh --help This tool helps to continuously copy data between two Kafka clusters. Option Description ------ ----------- --abort.on.send.failure <String: Stop Configure the mirror maker to exit onthe entire mirror maker when a send a failed send. (default: true)failure occurs> --consumer.config <String: config file> Embedded consumer config for consumingfrom the source cluster. --consumer.rebalance.listener <String: The consumer rebalance listener to useA custom rebalance listener of type for mirror maker consumer.ConsumerRebalanceListener> --help Print usage information. --message.handler <String: A custom Message handler which will processmessage handler of type every record in-between consumer andMirrorMakerMessageHandler> producer. --message.handler.args <String: Arguments used by custom messageArguments passed to message handler handler for mirror maker.constructor.> --new.consumer DEPRECATED Use new consumer in mirrormaker (this is the default so thisoption will be removed in a futureversion). --num.streams <Integer: Number of Number of consumption streams.threads> (default: 1) --offset.commit.interval.ms <Integer: Offset commit interval in ms.offset commit interval in (default: 60000)millisecond> --producer.config <String: config file> Embedded producer config. --rebalance.listener.args <String: Arguments used by custom rebalanceArguments passed to custom rebalance listener for mirror maker consumer.listener constructor as a string.> --version Display Kafka version. --whitelist <String: Java regex Whitelist of topics to mirror.(String)>

?

運行后會提示這么一句話

WARNING: The default partition assignment strategy of the mirror maker will change from 'range' to 'roundrobin' in an upcoming release (so that better load balancing can be achieved). If you prefer to make this switch in advance of that release add the following to the corresponding config: 'partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor'

?主要是說是MirrorMaker使用的消費策略是 Range ，以后可能改成 “輪訓(xùn)策略” ，我們可以手動指定“輪詢策略”

在Consuemr.properties中設(shè)置即可

partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor

?

?

此時我們已經(jīng)運行成功，直接去查看是否可以消費成功即可。

注意：同步前最后將目標集群的Topic創(chuàng)建好，否則會使用Broker默認配置

總結(jié)

以上是生活随笔為你收集整理的Kafka 集群数据备份 MirrorMaker 详解的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。