漫游kafka实战篇之搭建Kafka开发环境
生活随笔
收集整理的這篇文章主要介紹了
漫游kafka实战篇之搭建Kafka开发环境
小編覺(jué)得挺不錯(cuò)的,現(xiàn)在分享給大家,幫大家做個(gè)參考.
轉(zhuǎn)載注明出處：http://blog.csdn.net/honglei915/article/details/37563647
Kafka視頻教程同步首發(fā)，歡迎觀看！
上篇文章中我們搭建了kafka的服務(wù)器，并可以使用Kafka的命令行工具創(chuàng)建topic，發(fā)送和接收消息。下面我們來(lái)搭建kafka的開(kāi)發(fā)環(huán)境。
添加依賴
搭建開(kāi)發(fā)環(huán)境需要引入kafka的jar包，一種方式是將Kafka安裝包中l(wèi)ib下的jar包加入到項(xiàng)目的classpath中，這種比較簡(jiǎn)單了。不過(guò)我們使用另一種更加流行的方式：使用maven管理jar包依賴。 創(chuàng)建好maven項(xiàng)目后，在pom.xml中添加以下依賴：
[html]?view plaincopy <dependency>?? ????????????<groupId>org.apache.kafka</groupId>?? ????????????<artifactId>kafka_2.10</artifactId>?? ????????????<version>0.8.2.2</version>?? ????????</dependency>??
添加依賴后如果有兩個(gè)jar包的依賴找不到。點(diǎn)擊這里下載這兩個(gè)jar包，解壓后你有兩種選擇，第一種是使用mvn的install命令將jar包安裝到本地倉(cāng)庫(kù)，另一種是直接將解壓后的文件夾拷貝到mvn本地倉(cāng)庫(kù)的com文件夾下，比如我的本地倉(cāng)庫(kù)是d:\mvn,完成后我的目錄結(jié)構(gòu)是這樣的：
配置程序
更新更全的API編程實(shí)例點(diǎn)這里：http://blog.csdn.net/honglei915/article/details/37697655
首先是一個(gè)充當(dāng)配置文件作用的接口,配置了Kafka的各種連接參數(shù)： [java]?view plaincopy package?com.sohu.kafkademon;?? ?? public?interface?KafkaProperties?? {?? ????final?static?String?zkConnect?=?"10.22.10.139:2181";?? ????final?static?String?groupId?=?"group1";?? ????final?static?String?topic?=?"topic1";?? ????final?static?String?kafkaServerURL?=?"10.22.10.139";?? ????final?static?int?kafkaServerPort?=?9092;?? ????final?static?int?kafkaProducerBufferSize?=?64?*?1024;?? ????final?static?int?connectionTimeOut?=?20000;?? ????final?static?int?reconnectInterval?=?10000;?? ????final?static?String?topic2?=?"topic2";?? ????final?static?String?topic3?=?"topic3";?? ????final?static?String?clientId?=?"SimpleConsumerDemoClient";?? }??
producer
[java]?view plaincopy package?com.sohu.kafkademon;?? ?? import?java.util.Properties;?? ?? import?kafka.producer.KeyedMessage;?? import?kafka.producer.ProducerConfig;?? ?? /**? ?*?@author?leicui?bourne_cui@163.com? ?*/?? public?class?KafkaProducer?extends?Thread?? {?? ????private?final?kafka.javaapi.producer.Producer<Integer,?String>?producer;?? ????private?final?String?topic;?? ????private?final?Properties?props?=?new?Properties();?? ?? ????public?KafkaProducer(String?topic)?? ????{?? ????????props.put("serializer.class",?"kafka.serializer.StringEncoder");?? ????????props.put("metadata.broker.list",?"10.22.10.139:9092");?? ????????producer?=?new?kafka.javaapi.producer.Producer<Integer,?String>(new?ProducerConfig(props));?? ????????this.topic?=?topic;?? ????}?? ?? ????@Override?? ????public?void?run()?{?? ????????int?messageNo?=?1;?? ????????while?(true)?? ????????{?? ????????????String?messageStr?=?new?String("Message_"?+?messageNo);?? ????????????System.out.println("Send:"?+?messageStr);?? ????????????producer.send(new?KeyedMessage<Integer,?String>(topic,?messageStr));?? ????????????messageNo++;?? ????????????try?{?? ????????????????sleep(3000);?? ????????????}?catch?(InterruptedException?e)?{?? ????????????????//?TODO?Auto-generated?catch?block?? ????????????????e.printStackTrace();?? ????????????}?? ????????}?? ????}?? ?? }??
consumer
[java]?view plaincopy package?com.sohu.kafkademon;?? ?? import?java.util.HashMap;?? import?java.util.List;?? import?java.util.Map;?? import?java.util.Properties;?? ?? import?kafka.consumer.ConsumerConfig;?? import?kafka.consumer.ConsumerIterator;?? import?kafka.consumer.KafkaStream;?? import?kafka.javaapi.consumer.ConsumerConnector;?? ?? /**? ?*?@author?leicui?bourne_cui@163.com? ?*/?? public?class?KafkaConsumer?extends?Thread?? {?? ????private?final?ConsumerConnector?consumer;?? ????private?final?String?topic;?? ?? ????public?KafkaConsumer(String?topic)?? ????{?? ????????consumer?=?kafka.consumer.Consumer.createJavaConsumerConnector(?? ????????????????createConsumerConfig());?? ????????this.topic?=?topic;?? ????}?? ?? ????private?static?ConsumerConfig?createConsumerConfig()?? ????{?? ????????Properties?props?=?new?Properties();?? ????????props.put("zookeeper.connect",?KafkaProperties.zkConnect);?? ????????props.put("group.id",?KafkaProperties.groupId);?? ????????props.put("zookeeper.session.timeout.ms",?"40000");?? ????????props.put("zookeeper.sync.time.ms",?"200");?? ????????props.put("auto.commit.interval.ms",?"1000");?? ????????return?new?ConsumerConfig(props);?? ????}?? ?? ????@Override?? ????public?void?run()?{?? ????????Map<String,?Integer>?topicCountMap?=?new?HashMap<String,?Integer>();?? ????????topicCountMap.put(topic,?new?Integer(1));?? ????????Map<String,?List<KafkaStream<byte[],?byte[]>>>?consumerMap?=?consumer.createMessageStreams(topicCountMap);?? ????????KafkaStream<byte[],?byte[]>?stream?=?consumerMap.get(topic).get(0);?? ????????ConsumerIterator<byte[],?byte[]>?it?=?stream.iterator();?? ????????while?(it.hasNext())?{?? ????????????System.out.println("receive："?+?new?String(it.next().message()));?? ????????????try?{?? ????????????????sleep(3000);?? ????????????}?catch?(InterruptedException?e)?{?? ????????????????e.printStackTrace();?? ????????????}?? ????????}?? ????}?? }??
簡(jiǎn)單的發(fā)送接收
運(yùn)行下面這個(gè)程序，就可以進(jìn)行簡(jiǎn)單的發(fā)送接收消息了： [java]?view plaincopy package?com.sohu.kafkademon;?? ?? /**? ?*?@author?leicui?bourne_cui@163.com? ?*/?? public?class?KafkaConsumerProducerDemo?? {?? ????public?static?void?main(String[]?args)?? ????{?? ????????KafkaProducer?producerThread?=?new?KafkaProducer(KafkaProperties.topic);?? ????????producerThread.start();?? ?? ????????KafkaConsumer?consumerThread?=?new?KafkaConsumer(KafkaProperties.topic);?? ????????consumerThread.start();?? ????}?? }??
高級(jí)別的consumer
下面是比較負(fù)載的發(fā)送接收的程序： [java]?view plaincopy package?com.sohu.kafkademon;?? ?? import?java.util.HashMap;?? import?java.util.List;?? import?java.util.Map;?? import?java.util.Properties;?? ?? import?kafka.consumer.ConsumerConfig;?? import?kafka.consumer.ConsumerIterator;?? import?kafka.consumer.KafkaStream;?? import?kafka.javaapi.consumer.ConsumerConnector;?? ?? /**? ?*?@author?leicui?bourne_cui@163.com? ?*/?? public?class?KafkaConsumer?extends?Thread?? {?? ????private?final?ConsumerConnector?consumer;?? ????private?final?String?topic;?? ?? ????public?KafkaConsumer(String?topic)?? ????{?? ????????consumer?=?kafka.consumer.Consumer.createJavaConsumerConnector(?? ????????????????createConsumerConfig());?? ????????this.topic?=?topic;?? ????}?? ?? ????private?static?ConsumerConfig?createConsumerConfig()?? ????{?? ????????Properties?props?=?new?Properties();?? ????????props.put("zookeeper.connect",?KafkaProperties.zkConnect);?? ????????props.put("group.id",?KafkaProperties.groupId);?? ????????props.put("zookeeper.session.timeout.ms",?"40000");?? ????????props.put("zookeeper.sync.time.ms",?"200");?? ????????props.put("auto.commit.interval.ms",?"1000");?? ????????return?new?ConsumerConfig(props);?? ????}?? ?? ????@Override?? ????public?void?run()?{?? ????????Map<String,?Integer>?topicCountMap?=?new?HashMap<String,?Integer>();?? ????????topicCountMap.put(topic,?new?Integer(1));?? ????????Map<String,?List<KafkaStream<byte[],?byte[]>>>?consumerMap?=?consumer.createMessageStreams(topicCountMap);?? ????????KafkaStream<byte[],?byte[]>?stream?=?consumerMap.get(topic).get(0);?? ????????ConsumerIterator<byte[],?byte[]>?it?=?stream.iterator();?? ????????while?(it.hasNext())?{?? ????????????System.out.println("receive："?+?new?String(it.next().message()));?? ????????????try?{?? ????????????????sleep(3000);?? ????????????}?catch?(InterruptedException?e)?{?? ????????????????e.printStackTrace();?? ????????????}?? ????????}?? ????}?? }??
Kafka視頻教程同步首發(fā)，歡迎觀看！
上篇文章中我們搭建了kafka的服務(wù)器，并可以使用Kafka的命令行工具創(chuàng)建topic，發(fā)送和接收消息。下面我們來(lái)搭建kafka的開(kāi)發(fā)環(huán)境。
添加依賴
搭建開(kāi)發(fā)環(huán)境需要引入kafka的jar包，一種方式是將Kafka安裝包中l(wèi)ib下的jar包加入到項(xiàng)目的classpath中，這種比較簡(jiǎn)單了。不過(guò)我們使用另一種更加流行的方式：使用maven管理jar包依賴。 創(chuàng)建好maven項(xiàng)目后，在pom.xml中添加以下依賴：
[html]?view plaincopy
添加依賴后如果有兩個(gè)jar包的依賴找不到。點(diǎn)擊這里下載這兩個(gè)jar包，解壓后你有兩種選擇，第一種是使用mvn的install命令將jar包安裝到本地倉(cāng)庫(kù)，另一種是直接將解壓后的文件夾拷貝到mvn本地倉(cāng)庫(kù)的com文件夾下，比如我的本地倉(cāng)庫(kù)是d:\mvn,完成后我的目錄結(jié)構(gòu)是這樣的：
配置程序
更新更全的API編程實(shí)例點(diǎn)這里：http://blog.csdn.net/honglei915/article/details/37697655
首先是一個(gè)充當(dāng)配置文件作用的接口,配置了Kafka的各種連接參數(shù)： [java]?view plaincopy
producer
[java]?view plaincopy
consumer
[java]?view plaincopy
簡(jiǎn)單的發(fā)送接收
運(yùn)行下面這個(gè)程序，就可以進(jìn)行簡(jiǎn)單的發(fā)送接收消息了： [java]?view plaincopy
高級(jí)別的consumer
下面是比較負(fù)載的發(fā)送接收的程序： [java]?view plaincopy
總結(jié)
以上是生活随笔為你收集整理的漫游kafka实战篇之搭建Kafka开发环境的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。
- 上一篇： 漫游Kafka实战篇之搭建Kafka运行
- 下一篇： 漫游Kafka设计篇之数据持久化