生活随笔
收集整理的這篇文章主要介紹了
ELK学习9_ELK数据流传输过程_问题总结2
小編覺得挺不錯的,現(xiàn)在分享給大家,幫大家做個參考.
目的：使用Java程序把一個.csv文件發(fā)送給Kafka，再發(fā)送給ELK
實驗步驟：見以前的博客
遇到的問題：
一、
直接發(fā)送比較大的.csv文件，ELK接到的漢字內容是亂碼的
發(fā)送比較小的.csv文件，ELK接到的漢字內容正常
猜測：是否是Kafka出錯
二、
把大的.csv文件拆分成多個小.csv文件發(fā)送，發(fā)送過程中，會不定期報錯
報錯內容如下
[html]?view plaincopy
[2015-12-13?16:02:14,273]?ERROR?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?(kafka.utils.Utils$:106)?? kafka.common.KafkaException:?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?? ????at?kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)?? ????at?kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)?? ????at?kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67)?? ????at?kafka.utils.Utils$.swallow(Utils.scala:172)?? ????at?kafka.utils.Logging$class.swallowError(Logging.scala:106)?? ????at?kafka.utils.Utils$.swallowError(Utils.scala:45)?? ????at?kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)?? ????at?kafka.producer.Producer.send(Producer.scala:77)?? ????at?kafka.javaapi.producer.Producer.send(Producer.scala:33)?? ????at?com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33)?? ????at?com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73)?? ????at?com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17)?? Caused?by:?java.nio.channels.ClosedChannelException?? ????at?kafka.network.BlockingChannel.send(BlockingChannel.scala:100)?? ????at?kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)?? ????at?kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)?? ????at?kafka.producer.SyncProducer.send(SyncProducer.scala:113)?? ????at?kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)?? ????...?11?more?? [2015-12-13?16:02:14,283]?ERROR?Failed?to?collate?messages?by?topic,?partition?due?to:?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?(kafka.producer.async.DefaultEventHandler:97)?? [2015-12-13?16:02:14,394]?ERROR?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?(kafka.utils.Utils$:106)?? kafka.common.KafkaException:?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?? ????at?kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)?? ????at?kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)?? ????at?kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78)?? ????at?kafka.utils.Utils$.swallow(Utils.scala:172)?? ????at?kafka.utils.Logging$class.swallowError(Logging.scala:106)?? ????at?kafka.utils.Utils$.swallowError(Utils.scala:45)?? ????at?kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)?? ????at?kafka.producer.Producer.send(Producer.scala:77)?? ????at?kafka.javaapi.producer.Producer.send(Producer.scala:33)?? ????at?com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33)?? ????at?com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73)?? ????at?com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17)?? Caused?by:?java.nio.channels.ClosedChannelException?? ????at?kafka.network.BlockingChannel.send(BlockingChannel.scala:100)?? ????at?kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)?? ????at?kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)?? ????at?kafka.producer.SyncProducer.send(SyncProducer.scala:113)?? ????at?kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)?? ????...?11?more?? [2015-12-13?16:02:14,404]?ERROR?Failed?to?collate?messages?by?topic,?partition?due?to:?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?(kafka.producer.async.DefaultEventHandler:97)?? [2015-12-13?16:02:14,514]?ERROR?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?(kafka.utils.Utils$:106)?? kafka.common.KafkaException:?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?? ????at?kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)?? ????at?kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)?? ????at?kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78)?? ????at?kafka.utils.Utils$.swallow(Utils.scala:172)?? ????at?kafka.utils.Logging$class.swallowError(Logging.scala:106)?? ????at?kafka.utils.Utils$.swallowError(Utils.scala:45)?? ????at?kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)?? ????at?kafka.producer.Producer.send(Producer.scala:77)?? ????at?kafka.javaapi.producer.Producer.send(Producer.scala:33)?? ????at?com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33)?? ????at?com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73)?? ????at?com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17)?? Caused?by:?java.nio.channels.ClosedChannelException?? ????at?kafka.network.BlockingChannel.send(BlockingChannel.scala:100)?? ????at?kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)?? ????at?kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)?? ????at?kafka.producer.SyncProducer.send(SyncProducer.scala:113)?? ????at?kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)?? ????...?11?more?? [2015-12-13?16:02:14,524]?ERROR?Failed?to?collate?messages?by?topic,?partition?due?to:?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?(kafka.producer.async.DefaultEventHandler:97)?? [2015-12-13?16:02:14,644]?ERROR?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?(kafka.utils.Utils$:106)?? kafka.common.KafkaException:?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?? ????at?kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)?? ????at?kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)?? ????at?kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78)?? ????at?kafka.utils.Utils$.swallow(Utils.scala:172)?? ????at?kafka.utils.Logging$class.swallowError(Logging.scala:106)?? ????at?kafka.utils.Utils$.swallowError(Utils.scala:45)?? ????at?kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)?? ????at?kafka.producer.Producer.send(Producer.scala:77)?? ????at?kafka.javaapi.producer.Producer.send(Producer.scala:33)?? ????at?com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33)?? ????at?com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73)?? ????at?com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17)?? Caused?by:?java.nio.channels.ClosedChannelException?? ????at?kafka.network.BlockingChannel.send(BlockingChannel.scala:100)?? ????at?kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)?? ????at?kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)?? ????at?kafka.producer.SyncProducer.send(SyncProducer.scala:113)?? ????at?kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)?? ????...?11?more?? [2015-12-13?16:02:14,654]?ERROR?Failed?to?collate?messages?by?topic,?partition?due?to:?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?(kafka.producer.async.DefaultEventHandler:97)?? [2015-12-13?16:02:14,764]?ERROR?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?(kafka.utils.Utils$:106)?? kafka.common.KafkaException:?fetching?topic?metadata?for?topics?[Set(MyPattern2)]?from?broker?[ArrayBuffer(id:0,host:10.10.16.14,port:9092)]?failed?? ????at?kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72)?? ????at?kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)?? ????at?kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78)?? ????at?kafka.utils.Utils$.swallow(Utils.scala:172)?? ????at?kafka.utils.Logging$class.swallowError(Logging.scala:106)?? ????at?kafka.utils.Utils$.swallowError(Utils.scala:45)?? ????at?kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)?? ????at?kafka.producer.Producer.send(Producer.scala:77)?? ????at?kafka.javaapi.producer.Producer.send(Producer.scala:33)?? ????at?com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33)?? ????at?com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73)?? ????at?com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17)?? Caused?by:?java.nio.channels.ClosedChannelException?? ????at?kafka.network.BlockingChannel.send(BlockingChannel.scala:100)?? ????at?kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)?? ????at?kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)?? ????at?kafka.producer.SyncProducer.send(SyncProducer.scala:113)?? ????at?kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)?? ????...?11?more?? [2015-12-13?16:02:14,764]?ERROR?Failed?to?send?requests?for?topics?MyPattern2?with?correlation?ids?in?[0,8]?(kafka.producer.async.DefaultEventHandler:97)?? kafka.common.FailedToSendMessageException:?Failed?to?send?messages?after?3?tries.error?? ?? ????at?kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)?? ????at?kafka.producer.Producer.send(Producer.scala:77)?? ????at?kafka.javaapi.producer.Producer.send(Producer.scala:33)?? ????at?com.test.MessageSender.sendToKafkaTopic(MessageSender.java:33)?? ????at?com.test.ReadFile2Kafka.bianli(ReadFile2Kafka.java:73)?? ????at?com.test.ReadFile2Kafka.main(ReadFile2Kafka.java:17)??
沒能找到出錯原因
三、使用服務器本地安裝的Eclipse執(zhí)行程序，中間不會報錯，但是，接收到的中文內容還是亂碼
來源:http://blog.csdn.net/wang_zhenwei/article/details/50285345
總結
以上是生活随笔為你收集整理的ELK学习9_ELK数据流传输过程_问题总结2的全部內容，希望文章能夠幫你解決所遇到的問題。
如果覺得生活随笔網(wǎng)站內容還不錯，歡迎將生活随笔推薦給好友。
