本文轉載于?http://bbotte.com/? ELK日志服務使用-kafka傳輸日志
對于日志傳輸，rsyslog或者logstash也就夠用了，一般的redis，kafka，主要是作為緩沖或序列防止logstash掛掉后，日志中斷，下面寫kafka的使用。
在用kafka之前，需要rsyslog模塊支持，所以需要對rsyslog升級，并且啟用omkafka，下面對rsyslog升級同時加入omhiredis：
下載https://github.com/rsyslog/rsyslog/archive/master.zip
rsyslog編譯安裝過程中會遇到種種問題，如下：
checking?for?LIBLOGGING_STDLOG...?no
configure:?error:?Package?requirements?(liblogging-stdlog?>=?1.0.3)?were?not?met:
No?package?'liblogging-stdlog'?found
Consider?adjusting?the?PKG_CONFIG_PATH?environment?variable?if?you
installed?software?in?a?non-standard?prefix.
Alternatively,?you?may?set?the?environment?variables?LIBLOGGING_STDLOG_CFLAGS
and?LIBLOGGING_STDLOG_LIBS?to?avoid?the?need?to?call?pkg-config
checking?for?HIREDIS...?configure:?error:?Package?requirements?(hiredis?>=?0.10.1)?were?not?met:
No?package?'hiredis'?found
Consider?adjusting?the?PKG_CONFIG_PATH?environment?variable?if?you
installed?software?in?a?non-standard?prefix.
Alternatively,?you?may?set?the?environment?variables?HIREDIS_CFLAGS
and?HIREDIS_LIBS?to?avoid?the?need?to?call?pkg-config.
See?the?pkg-config?man?page?for?more?details.
下面為rsyslog安裝升級步驟：
#?yum?install?autoconf?liblogging-devel
#?rpm?-ivh?http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm?
#?yum?-y?install?hiredis?hiredis-devel?libgt-devel?libksi-devel?librelp-devel?liblognorm1-devel?liblognorm1-devel?liblognorm1-devel?libmongo-client-devel?adiscon-librdkafka-devel?libnet-devel
#?yum?-y?install?pkgconfig?libuuid*?libgcrypt-devel?zlib-devel?byacc?mysql-devel
#?wget?http://libestr.adiscon.com/files/download/libestr-0.1.9.tar.gz?
#?tar?xvf?libestr-0.1.9.tar.gz?
#?cd?libestr
#?./configure?&&?make?&&?make?install
#?wget?https://github.com/downloads/json-c/json-c/json-c-0.10.tar.gz
#?tar?xvf?json-c-0.10.tar.gz
#?cd?json-c
#?./configure?&&?make?&&?make?install?
#?cp?../json-c-0.10/json_object_iterator.h?/usr/local/include/json/
#?wget?https://pypi.python.org/packages/source/d/docutils/docutils-0.11.tar.gz?--no-check-certificate?
#?python?setup.py?install?
#?ln?-s?/usr/bin/rst2man.py?/usr/bin/rst2man
wget?https://github.com/redis/hiredis/archive/master.zip
cd?../hiredis-master
make
Generating?hiredis.pc?for?pkgconfig...?生成了hiredis.pc文件
#?cat?/usr/lib64/pkgconfig/hiredis.pc?
prefix=/usr/local
exec_prefix=${prefix}
libdir=/usr/local/lib
includedir=/usr/local/include/hiredis
Name:?hiredis
Description:?Minimalistic?C?client?library?for?Redis.
Version:?0.13.3
Libs:?-L${libdir}?-lhiredis
Cflags:?-I${includedir}?-D_FILE_OFFSET_BITS=64
cp?hiredis.pc?/usr/lib64/pkgconfig/
./configure?--enable-omkafka?--enable-omhiredis?PKG_CONFIG_PATH=/usr/lib64/pkgconfig?--libdir=/usr/lib64
或者更詳細的編譯操作
./configure?--build=x86_64-redhat-linux-gnu?--host=x86_64-redhat-linux-gnu?--target=x86_64-redhat-linux-gnu?\
--program-prefix=?--prefix=/usr?--exec-prefix=?--bindir=/bin?--sbindir=/sbin?--sysconfdir=/etc?\
--datadir=/usr/share?--includedir=/usr/include?--libdir=/lib64?--libexecdir=/libexec?--localstatedir=/var?\
--sharedstatedir=/var/lib?--mandir=/usr/share/man?--infodir=/usr/share/info?--disable-static?\
--disable-testbench?--enable-uuid?--enable-elasticsearch?--enable-ommongodb?--enable-omkafka?\
--enable-usertools?--enable-gnutls?--enable-imfile?--enable-impstats?--enable-imptcp?--enable-libdbi?\
--enable-mail?--enable-mysql?--enable-omprog?--enable-omudpspoof?--enable-omuxsock?--enable-pgsql?\
--enable-pmlastmsg?--enable-relp?--enable-snmp?--enable-unlimited-select?--enable-mmjsonparse?\
--enable-mmnormalize?--enable-mmanon?--enable-mmutf8fix?--enable-mail?--enable-mmfields?\
--enable-mmpstrucdata?--enable-mmsequence?--enable-pmaixforwardedfrom?--enable-pmciscoios?\
--enable-guardtime?--enable-omhiredis?--enable-omhttpfs
---{?database?support?}---
MySql?support?enabled:?no
libdbi?support?enabled:?no
PostgreSQL?support?enabled:?no
mongodb?support?enabled:?no
hiredis?support?enabled:?yes
---{?output?plugins?}---
Mail?support?enabled:?no
omkafka?module?will?be?compiled:?yes
omprog?module?will?be?compiled:?no
omstdout?module?will?be?compiled:?no
omjournal?module?will?be?compiled:?no
omhdfs?module?will?be?compiled:?no
omelasticsearch?module?will?be?compiled:?no
make?&&?make?install
#kafka的rsyslog模塊http://bbotte.com/
ls /usr/lib64/rsyslog/omkafka.so
# ls /usr/lib64/rsyslog
ln?-s?/usr/lib64/rsyslog/omhiredis.so?/lib64/rsyslog/omhiredis.so
ln?-s?/usr/lib64/rsyslog/omkafka.so?/lib64/rsyslog/omkafka.so
service?rsyslog?restart
#?rsyslogd?-v
rsyslogd?8.15.0,?compiled?with:
PLATFORM:?x86_64-redhat-linux-gnu
PLATFORM?(lsb_release?-d):?
FEATURE_REGEXP:?Yes
GSSAPI?Kerberos?5?support:?No
FEATURE_DEBUG?(debug?build,?slow?code):?No
32bit?Atomic?operations?supported:?Yes
64bit?Atomic?operations?supported:?Yes
memory?allocator:?system?default
Runtime?Instrumentation?(slow?code):?No
uuid?support:?Yes
Number?of?Bits?in?RainerScript?integers:?64
See?http://www.rsyslog.com?for?more?information.
發送日志服務器：
#?egrep?-v?"^$|^#"?/etc/rsyslog.conf?
module(load="imuxsock")?#?provides?support?for?local?system?logging?(e.g.?via?logger?command)
module(load="imklog")???#?provides?kernel?logging?support?(previously?done?by?rklogd)
module(load="imudp")?#?needs?to?be?done?just?once
input(type="imudp"?port="1514")
module(load="imfile")
module(load="omhiredis")?#?lets?you?send?to?Redis
module(load="omkafka")???#?lets?you?send?to?Kafka
module(load="imtcp")?#?needs?to?be?done?just?once
input(type="imtcp"?port="514")
$ActionFileDefaultTemplate?RSYSLOG_TraditionalFileFormat
$IncludeConfig?/etc/rsyslog.d/*.conf
*.info;mail.none;authpriv.none;cron.none????????????????/var/log/messages
authpriv.*??????????????????????????????????????????????/var/log/secure
mail.*??????????????????????????????????????????????????/var/log/maillog
cron.*??????????????????????????????????????????????????/var/log/cron
*.emerg?????????????????????????????????????????????????:omusrmsg:*
uucp,news.crit??????????????????????????????????????????/var/log/spooler
local7.*????????????????????????????????????????????????/var/log/boot.log#?egrep?-v?"^$|^#"?/etc/rsyslog.d/kafka.conf?
input(type="imfile"file="/tmp/example.log"tag="examplelogs"facility="local8"
)
template(name="json_lines"?type="list"?option.json="on")?{constant(value="{")constant(value="\"timestamp\":\"")property(name="timereported"?dateFormat="rfc3339")constant(value="\",\"message\":\"")property(name="msg")constant(value="\",\"host\":\"")property(name="hostname")constant(value="\",\"severity\":\"")property(name="syslogseverity-text")constant(value="\",\"facility\":\"")property(name="syslogfacility-text")constant(value="\",\"syslog-tag\":\"")property(name="syslogtag")constant(value="\"}")
}
main_queue(queue.workerthreads="1"??????#?threads?to?work?on?the?queuequeue.dequeueBatchSize="100"?#?max?number?of?messages?to?process?at?oncequeue.size="10000"???????????#?max?queue?size
)
action(broker=["192.168.71.37:9092"]type="omkafka"topic="kafka1"template="json_lines"
)
if?facility="local8"?then?~
kafka的設置：
kafka幫助文檔?https://kafka.apache.org/documentation.html#quickstart
#?tar?-xzf?kafka_2.10-0.9.0.0.tgz
#?cd?/usr/local/kafka_2.10-0.9.0.0/
#?./bin/zookeeper-server-start.sh?config/zookeeper.properties?&
#?./bin/kafka-server-start.sh?config/server.properties?&
#?./bin/kafka-console-consumer.sh?--zookeeper?localhost:2181?--from-beginning?--topic?kafka1?&
接收日志服務器：
#?cd?/usr/local/kafka_2.10-0.9.0.0/
#?./bin/zookeeper-server-start.sh?config/zookeeper.properties?&
#?./bin/kafka-server-start.sh?config/server.properties?&
發送方寫入一條日志：
#?vim?/tmp/example.log
2016-02-26?16:30:29,043?(OrderInvestmentServiceImpl.java:229)?INFO?[DubboServerHandler-10.117.30.32:20886-thread-182]?201602261630
從兩個方面驗證日志的傳輸
#?./bin/kafka-console-consumer.sh?--zookeeper?192.168.71.37:2181?--from-beginning?--topic?kafka1
{"timestamp":"2016-02-26T17:23:00.101552+08:00","message":"2016-02-26?16:30:29,043?(OrderInvestmentServiceImpl.java:229)?INFO?[DubboServerHandler-10.117.30.32:20886-thread-182]?201602261630","host":"localhost","severity":"debug","facility":"invld","syslog-tag":"examplelogs"}#?/opt/logstash/bin/logstash?-f?kafka.conf?-v
{"timestamp"?=>?"2016-02-26T17:23:00.101552+08:00","message"?=>?"2016-02-26?16:30:29,043?(OrderInvestmentServiceImpl.java:229)?INFO?[DubboServerHandler-10.117.30.32:20886-thread-182]?201602261630","host"?=>?"localhost","severity"?=>?"debug","facility"?=>?"invld","syslog-tag"?=>?"examplelogs","@version"?=>?"1","@timestamp"?=>?"2016-02-26T09:23:00.430Z"
}#?vim?kafka.conf?
input?{kafka?{zk_connect?=>?"192.168.71.37:2181"topic_id?=>?"kafka1"}
}
output?{stdout?{codec=>"rubydebug"}
}
參考文檔：
http://blog.sematext.com/tag/rsyslog/
http://www.rsyslog.com/doc/v8-stable/installation/install_from_source.html
by?http://bbotte.com/
轉載于:https://blog.51cto.com/bbotte/1747143
總結
以上是生活随笔為你收集整理的ELK日志服务使用-kafka传输日志(bbotte.com)的全部內容，希望文章能夠幫你解決所遇到的問題。
如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。
