啟動(dòng)多個(gè)進(jìn)程收集日志。直接output到kafka，output到不同的topic
cat filebeat.yml
filebeat.prospectors:
- input_type: log

paths:

- /home/test/job/logs/job-20*.log
document_type: job
tail_files: true
multiline.pattern: ^\[[0-9]{4}-[0-9]{2}-[0-9]{2}

multiline.negate: true

multiline.match: after

output.kafka:

hosts: ["192.168.1.99:9092"]
topic: job
partition.round_robin:
reachable_only: false

required_acks: 1
compression: gzip
max_message_bytes: 1000000
通過(guò)logstash節(jié)點(diǎn)會(huì)實(shí)時(shí)去kafka broker集群拉數(shù)據(jù)，轉(zhuǎn)ES
[root@centos199 config]# cat job.conf
input {
kafka {
bootstrap_servers => "192.168.1.99:9092"
topics => ["job"]
codec => "json"
}
}
output {
elasticsearch {
hosts => ["192.168.1.99:9200"]
index => "job-%{+YYYY.MM.dd}"
template_overwrite => true
}
}

轉(zhuǎn)載于:https://www.cnblogs.com/haoge92/p/9317903.html

總結(jié)

以上是生活随笔為你收集整理的ELK之filebeat、logstash多个topic配置的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問(wèn)題。

如果覺(jué)得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。