Flume環(huán)境搭建_五種案例

http://flume.apache.org/FlumeUserGuide.html

A simple example

Here, we give an example configuration file, describing a single-node Flume deployment. This configuration lets a user generate events and subsequently logs them to the console.

# example.conf: A single-node Flume configuration# Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444 # Describe the sink a1.sinks.k1.type = logger # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1

This configuration defines a single agent named a1. a1 has a source that listens for data on port 44444, a channel that buffers event data in memory, and a sink that logs event data to the console. The configuration file names the various components, then describes their types and configuration parameters. A given configuration file might define several named agents; when a given Flume process is launched a flag is passed telling it which named agent to manifest.

Given this configuration file, we can start Flume as follows:

$ bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=INFO,console

Note that in a full deployment we would typically include one more option:?--conf=<conf-dir>. The?<conf-dir>?directory would include a shell script?flume-env.sh?and potentially a log4j properties file. In this example, we pass a Java option to force Flume to log to the console and we go without a custom environment script.

From a separate terminal, we can then telnet port 44444 and send Flume an event:

$ telnet localhost 44444 Trying 127.0.0.1... Connected to localhost.localdomain (127.0.0.1). Escape character is '^]'. Hello world! <ENTER> OK

The original Flume terminal will output the event in a log message.

12/06/19 15:32:19 INFO source.NetcatSource: Source starting 12/06/19 15:32:19 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444] 12/06/19 15:32:34 INFO sink.LoggerSink: Event: { headers:{} body: 48 65 6C 6C 6F 20 77 6F 72 6C 64 21 0D Hello world!. }

Congratulations - you’ve successfully configured and deployed a Flume agent! Subsequent sections cover agent configuration in much more detail.

?

?

?

以下為具體搭建流程

Flume搭建_案例一：單個(gè)Flume

?

?

安裝node2上

1.? ?上傳到/home/tools，解壓，解壓后移動(dòng)到/home下

?

2.? ?重命名，并修改flume-env.sh

vi flume-env.sh ? 3.? ?配置Flume的環(huán)境變量 vi /etc/profile source /etc/profile 查看Flume的版本，看Flume的環(huán)境變量是否配置成功 4. ? ?在/home下創(chuàng)建tests_flume, 并創(chuàng)建flume配置文件 cd test_flume vi flume1 5. ? ?命令測(cè)試Flume是否安裝成功 flume-ng agent --conf /home/test_flume --conf-file /home/test_flume/flume1 --name a1 -Dflume.root.logger=INFO,console 安裝telnet 隨意輸入? hi flume 切換窗口查看 退出?ctrl+] ?quit

?

?

Flume搭建_案例二：兩個(gè)Flume做集群

安裝node1，node2上

MemoryChanel配置

capacity：默認(rèn)該通道中最大的可以存儲(chǔ)的event數(shù)量是100，

trasactionCapacity：每次最大可以從source中拿到或者送到sink中的event數(shù)量也是100

keep-alive：event添加到通道中或者移出的允許時(shí)間

byte**：即event的字節(jié)量的限制，只包括eventbody

1.? ?node1,node2，上傳壓縮包到/home/tools下，解壓， 2.? ??? 修改conf下的flume-env.sh中的java環(huán)境變量， 3. ?? ? 在/etc/profile下? 配置Flume的環(huán)境變量 4. ? ? ? node1,node2下創(chuàng)建測(cè)試目錄test_flume，并分別在node1,node2下創(chuàng)建配置文件——flume21，flume22 node1下創(chuàng)建flume21 node2下創(chuàng)建flume22 5.? ?node1,node2分別啟動(dòng)flume(注意因?yàn)閚ode2在后面，所以先啟動(dòng)node2中flume,再啟動(dòng)node1中flume)

先啟動(dòng)node02的Flume

flume-ng agent?-n a1?-c conf?-f avro.conf?-Dflume.root.logger=INFO,console

flume-ng agent?-n a1?-c conf?-f?/home/test_flume/flume22?-Dflume.root.logger=INFO,console

再啟動(dòng)node01的Flume

flume-ng agent?-n a1?-c conf?-f simple.conf2?-Dflume.root.logger=INFO,console

flume-ng agent?-n a1?-c conf?-f?/home/test_flume/flume21?-Dflume.root.logger=INFO,console

node2: node1: 6.? ?打開telnet測(cè)試，node2輸出結(jié)果 ?

?

?

?

?

?

Flume搭建_案例三：如何監(jiān)控一個(gè)文件的變化？

安裝node2上 1.? ?node2，上傳壓縮包到/home/tools下，解壓， 2.? ??? 修改conf下的flume-env.sh中的java環(huán)境變量， 3. ?? ? 在/etc/profile下? 配置Flume的環(huán)境變量 4. ? ? ?? node2下創(chuàng)建測(cè)試目錄test_flume，node2下創(chuàng)建配置文件——flume3 mkdir test_flume vi flume3 5. ? ?node2啟動(dòng)flume

啟動(dòng)Flume

flume-ng agent?-n a1?-c conf?-f exec.conf?-Dflume.root.logger=INFO,console

flume-ng agent?-n a1?-c conf?-f?/home/test_flume/flume3?-Dflume.root.logger=INFO,console

6. ? ?測(cè)試 在/home/test_flume下創(chuàng)建空文件演示 touch flume.exec.log 循環(huán)添加數(shù)據(jù) for i in {1..50}; do echo "$i hi flume" >> flume.exec.log ; sleep 0.1; done

Flume搭建_案例四： 如何監(jiān)控一個(gè)文件：目錄的變化？

安裝node2上 1.? ?node2，上傳壓縮包到/home/tools下，解壓， 2.? ??? 修改conf下的flume-env.sh中的java環(huán)境變量， 3. ?? ? 在/etc/profile下? 配置Flume的環(huán)境變量 4. ? ? ??? node2下創(chuàng)建測(cè)試目錄test_flume，node2下創(chuàng)建配置文件——flume4 mkdir test_flume vi flume4 5. ? ?node2啟動(dòng)flume 6. ? ?測(cè)試

?

Flume搭建_案例五： 如何定義一個(gè)HDFS類型的Sink?

安裝node2上

Flume搭建_案例五_配置項(xiàng)解讀

1.? ?Flume中日期的格式 什么時(shí)候會(huì)用？ Flume收集的時(shí)候根據(jù)時(shí)間來創(chuàng)建，比如今天的產(chǎn)生的數(shù)據(jù)就創(chuàng)建20170216，昨天的就放在20170215下 ！注意 2.? ?Flume是如何找到HDFS？ Flume如果配置的是hdfs,它會(huì)根據(jù)系統(tǒng)中配置的環(huán)境變量去找
3.? ?Flume什么時(shí)候滾動(dòng)生成新文件？ 滾動(dòng)的間隔，大小，數(shù)量

hdfs.rollInterval	30	Number of seconds to wait before rolling current file (0 = never roll based on time interval)
hdfs.rollSize	1024	File size to trigger roll, in bytes (0: never roll based on file size)
hdfs.rollCount	10	Number of events written to file before it rolled (0 = never roll based on number of events)

?

4.? ?多長時(shí)間沒有操作，Flume將一個(gè)臨時(shí)文件生成新文件？

hdfs.idleTimeout

0

Timeout after which inactive files get closed (0 = disable automatic closing of idle files)

?

5.? ?多長時(shí)間生成一個(gè)新的目錄？（比如每10s生成一個(gè)新的目錄）

? ?? ?四舍五入，沒有五入，只有四舍

? ?? ?(比如57分劃分為55分，5,6,7,8,9在一個(gè)目錄，10,11,12,13,14在一個(gè)目錄)

hdfs.round	false	Should the timestamp be rounded down (if true, affects all time based escape sequences except %t)
hdfs.roundValue	1	Rounded down to the highest multiple of this (in the unit configured using?hdfs.roundUnit), less than current time.
hdfs.roundUnit	second	The unit of the round down value -?second,?minute?or?hour.

?

1.? ?node2，上傳壓縮包到/home/tools下，解壓， 2.? ??? 修改conf下的flume-env.sh中的java環(huán)境變量， 3. ?? ? 在/etc/profile下? 配置Flume的環(huán)境變量 4. ? ? ??? node2下創(chuàng)建測(cè)試目錄test_flume，node2下創(chuàng)建配置文件——flume5 mkdir test_flume

?

vi flume5

?

超越永無止境

轉(zhuǎn)載于:https://www.cnblogs.com/wangbin/p/8192950.html

總結(jié)

以上是生活随笔為你收集整理的Flume环境搭建_五种案例(转)的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。