2019獨角獸企業重金招聘Python工程師標準>>>

Goal:

Use Flume to?pool a folder on local file system and write it to HDFS.

Version Information:

hadoop-0.22.0

apache-flume-1.3.1

Flume Configuration:

Edit file? flume-env.sh?under ?/$FLUME_HOME$/conf:

export JAVA_HOME=your jave home export FLUME_CLASSPATH=your flume home export HADOOP_CLASSPATH=your hadoop home

Edit file?flume-conf.properties?under?/$FLUME_HOME$/conf:

# Configure the agent agent.sources = spooldirSource agent.channels = memoryChannel agent.sinks = hdfsSink# Configure the source agent.sources.spooldirSource.type = spooldir agent.sources.spooldirSource.spoolDir = /tmp/flume/ agent.sources.spooldirSource.channels = memoryChannel# Configure the sink agent.sinks.hdfsSink.type = hdfs agent.sinks.hdfsSink.hdfs.path = hdfs://masternode:9000/flume/events agent.sinks.hdfsSink.hdfs.filePrefix = events- agent.sinks.hdfsSink.channel = memoryChannel# Configure the channel agent.channels.memoryChannel.type = memory agent.channels.memoryChannel.capacity = 100

Copy Hadoop Jars to Flume lib directory:

Copy?hadoop-hdfs-0.22.0.jar and?hadoop-common-0.22.0.jar to?/$FLUME_HOME$/lib.

Start Flume Agent:

./bin/flume-ng agent -n agent -c conf -f conf/flume-conf.properties

Write File:

echo "Hello World">>/tmp/flume/test

View Logs:

Under?/$FLUME_HOME$/logs

轉載于:https://my.oschina.net/xiangchen/blog/101111

《新程序員》：云原生和全面數字化實踐50位技術專家共同創作，文字、視頻、音頻交互閱讀

總結

以上是生活随笔為你收集整理的Flume向HDFS写数据实例的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。