先創(chuàng)建一個目錄

?

在這個job目錄下創(chuàng)建upload.sh文件

[hadoop@node1 ~]$ pwd /home/hadoop [hadoop@node1 ~]$ mkdir job [hadoop@node1 ~]$ ls Desktop Downloads job Music Pictures Templates Documents hive logs mysql-community-release-el7-5.noarch.rpm Public Videos [hadoop@node1 ~]$ cd job/ [hadoop@node1 job]$ vim upload.sh

?

對upload.sh進(jìn)行編輯

#!/bin/bash#set java env export JAVA_HOME=/opt/modules/jdk1.8.0_65 export JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:$PATH#set hadoop env export HADOOP_HOME=/opt/modules/hadoop-2.6.0 export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATHlog_src_dir=/home/hadoop/logs/log/ log_toupload_dir=/home/hadoop/logs/toupload/ hdfs_root_dir=/data/clickLog/20190620/echo "log_src_dir:"$log_src_dir ls $log_src_dir | while read fileName doif [[ "$fileName" == access.log ]]; then# if [ "access.log" = "$fileName" ];thendate=`date +%Y_%m_%d_%H_%M_%S`#將文件移動到待上傳目錄并重命名#打印信息echo "moving $log_src_dir$fileName to $log_toupload_dir"xxxxx_click_log_$fileName"$date"mv $log_src_dir$fileName $log_toupload_dir"xxxxx_click_log_$fileName"$date#將待上傳的文件path寫入一個列表文件willDoingecho $log_toupload_dir"xxxxx_click_log_$fileName"$date >> $log_toupload_dir"willDoing."$datefidone#找到列表文件willDoing ls $log_toupload_dir | grep will |grep -v "_COPY_" | grep -v "_DONE_" | while read line do#打印信息echo "toupload is in file:"$line#將待上傳文件列表willDoing改名為willDoing_COPY_mv $log_toupload_dir$line $log_toupload_dir$line"_COPY_"#讀列表文件willDoing_COPY_的內(nèi)容（一個一個的待上傳文件名） ,此處的line 就是列表中的一個待上傳文件的pathcat $log_toupload_dir$line"_COPY_" |while read linedo#打印信息echo "puting...$line to hdfs path.....$hdfs_root_dir"hadoop fs -put $line $hdfs_root_dirdone mv $log_toupload_dir$line"_COPY_" $log_toupload_dir$line"_DONE_" done

?

然后新建目錄，并上傳日志文件

?

給腳本賦予權(quán)限

[hadoop@node1 job]$ ls upload.sh [hadoop@node1 job]$ pwd /home/hadoop/job [hadoop@node1 job]$ ll total 4 -rw-rw-r--. 1 hadoop hadoop 1734 Jun 20 10:57 upload.sh [hadoop@node1 job]$ chmod 777 upload.sh [hadoop@node1 job]$ ll total 4 -rwxrwxrwx. 1 hadoop hadoop 1734 Jun 20 10:57 upload.sh [hadoop@node1 job]$

?

?

在HDFS上新建目錄

?

?執(zhí)行腳本

?

?可以看到結(jié)果

?

轉(zhuǎn)載于:https://www.cnblogs.com/braveym/p/11051483.html

總結(jié)

以上是生活随笔為你收集整理的18.linux日志收集数据到hdfs上面的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯，歡迎將生活随笔推薦給好友。