分享一個自己很早以前先寫的日志存儲方案，簡單實用，業務端上傳完日志后自己清除已經上傳過的日志，連清空間的工作都省了。（未完）

找出所有部署目錄下的日志文件夾,壓縮指定文件與文件夾,不改變日志壓縮路徑(排除gzip)
上傳指定名稱的日志壓縮包到rsync服務器,根據ip自動生成目錄,并上傳后保持原來的目錄結構.
上傳文件權限進行自動修改,方便log用戶訪問.
自動創建用戶,安裝rsync,鏈接日志目錄到log用戶.


日志服務器:192.168.165.88,
同步服務器運行軟件：rsync
日志上傳到/back/chss并自動創建基于業務服務器IP的目錄實現日志歸類

點擊(此處)折疊或打開

[root@192_168_165_88 ~]# cat /etc/rsyncd/rsyncd.conf

# Minimal configuration file for rsync daemon

# See rsync(1) and rsyncd.conf(5) man pages for help

# This line is required by the /etc/init.d/rsyncd script

pid file = /var/run/rsyncd.pid

port = 873

address = 192.168.165.88

uid=root

gid=root

auth users = admin

secrets file = /etc/rsyncd/rsyncd.secrets

#incoming chmod = u+rwx,g+rwx,o+rx

use chroot = yes

read only = no

#limit access to private LANs

hosts allow=192.168.0.0/255.255.0.0

hosts deny=*

max connections = 500

#This will give you a separate log file

log file = /var/log/rsync.log

#This will log every file transferred - up to 85,000+ per user, per sync

transfer logging = yes

log format = %t %a %m %f %b

syslog facility = local3

timeout = 300

#ignore errors

[chss-t]

path = /back/chss

list = yes

[chss-tc]

path = /back/chss_config

list = yes

每個目錄下都是跟業務服務器上的部署路徑完全一樣的完整的存儲路徑，這里不就不貼出來了。

點擊(此處)折疊或打開

[root@192_168_165_88 chss]# pwd

/back/chss

[root@192_168_165_88 chss]# ls

XXX XXX XXX XXX XXX ...

[root@192_168_165_88 chss]# cd chss-aaaaaa/

[root@192_168_165_88 chss-aaaaaa]# ls

192.168.171.72? 192.168.174.45? 192.168.174.46

各業務服務器中使用腳本處理過濾壓縮后上傳到日志服務器
把以下2個腳本放置在服務器的bin目錄下
執行

點擊(此處)折疊或打開

[root@S192-168-171-134 ~]# nohup /bin/while.t &

點擊(此處)折疊或打開

[root@S192-168-171-134 tmp]# ls

find_192.168.171.134.log? rsync.exe.log nohup.out rsync_192.168.171.134.log

/bin/while.t

點擊(此處)折疊或打開

#!/bin/bash

while :

do

/bin/bash /bin/rsync.t

sleep 0.5h

done

/bin/rsync.t

點擊(此處)折疊或打開

#!/bin/env bash

#set -x

#2013-08-18: 修復直接壓縮201目錄的bug

#2013-09-28: 新加項目分類目錄,新加多重判斷

#2015-10-19: 修復與glusterfs沖突問題.

export RSYNC_PASSWORD=skymobi

LOGPATH="opt app data app1"

SLEEPTIME=10s

IP=$(LC_ALL=C /sbin/ifconfig|grep "inet addr:"|grep -v "127.0.0.1"|cut -d: -f2|awk '{print $1}'|sed ':a;$!N;s/\n/_/g;ta;P;D')

USER=admin

RSYNCIP=192.168.165.88

PROJECT=chss-mis

RSYNCMODULE=chss-t

RSYNCCONFMODULE=chss-tc

RSYNCDIR=${PROJECT}/"$IP"

FINDLOGFILE=/tmp/find_"$IP".log

RSYNCLOGFILE=/tmp/rsync_"$IP".log

RSYNCEXELOGFILE=/tmp/rsync.exe.log

DATETIME=$(date +%F_%T)

mkdir -p /tmp/$RSYNCDIR

for rspath in $RSYNCMODULE $RSYNCCONFMODULE

do

rsync -atv --log-file=${RSYNCLOGFILE} /tmp/${PROJECT} ${USER}@"$RSYNCIP"::${rspath}

done

for Path in $LOGPATH

do

echo $DATETIME

if [ -d /$Path ];then

FINDER(){

find /${Path} -regex ".*logs/.*log.*201.*" -a ! -regex '.*tar.gz' -a -mmin +5 -exec date '+%F %T' \; -exec tar -zcvf `basename {}`_t.tar.gz {} --remove-files --exclude=*tar.gz \; >> $FINDLOGFILE

}

SPACE=$(df -P /${Path} |tail -n 1 |awk '{print $5}'|awk -F% '{print $1}')

if [ $SPACE -gt 80 ];then

find /${Path} -regex ".*logs/.*log.*201.*" -mtime +"${DAYS:=1}" -exec date '+%F %T' \; -exec rm -rfv {} \; >> ${FINDLOGFILE}

else

FINDER

fi

sleep $SLEEPTIME

rsync -aERtvz --no-p --chmod=Do+rX,Fo-X --log-file=${RSYNCLOGFILE} --remove-source-files --skip-compress=*.gz $(find /$Path -type f -name '*_t.tar.gz' -a ! -regex '.*/brick/.*') ${USER}@"$RSYNCIP"::"$RSYNCMODULE"/${RSYNCDIR}

rsync -aERtvz --no-p --chmod=Do+rX,Fo-X --log-file=${RSYNCLOGFILE} --remove-source-files --skip-compress=*.gz $(find /$Path -type f -name '*_tc.tar.gz' -a ! -regex '.*/brick/.*') ${USER}@"$RSYNCIP"::"$RSYNCCONFMODULE"/${RSYNCDIR}

else

echo /${Path} is not exist...

fi

done >> ${RSYNCEXELOGFILE}

在日志服務器上執行crontab任務
定期清理空間

點擊(此處)折疊或打開

#!/bin/env bash

set -x

LOGPATH="back/chss/chss-mis back/chss/zm-adv-mis opt app data app1 back"

IP=$(LC_ALL=C /sbin/ifconfig|grep "inet addr:"|grep -v "127.0.0.1"|cut -d: -f2|awk '{print $1}'|sed ':a;$!N;s/\n/_/g;ta;P;D')

FINDLOGFILE=/tmp/find_"$IP".log

RSYNCLOGFILE=/tmp/rm_"$IP".log

RSYNCEXELOGFILE=/tmp/rm.exe.log

DATETIME=$(date +%F_%T)

echo _start_ `date +%F_%T`

for Path in $LOGPATH

do

??echo $DATETIME

??if [ -d /$Path ];then

????SPACE=$(df -P /${Path} |tail -n 1 |awk '{print $5}'|awk -F% '{print $1}')

????if [ $SPACE -gt 80 ];then

??????if [ ${Path} == 'back/chss/zm-adv-mis' ];then

????????DATENUM=0

????????MINNUM=360

??????elif [ ${Path} == 'back/chss/chss-mis' ];then

????????DATENUM=0

????????MINNUM=7200

??????else

????????DATENUM=10

????????MINNUM=10080

??????fi

????else

??????DATENUM=50

??????MINNUM=43200

????fi

????FINDER(){

??????#days

??????#find /$Path -mtime +"${DATENUM:=20}" -a -regex '.*logs.*' -a ! -regex '.*chss_config' -a ! -empty -exec date +%F_%T \; -exec rm -rv {} \; >> $FINDLOGFILE

??????#Mins

??????find /$Path -mmin +"${MINNUM:=43200}" -a -regex '.*logs.*' -a ! -regex '.*chss_config' -a ! -empty -exec date +%F_%T \; -exec rm -rv {} \; >> $FINDLOGFILE

????}

????FINDER

????#echo ---- /${Path} $DATENUM days rm ok

????echo ---- /${Path} $(( $MINNUM / 60 / 24 )) days rm ok

??else

????echo ---- /${Path} is not exist...

??fi

done >> ${RSYNCEXELOGFILE}

《新程序員》：云原生和全面數字化實踐50位技術專家共同創作，文字、視頻、音頻交互閱讀

總結

以上是生活随笔為你收集整理的巧用shell+rsync服务实现日志自动过滤处理压缩并上传日志服务器，自动分类的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。