故障現(xiàn)象：

外網(wǎng)訪問頁面顯示不正常，Template Error!

?

接到反饋后，遠(yuǎn)程到服務(wù)器，執(zhí)行service nginx restart ，提示磁盤滿，無法繼續(xù)。

查看磁盤占用情況，磁盤/目錄占用100%,磁盤空間滿。

執(zhí)行

find / -size +100M -exec ls -lh {} \;

查找100M以上的文件，發(fā)現(xiàn)主要大的是NGINX的日志：

一天的NGINX日志大到30多個(gè)G，太離譜了！！！

?

切割文件的最后Ｎ行到新的文件中

tail -n 1000 access.log >> 111.log

這樣做，主要是太大的文件我用VI無法打開。

查看最新的日志：

{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}

對比正常的其它服務(wù)器：
http://10.10.14.199/dsideal_yy/golbal/getValueByKey?key=common.rongyun.suffix
{"common.rongyun.suffix":"_199"}

繼續(xù)查看此服務(wù)器：
http://10.24.0.7:7777/dsideal_yy/golbal/getValueByKey?key=common.rongyun.suffix
{"common.rongyun.suffix":"_zztez"}

也不是404啊！！！！

?

訪問一下http://10.24.0.7,居然可以訪問的到！！！！！！還顯示welcome to Nginx!!!!我們是7777的端口啊！！！

查看一下80端口是什么東東在使用：

yum install lsof
lsof -i tcp:80

[root@bogon conf]# lsof -i tcp:80 ?
COMMAND??? PID USER?? FD?? TYPE???? DEVICE SIZE/OFF NODE NAME
nginx?? 243175 root?? 10u? IPv4 2358009951????? 0t0? TCP *:http (LISTEN)
nginx?? 243176 root?? 10u? IPv4 2358009951????? 0t0? TCP *:http (LISTEN)
nginx?? 243178 root?? 10u? IPv4 2358009951????? 0t0? TCP *:http (LISTEN)
nginx?? 243179 root?? 10u? IPv4 2358009951????? 0t0? TCP *:http (LISTEN)
nginx?? 243180 root?? 10u? IPv4 2358009951????? 0t0? TCP *:http (LISTEN)

根據(jù)pid查文件位置：
ps 243175

[root@bogon conf]# ps 243175
?? PID TTY????? STAT?? TIME COMMAND
243175 ???????? Ss???? 0:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -c /usr/local/openresty/nginx/conf/nginx.conf

到這里很顯示了，還是這個(gè)配置文件有問題，里面配置了兩個(gè)端口：7777和80

?

server {listen 80;server_name office.edusoa.com;location ^~ /yjj/ {proxy_pass http://221.194.113.150/; }location ^~ /pingli/ {proxy_pass http://61.134.47.35:9999/; }}

這個(gè)東東明顯就是垃圾，注釋掉，并手工刪除超大的日志文件，世界清靜了~

?

回頭想想，這是兩個(gè)問題：

1、即然配置了7777，為什么還保留了80，這是明確的錯(cuò)誤。

2、就算是同時(shí)配置了80和7777，那個(gè)大量的404錯(cuò)誤從哪里來？？

{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}

很明顯，是10.24.0.6這臺(tái)機(jī)器上來的，這是一臺(tái)windows主機(jī)，是處理程序發(fā)過來的請求，它在干什么？為什么不去訪問7777的端口？？

補(bǔ)充：

1、在訪問時(shí)，發(fā)現(xiàn)訪問不了，直接while true了，這里應(yīng)該是停止或者休息一會(huì)。

2、處理程序在訪問全局變量時(shí)居然沒有加上端口！！！！！造成繼續(xù)訪問80，這就是幾個(gè)錯(cuò)誤集中在一起，導(dǎo)致了問題出現(xiàn)，血的教訓(xùn)！！！

轉(zhuǎn)載于:https://www.cnblogs.com/littlehb/p/8274073.html

總結(jié)

以上是生活随笔為你收集整理的山东省枣庄市台儿庄区云平台运维故障处理一例的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。