某次cloudera-scm-agnet系統(tǒng)啟動(dòng)不了

前幾天添加磁盤的時(shí)候系統(tǒng)以外重啟了，那個(gè)時(shí)候并沒stop cloudera-scm-server和cloudera-scm-agent兩個(gè)進(jìn)程，導(dǎo)致了重新啟動(dòng)的時(shí)候cloudera-scm-server可以啟動(dòng)而cloudera-scm-agent無法啟動(dòng)。
使用命令: systemctl status cloudera-scm-agent得到的反饋

● cloudera-scm-agent.service - LSB: Cloudera SCM AgentLoaded: loaded (/etc/rc.d/init.d/cloudera-scm-agent; bad; vendor preset: disabled)Active: failed (Result: exit-code) since Thu 2019-08-22 22:52:37 CST; 10h agoDocs: man:systemd-sysv-generator(8)Process: 16911 ExecStop=/etc/rc.d/init.d/cloudera-scm-agent stop (code=exited, status=0/SUCCESS)Process: 18991 ExecStart=/etc/rc.d/init.d/cloudera-scm-agent start (code=exited, status=1/FAILURE)Aug 22 22:52:37 test cloudera-scm-agent[18991]: install: cannot create directory ‘/var/run’: File exists Aug 22 22:52:37 test su[19009]: (to root) root on none Aug 22 22:52:37 test su[19009]: pam_systemd(su:session): Failed to connect to system bus: No such file or directory Aug 22 22:52:37 test su[19009]: pam_unix(su:session): session opened for user root by (uid=0) Aug 22 22:52:37 test su[19009]: pam_unix(su:session): session closed for user root Aug 22 22:52:37 test cloudera-scm-agent[18991]: Starting cloudera-scm-agent: [FAILED] Aug 22 22:52:37 test systemd[1]: cloudera-scm-agent.service: control process exited, code=exited status=1 Aug 22 22:52:37 test systemd[1]: Failed to start LSB: Cloudera SCM Agent. Aug 22 22:52:37 test systemd[1]: Unit cloudera-scm-agent.service entered failed state. Aug 22 22:52:37 test systemd[1]: cloudera-scm-agent.service failed.

查看日志

一開始看著這一堆問題的時(shí)候完全不知道發(fā)生了什么事情，決定去看日志
使用命令:tail -1000f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

Traceback (most recent call last):File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.0-py2.7.egg/cmf/monitor/firehose.py", line 116, in _sendself._port)File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 469, in __init__self.conn.connect()File "/usr/lib64/python2.7/httplib.py", line 824, in connectself.timeout, self.source_address)File "/usr/lib64/python2.7/socket.py", line 571, in create_connectionraise err error: [Errno 111] Connection refused

tm還是沒有看出些什么來，然后又看另外一個(gè)日志
使用命令:tail -100f /var/log/cloudera-scm-agent/cloudera-scm-agent.out

[root@test cloudera-scm-agent]# tail -10000f cloudera-scm-agent.out [13/Aug/2019 13:09:49 +0000] 16976 MainThread agent INFO SCM Agent Version: 5.14.0 [13/Aug/2019 13:09:49 +0000] 16976 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [13/Aug/2019 13:09:49 +0000] 16976 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent [13/Aug/2019 14:24:07 +0000] 32268 MainThread agent INFO SCM Agent Version: 5.14.0 [13/Aug/2019 14:24:07 +0000] 32268 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [13/Aug/2019 14:24:07 +0000] 32268 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent [22/Aug/2019 13:51:57 +0000] 1475 MainThread agent INFO SCM Agent Version: 5.14.0 [22/Aug/2019 13:51:57 +0000] 1475 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [22/Aug/2019 13:51:57 +0000] 1475 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent [22/Aug/2019 22:35:59 +0000] 17962 MainThread agent INFO SCM Agent Version: 5.14.0 [22/Aug/2019 22:35:59 +0000] 17962 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [22/Aug/2019 22:35:59 +0000] 17962 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent Unable to create the pidfile. [22/Aug/2019 22:52:37 +0000] 19011 MainThread agent INFO SCM Agent Version: 5.14.0 [22/Aug/2019 22:52:37 +0000] 19011 MainThread agent WARNING Expected mode 0751 for /run/cloudera-scm-agent but was 0755 [22/Aug/2019 22:52:37 +0000] 19011 MainThread agent INFO Re-using pre-existing directory: /run/cloudera-scm-agent Unable to create the pidfile. [23/Aug/2019 09:13:53 +0000] 23991 MainThread agent INFO SCM Agent Version: 5.14.0 [23/Aug/2019 09:13:53 +0000] 23991 MainThread agent WARNING Expected mode 0751 for /var/run/cloudera-scm-agent but was 0755 [23/Aug/2019 09:13:53 +0000] 23991 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent

現(xiàn)在有一些線索了，這里有這么一句Unable to create the pidfile，然后去百度了一下大部分的博客都是說在/run目錄下沒有cloudera-scm-agent這個(gè)目錄，只要?jiǎng)?chuàng)建一下就有了，然而我這里卻是有的。
完了，現(xiàn)在線索好像又?jǐn)嗟袅恕?/p>

事情還是不要想的太復(fù)雜

再次執(zhí)行systemctl status cloudera-scm-agent這個(gè)命令，卻在這里找到一個(gè)新線索

Process: 18991 ExecStart=/etc/rc.d/init.d/cloudera-scm-agent start (code=exited, status=1/FAILURE)Aug 22 22:52:37 test cloudera-scm-agent[18991]: install: cannot create directory ‘/var/run’: File exists Aug 22 22:52:37 test su[19009]: (to root) root on none Aug 22 22:52:37 test su[19009]: pam_systemd(su:session): Failed to connect to system bus: No such file or directory

這里說cannot create directory ‘/var/run’: File exists，那我就去/var下看了看，這個(gè)地方有一個(gè)軟連接由:
/var/run -> /run的，然后又百度了一下/run這個(gè)目錄是干嘛的，就一個(gè)臨時(shí)目錄，重啟后會(huì)刪掉的。
好吧，這時(shí)果斷把軟連接刪除掉。然后cloudera-scm-agent竟然啟動(dòng)成功了，看來問題就是出在這里。

事情還沒結(jié)束

進(jìn)入到cm后，在啟動(dòng)HDFS時(shí)又出問題了，namenode啟動(dòng)不了。查看日志上顯示:

Failed to start namenode. java.io.IOException: NameNode is not formatted.

好吧，我手動(dòng)到集群里面格式化一下namenode吧

接著，我啟動(dòng)Hive和Spark的過程中都發(fā)生錯(cuò)誤了，顯示/user目錄的讀寫權(quán)限變了，我需要重寫變更，接著就是Spark的History Server啟動(dòng)不了，查看日志是:

File does not exist: hdfs://xxx:8020/user/spark/spark2ApplicationHistory

那么沒有就手動(dòng)創(chuàng)建一下吧，終于所有事情都解決掉了。
哎…心累

總結(jié)

以上是生活随笔為你收集整理的大数据开发之CDH篇----cloudera-scm-agent启动不了后的一堆事的全部?jī)?nèi)容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網(wǎng)站內(nèi)容還不錯(cuò)，歡迎將生活随笔推薦給好友。