【Oracle】RAC11gR2Grid启动顺序及启动故障诊断思路
轉(zhuǎn)自：https://www.2cto.com/database/201605/507269.html
圖片來自：https://blog.csdn.net/zwjzqqb/article/details/80679508
從11gR2開始，Oracle RAC的架構(gòu)有了比較大的變化，集群層面相交于之前的版本有了比較大的變動(dòng)，原來的rac架構(gòu)基本上屬于cssd、crsd、evmd三大光禿禿的主干進(jìn)程，日志數(shù)量較少，對于rac無法啟動(dòng)原因，采用最原始的方法逐一查看各個(gè)進(jìn)程的日志也可找到無法啟動(dòng)的原因。然而從11gR2之后，集群層發(fā)生了比較大的變動(dòng)，以下是$GRID_HOME/log/rac1/下的目錄情況：
[grid@rac1 rac1]$ ls
acfs acfsrepl acfssec agent client crfmond cssd cvu evmd gnsd mdnsd racg
acfslog acfsreplroot admin alertrac1.log crflogd crsd ctssd diskmon gipcd gpnpd ohasd srvm
可以看到在這個(gè)目錄中的文件夾非常多，在rac無法啟動(dòng)的情況下，如果去所有日志下查看無法啟動(dòng)的原因無疑效率極低。所以我們需要一個(gè)比較明確的診斷思路。
OK，接下來進(jìn)入正題，希望可以為大家的日常診斷提供幫助。
第一步，在診斷Grid無法啟動(dòng)的情況之前我們需要先了解11gR2中Grid的啟動(dòng)流程，下面這張圖比較清晰的說明了現(xiàn)在Grid的啟動(dòng)順序：
?
?
從圖中我們可以看到，相比的原來Oracle 10g的集群架構(gòu)，11gR2有了比較大的改動(dòng)。具體的進(jìn)程作用在這里不再贅述，不了解的可以自己去惡補(bǔ)一下，這里只說進(jìn)程啟動(dòng)順序相關(guān)的內(nèi)容。在啟動(dòng)集群的過程中首先啟動(dòng)的是ohasd進(jìn)程，在ohasd進(jìn)程啟動(dòng)之后會(huì)啟動(dòng)4個(gè)agent：
1.cssd agent
以root用戶權(quán)限啟動(dòng)，負(fù)責(zé)啟動(dòng)cssd進(jìn)程。
2.orarootagent
以root用戶權(quán)限啟動(dòng)，負(fù)責(zé)啟動(dòng)以下這些守護(hù)進(jìn)程：crsd進(jìn)程、ctssd進(jìn)程、Diskmon進(jìn)程、acfs進(jìn)程。這些進(jìn)程也都是以root用戶權(quán)限啟動(dòng)。
3.oraagent
以grid用戶權(quán)限啟動(dòng)，負(fù)責(zé)mdnsd進(jìn)程、gipcd進(jìn)程、gpnpd進(jìn)程、evmd進(jìn)程、asm進(jìn)程（11gR2之后的asm在集群中被放置到了更底層，和之前版本區(qū)別較大）。
4.cssdmonitor。
以root用戶權(quán)限啟動(dòng)，負(fù)責(zé)cssdmonitor進(jìn)程的啟動(dòng)。
從圖中我們可以看到之后又由crsd進(jìn)程負(fù)責(zé)啟動(dòng)了兩個(gè)agent：orarootagent和oraagent（最后的進(jìn)程中我們可以看到兩個(gè)oraagent進(jìn)程，就是之前啟動(dòng)的那個(gè)加上這個(gè)），之后再由orarootagent和oraagent去負(fù)責(zé)啟動(dòng)之后的用戶資源，進(jìn)程啟動(dòng)到這里我認(rèn)為grid底層啟動(dòng)完畢，之后再由orarootagent和oraagent啟動(dòng)的資源出現(xiàn)的問題不再本文的討論范圍內(nèi)了。
第二步，我們已經(jīng)對grid的進(jìn)程啟動(dòng)順序進(jìn)行了梳理，之后對于grid無法啟動(dòng)的診斷也就變得簡單。我們只要通過ps -ef|grep /oracle/app/grid/product/11.2.0（$GRID_HOME）就可以了解到grid已經(jīng)啟動(dòng)到哪一步，哪些進(jìn)程已經(jīng)啟動(dòng)，哪些進(jìn)程還未啟動(dòng)，卡在了哪個(gè)進(jìn)程上，這樣我們就能快速找到應(yīng)該查看的日志。比如crsd進(jìn)程沒有啟動(dòng)，我們就可以通過查看$GRID_HOME/log/rac1/crsd目錄下的crsd.log來進(jìn)行查看，究竟在crsd進(jìn)程啟動(dòng)過程中遭遇了哪些錯(cuò)誤導(dǎo)致進(jìn)程無法正常啟動(dòng)。
舉例：
[grid@rac1 crsd]$ ps -ef|grep /oracle
root 15235 1 0 14:12 ? 00:00:06 /oracle/app/grid/product/11.2.0/bin/ohasd.bin reboot
grid 15356 1 0 14:12 ? 00:00:00 /oracle/app/grid/product/11.2.0/bin/oraagent.bin
grid 15367 1 0 14:12 ? 00:00:00 /oracle/app/grid/product/11.2.0/bin/mdnsd.bin
grid 15378 1 0 14:12 ? 00:00:02 /oracle/app/grid/product/11.2.0/bin/gpnpd.bin
grid 15388 1 2 14:12 ? 00:00:19 /oracle/app/grid/product/11.2.0/bin/gipcd.bin
root 15390 1 0 14:12 ? 00:00:00 /oracle/app/grid/product/11.2.0/bin/orarootagent.bin
root 15403 1 0 14:12 ? 00:00:08 /oracle/app/grid/product/11.2.0/bin/osysmond.bin
root 15477 1 0 14:12 ? 00:00:02 /oracle/app/grid/product/11.2.0/bin/ologgerd -M -d /oracle/app/grid/product/11.2.0/crf/db/rac1
root 15637 1 0 14:22 ? 00:00:00 /oracle/app/grid/product/11.2.0/bin/cssdmonitor
root 15665 1 0 14:22 ? 00:00:00 /oracle/app/grid/product/11.2.0/bin/cssdagent
grid 15676 1 0 14:22 ? 00:00:00 /oracle/app/grid/product/11.2.0/bin/ocssd.bin
grid 15730 13826 0 14:27 pts/1 00:00:00 grep /oracle
從以上的輸出我們就可以看到，此時(shí)grid無法啟動(dòng)的原因在于cssd進(jìn)程無法啟動(dòng)，所以我們直接查看ocssd.log，查看無法啟動(dòng)的原因，在日志中找到以下內(nèi)容：
2016-05-09 14:30:26.476: [ CSSD][1104030016]clssnmvDHBValidateNcopy: node 2, rac2, has a disk HB, but no network HB, DHB has rcfg 358258450, wrtcnt, 177436, LATS 10923264, lastSeqNo 177435, uniqueness 1462763679, timestamp 1462775426/10874194
可以看到是因?yàn)樗骄W(wǎng)出現(xiàn)了問題，導(dǎo)出有disk HB，而沒有network HB，修復(fù)私網(wǎng)問題后，集群可以正常啟動(dòng)。
第三步，附送一篇MOS文章：ID 1623340.1，里邊羅列了grid各個(gè)進(jìn)程無法啟動(dòng)的常見原因以及對應(yīng)的日志：
1.1.1. 集群狀態(tài)
查詢集群和守護(hù)進(jìn)程的狀態(tài):
$GRID_HOME/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
$GRID_HOME/bin/crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE rac1 Started
ora.crsd
1 ONLINE ONLINE rac1
ora.cssd
1 ONLINE ONLINE rac1
ora.cssdmonitor
1 ONLINE ONLINE rac1
ora.ctssd
1 ONLINE ONLINE rac1 OBSERVER
ora.diskmon
1 ONLINE ONLINE rac1
ora.drivers.acfs
1 ONLINE ONLINE rac1
ora.evmd
1 ONLINE ONLINE rac1
ora.gipcd
1 ONLINE ONLINE rac1
ora.gpnpd
1 ONLINE ONLINE rac1
ora.mdnsd
1 ONLINE ONLINE rac1
對于11.2.0.2 和以上的版本，會(huì)有以下兩個(gè)額外的進(jìn)程：
ora.cluster_interconnect.haip
1 ONLINE ONLINE rac1
ora.crf
1 ONLINE ONLINE rac1
對于11.2.0.3 以上的非EXADATA的系統(tǒng)，ora.diskmon會(huì)處于offline的狀態(tài)，如下：
ora.diskmon
1 OFFLINE OFFLINE rac1
對于 12c 以上的版本, 會(huì)出現(xiàn)ora.storage資源：
ora.storage
1 ONLINE ONLINE racnode1 STABLE
如果守護(hù)進(jìn)程 offline 我們可以通過以下命令啟動(dòng):
$GRID_HOME/bin/crsctl start res ora.crsd -init
1.1.2. 問題 1: OHASD 無法啟動(dòng)
由于 ohasd.bin 的責(zé)任是直接或者間接的啟動(dòng)集群所有的其它進(jìn)程，所以只有這個(gè)進(jìn)程正常啟動(dòng)了，其它的進(jìn)程才能起來，如果 ohasd.bin 的進(jìn)程沒有起來，當(dāng)我們檢查資源狀態(tài)的時(shí)候會(huì)報(bào)錯(cuò) CRS-4639 (Could not contact Oracle High Availability Services); 如果 ohasd.bin 已經(jīng)啟動(dòng)了，而再次嘗試啟時(shí)，錯(cuò)誤 CRS-4640 會(huì)出現(xiàn)；如果它啟動(dòng)失敗了，那么我們會(huì)看到以下的錯(cuò)誤信息：
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
自動(dòng)啟動(dòng) ohasd.bin 依賴于以下的配置：
1. 操作系統(tǒng)配置了正確的 run level:
OS 需要在 CRS 啟動(dòng)之前設(shè)置成指定的 run level 來確保 CRS 的正常啟動(dòng)。
我們可以通過以下方式找到 CRS 需要 OS 設(shè)置的 run level：
cat /etc/inittab|grep init.ohasd
h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1
以上例子展示了，CRS 需要 OS 運(yùn)行在 run level 3 或 5；請注意，由于操作系統(tǒng)的不同，CRS 啟動(dòng)需要的 OS 的 run level 也會(huì)不同。
找到當(dāng)前 OS 正在運(yùn)行的 run level:
who -r
2. "init.ohasd run" 啟動(dòng)
在 Linux/Unix 平臺(tái)上，由于"init.ohasd run" 是配置在 /etc/inittab中，進(jìn)程 init（進(jìn)程id 1，linux，Solars和HP-UX上為/sbin/init ，Aix上為/usr/sbin/init）會(huì)啟動(dòng)并且產(chǎn)生"init.ohasd run"進(jìn)程，如果這個(gè)過程失敗了，就不會(huì)有"init.ohasd run"的啟動(dòng)和運(yùn)行，ohasd.bin 也是無法啟動(dòng)的：
ps -ef|grep init.ohasd|grep -v grep
root 2279 1 0 18:14 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run
注意：Oracle Linux (OL6)以及 Red Hat Linux 6 (RHEL6) 已經(jīng)不再支持 inittab 了，所以 init.ohasd 會(huì)被配置在 /etc/init 中，并被 /etc/init 啟動(dòng)，盡管如此，我們還是應(yīng)該能看到進(jìn)程 "/etc/init.d/init.ohasd run" 被啟動(dòng)；
如果任何 rc Snncommand 的腳本（在 rcn.d 中，如 S98gcstartup）在啟動(dòng)的過程中掛死，此時(shí) init 的進(jìn)程可能無法啟動(dòng)"/etc/init.d/init.ohasd run"；您需要尋求 OS 廠商的幫助，找到為什么 Snncommand 腳本掛死或者無法正常啟動(dòng)的原因；
錯(cuò)誤"[ohasd()] CRS-0715:Oracle High Availability Service has timed out waiting for init.ohasd to be started." 可能會(huì)在 init.ohasd 無法在指定時(shí)間內(nèi)啟動(dòng)后出現(xiàn)
如果系統(tǒng)管理員無法在短期內(nèi)找到 init.ohasd 無法啟動(dòng)的原因，以下辦法可以作為一個(gè)臨時(shí)的解決辦法：
cd
nohup ./init.ohasd run &
3. Clusterware 自動(dòng)啟動(dòng)；--自動(dòng)啟動(dòng)默認(rèn)是開啟的
默認(rèn)情況下 CRS 自動(dòng)啟動(dòng)是開啟的，我們可以通過以下方式開啟：
$GRID_HOME/bin/crsctl enable crs
檢查這個(gè)功能是否被開啟:
$GRID_HOME/bin/crsctl config crs
如果以下信息被輸出在OS的日志中
Feb 29 16:20:36 racnode1 logger: Oracle Cluster Ready Services startup disabled.
Feb 29 16:20:36 racnode1 logger: Could not access /var/opt/oracle/scls_scr/racnode1/root/ohasdstr
原因是由于這個(gè)文件不存在或者不可訪問，產(chǎn)生這個(gè)問題的原因一般是人為的修改或者是打 GI 補(bǔ)丁的過程中使用了錯(cuò)誤的 opatch (如：使用 Solaris 平臺(tái)上的 opatch 在 Linux 上打補(bǔ)丁)
4. syslogd 啟動(dòng)并且 OS 能夠執(zhí)行 init 腳本 S96ohasd
節(jié)點(diǎn)啟動(dòng)之后，OS 可能停滯在一些其它的 Snn 的腳本上，所以可能沒有機(jī)會(huì)執(zhí)行到腳本 S96ohasd；如果是這種情況，我們不會(huì)在 OS 日志中看到以下信息
（aix /var/adm/syslog linux /var/log/messages）
Jan 20 20:46:51 rac1 logger: Oracle HA daemon is enabled for autostart.
如果在 OS 日志里看不到上面的信息，還有一種可能是 syslogd（(/usr/sbin/syslogd）沒有被完全啟動(dòng)。GRID 在這種情況下也是無法正常啟動(dòng)的，這種情況不適用于 AIX 的平臺(tái)。
為了了解 OS 啟動(dòng)之后是否能夠執(zhí)行 S96ohasd 腳本，可以按照以下的方法修改該腳本：
From:
case `$CAT $AUTOSTARTFILE` in
enable*)
$LOGERR "Oracle HA daemon is enabled for autostart."
To:
case `$CAT $AUTOSTARTFILE` in
enable*)
/bin/touch /tmp/ohasd.start."`date`"
$LOGERR "Oracle HA daemon is enabled for autostart."
重啟節(jié)點(diǎn)后，如果您沒有看到文件 /tmp/ohasd.start.timestamp 被創(chuàng)建，那么就是說 OS 停滯在其它的 Snn 的腳本上。如果您能看到 /tmp/ohasd.start.timestamp 生成了，但是"Oracle HA daemon is enabled for autostart"沒有寫入到messages 文件里，就是 syslogd 沒有被完全啟動(dòng)了。以上的兩種情況，您都需要尋求系統(tǒng)管理員的幫助，從 OS 的層面找到問題的原因，對于后一種情況，有個(gè)臨時(shí)的解決辦法是“休眠”2分鐘， 按照以下的方法修改 ohasd 腳本：
From:
case `$CAT $AUTOSTARTFILE` in
enable*)
$LOGERR "Oracle HA daemon is enabled for autostart."
To:
case `$CAT $AUTOSTARTFILE` in
enable*)
/bin/sleep 120
$LOGERR "Oracle HA daemon is enabled for autostart."
5. GRID_HOME 所在的文件系統(tǒng)在執(zhí)行初始化腳本 S96ohasd 的時(shí)候在線；正常情況下一旦 S96ohasd 執(zhí)行結(jié)束，我們會(huì)在 OS message 里看到以下信息：
Jan 20 20:46:51 rac1 logger: Oracle HA daemon is enabled for autostart.
..
Jan 20 20:46:57 rac1 logger: exec /ocw/grid/perl/bin/perl -I/ocw/grid/perl/lib /ocw/grid/bin/crswrapexece.pl /ocw/grid/crs/install/s_crsconfig_rac1_env.txt /ocw/grid/bin/ohasd.bin "reboot"
如果您只看到了第一行，沒有看到最后一行的信息，很可能是 GRID_HOME 所在的文件系統(tǒng)在腳本 S96ohasd 執(zhí)行的時(shí)候還沒有正常掛載。
6. Oracle Local Registry (OLR, $GRID_HOME/cdata/${HOSTNAME}.olr) 有效并可以正常讀寫
ls -l $GRID_HOME/cdata/*.olr
-rw------- 1 root oinstall 272756736 Feb 2 18:20 rac1.olr
如果 OLR 是不可讀寫的或者損壞的，我們會(huì)在 ohasd.log 中看到以下的相關(guān)信息
..
2010-01-24 22:59:10.470: [ default][1373676464] Initializing OLR
2010-01-24 22:59:10.472: [ OCROSD][1373676464]utopen:6m':failed in stat OCR file/disk /ocw/grid/cdata/rac1.olr, errno=2, os err string=No such file or directory
2010-01-24 22:59:10.472: [ OCROSD][1373676464]utopen:7:failed to open any OCR file/disk, errno=2, os err string=No such file or directory
2010-01-24 22:59:10.473: [ OCRRAW][1373676464]proprinit: Could not open raw device
2010-01-24 22:59:10.473: [ OCRAPI][1373676464]a_init:16!: Backend init unsuccessful : [26]
2010-01-24 22:59:10.473: [ CRSOCR][1373676464] OCR context init failure. Error: PROCL-26: Error while accessing the physical storage Operating System error [No such file or directory] [2]
2010-01-24 22:59:10.473: [ default][1373676464] OLR initalization failured, rc=26
2010-01-24 22:59:10.474: [ default][1373676464]Created alert : (:OHAS00106:) : Failed to initialize Oracle Local Registry
2010-01-24 22:59:10.474: [ default][1373676464][PANIC] OHASD exiting; Could not init OLR
或者
..
2010-01-24 23:01:46.275: [ OCROSD][1228334000]utread:3: Problem reading buffer 1907f000 buflen 4096 retval 0 phy_offset 102400 retry 5
2010-01-24 23:01:46.275: [ OCRRAW][1228334000]propriogid:1_1: Failed to read the whole bootblock. Assumes invalid format.
2010-01-24 23:01:46.275: [ OCRRAW][1228334000]proprioini: all disks are not OCR/OLR formatted
2010-01-24 23:01:46.275: [ OCRRAW][1228334000]proprinit: Could not open raw device
2010-01-24 23:01:46.275: [ OCRAPI][1228334000]a_init:16!: Backend init unsuccessful : [26]
2010-01-24 23:01:46.276: [ CRSOCR][1228334000] OCR context init failure. Error: PROCL-26: Error while accessing the physical storage
2010-01-24 23:01:46.276: [ default][1228334000] OLR initalization failured, rc=26
2010-01-24 23:01:46.276: [ default][1228334000]Created alert : (:OHAS00106:) : Failed to initialize Oracle Local Registry
2010-01-24 23:01:46.277: [ default][1228334000][PANIC] OHASD exiting; Could not init OLR
或者
..
2010-11-07 03:00:08.932: [ default][1] Created alert : (:OHAS00102:) : OHASD is not running as privileged user
2010-11-07 03:00:08.932: [ default][1][PANIC] OHASD exiting: must be run as privileged user
或者
ohasd.bin comes up but output of "crsctl stat res -t -init"shows no resource, and "ocrconfig -local -manualbackup" fails
或者
..
2010-08-04 13:13:11.102: [ CRSPE][35] Resources parsed
2010-08-04 13:13:11.103: [ CRSPE][35] Server [] has been registered with the PE data model
2010-08-04 13:13:11.103: [ CRSPE][35] STARTUPCMD_REQ = false:
2010-08-04 13:13:11.103: [ CRSPE][35] Server [] has changed state from [Invalid/unitialized] to [VISIBLE]
2010-08-04 13:13:11.103: [ CRSOCR][31] Multi Write Batch processing...
2010-08-04 13:13:11.103: [ default][35] Dump State Starting ...
..
2010-08-04 13:13:11.112: [ CRSPE][35] SERVERS:
:VISIBLE:address{{Absolute|Node:0|Process:-1|Type:1}}; recovered state:VISIBLE. Assigned to no pool
------------- SERVER POOLS:
Free [min:0][max:-1][importance:0] NO SERVERS ASSIGNED
2010-08-04 13:13:11.113: [ CRSPE][35] Dumping ICE contents...:ICE operation count: 0
2010-08-04 13:13:11.113: [ default][35] Dump State Done.
解決辦法就是使用下面的命令，恢復(fù)一個(gè)好的備份 "ocrconfig -local -restore "。
默認(rèn)情況下，OLR 在系統(tǒng)安裝結(jié)束后會(huì)自動(dòng)的備份在 $GRID_HOME/cdata/$HOST/backup_$TIME_STAMP.olr 。
7. ohasd.bin可以正常的訪問到網(wǎng)絡(luò)的 socket 文件:
2010-06-29 10:31:01.570: [ COMMCRS][1206901056]clsclisten: Permission denied for (ADDRESS=(PROTOCOL=ipc)(KEY=procr_local_conn_0_PROL))
2010-06-29 10:31:01.571: [ OCRSRV][1217390912]th_listen: CLSCLISTEN failed clsc_ret= 3, addr= [(ADDRESS=(PROTOCOL=ipc)(KEY=procr_local_conn_0_PROL))]
2010-06-29 10:31:01.571: [ OCRSRV][3267002960]th_init: Local listener did not reach valid state
在 Grid Infrastructure 環(huán)境中，和 ohasd 有關(guān)的 socket 文件屬主應(yīng)該是 root 用戶，但是在 Oracle Restart 的環(huán)境中，他們應(yīng)該是屬于 grid 用戶的，關(guān)于更多的關(guān)于網(wǎng)絡(luò) socket 文件權(quán)限和屬主，請參考章節(jié)"網(wǎng)絡(luò) socket 文件，屬主和權(quán)限" 給出的例子.
8. ohasd.bin 能夠訪問日志文件的位置:
OS messages/syslog 顯示以下信息:
Feb 20 10:47:08 racnode1 OHASD[9566]: OHASD exiting; Directory /ocw/grid/log/racnode1/ohasd not found.
請參考章節(jié)"日志位置, 屬主和權(quán)限"部分的例子，并確定這些必要的目錄是否有丟失的，并且是按照正確的權(quán)限和屬主創(chuàng)建的。
9. 節(jié)點(diǎn)啟動(dòng)后，在 SUSE Linux 的系統(tǒng)上，ohasd 可能無法啟動(dòng)，此問題請參考 note 1325718.1 - OHASD not Starting After Reboot on SLES
10. OHASD 無法啟動(dòng)，使用 "ps -ef| grep ohasd.bin" 顯示 ohasd.bin 的進(jìn)程已經(jīng)啟動(dòng)，但是 $GRID_HOME/log//ohasd/ohasd.log 在好幾分鐘之后都沒有任何信息更新,使用 OS 的 truss 工具 可以看到該進(jìn)程一致在循環(huán)的執(zhí)行關(guān)閉從未被打開的文件句柄的操作:
..
15058/1: 0.1995 close(2147483646) Err#9 EBADF
15058/1: 0.1996 close(2147483645) Err#9 EBADF
..
通過 ohasd.bin 的 Call stack ，可以看到以下信息:
_close sclssutl_closefiledescriptors main ..
這是由于 bug 11834289 導(dǎo)致的， 該問題在 11.2.0.3 和之上的版本已經(jīng)被修復(fù),該 bug 的其它癥狀還有：集群的進(jìn)程無法啟動(dòng)，而且做 call stack 和 truss 查看的時(shí)候也會(huì)看到相同的情況(循環(huán)的執(zhí)行 OS 函數(shù) "close") . 如果該 bug 發(fā)生在啟動(dòng)其它的資源時(shí)，我們會(huì)看到錯(cuò)誤信息： "CRS-5802: Unable to start the agent process" 提示。
11. 其它的一些潛在的原因和解決辦法請參見 note 1069182.1 - OHASD Failed to Start: Inappropriate ioctl for device
12. ohasd.bin 正常啟動(dòng)，但是, "crsctl check crs" 只顯示以下一行信息:
CRS-4638: Oracle High Availability Services is online
并且命令 "crsctl stat res -p -init" 無法顯示任何信息
這個(gè)問題是由于 OLR 損壞導(dǎo)致的，請參考 note 1193643.1 進(jìn)行恢復(fù)。
13. 如果 ohasd 仍然無法啟動(dòng)，請參見 ohasd 的日志 /log//ohasd/ohasd.log 和 ohasdOUT.log 來獲取更多的信息；
1.1.3. 問題 2: OHASD Agents 未啟動(dòng)
OHASD.BIN 會(huì)啟動(dòng) 4 個(gè) agents/monitors 來啟動(dòng)其它的資源:
oraagent: 負(fù)責(zé)啟動(dòng) ora.asm, ora.evmd, ora.gipcd, ora.gpnpd, ora.mdnsd 等
orarootagent: 負(fù)責(zé)啟動(dòng) ora.crsd, ora.ctssd, ora.diskmon, ora.drivers.acfs 等
cssdagent / cssdmonitor: 負(fù)責(zé)啟動(dòng) ora.cssd(對應(yīng) ocssd.bin) 和 ora.cssdmonitor(對應(yīng) cssdmonitor)
如果 ohasd.bin 不能正常地啟動(dòng)以上任何一個(gè) agents，集群都無法運(yùn)行在正常的狀態(tài)。
1. 通常情況下，agents 無法啟動(dòng)的原因是 agent 的日志或者日志所在的目錄沒有正確設(shè)置屬主和權(quán)限。
關(guān)于日志文件和文件夾的權(quán)限和屬主設(shè)置，請參見章節(jié) "日志文件位置, 屬主和權(quán)限" 中的介紹。
2. 如果 agent 的二進(jìn)制文件（oraagent.bin 或者 orarootagent.bin 等）損壞, agent 也將無法啟動(dòng)，從而導(dǎo)致相關(guān)的資源也無法啟動(dòng)：
2011-05-03 11:11:13.189
[ohasd(25303)]CRS-5828:Could not start agent '/ocw/grid/bin/orarootagent_grid'. Details at (:CRSAGF00130:) {0:0:2} in /ocw/grid/log/racnode1/ohasd/ohasd.log.
2011-05-03 12:03:17.491: [ AGFW][1117866336] {0:0:184} Created alert : (:CRSAGF00130:) : Failed to start the agent /ocw/grid/bin/orarootagent_grid
2011-05-03 12:03:17.491: [ AGFW][1117866336] {0:0:184} Agfw Proxy Server sending the last reply to PE for message:RESOURCE_START[ora.diskmon 1 1] ID 4098:403
2011-05-03 12:03:17.491: [ AGFW][1117866336] {0:0:184} Can not stop the agent: /ocw/grid/bin/orarootagent_grid because pid is not initialized
..
2011-05-03 12:03:17.492: [ CRSPE][1128372576] {0:0:184} Fatal Error from AGFW Proxy: Unable to start the agent process
2011-05-03 12:03:17.492: [ CRSPE][1128372576] {0:0:184} CRS-2674: Start of 'ora.diskmon' on 'racnode1' failed
..
2011-06-27 22:34:57.805: [ AGFW][1131669824] {0:0:2} Created alert : (:CRSAGF00123:) : Failed to start the agent process: /ocw/grid/bin/cssdagent Category: -1 Operation: fail Loc: canexec2 OS error: 0 Other : no exe permission, file [/ocw/grid/bin/cssdagent]
2011-06-27 22:34:57.805: [ AGFW][1131669824] {0:0:2} Created alert : (:CRSAGF00126:) : Agent start failed
..
2011-06-27 22:34:57.806: [ AGFW][1131669824] {0:0:2} Created alert : (:CRSAGF00123:) : Failed to start the agent process: /ocw/grid/bin/cssdmonitor Category: -1 Operation: fail Loc: canexec2 OS error: 0 Other : no exe permission, file [/ocw/grid/bin/cssdmonitor]
解決辦法： 您可以和正常節(jié)點(diǎn)上的 agent 文件進(jìn)行比較，并且恢復(fù)一個(gè)好的副本回來。
1.1.4. 問題 3: OCSSD.BIN 無法啟動(dòng)
cssd.bin 的正常啟動(dòng)依賴于以下幾個(gè)必要的條件:
1. GPnP profile 可正常讀寫 - gpnpd 需要完全正常啟動(dòng)來為profile服務(wù)。
如果 ocssd.bin 能夠正常的獲取 profile，通常情況下，我們會(huì)在 ocssd.log 中看到以下類似的信息：
2010-02-02 18:00:16.251: [ GPnP][408926240]clsgpnpm_exchange: [at clsgpnpm.c:1175] Calling "ipc://GPNPD_rac1", try 4 of 500...
2010-02-02 18:00:16.263: [ GPnP][408926240]clsgpnp_profileVerifyForCall: [at clsgpnp.c:1867] Result: (87) CLSGPNP_SIG_VALPEER. Profile verified. prf=0x165160d0
2010-02-02 18:00:16.263: [ GPnP][408926240]clsgpnp_profileGetSequenceRef: [at clsgpnp.c:841] Result: (0) CLSGPNP_OK. seq of p=0x165160d0 is '6'=6
2010-02-02 18:00:16.263: [ GPnP][408926240]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2186] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_rac1" disco ""
否則，我們會(huì)看到以下信息顯示在 ocssd.log 中。
2010-02-03 22:26:17.057: [ GPnP][3852126240]clsgpnpm_connect: [at clsgpnpm.c:1100] GIPC gipcretConnectionRefused (29) gipcConnect(ipc-ipc://GPNPD_rac1)
2010-02-03 22:26:17.057: [ GPnP][3852126240]clsgpnpm_connect: [at clsgpnpm.c:1101] Result: (48) CLSGPNP_COMM_ERR. Failed to connect to call url "ipc://GPNPD_rac1"
2010-02-03 22:26:17.057: [ GPnP][3852126240]clsgpnp_getProfileEx: [at clsgpnp.c:546] Result: (13) CLSGPNP_NO_DAEMON. Can't get GPnP service profile from local GPnP daemon
2010-02-03 22:26:17.057: [ default][3852126240]Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2010-02-03 22:26:17.057: [ CSSD][3852126240]clsgpnp_getProfile failed, rc(13)
2. Voting Disk 可以正常讀寫
在 11gR2 的版本中， ocssd.bin 通過 GPnP profile 中的記錄獲取 Voting disk 的信息, 如果沒有足夠多的選舉盤是可讀寫的，那么 ocssd.bin 會(huì)終止掉自己。
2010-02-03 22:37:22.212: [ CSSD][2330355744]clssnmReadDiscoveryProfile: voting file discovery string(/share/storage/di*)
..
2010-02-03 22:37:22.227: [ CSSD][1145538880]clssnmvDiskVerify: Successful discovery of 0 disks
2010-02-03 22:37:22.227: [ CSSD][1145538880]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2010-02-03 22:37:22.227: [ CSSD][1145538880]clssnmvFindInitialConfigs: No voting files found
2010-02-03 22:37:22.228: [ CSSD][1145538880]###################################
2010-02-03 22:37:22.228: [ CSSD][1145538880]clssscExit: CSSD signal 11 in thread clssnmvDDiscThread
如果所有節(jié)點(diǎn)上的 ocssd.bin 因?yàn)橐韵洛e(cuò)誤無法啟動(dòng)，這是因?yàn)?voting file 正在被修改:
2010-05-02 03:11:19.033: [ CSSD][1197668093]clssnmCompleteInitVFDiscovery: Detected voting file add in progress for CIN 0:1134513465:0, waiting for configuration to complete 0:1134513098:0
解決的辦法是，參照 note 1364971.1 中的步驟，以 exclusive 模式啟動(dòng) ocssd.bin。
如果選舉盤的位置是非 ASM 的設(shè)備，它的權(quán)限和屬主應(yīng)該是如下顯示：
-rw-r----- 1 ogrid oinstall 21004288 Feb 4 09:13 votedisk1
3. 網(wǎng)絡(luò)功能是正常的，并且域名解析能夠正常工作:
如果 ocssd.bin 無法正常的綁定到任何網(wǎng)絡(luò)上，我們會(huì)在 ocssd.log 中看到以下類似的日志信息：
2010-02-03 23:26:25.804: [GIPCXCPT][1206540320]gipcmodGipcPassInitializeNetwork: failed to find any interfaces in clsinet, ret gipcretFail (1)
2010-02-03 23:26:25.804: [GIPCGMOD][1206540320]gipcmodGipcPassInitializeNetwork: EXCEPTION[ ret gipcretFail (1) ] failed to determine host from clsinet, using default
..
2010-02-03 23:26:25.810: [ CSSD][1206540320]clsssclsnrsetup: gipcEndpoint failed, rc 39
2010-02-03 23:26:25.811: [ CSSD][1206540320]clssnmOpenGIPCEndp: failed to listen on gipc addr gipc://rac1:nm_eotcs- ret 39
2010-02-03 23:26:25.811: [ CSSD][1206540320]clssscmain: failed to open gipc endp
如果私網(wǎng)上出現(xiàn)了聯(lián)通性的故障（包含多播功能關(guān)閉），我們會(huì)在 ocssd.log 中看到以下類似的日志信息：
2010-09-20 11:52:54.014: [ CSSD][1103055168]clssnmvDHBValidateNCopy: node 1, racnode1, has a disk HB, but no network HB, DHB has rcfg 180441784, wrtcnt, 453, LATS 328297844, lastSeqNo 452, uniqueness 1284979488, timestamp 1284979973/329344894
2010-09-20 11:52:54.016: [ CSSD][1078421824]clssgmWaitOnEventValue: after CmInfo State val 3, eval 1 waited 0
.. >>>> after a long delay
2010-09-20 12:02:39.578: [ CSSD][1103055168]clssnmvDHBValidateNCopy: node 1, racnode1, has a disk HB, but no network HB, DHB has rcfg 180441784, wrtcnt, 1037, LATS 328883434, lastSeqNo 1036, uniqueness 1284979488, timestamp 1284980558/329930254
2010-09-20 12:02:39.895: [ CSSD][1107286336]clssgmExecuteClientRequest: MAINT recvd from proc 2 (0xe1ad870)
2010-09-20 12:02:39.895: [ CSSD][1107286336]clssgmShutDown: Received abortive shutdown request from client.
2010-09-20 12:02:39.895: [ CSSD][1107286336]###################################
2010-09-20 12:02:39.895: [ CSSD][1107286336]clssscExit: CSSD aborting from thread GMClientListener
2010-09-20 12:02:39.895: [ CSSD][1107286336]###################################
驗(yàn)證網(wǎng)絡(luò)是否正常，請參見：note 1054902.1
$GRID_HOME/bin/lsnodes -n
racnode1 1
racnode1 0
如果第三方的集群管理軟件沒有完全正常啟動(dòng)，我們在 ocssd.log 中看到以下類似的日志信息：
2010-08-30 18:28:13.207: [ CSSD][36]clssnm_skgxninit: skgxncin failed, will retry
2010-08-30 18:28:14.207: [ CSSD][36]clssnm_skgxnmon: skgxn init failed
2010-08-30 18:28:14.208: [ CSSD][36]###################################
2010-08-30 18:28:14.208: [ CSSD][36]clssscExit: CSSD signal 11 in thread skgxnmon
未安裝集群管理軟件之前，請使用 grid 用戶執(zhí)行以下操作驗(yàn)證:
$INSTALL_SOURCE/install/lsnodes -v
5. 在錯(cuò)誤的 GRID_HOME 下執(zhí)行命令"crsctl"
命令"crsctl" 必須在正確的 GRID_HOME 下執(zhí)行，才能正常啟動(dòng)其它進(jìn)程,否則我們會(huì)看到以下的錯(cuò)誤信息提示：
2012-11-14 10:21:44.014: [ CSSD][1086675264]ASSERT clssnm1.c 3248
2012-11-14 10:21:44.014: [ CSSD][1086675264](:CSSNM00056:)clssnmvStartDiscovery: Terminating because of the release version(11.2.0.2.0) of this node being lesser than the active version(11.2.0.3.0) that the cluster is at
2012-11-14 10:21:44.014: [ CSSD][1086675264]###################################
2012-11-14 10:21:44.014: [ CSSD][1086675264]clssscExit: CSSD aborting from thread clssnmvDDiscThread#
1.1.5. 問題 4: CRSD.BIN 無法啟動(dòng)
crsd.bin 的正常啟動(dòng)依賴于以下幾個(gè)必要的條件:
1. ocssd 已經(jīng)完全正常啟動(dòng)
如果 ocssd.bin 沒有完全正常啟動(dòng)，我們會(huì)在 crsd.log 中看到以下提示信息:
2010-02-03 22:37:51.638: [ CSSCLNT][1548456880]clssscConnect: gipc request failed with 29 (0x16)
2010-02-03 22:37:51.638: [ CSSCLNT][1548456880]clsssInitNative: connect failed, rc 29
2010-02-03 22:37:51.639: [ CRSRTI][1548456880] CSS is not ready. Received status 3 from CSS. Waiting for good status ..
2. OCR 可以正常讀寫
如果 OCR 保存在 ASM 中,那么 ora.asm 資源(ASM 實(shí)例) 必須已經(jīng)啟動(dòng)而且 OCR 所在的磁盤組必須已經(jīng)被掛載,否則我們在 crsd.log 會(huì)看到以下的類似信息：
2010-02-03 22:22:55.186: [ OCRASM][2603807664]proprasmo: Error in open/create file in dg [GI]
[ OCRASM][2603807664]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
ORA-15077: could not locate ASM instance serving a required diskgroup
2010-02-03 22:22:55.189: [ OCRASM][2603807664]proprasmo: kgfoCheckMount returned [7]
2010-02-03 22:22:55.189: [ OCRASM][2603807664]proprasmo: The ASM instance is down
2010-02-03 22:22:55.190: [ OCRRAW][2603807664]proprioo: Failed to open [+GI]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2010-02-03 22:22:55.190: [ OCRRAW][2603807664]proprioo: No OCR/OLR devices are usable
2010-02-03 22:22:55.190: [ OCRASM][2603807664]proprasmcl: asmhandle is NULL
2010-02-03 22:22:55.190: [ OCRRAW][2603807664]proprinit: Could not open raw device
2010-02-03 22:22:55.190: [ OCRASM][2603807664]proprasmcl: asmhandle is NULL
2010-02-03 22:22:55.190: [ OCRAPI][2603807664]a_init:16!: Backend init unsuccessful : [26]
2010-02-03 22:22:55.190: [ CRSOCR][2603807664] OCR context init failure. Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
ORA-15077: could not locate ASM instance serving a required diskgroup
] [7]
2010-02-03 22:22:55.190: [ CRSD][2603807664][PANIC] CRSD exiting: Could not init OCR, code: 26
注意：在11.2 的版本中 ASM 會(huì)比 crsd.bin 先啟動(dòng)，并且會(huì)把含有 OCR 的磁盤組自動(dòng)掛載。
如果您的 OCR 在非 ASM 的存儲(chǔ)中，該文件的屬主和權(quán)限如下：
-rw-r----- 1 root oinstall 272756736 Feb 3 23:24 ocr
如果 OCR 是在非 ASM 的存儲(chǔ)中，并且不能被正常訪問，在 crsd.log 會(huì)看到以下的類似信息
2010-02-03 23:14:33.583: [ OCROSD][2346668976]utopen:7:failed to open any OCR file/disk, errno=2, os err string=No such file or directory
2010-02-03 23:14:33.583: [ OCRRAW][2346668976]proprinit: Could not open raw device
2010-02-03 23:14:33.583: [ default][2346668976]a_init:7!: Backend init unsuccessful : [26]
2010-02-03 23:14:34.587: [ OCROSD][2346668976]utopen:6m':failed in stat OCR file/disk /share/storage/ocr, errno=2, os err string=No such file or directory
2010-02-03 23:14:34.587: [ OCROSD][2346668976]utopen:7:failed to open any OCR file/disk, errno=2, os err string=No such file or directory
2010-02-03 23:14:34.587: [ OCRRAW][2346668976]proprinit: Could not open raw device
2010-02-03 23:14:34.587: [ default][2346668976]a_init:7!: Backend init unsuccessful : [26]
2010-02-03 23:14:35.589: [ CRSD][2346668976][PANIC] CRSD exiting: OCR device cannot be initialized, error: 1:26
如果 OCR 是壞掉了，在 crsd.log 會(huì)看到以下的類似信息:
2010-02-03 23:19:38.417: [ default][3360863152]a_init:7!: Backend init unsuccessful : [26]
2010-02-03 23:19:39.429: [ OCRRAW][3360863152]propriogid:1_2: INVALID FORMAT
2010-02-03 23:19:39.429: [ OCRRAW][3360863152]proprioini: all disks are not OCR/OLR formatted
2010-02-03 23:19:39.429: [ OCRRAW][3360863152]proprinit: Could not open raw device
2010-02-03 23:19:39.429: [ default][3360863152]a_init:7!: Backend init unsuccessful : [26]
2010-02-03 23:19:40.432: [ CRSD][3360863152][PANIC] CRSD exiting: OCR device cannot be initialized, error: 1:26
如果您的 grid 用戶的權(quán)限或者所在組發(fā)生了變化，盡管 ASM 還是可以訪問的，在 crsd.log 會(huì)看到以下的類似信息:
2010-03-10 11:45:12.510: [ OCRASM][611467760]proprasmo: Error in open/create file in dg [SYSTEMDG]
[ OCRASM][611467760]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=1031, loc=kgfokge
ORA-01031: insufficient privileges
2010-03-10 11:45:12.528: [ OCRASM][611467760]proprasmo: kgfoCheckMount returned [7]
2010-03-10 11:45:12.529: [ OCRASM][611467760]proprasmo: The ASM instance is down
2010-03-10 11:45:12.529: [ OCRRAW][611467760]proprioo: Failed to open [+SYSTEMDG]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2010-03-10 11:45:12.529: [ OCRRAW][611467760]proprioo: No OCR/OLR devices are usable
2010-03-10 11:45:12.529: [ OCRASM][611467760]proprasmcl: asmhandle is NULL
2010-03-10 11:45:12.529: [ OCRRAW][611467760]proprinit: Could not open raw device
2010-03-10 11:45:12.529: [ OCRASM][611467760]proprasmcl: asmhandle is NULL
2010-03-10 11:45:12.529: [ OCRAPI][611467760]a_init:16!: Backend init unsuccessful : [26]
2010-03-10 11:45:12.530: [ CRSOCR][611467760] OCR context init failure. Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=1031, loc=kgfokge
ORA-01031: insufficient privileges
] [7]
如果 GRID_HOME 下的 oracle 二進(jìn)制文件的屬主或者權(quán)限錯(cuò)誤，盡管 ASM 正常啟動(dòng)并運(yùn)行，在 crsd.log 會(huì)看到以下的類似信息：
2012-03-04 21:34:23.139: [ OCRASM][3301265904]proprasmo: Error in open/create file in dg [OCR]
[ OCRASM][3301265904]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=12547, loc=kgfokge
2012-03-04 21:34:23.139: [ OCRASM][3301265904]ASM Error Stack : ORA-12547: TNS:lost contact
2012-03-04 21:34:23.633: [ OCRASM][3301265904]proprasmo: kgfoCheckMount returned [7]
2012-03-04 21:34:23.633: [ OCRASM][3301265904]proprasmo: The ASM instance is down
2012-03-04 21:34:23.634: [ OCRRAW][3301265904]proprioo: Failed to open [+OCR]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2012-03-04 21:34:23.634: [ OCRRAW][3301265904]proprioo: No OCR/OLR devices are usable
2012-03-04 21:34:23.635: [ OCRASM][3301265904]proprasmcl: asmhandle is NULL
2012-03-04 21:34:23.636: [ GIPC][3301265904] gipcCheckInitialization: possible incompatible non-threaded init from [prom.c : 690], original from [clsss.c : 5326]
2012-03-04 21:34:23.639: [ default][3301265904]clsvactversion:4: Retrieving Active Version from local storage.
2012-03-04 21:34:23.643: [ OCRRAW][3301265904]proprrepauto: The local OCR configuration matches with the configuration published by OCR Cache Writer. No repair required.
2012-03-04 21:34:23.645: [ OCRRAW][3301265904]proprinit: Could not open raw device
2012-03-04 21:34:23.646: [ OCRASM][3301265904]proprasmcl: asmhandle is NULL
2012-03-04 21:34:23.650: [ OCRAPI][3301265904]a_init:16!: Backend init unsuccessful : [26]
2012-03-04 21:34:23.651: [ CRSOCR][3301265904] OCR context init failure. Error: PROC-26: Error while accessing the physical storage
ORA-12547: TNS:lost contact
2012-03-04 21:34:23.652: [ CRSMAIN][3301265904] Created alert : (:CRSD00111:) : Could not init OCR, error: PROC-26: Error while accessing the physical storage
ORA-12547: TNS:lost contact
2012-03-04 21:34:23.652: [ CRSD][3301265904][PANIC] CRSD exiting: Could not init OCR, code: 26
正常的 GRID_HOME 下該文件的屬主和權(quán)限應(yīng)該是如下顯示：
-rwsr-s--x 1 grid oinstall 184431149 Feb 2 20:37 /ocw/grid/bin/oracle
如果 OCR 文件或者它的鏡像文件無法正常訪問 (可能是 ASM 已經(jīng)啟動(dòng), 但是 OCR/mirror 所在的磁盤組沒有掛載),在 crsd.log 會(huì)看到以下的類似信息:
2010-05-11 11:16:38.578: [ OCRASM][18]proprasmo: Error in open/create file in dg [OCRMIR]
[ OCRASM][18]SLOS : SLOS: cat=8, opn=kgfoOpenFile01, dep=15056, loc=kgfokge
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +OCRMIR.255.4294967295
ORA-17503: ksfdopn:2 Failed to open file +OCRMIR.255.4294967295
ORA-15001: diskgroup "OCRMIR
..
2010-05-11 11:16:38.647: [ OCRASM][18]proprasmo: kgfoCheckMount returned [6]
2010-05-11 11:16:38.648: [ OCRASM][18]proprasmo: The ASM disk group OCRMIR is not found or not mounted
2010-05-11 11:16:38.648: [ OCRASM][18]proprasmdvch: Failed to open OCR location [+OCRMIR] error [26]
2010-05-11 11:16:38.648: [ OCRRAW][18]propriodvch: Error [8] returned device check for [+OCRMIR]
2010-05-11 11:16:38.648: [ OCRRAW][18]dev_replace: non-master could not verify the new disk (8)
[ OCRSRV][18]proath_invalidate_action: Failed to replace [+OCRMIR] [8]
[ OCRAPI][18]procr_ctx_set_invalid_no_abort: ctx set to invalid
..
2010-05-11 11:16:46.587: [ OCRMAS][19]th_master:91: Comparing device hash ids between local and master failed
2010-05-11 11:16:46.587: [ OCRMAS][19]th_master:91 Local dev (1862408427, 1028247821, 0, 0, 0)
2010-05-11 11:16:46.587: [ OCRMAS][19]th_master:91 Master dev (1862408427, 1859478705, 0, 0, 0)
2010-05-11 11:16:46.587: [ OCRMAS][19]th_master:9: Shutdown CacheLocal. my hash ids don't match
[ OCRAPI][19]procr_ctx_set_invalid_no_abort: ctx set to invalid
[ OCRAPI][19]procr_ctx_set_invalid: aborting...
2010-05-11 11:16:46.587: [ CRSD][19] Dump State Starting ...
3. crsd.bin 的進(jìn)程號文件（/crs/init/<節(jié)點(diǎn)名>.pid）存在，但是卻指向其它的進(jìn)程
如果進(jìn)程號文件不存在,在日志 $GRID_HOME/log/$HOST/agent/ohasd/orarootagent_root/orarootagent_root.log 我們會(huì)看到以下的提示信息:
2010-02-14 17:40:57.927: [ora.crsd][1243486528] [check] PID FILE doesn't exist.
..
2010-02-14 17:41:57.927: [ clsdmt][1092499776]Creating PID [30269] file for home /ocw/grid host racnode1 bin crs to /ocw/grid/crs/init/
2010-02-14 17:41:57.927: [ clsdmt][1092499776]Error3 -2 writing PID [30269] to the file []
2010-02-14 17:41:57.927: [ clsdmt][1092499776]Failed to record pid for CRSD
2010-02-14 17:41:57.927: [ clsdmt][1092499776]Terminating process
2010-02-14 17:41:57.927: [ default][1092499776] CRSD exiting on stop request from clsdms_thdmai
解決辦法，我們可以手工創(chuàng)建一個(gè)進(jìn)程號文件：使用 grid 用戶執(zhí)行 "touch" 命令，然后重新啟動(dòng) ora.crsd 資源。
如果進(jìn)程號文件存在，但是記錄的 PID 是指向了其它的進(jìn)程，而不是 crsd.bin 的進(jìn)程，在日志 $GRID_HOME/log/$HOST/agent/ohasd/orarootagent_root/orarootagent_root.log 我們會(huì)看到以下的提示信息:
2011-04-06 15:53:38.777: [ora.crsd][1160390976] [check] PID will be looked for in /ocw/grid/crs/init/racnode1.pid
2011-04-06 15:53:38.778: [ora.crsd][1160390976] [check] PID which will be monitored will be 1535 >> 1535 is output of "cat /ocw/grid/crs/init/racnode1.pid"
2011-04-06 15:53:38.965: [ COMMCRS][1191860544]clsc_connect: (0x2aaab400b0b0) no listener at (ADDRESS=(PROTOCOL=ipc)(KEY=racnode1DBG_CRSD))
[ clsdmc][1160390976]Fail to connect (ADDRESS=(PROTOCOL=ipc)(KEY=racnode1DBG_CRSD)) with status 9
2011-04-06 15:53:38.966: [ora.crsd][1160390976] [check] Error = error 9 encountered when connecting to CRSD
2011-04-06 15:53:39.023: [ora.crsd][1160390976] [check] Calling PID check for daemon
2011-04-06 15:53:39.023: [ora.crsd][1160390976] [check] Trying to check PID = 1535
2011-04-06 15:53:39.203: [ora.crsd][1160390976] [check] PID check returned ONLINE CLSDM returned OFFLINE
2011-04-06 15:53:39.203: [ora.crsd][1160390976] [check] DaemonAgent::check returned 5
2011-04-06 15:53:39.203: [ AGFW][1160390976] check for resource: ora.crsd 1 1 completed with status: FAILED
2011-04-06 15:53:39.203: [ AGFW][1170880832] ora.crsd 1 1 state changed from: UNKNOWN to: FAILED
..
2011-04-06 15:54:10.511: [ AGFW][1167522112] ora.crsd 1 1 state changed from: UNKNOWN to: CLEANING
..
2011-04-06 15:54:10.513: [ora.crsd][1146542400] [clean] Trying to stop PID = 1535
..
2011-04-06 15:54:11.514: [ora.crsd][1146542400] [clean] Trying to check PID = 1535
在 OS 層面檢查該問題:
ls -l /ocw/grid/crs/init/*pid
-rwxr-xr-x 1 ogrid oinstall 5 Feb 17 11:00 /ocw/grid/crs/init/racnode1.pid
cat /ocw/grid/crs/init/*pid
1535
ps -ef| grep 1535
root 1535 1 0 Mar30 ? 00:00:00 iscsid >> 注意：進(jìn)程 1535 不是 crsd.bin
解決辦法是，使用 root 用戶，創(chuàng)建一個(gè)空的進(jìn)程號文件，然后重啟資源 ora.crsd：
# > $GRID_HOME/crs/init/.pid
# $GRID_HOME/bin/crsctl stop res ora.crsd -init
# $GRID_HOME/bin/crsctl start res ora.crsd -init
4. 網(wǎng)絡(luò)功能是正常的，并且域名解析能夠正常工作：
如果網(wǎng)絡(luò)功能不正常，ocssd.bin 進(jìn)程仍然可能被啟動(dòng), 但是 crsd.bin 可能會(huì)失敗，同時(shí)在 crsd.log 中會(huì)提示以下信息:
2010-02-03 23:34:28.412: [ GPnP][2235814832]clsgpnp_Init: [at clsgpnp0.c:837] GPnP client pid=867, tl=3, f=0
2010-02-03 23:34:28.428: [ OCRAPI][2235814832]clsu_get_private_ip_addresses: no ip addresses found.
..
2010-02-03 23:34:28.434: [ OCRAPI][2235814832]a_init:13!: Clusterware init unsuccessful : [44]
2010-02-03 23:34:28.434: [ CRSOCR][2235814832] OCR context init failure. Error: PROC-44: Error in network address and interface operations Network address and interface operations error [7]
2010-02-03 23:34:28.434: [ CRSD][2235814832][PANIC] CRSD exiting: Could not init OCR, code: 44
或者:
2009-12-10 06:28:31.974: [ OCRMAS][20]proath_connect_master:1: could not connect to master clsc_ret1 = 9, clsc_ret2 = 9
2009-12-10 06:28:31.974: [ OCRMAS][20]th_master:11: Could not connect to the new master
2009-12-10 06:29:01.450: [ CRSMAIN][2] Policy Engine is not initialized yet!
2009-12-10 06:29:31.489: [ CRSMAIN][2] Policy Engine is not initialized yet!
或者:
2009-12-31 00:42:08.110: [ COMMCRS][10]clsc_receive: (102b03250) Error receiving, ns (12535, 12560), transport (505, 145, 0)
關(guān)于網(wǎng)絡(luò)和域名解析的驗(yàn)證，請參考：note 1054902.1
5. crsd 可執(zhí)行文件(crsd.bin 和 crsd in GRID_HOME/bin) 的權(quán)限或者屬主正確并且沒有進(jìn)行過手工的修改, 一個(gè)簡單可行的檢查辦法是對比好的節(jié)點(diǎn)和壞節(jié)點(diǎn)的以下命令輸出 "ls -l /bin/crsd /bin/crsd.bin".
6. 關(guān)于CRSD進(jìn)程啟動(dòng)問題的進(jìn)一步深入診斷，請參考 note 1323698.1 - Troubleshooting CRSD Start up Issue
1.1.6. 問題 5: GPNPD.BIN 無法啟動(dòng)
1. 網(wǎng)絡(luò)的域名解析不正常
gpnpd.bin 進(jìn)程啟動(dòng)失敗，以下信息提示在 gpnpd.log 中:
2010-05-13 12:48:11.540: [ GPnP][1171126592]clsgpnpm_exchange: [at clsgpnpm.c:1175] Calling "tcp://node2:9393", try 1 of 3...
2010-05-13 12:48:11.540: [ GPnP][1171126592]clsgpnpm_connect: [at clsgpnpm.c:1015] ENTRY
2010-05-13 12:48:11.541: [ GPnP][1171126592]clsgpnpm_connect: [at clsgpnpm.c:1066] GIPC gipcretFail (1) gipcConnect(tcp-tcp://node2:9393)
2010-05-13 12:48:11.541: [ GPnP][1171126592]clsgpnpm_connect: [at clsgpnpm.c:1067] Result: (48) CLSGPNP_COMM_ERR. Failed to connect to call url "tcp://node2:9393"
以上的例子，請確定當(dāng)前節(jié)點(diǎn)能夠正常的 ping 到“node2” ，并且確認(rèn)兩個(gè)節(jié)點(diǎn)之間沒有任何防火墻。
2. bug 10105195
由于 bug 10105195, gpnp 的調(diào)度線程（dispatch thread）可能被阻斷，例如：網(wǎng)絡(luò)掃描。這個(gè) bug 在 11.2.0.2 GI PSU2，11.2.0.3 及以上版本被修復(fù)，具體信息，請參見 note 10105195.8。
1.1.7. 問題 6: 其它的一些守護(hù)進(jìn)程無法啟動(dòng)
常見原因:
1. 守護(hù)進(jìn)程的日志文件或者日志所在的路徑權(quán)限或者屬主不正確。
如果日志文件或者日志文件所在的路徑權(quán)限或者屬主設(shè)置有問題，通常我們會(huì)看到進(jìn)程嘗試啟動(dòng)，但是日志里的信息卻始終沒有更新.
關(guān)于日志位置和權(quán)限屬主的限制，請參見 "日志文件位置, 屬主和權(quán)限" 獲取更多的信息。
2. 網(wǎng)絡(luò)的 socket 文件權(quán)限或者屬主錯(cuò)誤
這種情況下，守護(hù)進(jìn)程的日志會(huì)顯示以下信息:
2010-02-02 12:55:20.485: [ COMMCRS][1121433920]clsclisten: Permission denied for (ADDRESS=(PROTOCOL=ipc)(KEY=rac1DBG_GIPCD))
2010-02-02 12:55:20.485: [ clsdmt][1110944064]Fail to listen to (ADDRESS=(PROTOCOL=ipc)(KEY=rac1DBG_GIPCD))
3. OLR 文件損壞
這種情況下，守護(hù)進(jìn)程的日志會(huì)顯示以下信息(以下是個(gè) ora.ctssd 無法啟動(dòng)的例子):
2012-07-22 00:15:16.565: [ default][1]clsvactversion:4: Retrieving Active Version from local storage.
2012-07-22 00:15:16.575: [ CTSS][1]clsctss_r_av3: Invalid active version [] retrieved from OLR. Returns [19].
2012-07-22 00:15:16.585: [ CTSS][1](:ctss_init16:): Error [19] retrieving active version. Returns [19].
2012-07-22 00:15:16.585: [ CTSS][1]ctss_main: CTSS init failed [19]
2012-07-22 00:15:16.585: [ CTSS][1]ctss_main: CTSS daemon aborting [19].
2012-07-22 00:15:16.585: [ CTSS][1]CTSS daemon aborting
解決辦法，請恢復(fù)一個(gè)好的OLR的副本，具體辦法請參見 note 1193643.1
1.1.8. 問題 7: CRSD Agents 無法啟動(dòng)
CRSD.BIN 會(huì)負(fù)責(zé)衍生出兩個(gè) agents 進(jìn)程來啟動(dòng)用戶的資源，這兩個(gè) agents 的名字和 ohasd.bin 的 agents 的名字相同：
orarootagent: 負(fù)責(zé)啟動(dòng) ora.netn.network, ora.nodename.vip, ora.scann.vip 和 ora.gns
oraagent: 負(fù)責(zé)啟動(dòng) ora.asm, ora.eons, ora.ons, listener, SCAN listener, diskgroup, database, service 等資源
我們可以通過以下命令查看用戶的資源狀態(tài):
$GRID_HOME/crsctl stat res -t
如果 crsd.bin 無法正常啟動(dòng)以上任何一個(gè) agent，用戶的資源都將無法正常啟動(dòng).
1. 通常這些 agent 無法啟動(dòng)的常見原因是 agent 的日志或者日志所在的路徑?jīng)]有設(shè)置合適的權(quán)限或者屬主。
請參見以下 "日志文件位置, 屬主和權(quán)限" 部分關(guān)于日志權(quán)限的設(shè)置。
2. agent 可能因?yàn)?bug 11834289 無法啟動(dòng)，此時(shí)我們會(huì)看到 "CRS-5802: Unable to start the agent process"錯(cuò)誤信息，請參見 "OHASD 無法啟動(dòng)" #10 獲取更多信息。
1.1.9. 問題 8: HAIP 無法啟動(dòng)
HAIP 無法啟動(dòng)的原因有很多,例如：
[ohasd(891)]CRS-2807:Resource 'ora.cluster_interconnect.haip' failed to start automatically.
請參見 note?1210883.1 獲取更多關(guān)于 HAIP 的信息。
1.1.10. 網(wǎng)絡(luò)和域名解析的驗(yàn)證
CRS 的啟動(dòng)，依賴于網(wǎng)絡(luò)功能和域名解析的正常工作，如果網(wǎng)絡(luò)功能或者域名解析不能正常工作，CRS 將無法正常啟動(dòng)。
關(guān)于網(wǎng)絡(luò)和域名解析的驗(yàn)證，請參考： note 1054902.1
1.1.11. 日志文件位置, 屬主和權(quán)限
正確的設(shè)置 $GRID_HOME/log 和這里的子目錄以及文件對 CRS 組件的正常啟動(dòng)是至關(guān)重要的。
1.1.11.1. 在 Grid Infrastructure 的環(huán)境中:
我們假設(shè)一個(gè) Grid Infrastructure 環(huán)境，節(jié)點(diǎn)名字為 rac1, CRS 的屬主是 grid, 并且有兩個(gè)單獨(dú)的 RDBMS 屬主分別為： rdbmsap 和 rdbmsar，以下是 $GRID_HOME/log 中正常的設(shè)置情況:
drwxrwxr-x 5 grid oinstall 4096 Dec 6 09:20 log
drwxr-xr-x 2 grid oinstall 4096 Dec 6 08:36 crs
drwxr-xr-t 17 root oinstall 4096 Dec 6 09:22 rac1
drwxr-x--- 2 grid oinstall 4096 Dec 6 09:20 admin
drwxrwxr-t 4 root oinstall 4096 Dec 6 09:20 agent
drwxrwxrwt 7 root oinstall 4096 Jan 26 18:15 crsd
drwxr-xr-t 2 grid oinstall 4096 Dec 6 09:40 application_grid
drwxr-xr-t 2 grid oinstall 4096 Jan 26 18:15 oraagent_grid
drwxr-xr-t 2 rdbmsap oinstall 4096 Jan 26 18:15 oraagent_rdbmsap
drwxr-xr-t 2 rdbmsar oinstall 4096 Jan 26 18:15 oraagent_rdbmsar
drwxr-xr-t 2 grid oinstall 4096 Jan 26 18:15 ora_oc4j_type_grid
drwxr-xr-t 2 root root 4096 Jan 26 20:09 orarootagent_root
drwxrwxr-t 6 root oinstall 4096 Dec 6 09:24 ohasd
drwxr-xr-t 2 grid oinstall 4096 Jan 26 18:14 oraagent_grid
drwxr-xr-t 2 root root 4096 Dec 6 09:24 oracssdagent_root
drwxr-xr-t 2 root root 4096 Dec 6 09:24 oracssdmonitor_root
drwxr-xr-t 2 root root 4096 Jan 26 18:14 orarootagent_root
-rw-rw-r-- 1 root root 12931 Jan 26 21:30 alertrac1.log
drwxr-x--- 2 grid oinstall 4096 Jan 26 20:44 client
drwxr-x--- 2 root oinstall 4096 Dec 6 09:24 crsd
drwxr-x--- 2 grid oinstall 4096 Dec 6 09:24 cssd
drwxr-x--- 2 root oinstall 4096 Dec 6 09:24 ctssd
drwxr-x--- 2 grid oinstall 4096 Jan 26 18:14 diskmon
drwxr-x--- 2 grid oinstall 4096 Dec 6 09:25 evmd
drwxr-x--- 2 grid oinstall 4096 Jan 26 21:20 gipcd
drwxr-x--- 2 root oinstall 4096 Dec 6 09:20 gnsd
drwxr-x--- 2 grid oinstall 4096 Jan 26 20:58 gpnpd
drwxr-x--- 2 grid oinstall 4096 Jan 26 21:19 mdnsd
drwxr-x--- 2 root oinstall 4096 Jan 26 21:20 ohasd
drwxrwxr-t 5 grid oinstall 4096 Dec 6 09:34 racg
drwxrwxrwt 2 grid oinstall 4096 Dec 6 09:20 racgeut
drwxrwxrwt 2 grid oinstall 4096 Dec 6 09:20 racgevtf
drwxrwxrwt 2 grid oinstall 4096 Dec 6 09:20 racgmain
drwxr-x--- 2 grid oinstall 4096 Jan 26 20:57 srvm
請注意，絕大部分的子目錄都繼承了父目錄的屬主和權(quán)限，以上僅作為一個(gè)參考，來判斷 CRS HOME 中是否有一些遞歸的權(quán)限和屬主改變，如果您已經(jīng)有一個(gè)相同版本的正在運(yùn)行的工作節(jié)點(diǎn)，您可以把該運(yùn)行的節(jié)點(diǎn)作為參考。
1.1.11.2. 在 Oracle Restart 的環(huán)境中:
這里顯示了在 Oracle Restart 環(huán)境中 $GRID_HOME/log 目錄下的權(quán)限和屬主設(shè)置:
drwxrwxr-x 5 grid oinstall 4096 Oct 31 2009 log
drwxr-xr-x 2 grid oinstall 4096 Oct 31 2009 crs
drwxr-xr-x 3 grid oinstall 4096 Oct 31 2009 diag
drwxr-xr-t 17 root oinstall 4096 Oct 31 2009 rac1
drwxr-x--- 2 grid oinstall 4096 Oct 31 2009 admin
drwxrwxr-t 4 root oinstall 4096 Oct 31 2009 agent
drwxrwxrwt 2 root oinstall 4096 Oct 31 2009 crsd
drwxrwxr-t 8 root oinstall 4096 Jul 14 08:15 ohasd
drwxr-xr-x 2 grid oinstall 4096 Aug 5 13:40 oraagent_grid
drwxr-xr-x 2 grid oinstall 4096 Aug 2 07:11 oracssdagent_grid
drwxr-xr-x 2 grid oinstall 4096 Aug 3 21:13 orarootagent_grid
-rwxr-xr-x 1 grid oinstall 13782 Aug 1 17:23 alertrac1.log
drwxr-x--- 2 grid oinstall 4096 Nov 2 2009 client
drwxr-x--- 2 root oinstall 4096 Oct 31 2009 crsd
drwxr-x--- 2 grid oinstall 4096 Oct 31 2009 cssd
drwxr-x--- 2 root oinstall 4096 Oct 31 2009 ctssd
drwxr-x--- 2 grid oinstall 4096 Oct 31 2009 diskmon
drwxr-x--- 2 grid oinstall 4096 Oct 31 2009 evmd
drwxr-x--- 2 grid oinstall 4096 Oct 31 2009 gipcd
drwxr-x--- 2 root oinstall 4096 Oct 31 2009 gnsd
drwxr-x--- 2 grid oinstall 4096 Oct 31 2009 gpnpd
drwxr-x--- 2 grid oinstall 4096 Oct 31 2009 mdnsd
drwxr-x--- 2 grid oinstall 4096 Oct 31 2009 ohasd
drwxrwxr-t 5 grid oinstall 4096 Oct 31 2009 racg
drwxrwxrwt 2 grid oinstall 4096 Oct 31 2009 racgeut
drwxrwxrwt 2 grid oinstall 4096 Oct 31 2009 racgevtf
drwxrwxrwt 2 grid oinstall 4096 Oct 31 2009 racgmain
drwxr-x--- 2 grid oinstall 4096 Oct 31 2009 srvm
1.1.12. 網(wǎng)絡(luò)socket文件的位置，屬主和權(quán)限
網(wǎng)絡(luò)的 socket 文件可能位于目錄： /tmp/.oracle, /var/tmp/.oracle or /usr/tmp/.oracle 中。
當(dāng)網(wǎng)絡(luò)的 socket 文件權(quán)限或者屬主設(shè)置不正確的時(shí)候，我們通常會(huì)在守護(hù)進(jìn)程的日志中看到以下類似的信息:
2011-06-18 14:07:28.545: [ COMMCRS][772]clsclisten: Permission denied for (ADDRESS=(PROTOCOL=ipc)(KEY=racnode1DBG_EVMD))
2011-06-18 14:07:28.545: [ clsdmt][515]Fail to listen to (ADDRESS=(PROTOCOL=ipc)(KEY=lena042DBG_EVMD))
2011-06-18 14:07:28.545: [ clsdmt][515]Terminating process
2011-06-18 14:07:28.559: [ default][515] EVMD exiting on stop request from clsdms_thdmai
以下錯(cuò)誤也有可能提示:
CRS-5017: The resource action "ora.evmd start" encountered the following error:
CRS-2674: Start of 'ora.evmd' on 'racnode1' failed
..
解決的辦法：請使用 root 用戶停掉 GI，刪除這些 socket 文件，并重新啟動(dòng) GI。
我們假設(shè)一個(gè) Grid Infrastructure 環(huán)境，節(jié)點(diǎn)名為 rac1, CRS 的屬主是 grid，以下是 socket 文件夾(../.oracle)正常的設(shè)置情況:
1.1.12.1. 在 Grid Infrastructure cluster 環(huán)境中:
以下例子是集群環(huán)境中的例子：
drwxrwxrwt 2 root oinstall 4096 Feb 2 21:25 .oracle
./.oracle:
drwxrwxrwt 2 root oinstall 4096 Feb 2 21:25 .
srwxrwx--- 1 grid oinstall 0 Feb 2 18:00 master_diskmon
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:00 mdnsd
-rw-r--r-- 1 grid oinstall 5 Feb 2 18:00 mdnsd.pid
prw-r--r-- 1 root root 0 Feb 2 13:33 npohasd
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:00 ora_gipc_GPNPD_rac1
-rw-r--r-- 1 grid oinstall 0 Feb 2 13:34 ora_gipc_GPNPD_rac1_lock
srwxrwxrwx 1 grid oinstall 0 Feb 2 13:39 s#11724.1
srwxrwxrwx 1 grid oinstall 0 Feb 2 13:39 s#11724.2
srwxrwxrwx 1 grid oinstall 0 Feb 2 13:39 s#11735.1
srwxrwxrwx 1 grid oinstall 0 Feb 2 13:39 s#11735.2
srwxrwxrwx 1 grid oinstall 0 Feb 2 13:45 s#12339.1
srwxrwxrwx 1 grid oinstall 0 Feb 2 13:45 s#12339.2
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:01 s#6275.1
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:01 s#6275.2
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:01 s#6276.1
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:01 s#6276.2
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:01 s#6278.1
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:01 s#6278.2
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:00 sAevm
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:00 sCevm
srwxrwxrwx 1 root root 0 Feb 2 18:01 sCRSD_IPC_SOCKET_11
srwxrwxrwx 1 root root 0 Feb 2 18:01 sCRSD_UI_SOCKET
srwxrwxrwx 1 root root 0 Feb 2 21:25 srac1DBG_CRSD
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:00 srac1DBG_CSSD
srwxrwxrwx 1 root root 0 Feb 2 18:00 srac1DBG_CTSSD
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:00 srac1DBG_EVMD
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:00 srac1DBG_GIPCD
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:00 srac1DBG_GPNPD
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:00 srac1DBG_MDNSD
srwxrwxrwx 1 root root 0 Feb 2 18:00 srac1DBG_OHASD
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:01 sLISTENER
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:01 sLISTENER_SCAN2
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:01 sLISTENER_SCAN3
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:00 sOCSSD_LL_rac1_
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:00 sOCSSD_LL_rac1_eotcs
-rw-r--r-- 1 grid oinstall 0 Feb 2 18:00 sOCSSD_LL_rac1_eotcs_lock
-rw-r--r-- 1 grid oinstall 0 Feb 2 18:00 sOCSSD_LL_rac1__lock
srwxrwxrwx 1 root root 0 Feb 2 18:00 sOHASD_IPC_SOCKET_11
srwxrwxrwx 1 root root 0 Feb 2 18:00 sOHASD_UI_SOCKET
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:00 sOracle_CSS_LclLstnr_eotcs_1
-rw-r--r-- 1 grid oinstall 0 Feb 2 18:00 sOracle_CSS_LclLstnr_eotcs_1_lock
srwxrwxrwx 1 root root 0 Feb 2 18:01 sora_crsqs
srwxrwxrwx 1 root root 0 Feb 2 18:00 sprocr_local_conn_0_PROC
srwxrwxrwx 1 root root 0 Feb 2 18:00 sprocr_local_conn_0_PROL
srwxrwxrwx 1 grid oinstall 0 Feb 2 18:00 sSYSTEM.evm.acceptor.auth
1.1.12.2. 在 Oracle Restart 環(huán)境中:
以下是 Oracle Restart 環(huán)境中的輸出例子:
drwxrwxrwt 2 root oinstall 4096 Feb 2 21:25 .oracle
./.oracle:
srwxrwx--- 1 grid oinstall 0 Aug 1 17:23 master_diskmon
prw-r--r-- 1 grid oinstall 0 Oct 31 2009 npohasd
srwxrwxrwx 1 grid oinstall 0 Aug 1 17:23 s#14478.1
srwxrwxrwx 1 grid oinstall 0 Aug 1 17:23 s#14478.2
srwxrwxrwx 1 grid oinstall 0 Jul 14 08:02 s#2266.1
srwxrwxrwx 1 grid oinstall 0 Jul 14 08:02 s#2266.2
srwxrwxrwx 1 grid oinstall 0 Jul 7 10:59 s#2269.1
srwxrwxrwx 1 grid oinstall 0 Jul 7 10:59 s#2269.2
srwxrwxrwx 1 grid oinstall 0 Jul 31 22:10 s#2313.1
srwxrwxrwx 1 grid oinstall 0 Jul 31 22:10 s#2313.2
srwxrwxrwx 1 grid oinstall 0 Jun 29 21:58 s#2851.1
srwxrwxrwx 1 grid oinstall 0 Jun 29 21:58 s#2851.2
srwxrwxrwx 1 grid oinstall 0 Aug 1 17:23 sCRSD_UI_SOCKET
srwxrwxrwx 1 grid oinstall 0 Aug 1 17:23 srac1DBG_CSSD
srwxrwxrwx 1 grid oinstall 0 Aug 1 17:23 srac1DBG_OHASD
srwxrwxrwx 1 grid oinstall 0 Aug 1 17:23 sEXTPROC1521
srwxrwxrwx 1 grid oinstall 0 Aug 1 17:23 sOCSSD_LL_rac1_
srwxrwxrwx 1 grid oinstall 0 Aug 1 17:23 sOCSSD_LL_rac1_localhost
-rw-r--r-- 1 grid oinstall 0 Aug 1 17:23 sOCSSD_LL_rac1_localhost_lock
-rw-r--r-- 1 grid oinstall 0 Aug 1 17:23 sOCSSD_LL_rac1__lock
srwxrwxrwx 1 grid oinstall 0 Aug 1 17:23 sOHASD_IPC_SOCKET_11
srwxrwxrwx 1 grid oinstall 0 Aug 1 17:23 sOHASD_UI_SOCKET
srwxrwxrwx 1 grid oinstall 0 Aug 1 17:23 sgrid_CSS_LclLstnr_localhost_1
-rw-r--r-- 1 grid oinstall 0 Aug 1 17:23 sgrid_CSS_LclLstnr_localhost_1_lock
srwxrwxrwx 1 grid oinstall 0 Aug 1 17:23 sprocr_local_conn_0_PROL
1.1.13. 診斷文件收集
如果通過本文沒有找到問題原因，請使用 root 用戶，在所有的節(jié)點(diǎn)上執(zhí)行 $GRID_HOME/bin/diagcollection.sh ，并上傳在當(dāng)前目錄下生成所有的 .gz 壓縮文件來做進(jìn)一步診斷。
轉(zhuǎn)載于:https://www.cnblogs.com/cnmarkao/p/10680299.html
總結(jié)
以上是生活随笔為你收集整理的【Oracle】RAC11gR2Grid启动顺序及启动故障诊断思路的全部內(nèi)容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： 剑指offer-按之字形顺序打印二叉树
- 下一篇： Flask 系列之 Blueprint