1、1：crs_stat -t查看集群信息报错[grid@qidong1 ~]$ crs_stat -tCRS-0184: Cannot communicate with the CRS daemon.[grid@qidong2 ~]$ crs_stat -tCRS-0184: Cannot communicate with the CRS daemon.两个节点执行crs_stat 报错。但是集群正常ASM正常，数据库正常。
2、2：asmcmd进入ASM中发现存储集群信息的DG消失asmcmd进入ASM中只能看到datadg和archdg。发现crsdg消失不见两个节点都是这样的情况
3、3：crsctl check crs 检测集群发现异常[grid@qidong2 ~]$ crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4535: Cannot communicate with Cluster Ready ServicesCRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online[grid@qidong2 ~]$
4、4：查看集睡连简郯群crsd.log日志在其中一个节点查看就行，应为ASM中crsdg消失不会，说明两个节点都会出现报错。crs的日志结构：根据CRS的安装目录，日志跟目录在$C伊怕锱鳏RS_HOME/log/[node],其中node是节点名。目录下包含的内容如下： 1）alert.log ：该日志类似于数据库的alert.log 一般作为检查的起点。 2）crsd，cssd，evmd ：分别对应3个目录，分别对应着CRS三个同名进程的日志，日志名字分别叫做：crsd.logocssd.log evmd.log 3) racg :这是个目录，里面放置着所有的nodeapp的日志，包括ONS、VIP，每个日志从名字上很容易辨别对应的nodeapp。 4）client ：该目录放置的是工具执行日志，Oracle Clusterware提供了许多命令行工具，比如ocrcheck 、ocrconfig、ocrdump、oifcfg clscfg 这些工具运行时产生的日志就放置在该目录下。查看crsd.log日志：$GRID_HOME/log/sid/crsd/crsd.log/u01/app/11.2.0.4/grid/log/qidong2/crsd:2015-10-24 11:27:29.017: [ CRSMAIN][1153496864] Checking the OCR device2015-10-24 11:27:29.017: [ CRSMAIN][1153496864] Sync-up with OCR2015-10-24 11:27:29.017: [ CRSMAIN][1153496864] Connecting to the CSS Daemon2015-10-24 11:27:29.017: [ CRSMAIN][1153496864] Getting local node number2015-10-24 11:27:29.018: [ CRSMAIN][1153496864] Initializing OCR[ CLWAL][1153496864]clsw_Initialize: OLR initlevel [70000]2015-10-24 11:27:29.347: [ OCRASM][1153496864]proprasmo: Error in open/create file in dg [crsdg][ OCRASM][1153496864]SLOS : SLOS: cat=8, opn=kgfoOpen01, dep=15056, loc=kgfokge2015-10-24 11:27:29.347: [ OCRASM][1153496864]ASM Error Stack :2015-10-24 11:27:29.395: [ OCRASM][1153496864]proprasmo: kgfoCheckMount returned [6]2015-10-24 11:27:29.395: [ OCRASM][1153496864]proprasmo: The ASM disk group crsdg is not found or not mounted2015-10-24 11:27:29.396: [ OCRRAW][1153496864]proprioo: Failed to open [+crsdg]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.2015-10-24 11:27:29.396: [ OCRRAW][1153496864]proprioo: No OCR/OLR devices are usable2015-10-24 11:27:29.396: [ OCRASM][1153496864]proprasmcl: asmhandle is NULL2015-10-24 11:27:29.397: [ GIPC][1153496864] gipcCheckInitialization: possible incompatible non-threaded init from [prom.c : 690], original from [clsss.c : 5343]2015-10-24 11:27:29.398: [ default][1153496864]clsvactversion:4: Retrieving Active Version from local storage.2015-10-24 11:27:29.401: [ CSSCLNT][1153496864]clssgsgrppubdata: group (ocr_qidong-cluster) not found2015-10-24 11:27:29.401: [ OCRRAW][1153496864]proprio_repairconf: Failed to retrieve the group public data. CSS ret code [20]2015-10-24 11:27:29.402: [ OCRRAW][1153496864]proprioo: Failed to auto repair the OCR configuration.2015-10-24 11:27:29.402: [ OCRRAW][1153496864]proprinit: Could not open raw device 2015-10-24 11:27:29.403: [ OCRASM][1153496864]proprasmcl: asmhandle is NULL2015-10-24 11:27:29.405: [ OCRAPI][1153496864]a_init:16!: Backend init unsuccessful : [26]2015-10-24 11:27:29.405: [ CRSOCR][1153496864] OCR context init failure. Error: PROC-26: Error while accessing the physical storage2015-10-24 11:27:29.405: [ CRSD][1153496864] Created alert : (:CRSD00111:) : Could not init OCR, error: PROC-26: Error while accessing the physical storage2015-10-24 11:27:29.405: [ CRSD][1153496864][PANIC] CRSD exiting: Could not init OCR, code: 262015-10-24 11:27:29.405: [ CRSD][1153496864] Done.根据crsd.log日志，可以判定由于无法读取到物理存储的信息，导致crsdg无法挂载。
5、5：查看磁盘信息通过fdisk -l在两个节点上查看发现存储都正常挂载到两个节点上，且udev唯一映射也正常。[root@qidong2 dev]# ll ora-*brwxrwxrw-. 1 grid asmadmin 8, 17 Dec 1 10:34 ora-crsa1brwxrwxrw-. 1 grid asmadmin 8, 33 Dec 1 10:34 ora-crsa2brwxrwxrw-. 1 grid asmadmin 8, 49 Dec 1 10:34 ora-crsa3brwxrwxrw-. 1 grid asmadmin 8, 65 Oct 24 11:26 ora-crsa4brwxrwxrw-. 1 oracle oinstall 8, 81 Oct 24 11:26 ora-dataa5brwxrwxrw-. 1 oracle oinstall 8, 97 Dec 1 10:34 ora-datab1brwxrwxrw-. 1 oracle oinstall 8, 113 Dec 1 10:34 ora-datab2brwxrwxrw-. 1 oracle oinstall 8, 129 Dec 1 10:34 ora-datab3brwxrwxrw-. 1 oracle oinstall 8, 145 Dec 1 10:34 ora-datab4brwxrwxrw-. 1 oracle oinstall 8, 161 Dec 1 10:34 ora-datab5brwxrwxrw-. 1 oracle oinstall 8, 177 Dec 1 10:27 ora-datab6
6、6：查看v$a衡痕贤伎sm_disk中集群信息[oracle@qidong1 ~]$[oracle@qidong1 ~]$ export ORA潮贾篡绐CLE_SID=+ASM1[oracle@qidong1 ~]$ export ORACLE_HOME=/u01/app/11.2.0.4/grid/[oracle@qidong1 ~]$ export PATH=$PATH:$ORACLE_HOME/bin[oracle@qidong1 ~]$ sqlplus sys as sysasmSQL*Plus: Release 11.2.0.4.0 Production on Tue Dec 1 10:49:38 2015Copyright (c) 1982, 2013, Oracle. All rights reserved.Enter password:Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit ProductionWith the Real Application Clusters and Automatic Storage Management optionsSQL> col name format a15SQL> col path format a17SQL> select name,path,header_status,mount_status,state from v$asm_disk;NAME PATH HEADER_STATU MOUNT_S STATE--------------- ----------------- ------------ ------- -------- /dev/ora-crsa3 MEMBER CLOSED NORMAL /dev/ora-crsa2 MEMBER CLOSED NORMAL /dev/ora-crsa1 MEMBER CLOSED NORMAL /dev/ora-dataa5 CANDIDATE CLOSED NORMAL /dev/ora-crsa4 CANDIDATE CLOSED NORMALDATADG_0002 /dev/ora-datab3 MEMBER CACHED NORMALDATADG_0003 /dev/ora-datab4 MEMBER CACHED NORMALDATADG_0004 /dev/ora-datab5 MEMBER CACHED NORMALARCHDG_0000 /dev/ora-datab6 MEMBER CACHED NORMALDATADG_0001 /dev/ora-datab2 MEMBER CACHED NORMALDATADG_0000 /dev/ora-datab1 MEMBER CACHED NORMALcrsdg 没有正常挂载注意：如何进入sqlplus sys as sysasm参考经验：sqlplus sys as sysasm进入ASM管理工具
7、7：asmcmd中 mount -a挂载所有DGASMCMD> mount -aORA-15032: not all alterations performedORA-15017: diskgroup "DATADG" cannot be mountedORA-15013: diskgroup "DATADG" is already mountedORA-15017: diskgroup "ARCHDG" cannot be mountedORA-15013: diskgroup "ARCHDG" is already mounted (DBD ERROR: OCIStmtExecute)ASMCMD> lsdgState Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files NameMOUNTED EXTERN N 512 4096 16777216 563184 450832 0 450832 0 N ARCHDG/MOUNTED NORMAL N 512 4096 16777216 61392 60000 20464 19768 0 Y CRSDG/MOUNTED EXTERN N 512 4096 16777216 2815920 1158096 0 1158096 0 N DATADG/ASMCMD>在一个节点执行即可，执行完之后，v$asm_disk中发现crsdg挂载正常了。asmcmd中执行lsdg也能看到crsdg信息了。注意：如果不想在asmcmd中使用mount -a,可以在第六步骤中，sqlplus sys as sysasm进入ASM中。在两个节点上分别执行alter diskgroup crsdg mount;
8、8：crs_stat -t查看集群信息仍然报错
9、9：crsctl query css votedisk和ocrcheck 都正常[grid@qidong1 ~]$ crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 5f065dff8dc04f4cbfadfb4e5d805957 (/dev/ora-crsa1) [CRSDG]2. ONLINE ab51b2fba8ad4fe2bf126f3d9ca8d9b4 (/dev/ora-crsa2) [CRSDG]3. ONLINE 8d555f4c64b24f63bf019e5e40320ecf (/dev/ora-crsa3) [CRSDG]Located 3 voting disk(s).[grid@qidong1 ~]$[grid@qidong1 ~]$ ocrcheckStatus of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 3100 Available space (kbytes) : 259020 ID : 2044633163 Device/File Name : +crsdg Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check bypassed due to non-privileged user
10、10：crsctl check crs 仍然报错[root@qidong1 grid]# crsctl check crsCRS-4638: Oracle High Availability Services is onlineCRS-4535: Cannot communicate with Cluster Ready ServicesCRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online
11、11：重启两个节点由于之前数据库一致正常运行，且集群也没有恢复正常，srvctl stop database不能正常执行，只能在每个节点执行shutdown immeidate关闭数据库。1：在每个节点执行shutdown immeidate关闭数据库2：crsctl stop crs 关闭两节点集群，如果不行crsctl stop crs -f 关闭3：reboot 重启两节点
12、12：重启之后发现crsdg正常，但是数据库无法启动重启之后，集群正常启动，但是数据库无法启动。定位发现datadg 和archdg 没有挂载。/dev/查看磁盘的各个主分区无法识别。但是通过fdisk -l 查看部分磁盘会显示主分区，但是有的磁盘分区不显示。由于使用udev单通道唯一映射需要对每个磁盘划分一个主分区才能进行udev映射。此时重启之后发现datadg和archdg所在磁盘分区无缘无故自动被删除。所以需要重新分区：1：crsctl stop crs 关闭两节点集群2：在节点1对datadg和archdg所在磁盘进行分区，所有磁盘分区完成之后，此时节点1虽然udev能够映射了，但是节点2不能够udev映射。这是建议重启节点1和节点2
13、13：重新重启之后集群正常，数据库能够正常启动
