Datastore verschwunden

[Christoph-SC]

Hallo zusammen,

wir haben hier ein massives Problem in einer vSphere-Umgebung.
(2 x ESX4.1; HP MSA-Storage via FC)

Nachdem gestern Abend auf dem Storage drei Platten innerhalb weniger Stunden augefallen sind (laut Storage war Rebulid erfolgreich) ist an keinem der beiden ESX unser Datastore mehr zusehen.
Die LUN selber ist sichtbar und verbunden - nur der daraufbefindliche Datastore wird nicht mehr angezeigt.

Wir haben die komplette Umgebung bereits mehrfach neu gestartet. Leider ohne Erfolg.
Das Storage meldet auch keinen Fehler.

Wie können wir den Datastore wieder einbinden/neu initialisieren?
Die "Neu einlesen"-Funktion im vSphere-Client bringt leider keine Lösung.

Viele Grüße
Christoph

[bla!zilla]

Ist die vDisk online oder in Quarantäne? Ansonsten würde ich mal vermuten, dass das Dateisystem hinüber ist.

Was für ein RAID Level ist konfiguriert? Bitte mal das Log der MSA hier posten.

[Christoph-SC]

vDisk ist online. RAID-Level 5.
MSA-Log folgt gleich...

[bla!zilla]

Wenn du drei Plattenausfälle hattest, und eine eine zweite Platte fiel aus, bevor der Rebuild fertig war, dann wird das Dateisystem wrack sein.

[Christoph-SC]

Laut Logs (der HP-Support hat diese auch bereits gecheckt) war der Rebulid jeweils fertig bevor die nächste HDD ausgefallen ist...

[bla!zilla]

Kannst du die betreffenden Teile vom Log hier posten? Also kurz vor dem ersten Plattenausfall bis kurz nach dem letzten Rebuild.

[Christoph-SC]

So, hier jetzt ein Auszug (kommentiert vom HP-Support):

I see that both the Vdisks are FTOL and online to Host at the moment
vd01 A 9 RAID5 2291439616 4 00c0ffdaabb90000e4b4ab4b00000000 FTOL OK
vd02 B 9 RAID5 2291439616 2 00c0ffdaa9e90000c728014c00000000 FTOL OK

Encl Slot ChA ChB SP Vendor Model Rev Serial Number Size How Used Type LED Status(#) memberIndex vdisk
1 4 0 1 HP EG0146FAWHU HPDD 3SD2SS6J00009030KQXM 146.8GB LEFTOVR SAS Fault (4) 0
1 20 0 1 HP EG0146FAWHU HPDD 3SD2XRRQ00009031K7B0 146.8GB LEFTOVR SAS Fault (4) 0
1 21 0 1 HP EG0146FAWHU HPDD 3SD2XRAP00009031K1SL 146.8GB LEFTOVR SAS Fault (4) 0
The problem started on the 18th around 18:24:42.
Disk in Enc 1 Slot 4 timedout
A1310 2011-01-18 18:24:42 58 INFORMATIONAL An error was detected by a disk drive. (disk: channel: 0, ID: 3, SN: 3SD2SS6J00009030KQXM, enclosure: 1, slot: 4)(Key,Code,Qual:0xB,0x4B,0x3)(CDB:Rd 025fec00 0080)(Info:0x025FEC45)(CmdSpc:0x0, FRU:0x6, SnsKeySpc:0x0)(Aborted Command, ack/nak timeout)
A1314 2011-01-18 18:24:43 58 INFORMATIONAL An error was detected by a disk drive. (disk: channel: 1, ID: 3, SN: 3SD2SS6J00009030KQXM, enclosure: 1, slot: 4)(Key,Code,Qual:0xB,0x4B,0x3)(CDB:Rd 025fec00 0080)(Info:0x025FEC45)(CmdSpc:0x0, FRU:0x5, SnsKeySpc:0x0)(Aborted Command, ack/nak timeout)
This caused the Vdisk to move into Critical state
A1317 2011-01-18 18:24:43 8 WARNING A disk that is part of a vdisk is down. (vdisk: vd01) (disk: channel: 0, ID: 3, SN: 3SD2SS6J00009030KQXM, enclosure: 1, slot: 4)
A1319 2011-01-18 18:24:43 1 WARNING A vdisk is critical. (vdisk: vd01, SN: 00c0ffdaabb90000e4b4ab4b00000000)

Spare Drive Kicked in
A1322 2011-01-18 18:24:50 9 INFORMATIONAL A spare disk kicked in for a critical vdisk. (vdisk: vd01, SN: 00c0ffdaabb90000e4b4ab4b00000000) (disk: channel: 0, ID: 20, SN: 3SD2XRAP00009031K1SL, enclosure: 1, slot: 21)
A1323 2011-01-18 18:24:50 37 INFORMATIONAL Vdisk reconstruct started. (vdisk: vd01, SN: 00c0ffdaabb90000e4b4ab4b00000000) (disk: channel: 0, ID: 20, SN: 3SD2XRAP00009031K1SL, enclosure: 1, slot: 21)
During the rebuild another Disk in Enclosure 1 Slot 20 timedout
A1324 2011-01-18 18:44:05 58 INFORMATIONAL An error was detected by a disk drive. (disk: channel: 0, ID: 19, SN: 3SD2XRRQ00009031K7B0, enclosure: 1, slot: 20)(Key,Code,Qual:0xB,0x4B,0x3)(CDB:Rd 0571d480 0800)(Info:0x0571DC41)(CmdSpc:0x0, FRU:0x6, SnsKeySpc:0x0)(Aborted Command, ack/nak timeout)
A1325 2011-01-18 18:44:05 58 WARNING An error was detected by a disk drive. (disk: channel: 0, ID: 19, SN: 3SD2XRRQ00009031K7B0, enclosure: 1, slot: 20)(Key,Code,Qual:0x4,0x41,0x8C)(CDB:Rd 0571d480 0800)(Info:0x0571DC42)(CmdSpc:0x0, FRU:0x1, SnsKeySpc:0x0)(Hardware, no decode for ASC/ASCQ)
A1330 2011-01-18 18:44:06 58 INFORMATIONAL An error was detected by a disk drive. (disk: channel: 1, ID: 19, SN: 3SD2XRRQ00009031K7B0, enclosure: 1, slot: 20)(Key,Code,Qual:0xB,0x4B,0x3)(CDB:Rd 0571d480 0800)(Info:0x0571DC41)(CmdSpc:0x0, FRU:0x5, SnsKeySpc:0x0)(Aborted Command, ack/nak timeout)
A1331 2011-01-18 18:44:06 58 INFORMATIONAL An error was detected by a disk drive. (disk: channel: 0, ID: 19, SN: 3SD2XRRQ00009031K7B0, enclosure: 1, slot: 20)(Key,Code,Qual:0xB,0x4B,0x3)(CDB:Rd 0571db00 0080)(Info:0x0571DB4E)(CmdSpc:0x0, FRU:0x5, SnsKeySpc:0x0)(Aborted Command, ack/nak timeout)
A1345 2011-01-18 18:44:17 58 INFORMATIONAL An error was detected by a disk drive. (disk: channel: 1, ID: 19, SN: 3SD2XRRQ00009031K7B0, enclosure: 1, slot: 20)(Key,Code,Qual:0xB,0x4B,0x3)(CDB:Rd 0571db00 0080)(Info:0x0571DB4D)(CmdSpc:0x0, FRU:0x6, SnsKeySpc:0x0)(Aborted Command, ack/nak timeout)
A1346 2011-01-18 18:44:17 8 WARNING A disk that is part of a vdisk is down. (vdisk: vd01) (disk: channel: 0, ID: 19, SN: 3SD2XRRQ00009031K7B0, enclosure: 1, slot: 20)
A1348 2011-01-18 18:44:17 44 WARNING Unwritable cache data exists for a volume. (volume: , SN: 00c0ffdaabb900009613114c01000000) It comprises 1% of cache space.
A1349 2011-01-18 18:44:17 44 WARNING Unwritable cache data exists for a volume. (volume: , SN: 00c0ffdaabb90000f14fed4b01000000) It comprises 2% of cache space.
A1350 2011-01-18 18:44:17 44 WARNING Unwritable cache data exists for a volume. (volume: , SN: 00c0ffdaabb900002c50ed4b02000000) It comprises 1% of cache space.
But luckly the Vdisk reconstruction completed
A1351 2011-01-18 19:28:11 18 INFORMATIONAL Vdisk reconstruct completed successfully. (vdisk: vd01, SN: 00c0ffdaabb90000e4b4ab4b00000000)
Since the Disk in Enc 1 Slot 20 was timedout the vdisk is in critical state
A1352 2011-01-18 19:28:11 1 WARNING A vdisk is critical. (vdisk: vd01, SN: 00c0ffdaabb90000e4b4ab4b00000000)
Another Spare Drive Kicked in
A1353 2011-01-18 19:28:16 9 INFORMATIONAL A spare disk kicked in for a critical vdisk. (vdisk: vd01, SN: 00c0ffdaabb90000e4b4ab4b00000000) (disk: channel: 0, ID: 22, SN: 3SD2XKR600009031P2ZA, enclosure: 1, slot: 23)
A1354 2011-01-18 19:28:16 37 INFORMATIONAL Vdisk reconstruct started. (vdisk: vd01, SN: 00c0ffdaabb90000e4b4ab4b00000000) (disk: channel: 0, ID: 22, SN: 3SD2XKR600009031P2ZA, enclosure: 1, slot: 23)
During the rebuild the 3rd Drive in Encl 1 Slot 21 timedout
A1376 2011-01-18 19:37:33 58 INFORMATIONAL An error was detected by a disk drive. (disk: channel: 1, ID: 20, SN: 3SD2XRAP00009031K1SL, enclosure: 1, slot: 21)(Key,Code,Qual:0xB,0x4B,0x3)(CDB:Rd 025fec00 0080)(Info:0x025FEC45)(CmdSpc:0x0, FRU:0x6, SnsKeySpc:0x0)(Aborted Command, ack/nak timeout)
A1377 2011-01-18 19:37:33 8 WARNING A disk that is part of a vdisk is down. (vdisk: vd01) (disk: channel: 0, ID: 20, SN: 3SD2XRAP00009031K1SL, enclosure: 1, slot: 21)
A1378 2011-01-18 19:37:33 314 ERROR There is a problem with a FRU. (FRU type: disk, enclosure: 1, device ID: 20, vendor: HP , product ID: EG0146FAWHU , SN: 3SD2XRAP00009031K1SL, version: HPDD, related event serial number: A1377, related event code:
A1379 2011-01-18 19:37:33 44 WARNING Unwritable cache data exists for a volume. (volume: , SN: 00c0ffdaabb900009613114c01000000) It comprises 1% of cache space.
A1380 2011-01-18 19:37:33 44 WARNING Unwritable cache data exists for a volume. (volume: , SN: 00c0ffdaabb90000f14fed4b01000000) It comprises 6% of cache space.
A1381 2011-01-18 19:37:33 44 WARNING Unwritable cache data exists for a volume. (volume: , SN: 00c0ffdaabb900002c50ed4b02000000) It comprises 1% of cache space.

But luckly the Vdisk reconstruction completed
A1382 2011-01-18 20:33:50 18 INFORMATIONAL Vdisk reconstruct completed successfully. (vdisk: vd01, SN: 00c0ffdaabb90000e4b4ab4b00000000)
Since the Disk in Enc 1 Slot 21 was timedout the vdisk is in critical state
A1383 2011-01-18 20:33:51 1 WARNING A vdisk is critical. (vdisk: vd01, SN: 00c0ffdaabb90000e4b4ab4b00000000)
Spare Kicked in vdisk Reconstruction is successfull
A1384 2011-01-18 20:33:55 9 INFORMATIONAL A spare disk kicked in for a critical vdisk. (vdisk: vd01, SN: 00c0ffdaabb90000e4b4ab4b00000000) (disk: channel: 0, ID: 23, SN: 3SD2SNQX00009029X400, enclosure: 1, slot: 24)
A1385 2011-01-18 20:33:55 37 INFORMATIONAL Vdisk reconstruct started. (vdisk: vd01, SN: 00c0ffdaabb90000e4b4ab4b00000000) (disk: channel: 0, ID: 23, SN: 3SD2SNQX00009029X400, enclosure: 1, slot: 24)
A1386 2011-01-18 21:40:45 18 INFORMATIONAL Vdisk reconstruct completed successfully. (vdisk: vd01, SN: 00c0ffdaabb90000e4b4ab4b00000000)

[Christoph-SC]

Kurze Info:

Der VMWare-Support ist letztendlich zu dem Ergebnis gekommen, dass der Datastore corrupted und nicht mehr rekonstruierbar ist

Vielen Dank für die Hilfe und viele Grüße
Christoph