Hi Leute,
ich hab mal eine "blöde" Frage:
Wir hatten am Wochenende das Problem, das eine virtuelle Maschine anscheinend einen Festplattenfehler hatte. Die VM ist mit folgender Meldung hängen geblieben. Es ging keine Anmeldung mehr am Server, es kam nur ein graues Bild:
Auf dem Volume "\\?\Volume{bbd05734-c883-11e5-9770-0050569f6580}" konnte der Transaktionsressourcen-Manager aufgrund eines nicht wiederholbaren Fehlers nicht gestartet werden. Der Fehlercode ist in den Daten enthalten.
Die Daten konnten nicht in das Transaktionsprotokoll verschoben werden. Die Daten sind möglicherweise beschädigt.
Ein Hardwaredefekt der physikalischen Festplatten ist auszuschließen, da die Netapp sonst was gemeldet hätte. Ist es also möglich, das virtuelle Platten auch einen defekt haben können?
Im Einsatz:
ESX 5.1
Netapp
Windows 2008 R2
grüße
Die Foren-SW läuft ohne erkennbare Probleme. Sollte doch etwas nicht funktionieren, bitte gerne hier jederzeit melden und wir kümmern uns zeitnah darum. Danke!
Können Virtuelle Disk auch Fehler haben, bzw. defekt sein?
-
kastlr
- Profi
- Beiträge: 993
- Registriert: 31.03.2008, 17:26
- Wohnort: Einzugsbereich des FC Schalke 04
- Kontaktdaten:
Hi,
neben simplen Hardware Defekten gibt es noch eine ganze Reihe von möglichen Fehlern im Zusammenhang mit Disk IO's.
Und ich bin mir ziemlich sicher, das bei den meisten dieser Fehler deine NetApp nichts melden wird, selbst wenn Sie Teil oder Verursacher des Problems sein sollte.
Ich würde an deiner Stelle mal in den vmkernel.logs des ESXi Servers nachsehen, ob sich dort Hinweise auf ein Problem finden lassen.
Eventuell wirst du zum Problemzeitraum einige SCSI Fehlermeldungen finden, vielleicht war auch der Kommunikationskanal zur NetApp nicht fehlerfrei.
Aus eigener Erfahrung kann ich dir nur raten, die Umgebung sowohl vom VMware als auch vom NetApp Support überprüfen zu lassen, solange die Logs das Zeitfenster mit dem Problem noch enthalten.
Gruß,
Ralf
neben simplen Hardware Defekten gibt es noch eine ganze Reihe von möglichen Fehlern im Zusammenhang mit Disk IO's.
Und ich bin mir ziemlich sicher, das bei den meisten dieser Fehler deine NetApp nichts melden wird, selbst wenn Sie Teil oder Verursacher des Problems sein sollte.
Ich würde an deiner Stelle mal in den vmkernel.logs des ESXi Servers nachsehen, ob sich dort Hinweise auf ein Problem finden lassen.
Eventuell wirst du zum Problemzeitraum einige SCSI Fehlermeldungen finden, vielleicht war auch der Kommunikationskanal zur NetApp nicht fehlerfrei.
Aus eigener Erfahrung kann ich dir nur raten, die Umgebung sowohl vom VMware als auch vom NetApp Support überprüfen zu lassen, solange die Logs das Zeitfenster mit dem Problem noch enthalten.
Gruß,
Ralf
Guten Abend,
die Netapp selbst kann solche Fehler gar nicht feststellen und somit Alarm schlagen. Dafür solltest du über ein Monitoring nachdenken. Um evtl. Schluckauf im Netzwerk oder evtl. Überbuchung festzustellen, würde ich über ein Monitoring wie Icinga oder PRTG nachdenken.
Meist sind solche Fehler auf das Netzwerk zurückzuführen. Gerade wenn du iSCSI gewählt hast, gibt es einige Fallstricke. Beispiel: Buffersize der Switches oder aber bei mehreren Netzwerkadaptern auf der ESXi ist kein Active/Active supportet. Dazu kommt noch im Guest die MPIO Konfiguration. Am Besten gleich die Logfiles von vCenter und Hosts einsammeln. Damit ist es am wahrscheinlichsten, dass alle Infos noch vorhanden sind und nichts überschrieben.
Gruß,
Dani
die Netapp selbst kann solche Fehler gar nicht feststellen und somit Alarm schlagen. Dafür solltest du über ein Monitoring nachdenken. Um evtl. Schluckauf im Netzwerk oder evtl. Überbuchung festzustellen, würde ich über ein Monitoring wie Icinga oder PRTG nachdenken.
Meist sind solche Fehler auf das Netzwerk zurückzuführen. Gerade wenn du iSCSI gewählt hast, gibt es einige Fallstricke. Beispiel: Buffersize der Switches oder aber bei mehreren Netzwerkadaptern auf der ESXi ist kein Active/Active supportet. Dazu kommt noch im Guest die MPIO Konfiguration. Am Besten gleich die Logfiles von vCenter und Hosts einsammeln. Damit ist es am wahrscheinlichsten, dass alle Infos noch vorhanden sind und nichts überschrieben.
Gruß,
Dani
Danke für die Antworten...
Die VMWare Server sind mit Fibre Channel mit der Netapp verbunden. Wir haben sogar PRTG im Einsatz. Ich konnte aber keine Auffälligkeiten feststellen. Wo hätten die Fehler vorkommen müssen? Auf dem FibreChannel Switch? Oder über Sensoren auf der Netapp? Evtl. hab ich ja genau diesen Sensor nicht eingerichtet!
Im Anhang ist ein Diagramm von dem Datastore auf dem die VM läuft. Zu dem Zeitpunkt wo der Ausfall war, sind peeks zu sehen. Was aber meiner Meinung nach erst mal nicht ungewöhnlich ist, ich hab zu anderen Zeiten auch höhere peeks gefunden... Ich weiß auch nicht, ab welcher höhe es ungewöhnlich wird, bzw. ein Alarm ausgegeben werden sollte!?
Die VMWare Server sind mit Fibre Channel mit der Netapp verbunden. Wir haben sogar PRTG im Einsatz. Ich konnte aber keine Auffälligkeiten feststellen. Wo hätten die Fehler vorkommen müssen? Auf dem FibreChannel Switch? Oder über Sensoren auf der Netapp? Evtl. hab ich ja genau diesen Sensor nicht eingerichtet!
Im Anhang ist ein Diagramm von dem Datastore auf dem die VM läuft. Zu dem Zeitpunkt wo der Ausfall war, sind peeks zu sehen. Was aber meiner Meinung nach erst mal nicht ungewöhnlich ist, ich hab zu anderen Zeiten auch höhere peeks gefunden... Ich weiß auch nicht, ab welcher höhe es ungewöhnlich wird, bzw. ein Alarm ausgegeben werden sollte!?
-
kastlr
- Profi
- Beiträge: 993
- Registriert: 31.03.2008, 17:26
- Wohnort: Einzugsbereich des FC Schalke 04
- Kontaktdaten:
Hallo,
leider können hier keine Attachments mehr angehängt warden, das ist hier schon länger ein leidiges Thema.
Wie auch immer, um mit der Fehlersuche zu beginnen würde ich mir auf dem ESXi Server mal folgende Logs ansehen.
Ich gehe aktuell auch nicht davon aus, dass du einen Hardware Fehler in deiner Umgebung hast.
Allerdings reicht es manchmal schon, wenn das Array nicht schnell genug auf die Anfragen antworten kann und die Applikation innerhalb einer VM dies nicht leiden mag.
Schon wird ein Fehler im Eventlog der VM protokolliert mit dem Verweis, dass es Probleme beim Zugriff auf die (virtuelle) Disk gibt.
Ohne Analyse der Logfiles wirst du so einen Fehler nicht finden können.
Gruß,
Ralf
leider können hier keine Attachments mehr angehängt warden, das ist hier schon länger ein leidiges Thema.
Wie auch immer, um mit der Fehlersuche zu beginnen würde ich mir auf dem ESXi Server mal folgende Logs ansehen.
- vmkernel.log
- vobd.log
Ich gehe aktuell auch nicht davon aus, dass du einen Hardware Fehler in deiner Umgebung hast.
Allerdings reicht es manchmal schon, wenn das Array nicht schnell genug auf die Anfragen antworten kann und die Applikation innerhalb einer VM dies nicht leiden mag.
Schon wird ein Fehler im Eventlog der VM protokolliert mit dem Verweis, dass es Probleme beim Zugriff auf die (virtuelle) Disk gibt.
Ohne Analyse der Logfiles wirst du so einen Fehler nicht finden können.
Gruß,
Ralf
-
continuum
- UNSTERBLICH(R.I.P.)
- Beiträge: 14759
- Registriert: 09.08.2003, 05:41
- Wohnort: sauerland
- Kontaktdaten:
Ist es also möglich, das virtuelle Platten auch einen defekt haben können?
Sicher - banalstes Beispiel wäre ein Syntaxfehler in der VMDK-beschreibungsdatei.
In diesem Fall kommen aber nur Fehler in Frage, die nicht zu einem sofortigen Ausfall führen.
Bei einer Delta-vmdk wird eine gewisse Anzahl von Graintable-fehlern toleriert, werden es zu viele kann die VMDK bei einer neu startenden VM nicht mehr benutzt werden.
Bei einer Flat-vmdk kommt es vor das einzelne Fragmente I/O-errors produzieren - bei Stellen die vom Gast nicht benutzt werden kann es sein dass diese Fehler nicht auffallen und man erst bei einem Cloneversuch feststellt, dass die VMDK sich nicht kopieren lässt.
Zur Bereinigung empfiehlt es sich, die VMDK per vmkfstools -i zu clonen.
Die neu erstellte VMDK kann man dann erstmal wieder als "clean" betrachten.
Sollte vmkfstools -i fehlschlagen ist die Sache ernst - aber in den meisten Fällen kann man auch dann noch etwas retten.
WIe es ausieht gibt es massve Probleme mit unserer Verbindung zwischen ESX Server über Fibre Channel mit dem Storage:
diese Meldungen sind im vobd.log zu finden und das nicht wenige!
3523 2016-02-07T01:41:12.536Z: [vmfsCorrelator] 10791076288458us: [esx.problem.vmfs.heartbeat.timedout] 4f99169f-12a01f11-cabc-d067e5eee823 xendatastore2
3524 2016-02-07T01:41:12.536Z: No correlator for vob.vmfs.heartbeat.timedout
3525 2016-02-07T01:41:12.536Z: [vmfsCorrelator] 10791076288629us: [esx.problem.vmfs.heartbeat.timedout] 4feaec43-24977e2a-c803-d067e5eeee6e xendatastore1
3526 2016-02-07T01:41:12.536Z: No correlator for vob.vmfs.heartbeat.timedout
3527 2016-02-07T01:41:57.552Z: No correlator for vob.vmfs.heartbeat.recovered
3528 2016-02-07T01:41:57.552Z: [vmfsCorrelator] 10791121304285us: [esx.problem.vmfs.heartbeat.recovered] 4feaec43-24977e2a-c803-d067e5eeee6e xendatastore1
3529 2016-02-07T01:41:57.703Z: [vmfsCorrelator] 10791121455984us: [esx.problem.vmfs.heartbeat.recovered] 4f99169f-12a01f11-cabc-d067e5eee823 xendatastore2
3530 2016-02-07T01:41:57.703Z: No correlator for vob.vmfs.heartbeat.recovered
3531 2016-02-07T01:43:15.604Z: [vmfsCorrelator] 10791199356469us: [esx.problem.vmfs.heartbeat.timedout] 4feaec43-24977e2a-c803-d067e5eeee6e xendatastore1
3532 2016-02-07T01:43:15.604Z: No correlator for vob.vmfs.heartbeat.timedout
3533 2016-02-07T01:43:15.604Z: [vmfsCorrelator] 10791199356677us: [esx.problem.vmfs.heartbeat.timedout] 4f99169f-12a01f11-cabc-d067e5eee823 xendatastore2
3534 2016-02-07T01:43:15.604Z: No correlator for vob.vmfs.heartbeat.timedout
3535 2016-02-07T01:43:23.692Z: No correlator for vob.vmfs.heartbeat.recovered
3536 2016-02-07T01:43:23.692Z: [vmfsCorrelator] 10791207445140us: [esx.problem.vmfs.heartbeat.recovered] 4f99169f-12a01f11-cabc-d067e5eee823 xendatastore2
3537 2016-02-07T01:43:23.764Z: No correlator for vob.vmfs.heartbeat.recovered
und
state in doubt; requested fast path state update...
5091 2016-02-07T01:41:12.464Z cpu6:4102)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
5092 2016-02-07T01:41:32.767Z cpu19:4115)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
5093 2016-02-07T01:41:32.791Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
5094 2016-02-07T01:41:52.643Z cpu8:4132)WARNING: HBX: 2697: Reclaiming timed out [HB state abcdef02 offset 3817472 gen 93 stampUS 10790655224671 uuid 5611f83b-2d8ebef8-8874-b8ac6f8c98ac jrnl <FB 346015> drv 14.54] on vol 'xendatastore1' failed: Timeou
5095 2016-02-07T01:41:52.643Z cpu14:4133)WARNING: HBX: 2697: Reclaiming timed out [HB state abcdef02 offset 3817472 gen 85 stampUS 10790655224442 uuid 5611f83b-2d8ebef8-8874-b8ac6f8c98ac jrnl <FB 602600> drv 14.54] on vol 'xendatastore2' failed: IO waESC[0$
5096 2016-02-07T01:42:55.523Z cpu8:4104)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
5097 2016-02-07T01:42:55.523Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
und
1040 2016-02-07T01:45:01.601Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003baf80) 0x2a, CmdSN 0x64a773 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possib
1041 2016-02-07T01:45:01.601Z cpu23:4291)<6>qla2xxx 0000:04:00.0: scsi(7:2:2): Abort command succeeded -- 1 155198234.
1042 2016-02-07T01:45:01.601Z cpu2:4098)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
1043 2016-02-07T01:45:01.601Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003eec00) 0x2a, CmdSN 0x40379e from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possib
1044 2016-02-07T01:45:03.600Z cpu23:4291)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 155198298.
1045 2016-02-07T01:45:03.600Z cpu6:4102)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
1046 2016-02-07T01:45:03.600Z cpu6:4102)ScsiDeviceIO: 2309: Cmd(0x4124003eec00) 0x2a, CmdSN 0x64a774 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possib
1047 2016-02-07T01:45:03.601Z cpu23:4291)<6>qla2xxx 0000:04:00.0: scsi(7:2:2): Abort command succeeded -- 1 155198299.
1048 2016-02-07T01:45:03.601Z cpu6:4102)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
1049 2016-02-07T01:45:03.601Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124003baf80) 0x2a, CmdSN 0x4037a0 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possib
1050 2016-02-07T01:45:03.671Z cpu13:4132)<6>qla2xxx 0000:04:00.0: scsi(7:2:2): Abort command succeeded -- 1 155198236.
1051 2016-02-07T01:45:03.671Z cpu22:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 155198243.
1052 2016-02-07T01:45:03.672Z cpu13:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 155198245.
Die Fehler treten Hauptsächlich Nachts auf. Kann mir aber nicht vorstellen, das es an der Sicherung hängt, da die Netapp ja eigentlich nicht den Fibrechannel nutz um Snapshots zu machen!?
diese Meldungen sind im vobd.log zu finden und das nicht wenige!
3523 2016-02-07T01:41:12.536Z: [vmfsCorrelator] 10791076288458us: [esx.problem.vmfs.heartbeat.timedout] 4f99169f-12a01f11-cabc-d067e5eee823 xendatastore2
3524 2016-02-07T01:41:12.536Z: No correlator for vob.vmfs.heartbeat.timedout
3525 2016-02-07T01:41:12.536Z: [vmfsCorrelator] 10791076288629us: [esx.problem.vmfs.heartbeat.timedout] 4feaec43-24977e2a-c803-d067e5eeee6e xendatastore1
3526 2016-02-07T01:41:12.536Z: No correlator for vob.vmfs.heartbeat.timedout
3527 2016-02-07T01:41:57.552Z: No correlator for vob.vmfs.heartbeat.recovered
3528 2016-02-07T01:41:57.552Z: [vmfsCorrelator] 10791121304285us: [esx.problem.vmfs.heartbeat.recovered] 4feaec43-24977e2a-c803-d067e5eeee6e xendatastore1
3529 2016-02-07T01:41:57.703Z: [vmfsCorrelator] 10791121455984us: [esx.problem.vmfs.heartbeat.recovered] 4f99169f-12a01f11-cabc-d067e5eee823 xendatastore2
3530 2016-02-07T01:41:57.703Z: No correlator for vob.vmfs.heartbeat.recovered
3531 2016-02-07T01:43:15.604Z: [vmfsCorrelator] 10791199356469us: [esx.problem.vmfs.heartbeat.timedout] 4feaec43-24977e2a-c803-d067e5eeee6e xendatastore1
3532 2016-02-07T01:43:15.604Z: No correlator for vob.vmfs.heartbeat.timedout
3533 2016-02-07T01:43:15.604Z: [vmfsCorrelator] 10791199356677us: [esx.problem.vmfs.heartbeat.timedout] 4f99169f-12a01f11-cabc-d067e5eee823 xendatastore2
3534 2016-02-07T01:43:15.604Z: No correlator for vob.vmfs.heartbeat.timedout
3535 2016-02-07T01:43:23.692Z: No correlator for vob.vmfs.heartbeat.recovered
3536 2016-02-07T01:43:23.692Z: [vmfsCorrelator] 10791207445140us: [esx.problem.vmfs.heartbeat.recovered] 4f99169f-12a01f11-cabc-d067e5eee823 xendatastore2
3537 2016-02-07T01:43:23.764Z: No correlator for vob.vmfs.heartbeat.recovered
und
state in doubt; requested fast path state update...
5091 2016-02-07T01:41:12.464Z cpu6:4102)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
5092 2016-02-07T01:41:32.767Z cpu19:4115)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
5093 2016-02-07T01:41:32.791Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
5094 2016-02-07T01:41:52.643Z cpu8:4132)WARNING: HBX: 2697: Reclaiming timed out [HB state abcdef02 offset 3817472 gen 93 stampUS 10790655224671 uuid 5611f83b-2d8ebef8-8874-b8ac6f8c98ac jrnl <FB 346015> drv 14.54] on vol 'xendatastore1' failed: Timeou
5095 2016-02-07T01:41:52.643Z cpu14:4133)WARNING: HBX: 2697: Reclaiming timed out [HB state abcdef02 offset 3817472 gen 85 stampUS 10790655224442 uuid 5611f83b-2d8ebef8-8874-b8ac6f8c98ac jrnl <FB 602600> drv 14.54] on vol 'xendatastore2' failed: IO waESC[0$
5096 2016-02-07T01:42:55.523Z cpu8:4104)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
5097 2016-02-07T01:42:55.523Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
und
1040 2016-02-07T01:45:01.601Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003baf80) 0x2a, CmdSN 0x64a773 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possib
1041 2016-02-07T01:45:01.601Z cpu23:4291)<6>qla2xxx 0000:04:00.0: scsi(7:2:2): Abort command succeeded -- 1 155198234.
1042 2016-02-07T01:45:01.601Z cpu2:4098)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
1043 2016-02-07T01:45:01.601Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003eec00) 0x2a, CmdSN 0x40379e from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possib
1044 2016-02-07T01:45:03.600Z cpu23:4291)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 155198298.
1045 2016-02-07T01:45:03.600Z cpu6:4102)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
1046 2016-02-07T01:45:03.600Z cpu6:4102)ScsiDeviceIO: 2309: Cmd(0x4124003eec00) 0x2a, CmdSN 0x64a774 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possib
1047 2016-02-07T01:45:03.601Z cpu23:4291)<6>qla2xxx 0000:04:00.0: scsi(7:2:2): Abort command succeeded -- 1 155198299.
1048 2016-02-07T01:45:03.601Z cpu6:4102)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
1049 2016-02-07T01:45:03.601Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124003baf80) 0x2a, CmdSN 0x4037a0 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possib
1050 2016-02-07T01:45:03.671Z cpu13:4132)<6>qla2xxx 0000:04:00.0: scsi(7:2:2): Abort command succeeded -- 1 155198236.
1051 2016-02-07T01:45:03.671Z cpu22:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 155198243.
1052 2016-02-07T01:45:03.672Z cpu13:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 155198245.
Die Fehler treten Hauptsächlich Nachts auf. Kann mir aber nicht vorstellen, das es an der Sicherung hängt, da die Netapp ja eigentlich nicht den Fibrechannel nutz um Snapshots zu machen!?
-
irix
- King of the Hill
- Beiträge: 13063
- Registriert: 02.08.2008, 15:06
- Wohnort: Hannover/Wuerzburg
- Kontaktdaten:
Also ESXi 5.1 ist ja nun schon aelter und du hast uns nicht deinen Patchstand verraten.
Des weiteren wissen wir nicht wie du Backups bzw. mit welchem Programm und mit welchem der vielen Modi. Alleine VADP bietet
- NBD (LAN Mode)
- HotAdd
- SAN
und dann die Programme welche die Daten direkt von einem SAN Snapshot holen koennen.
Gruss
Joerg
Des weiteren wissen wir nicht wie du Backups bzw. mit welchem Programm und mit welchem der vielen Modi. Alleine VADP bietet
- NBD (LAN Mode)
- HotAdd
- SAN
und dann die Programme welche die Daten direkt von einem SAN Snapshot holen koennen.
Gruss
Joerg
-
kastlr
- Profi
- Beiträge: 993
- Registriert: 31.03.2008, 17:26
- Wohnort: Einzugsbereich des FC Schalke 04
- Kontaktdaten:
Hallo zusammen,
zumindest meldet dein ESXi Host, dass Write IO's (0x2a) der VM/des Prozesses mit der World ID 4120 fehlschlagen.
Vermutlich ist das Array zu diesen Zeiten nicht in der Lage, die Write IO's in akzeptabler Zeit zu beantworten.
Und da der HBA Treiber involviert ist, haben diese IO's deinen Host in jedem Fall in Richtung Array verlassen.
Ich habe nur noch ESXi 5.5 Hosts, aber eventuell liefert dir ja der folgende Befehl weitere Erkenntnisse.
Was für eine NMP Policy habt Ihr denn in Verwendung?
Gruß,
Ralf
zumindest meldet dein ESXi Host, dass Write IO's (0x2a) der VM/des Prozesses mit der World ID 4120 fehlschlagen.
Vermutlich ist das Array zu diesen Zeiten nicht in der Lage, die Write IO's in akzeptabler Zeit zu beantworten.
Und da der HBA Treiber involviert ist, haben diese IO's deinen Host in jedem Fall in Richtung Array verlassen.
Ich habe nur noch ESXi 5.5 Hosts, aber eventuell liefert dir ja der folgende Befehl weitere Erkenntnisse.
Code: Alles auswählen
esxcli storage core adapter stats getWas für eine NMP Policy habt Ihr denn in Verwendung?
Gruß,
Ralf
Der Befehl liefert viele Fehler Erkenntnisse;)
vmhba0
Successful Commands: 7
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 350979
Failed Blocks Read: 6734
Failed Blocks Written: 0
Failed Read Operations: 6734
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba1
Successful Commands: 96227664
Blocks Read: 2174089036
Blocks Written: 2947600851
Read Operations: 40114429
Write Operations: 55401284
Reserve Operations: 2044
Reservation Conflicts: 0
Failed Commands: 27136691
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba2
Successful Commands: 7645635265
Blocks Read: 287686084180
Blocks Written: 84221185699
Read Operations: 6566270291
Write Operations: 1047905192
Reserve Operations: 720
Reservation Conflicts: 0
Failed Commands: 27451
Failed Blocks Read: 552
Failed Blocks Written: 116356
Failed Read Operations: 1518
Failed Write Operations: 6282
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba3
Successful Commands: 7586565149
Blocks Read: 284350587492
Blocks Written: 84177383561
Read Operations: 6515024467
Write Operations: 1045287137
Reserve Operations: 8
Reservation Conflicts: 0
Failed Commands: 626977
Failed Blocks Read: 1436
Failed Blocks Written: 155431
Failed Read Operations: 33
Failed Write Operations: 3755
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba4
Successful Commands: 6842961
Blocks Read: 659835033
Blocks Written: 3659
Read Operations: 5133267
Write Operations: 2004
Reserve Operations: 1
Reservation Conflicts: 0
Failed Commands: 4261
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba5
Successful Commands: 421822
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 4257
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
NMP Policy: Round Robin würde ich behaupten, wenn du das meinst, wo sehe ich das noch mal?
vmhba0
Successful Commands: 7
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 350979
Failed Blocks Read: 6734
Failed Blocks Written: 0
Failed Read Operations: 6734
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba1
Successful Commands: 96227664
Blocks Read: 2174089036
Blocks Written: 2947600851
Read Operations: 40114429
Write Operations: 55401284
Reserve Operations: 2044
Reservation Conflicts: 0
Failed Commands: 27136691
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba2
Successful Commands: 7645635265
Blocks Read: 287686084180
Blocks Written: 84221185699
Read Operations: 6566270291
Write Operations: 1047905192
Reserve Operations: 720
Reservation Conflicts: 0
Failed Commands: 27451
Failed Blocks Read: 552
Failed Blocks Written: 116356
Failed Read Operations: 1518
Failed Write Operations: 6282
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba3
Successful Commands: 7586565149
Blocks Read: 284350587492
Blocks Written: 84177383561
Read Operations: 6515024467
Write Operations: 1045287137
Reserve Operations: 8
Reservation Conflicts: 0
Failed Commands: 626977
Failed Blocks Read: 1436
Failed Blocks Written: 155431
Failed Read Operations: 33
Failed Write Operations: 3755
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba4
Successful Commands: 6842961
Blocks Read: 659835033
Blocks Written: 3659
Read Operations: 5133267
Write Operations: 2004
Reserve Operations: 1
Reservation Conflicts: 0
Failed Commands: 4261
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba5
Successful Commands: 421822
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 4257
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
NMP Policy: Round Robin würde ich behaupten, wenn du das meinst, wo sehe ich das noch mal?
-
kastlr
- Profi
- Beiträge: 993
- Registriert: 31.03.2008, 17:26
- Wohnort: Einzugsbereich des FC Schalke 04
- Kontaktdaten:
Na ja,
einige deiner vmhba's wirst du sicher so gut wie gar nicht verwenden.
Wahrscheinlich ist die NetApp über die vmhba's 2 + 3 angeschlossen.
Auffällig ist allerdings, das die Fehlerrate beim vmhba 3 deutlich höher ist als beim vmhba2 trotz annähernd vergleichbarer IO Aktivitäten.
Das deutet auch darauf hin, dass ihr die RR NMP einsetzt, das kannst du mit folgendem Kommando überprüfen.
Du kannst außerdem mal den vmkernel.log (und seine Archive unter /var/run/log) nach dem String H:0x durchsuchen lassen.
Damit erhältst du eine Übersicht über alle aufgetretenen SCSI Fehler.
Gruß,
Ralf
einige deiner vmhba's wirst du sicher so gut wie gar nicht verwenden.
Wahrscheinlich ist die NetApp über die vmhba's 2 + 3 angeschlossen.
Auffällig ist allerdings, das die Fehlerrate beim vmhba 3 deutlich höher ist als beim vmhba2 trotz annähernd vergleichbarer IO Aktivitäten.
Das deutet auch darauf hin, dass ihr die RR NMP einsetzt, das kannst du mit folgendem Kommando überprüfen.
Code: Alles auswählen
esxcli storage nmp device listDu kannst außerdem mal den vmkernel.log (und seine Archive unter /var/run/log) nach dem String H:0x durchsuchen lassen.
Damit erhältst du eine Übersicht über alle aufgetretenen SCSI Fehler.
Gruß,
Ralf
Hier mal ein kleiner Ausschnitt vom Befehl:
naa.60a9800044306931423f435734336474
Device Display Name: NETAPP Fibre Channel Disk (naa.60a9800044306931423f435734336474)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=1,TPG_state=ANO}{TPG_id=0,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=6: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T3:L3, vmhba2:C0:T2:L3, vmhba3:C0:T3:L3, vmhba3:C0:T2:L3
Is Local SAS Device: false
Is Boot USB Device: false
naa.60a9800044313450433f4357345a5162
Device Display Name: NETAPP Fibre Channel Disk (naa.60a9800044313450433f4357345a5162)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T0:L42, vmhba2:C0:T1:L42, vmhba3:C0:T0:L42, vmhba3:C0:T1:L42
Is Local SAS Device: false
Is Boot USB Device: false
naa.60a9800044306931423f435734343843
Device Display Name: NETAPP Fibre Channel Disk (naa.60a9800044306931423f435734343843)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=1,TPG_state=ANO}{TPG_id=0,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=4: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T3:L14, vmhba3:C0:T2:L14, vmhba3:C0:T3:L14, vmhba2:C0:T2:L14
Is Local SAS Device: false
Is Boot USB Device: false
naa.60a9800044313450433f43573458344d
Device Display Name: NETAPP Fibre Channel Disk (naa.60a9800044313450433f43573458344d)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=3: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba3:C0:T1:L33, vmhba2:C0:T0:L33, vmhba2:C0:T1:L33, vmhba3:C0:T0:L33
Is Local SAS Device: false
Is Boot USB Device: false
und ein Auschnitt aus dem vmkernel.log
2016-02-11T05:23:38.228Z cpu3:4725723)ScsiDeviceIO: 2331: Cmd(0x412401f1dd80) 0x1a, CmdSN 0x10b6905 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:28:38.240Z cpu8:18847081)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124030feec0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:28:38.240Z cpu8:18847081)ScsiDeviceIO: 2331: Cmd(0x4124030feec0) 0x1a, CmdSN 0x10b6956 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:33:38.256Z cpu5:4725726)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412402a40180, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:33:38.256Z cpu5:4725726)ScsiDeviceIO: 2331: Cmd(0x412402a40180) 0x1a, CmdSN 0x10b69a7 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:38:20.692Z cpu17:8416675)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x41244220ca40, 9279) to dev "naa.600508e0000000003669c8b032e65209" on path "vmhba1:C1:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:38:20.692Z cpu17:8416675)ScsiDeviceIO: 2331: Cmd(0x41244220ca40) 0x85, CmdSN 0xce62 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:38:20.693Z cpu17:8416675)ScsiDeviceIO: 2331: Cmd(0x41244220ca40) 0x4d, CmdSN 0xce63 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:38:20.693Z cpu17:8416675)ScsiDeviceIO: 2331: Cmd(0x41244220ca40) 0x1a, CmdSN 0xce64 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-11T05:38:38.272Z cpu5:18847081)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124024a25c0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:38:38.272Z cpu5:18847081)ScsiDeviceIO: 2331: Cmd(0x4124024a25c0) 0x1a, CmdSN 0x10b69fe from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:43:38.287Z cpu7:18838888)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124030f84c0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:43:38.287Z cpu7:18838888)ScsiDeviceIO: 2331: Cmd(0x4124030f84c0) 0x1a, CmdSN 0x10b6a4f from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:48:38.304Z cpu7:8887564)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412402a46880, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:48:38.304Z cpu7:8887564)ScsiDeviceIO: 2331: Cmd(0x412402a46880) 0x1a, CmdSN 0x10b6aa0 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:53:38.317Z cpu9:18847081)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412402dcab80, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:53:38.317Z cpu9:18847081)ScsiDeviceIO: 2331: Cmd(0x412402dcab80) 0x1a, CmdSN 0x10b6af1 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:58:38.333Z cpu19:8211)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412402441fc0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:58:38.333Z cpu19:8211)ScsiDeviceIO: 2331: Cmd(0x412402441fc0) 0x1a, CmdSN 0x10b6b42 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:03:38.349Z cpu21:8416673)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412401704dc0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:03:38.349Z cpu21:8416673)ScsiDeviceIO: 2331: Cmd(0x412401704dc0) 0x1a, CmdSN 0x10b6b9d from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:08:20.837Z cpu13:8205)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x4124403bc380, 9279) to dev "naa.600508e0000000003669c8b032e65209" on path "vmhba1:C1:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:08:20.837Z cpu13:8205)ScsiDeviceIO: 2331: Cmd(0x4124403bc380) 0x85, CmdSN 0xce65 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:08:20.838Z cpu13:8636)ScsiDeviceIO: 2331: Cmd(0x4124403bc380) 0x4d, CmdSN 0xce66 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:08:20.838Z cpu13:8636)ScsiDeviceIO: 2331: Cmd(0x4124403bc380) 0x1a, CmdSN 0xce67 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-11T06:08:38.366Z cpu12:8204)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412401c2ca00, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:08:38.366Z cpu12:8204)ScsiDeviceIO: 2331: Cmd(0x412401c2ca00) 0x1a, CmdSN 0x10b6bf4 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:13:38.386Z cpu23:18847486)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124003ba700, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:13:38.386Z cpu23:18847486)ScsiDeviceIO: 2331: Cmd(0x4124003ba700) 0x1a, CmdSN 0x10b6c45 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:18:38.399Z cpu7:4725723)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412401c01a80, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:18:38.399Z cpu7:4725723)ScsiDeviceIO: 2331: Cmd(0x412401c01a80) 0x1a, CmdSN 0x10b6c96 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:23:38.416Z cpu15:8207)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412401d8ea80, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:23:38.416Z cpu15:8207)ScsiDeviceIO: 2331: Cmd(0x412401d8ea80) 0x1a, CmdSN 0x10b6ce7 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:28:38.428Z cpu11:1405102)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412401ef52c0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:28:38.428Z cpu11:1405102)ScsiDeviceIO: 2331: Cmd(0x412401ef52c0) 0x1a, CmdSN 0x10b6d38 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:33:38.444Z cpu7:18838888)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124003b8f00, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:33:38.444Z cpu7:18838888)ScsiDeviceIO: 2331: Cmd(0x4124003b8f00) 0x1a, CmdSN 0x10b6d89 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:38:20.983Z cpu16:8416673)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x4124403f5f00, 9279) to dev "naa.600508e0000000003669c8b032e65209" on path "vmhba1:C1:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:38:20.983Z cpu16:8416673)ScsiDeviceIO: 2331: Cmd(0x4124403f5f00) 0x85, CmdSN 0xce68 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:38:20.983Z cpu16:8416673)ScsiDeviceIO: 2331: Cmd(0x4124403f5f00) 0x4d, CmdSN 0xce69 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:38:20.983Z cpu16:8416673)ScsiDeviceIO: 2331: Cmd(0x4124403f5f00) 0x1a, CmdSN 0xce6a from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-11T06:38:38.462Z cpu9:8201)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124003e2ac0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:38:38.462Z cpu9:8201)ScsiDeviceIO: 2331: Cmd(0x4124003e2ac0) 0x1a, CmdSN 0x10b6de0 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:43:38.482Z cpu14:8206)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412402019640, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:43:38.482Z cpu14:8206)ScsiDeviceIO: 2331: Cmd(0x412402019640) 0x1a, CmdSN 0x10b6e31 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
naa.60a9800044306931423f435734336474
Device Display Name: NETAPP Fibre Channel Disk (naa.60a9800044306931423f435734336474)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=1,TPG_state=ANO}{TPG_id=0,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=6: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T3:L3, vmhba2:C0:T2:L3, vmhba3:C0:T3:L3, vmhba3:C0:T2:L3
Is Local SAS Device: false
Is Boot USB Device: false
naa.60a9800044313450433f4357345a5162
Device Display Name: NETAPP Fibre Channel Disk (naa.60a9800044313450433f4357345a5162)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T0:L42, vmhba2:C0:T1:L42, vmhba3:C0:T0:L42, vmhba3:C0:T1:L42
Is Local SAS Device: false
Is Boot USB Device: false
naa.60a9800044306931423f435734343843
Device Display Name: NETAPP Fibre Channel Disk (naa.60a9800044306931423f435734343843)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=1,TPG_state=ANO}{TPG_id=0,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=4: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T3:L14, vmhba3:C0:T2:L14, vmhba3:C0:T3:L14, vmhba2:C0:T2:L14
Is Local SAS Device: false
Is Boot USB Device: false
naa.60a9800044313450433f43573458344d
Device Display Name: NETAPP Fibre Channel Disk (naa.60a9800044313450433f43573458344d)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=3,TPG_state=ANO}{TPG_id=2,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=3: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba3:C0:T1:L33, vmhba2:C0:T0:L33, vmhba2:C0:T1:L33, vmhba3:C0:T0:L33
Is Local SAS Device: false
Is Boot USB Device: false
und ein Auschnitt aus dem vmkernel.log
2016-02-11T05:23:38.228Z cpu3:4725723)ScsiDeviceIO: 2331: Cmd(0x412401f1dd80) 0x1a, CmdSN 0x10b6905 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:28:38.240Z cpu8:18847081)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124030feec0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:28:38.240Z cpu8:18847081)ScsiDeviceIO: 2331: Cmd(0x4124030feec0) 0x1a, CmdSN 0x10b6956 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:33:38.256Z cpu5:4725726)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412402a40180, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:33:38.256Z cpu5:4725726)ScsiDeviceIO: 2331: Cmd(0x412402a40180) 0x1a, CmdSN 0x10b69a7 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:38:20.692Z cpu17:8416675)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x41244220ca40, 9279) to dev "naa.600508e0000000003669c8b032e65209" on path "vmhba1:C1:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:38:20.692Z cpu17:8416675)ScsiDeviceIO: 2331: Cmd(0x41244220ca40) 0x85, CmdSN 0xce62 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:38:20.693Z cpu17:8416675)ScsiDeviceIO: 2331: Cmd(0x41244220ca40) 0x4d, CmdSN 0xce63 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:38:20.693Z cpu17:8416675)ScsiDeviceIO: 2331: Cmd(0x41244220ca40) 0x1a, CmdSN 0xce64 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-11T05:38:38.272Z cpu5:18847081)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124024a25c0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:38:38.272Z cpu5:18847081)ScsiDeviceIO: 2331: Cmd(0x4124024a25c0) 0x1a, CmdSN 0x10b69fe from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:43:38.287Z cpu7:18838888)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124030f84c0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:43:38.287Z cpu7:18838888)ScsiDeviceIO: 2331: Cmd(0x4124030f84c0) 0x1a, CmdSN 0x10b6a4f from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:48:38.304Z cpu7:8887564)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412402a46880, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:48:38.304Z cpu7:8887564)ScsiDeviceIO: 2331: Cmd(0x412402a46880) 0x1a, CmdSN 0x10b6aa0 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:53:38.317Z cpu9:18847081)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412402dcab80, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:53:38.317Z cpu9:18847081)ScsiDeviceIO: 2331: Cmd(0x412402dcab80) 0x1a, CmdSN 0x10b6af1 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T05:58:38.333Z cpu19:8211)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412402441fc0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T05:58:38.333Z cpu19:8211)ScsiDeviceIO: 2331: Cmd(0x412402441fc0) 0x1a, CmdSN 0x10b6b42 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:03:38.349Z cpu21:8416673)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412401704dc0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:03:38.349Z cpu21:8416673)ScsiDeviceIO: 2331: Cmd(0x412401704dc0) 0x1a, CmdSN 0x10b6b9d from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:08:20.837Z cpu13:8205)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x4124403bc380, 9279) to dev "naa.600508e0000000003669c8b032e65209" on path "vmhba1:C1:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:08:20.837Z cpu13:8205)ScsiDeviceIO: 2331: Cmd(0x4124403bc380) 0x85, CmdSN 0xce65 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:08:20.838Z cpu13:8636)ScsiDeviceIO: 2331: Cmd(0x4124403bc380) 0x4d, CmdSN 0xce66 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:08:20.838Z cpu13:8636)ScsiDeviceIO: 2331: Cmd(0x4124403bc380) 0x1a, CmdSN 0xce67 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-11T06:08:38.366Z cpu12:8204)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412401c2ca00, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:08:38.366Z cpu12:8204)ScsiDeviceIO: 2331: Cmd(0x412401c2ca00) 0x1a, CmdSN 0x10b6bf4 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:13:38.386Z cpu23:18847486)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124003ba700, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:13:38.386Z cpu23:18847486)ScsiDeviceIO: 2331: Cmd(0x4124003ba700) 0x1a, CmdSN 0x10b6c45 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:18:38.399Z cpu7:4725723)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412401c01a80, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:18:38.399Z cpu7:4725723)ScsiDeviceIO: 2331: Cmd(0x412401c01a80) 0x1a, CmdSN 0x10b6c96 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:23:38.416Z cpu15:8207)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412401d8ea80, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:23:38.416Z cpu15:8207)ScsiDeviceIO: 2331: Cmd(0x412401d8ea80) 0x1a, CmdSN 0x10b6ce7 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:28:38.428Z cpu11:1405102)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412401ef52c0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:28:38.428Z cpu11:1405102)ScsiDeviceIO: 2331: Cmd(0x412401ef52c0) 0x1a, CmdSN 0x10b6d38 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:33:38.444Z cpu7:18838888)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124003b8f00, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:33:38.444Z cpu7:18838888)ScsiDeviceIO: 2331: Cmd(0x4124003b8f00) 0x1a, CmdSN 0x10b6d89 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:38:20.983Z cpu16:8416673)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x85 (0x4124403f5f00, 9279) to dev "naa.600508e0000000003669c8b032e65209" on path "vmhba1:C1:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:38:20.983Z cpu16:8416673)ScsiDeviceIO: 2331: Cmd(0x4124403f5f00) 0x85, CmdSN 0xce68 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:38:20.983Z cpu16:8416673)ScsiDeviceIO: 2331: Cmd(0x4124403f5f00) 0x4d, CmdSN 0xce69 from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:38:20.983Z cpu16:8416673)ScsiDeviceIO: 2331: Cmd(0x4124403f5f00) 0x1a, CmdSN 0xce6a from world 9279 to dev "naa.600508e0000000003669c8b032e65209" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-11T06:38:38.462Z cpu9:8201)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124003e2ac0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:38:38.462Z cpu9:8201)ScsiDeviceIO: 2331: Cmd(0x4124003e2ac0) 0x1a, CmdSN 0x10b6de0 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2016-02-11T06:43:38.482Z cpu14:8206)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412402019640, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
2016-02-11T06:43:38.482Z cpu14:8206)ScsiDeviceIO: 2331: Cmd(0x412402019640) 0x1a, CmdSN 0x10b6e31 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Die anderen Ausgaben waren ziemlich lang, deswegen hatte ich nur auschnitte geschickt.
HBA Name Driver Link State UID Description
-------- --------- ---------- ------------------------------------ ------------------------------------------------------------------------
vmhba0 ata_piix link-n/a sata.vmhba0 (0:0:31.2) Intel Corporation 2 port SATA IDE Controller (ICH9)
vmhba1 mpt2sas link-n/a sas.5782bcb05c3d8600 (0:3:0.0) LSI Logic / Symbios Logic Dell PERC H200 Integrated
vmhba2 qla2xxx link-up fc.20000024ff05f30e:21000024ff05f30e (0:4:0.0) QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA
vmhba3 qla2xxx link-up fc.20000024ff05f247:21000024ff05f247 (0:5:0.0) QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA
vmhba4 qla2xxx link-up fc.2000001b3284500c:2100001b3284500c (0:6:0.0) QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA
vmhba5 qla2xxx link-up fc.2001001b32a4500c:2101001b32a4500c (0:6:0.1) QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA
vmhba32 ata_piix link-n/a sata.vmhba32 (0:0:31.2) Intel Corporation 2 port SATA IDE Controller (ICH9)
vmhba33 iscsi_vmk online iscsi.vmhba33 iSCSI Software Adapter
lspci | grep -i vmhba
00:00:1f.2 Mass storage controller: Intel Corporation 2 port SATA IDE Controller (ICH9) [vmhba0]
00:03:00.0 Mass storage controller: LSI Logic / Symbios Logic Dell PERC H200 Integrated [vmhba1]
00:04:00.0 Serial bus controller: QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA [vmhba2]
00:05:00.0 Serial bus controller: QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA [vmhba3]
00:06:00.0 Serial bus controller: QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA [vmhba4]
00:06:00.1 Serial bus controller: QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA [vmhba5]
HBA Name Driver Link State UID Description
-------- --------- ---------- ------------------------------------ ------------------------------------------------------------------------
vmhba0 ata_piix link-n/a sata.vmhba0 (0:0:31.2) Intel Corporation 2 port SATA IDE Controller (ICH9)
vmhba1 mpt2sas link-n/a sas.5782bcb05c3d8600 (0:3:0.0) LSI Logic / Symbios Logic Dell PERC H200 Integrated
vmhba2 qla2xxx link-up fc.20000024ff05f30e:21000024ff05f30e (0:4:0.0) QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA
vmhba3 qla2xxx link-up fc.20000024ff05f247:21000024ff05f247 (0:5:0.0) QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA
vmhba4 qla2xxx link-up fc.2000001b3284500c:2100001b3284500c (0:6:0.0) QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA
vmhba5 qla2xxx link-up fc.2001001b32a4500c:2101001b32a4500c (0:6:0.1) QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA
vmhba32 ata_piix link-n/a sata.vmhba32 (0:0:31.2) Intel Corporation 2 port SATA IDE Controller (ICH9)
vmhba33 iscsi_vmk online iscsi.vmhba33 iSCSI Software Adapter
lspci | grep -i vmhba
00:00:1f.2 Mass storage controller: Intel Corporation 2 port SATA IDE Controller (ICH9) [vmhba0]
00:03:00.0 Mass storage controller: LSI Logic / Symbios Logic Dell PERC H200 Integrated [vmhba1]
00:04:00.0 Serial bus controller: QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA [vmhba2]
00:05:00.0 Serial bus controller: QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA [vmhba3]
00:06:00.0 Serial bus controller: QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA [vmhba4]
00:06:00.1 Serial bus controller: QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA [vmhba5]
Ich hab noch mal von einem ESX alles zusammengefaßt, der auch die Probleme bringt, aber nicht so viele LUNs verbunden hat. Leider funktioniert ja das Anhängen von Dateien nicht, also hier die komplteet txt. Datei:
~ # esxcli storage core adapter stats get
vmhba0
Successful Commands: 3
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 37496
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba1
Successful Commands: 13019647
Blocks Read: 475321286
Blocks Written: 450392695
Read Operations: 7492015
Write Operations: 5352612
Reserve Operations: 18625
Reservation Conflicts: 0
Failed Commands: 9568884
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba2
Successful Commands: 314775389
Blocks Read: 586606724
Blocks Written: 7282562059
Read Operations: 10248659
Write Operations: 303788756
Reserve Operations: 480
Reservation Conflicts: 521
Failed Commands: 1868
Failed Blocks Read: 0
Failed Blocks Written: 52888
Failed Read Operations: 190
Failed Write Operations: 1516
Failed Reserve Operations: 107
Total Splits: 0
PAE Commands: 0
vmhba3
Successful Commands: 314965516
Blocks Read: 589293158
Blocks Written: 7282184408
Read Operations: 10248601
Write Operations: 303980219
Reserve Operations: 369
Reservation Conflicts: 638
Failed Commands: 1782
Failed Blocks Read: 0
Failed Blocks Written: 35248
Failed Read Operations: 182
Failed Write Operations: 1462
Failed Reserve Operations: 101
Total Splits: 0
PAE Commands: 0
vmhba32
Successful Commands: 0
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 0
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba33
Successful Commands: 0
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 0
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba34
Successful Commands: 0
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 0
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba35
Successful Commands: 0
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 0
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba36
Successful Commands: 0
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 256
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
~ # esxcli storage nmp device list
naa.600508e000000000d85b294d14b5f10d
Device Display Name: Dell Serial Attached SCSI Disk (naa.600508e000000000d85b294d14b5f10d)
Storage Array Type: VMW_SATP_LOCAL
Storage Array Type Device Config: SATP VMW_SATP_LOCAL does not support device configuration.
Path Selection Policy: VMW_PSP_FIXED
Path Selection Policy Device Config: {preferred=vmhba1:C1:T0:L0;current=vmhba1:C1:T0:L0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba1:C1:T0:L0
naa.60a9800044313450433f435734567638
Device Display Name: NETAPP Fibre Channel Disk (naa.60a9800044313450433f435734567638)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=2,TPG_state=AO}{TPG_id=3,TPG_state=ANO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba3:C0:T1:L0, vmhba3:C0:T0:L0, vmhba2:C0:T1:L0, vmhba2:C0:T0:L0
naa.60a980004335432d68346a6431745431
Device Display Name: NETAPP Fibre Channel Disk (naa.60a980004335432d68346a6431745431)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=1,TPG_state=ANO}{TPG_id=0,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=6: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T3:L3, vmhba2:C0:T2:L3, vmhba3:C0:T3:L3, vmhba3:C0:T2:L3
naa.60a980004335432d68346b7239454f67
Device Display Name: NETAPP Fibre Channel Disk (naa.60a980004335432d68346b7239454f67)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=1,TPG_state=ANO}{TPG_id=0,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=6: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T3:L2, vmhba2:C0:T2:L2, vmhba3:C0:T3:L2, vmhba3:C0:T2:L2
mpx.vmhba0:C0:T0:L0
Device Display Name: Local PLDS CD-ROM (mpx.vmhba0:C0:T0:L0)
Storage Array Type: VMW_SATP_LOCAL
Storage Array Type Device Config: SATP VMW_SATP_LOCAL does not support device configuration.
Path Selection Policy: VMW_PSP_FIXED
Path Selection Policy Device Config: {preferred=vmhba0:C0:T0:L0;current=vmhba0:C0:T0:L0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba0:C0:T0:L0
VMKernel.log
2016-02-07T01:41:57.827Z cpu13:4133)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore1' (held for 3589 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:42:20.572Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x4124403aa600) 0x12, CmdSN 0x75369 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T01:42:55.440Z cpu13:4290)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511532.
2016-02-07T01:42:55.440Z cpu10:4106)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412401ad6a80, 4120) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba2:C0:T3:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:42:55.440Z cpu10:4106)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:42:55.440Z cpu10:4106)ScsiDeviceIO: 2309: Cmd(0x412401ad6a80) 0x2a, CmdSN 0x4e8d0c from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:42:55.440Z cpu15:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511531.
2016-02-07T01:42:55.440Z cpu4:4100)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124003b0f80, 4120) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:42:55.440Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:42:55.440Z cpu4:4100)ScsiDeviceIO: 2309: Cmd(0x4124003b0f80) 0x2a, CmdSN 0x407805 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:42:55.618Z cpu15:6177)J3: 2611: Aborting txn (0x4100241c9840) callerID: 0xc1d00002 due to failure pre-committing: Optimistic lock acquired by another host
2016-02-07T01:42:55.817Z cpu15:6177)DLX: 3394: vol 'xendatastore1': [Req mode 1] Checking liveness of [type 10c00001 offset 66529280 v 12495, hb offset 3817472
gen 93, mode 1, owner 5611f83b-2d8ebef8-8874-b8ac6f8c98ac mtime 116468 nHld 0 nOvf 0]
2016-02-07T01:43:00.220Z cpu15:6177)DLX: 3394: vol 'xendatastore1': [Req mode 1] Checking liveness of [type 10c00001 offset 66529280 v 12497, hb offset 4173824
gen 157, mode 1, owner 5611ef3c-2a14acf6-cb0c-d067e5eee823 mtime 83644 nHld 0 nOvf 0]
2016-02-07T01:43:13.451Z cpu15:4290)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511727.
2016-02-07T01:43:13.451Z cpu1:4097)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:43:13.451Z cpu1:4097)ScsiDeviceIO: 2309: Cmd(0x41240112a5c0) 0x2a, CmdSN 0x4e8d3d from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:43:13.451Z cpu15:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511726.
2016-02-07T01:43:13.451Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:43:13.451Z cpu4:4100)ScsiDeviceIO: 2309: Cmd(0x4124003ead00) 0x2a, CmdSN 0x407826 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:43:15.451Z cpu15:4290)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511737.
2016-02-07T01:43:15.451Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:43:15.451Z cpu4:4100)ScsiDeviceIO: 2309: Cmd(0x41240112a5c0) 0x2a, CmdSN 0x4e8d3f from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:43:15.452Z cpu15:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511738.
2016-02-07T01:43:15.452Z cpu4:4755)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:43:15.452Z cpu4:4755)ScsiDeviceIO: 2309: Cmd(0x4124003ead00) 0x2a, CmdSN 0x407828 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511723.
2016-02-07T01:43:15.524Z cpu18:4131)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511721.
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511724.
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511733.
2016-02-07T01:43:15.524Z cpu18:4131)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511725.
2016-02-07T01:43:15.524Z cpu10:4106)ScsiDeviceIO: 2291: Cmd(0x412400f7c900) 0x2a, CmdSN 0x4e8d3c from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu12:4756)ScsiDeviceIO: 2291: Cmd(0x41244180e400) 0x2a, CmdSN 0x4e8d3b from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511734.
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511735.
2016-02-07T01:43:15.524Z cpu18:4131)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511722.
2016-02-07T01:43:15.524Z cpu14:4110)ScsiDeviceIO: 2291: Cmd(0x4124403b8ac0) 0x2a, CmdSN 0x80000026 from world 8751385 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511736.
2016-02-07T01:43:15.524Z cpu14:4132)ScsiDeviceIO: 2291: Cmd(0x4124403aa600) 0x2a, CmdSN 0x80000061 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)ScsiDeviceIO: 2291: Cmd(0x412440df29c0) 0x2a, CmdSN 0x8000001c from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)ScsiDeviceIO: 2291: Cmd(0x412440990200) 0x2a, CmdSN 0x80000068 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)ScsiDeviceIO: 2291: Cmd(0x412440d7e9c0) 0x2a, CmdSN 0x8000001f from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)ScsiDeviceIO: 2291: Cmd(0x412441c9e300) 0x2a, CmdSN 0x8000002c from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x41244171b0c0, 3107733) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511729.
2016-02-07T01:43:15.524Z cpu18:4131)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511730.
2016-02-07T01:43:15.524Z cpu16:4938)NMP: nmp_PathDetermineFailure:2084: SCSI cmd RESERVE failed on path vmhba3:C0:T2:L2, reservation state on device naa.60a980004335432d68346b7239454f67 is unknown.
2016-02-07T01:43:15.525Z cpu13:6177)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 129 stampUS 10791954687028 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 402000> drv 14.54] on vol 'xendatastore1'
2016-02-07T01:43:15.599Z cpu6:6099)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 121 stampUS 10791954686976 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 457201> drv 14.54] on vol 'xendatastore2'
2016-02-07T01:43:15.737Z cpu18:4131)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511732.
2016-02-07T01:43:15.737Z cpu4:4100)ScsiDeviceIO: 2291: Cmd(0x412400d89280) 0x2a, CmdSN 0x80000067 from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.737Z cpu4:4100)ScsiDeviceIO: 2291: Cmd(0x4124003b0f80) 0x2a, CmdSN 0x8000005b from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.737Z cpu18:4131)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511741.
2016-02-07T01:43:15.737Z cpu1:4097)ScsiDeviceIO: 2291: Cmd(0x4124003c1f80) 0x2a, CmdSN 0x8000002c from world 8751359 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:23.685Z cpu8:4132)HBX: 231: Reclaimed heartbeat for volume 4feaec43-24977e2a-c803-d067e5eeee6e (xendatastore1): [Timeout] [HB state abcdef02 offset 3727360 gen 129 stampUS 10791962846546 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB$
2016-02-07T01:43:23.685Z cpu8:4132)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore1' (held for 8058 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:43:23.820Z cpu20:4131)HBX: 231: Reclaimed heartbeat for volume 4f99169f-12a01f11-cabc-d067e5eee823 (xendatastore2): [Timeout] [HB state abcdef02 offset 3727360 gen 121 stampUS 10791962978862 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <F$
2016-02-07T01:43:23.820Z cpu12:4131)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore2' (held for 7980 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:44:37.504Z cpu15:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512447.
2016-02-07T01:44:37.504Z cpu8:4104)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124010ad100, 4120) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba2:C0:T2:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:44:37.504Z cpu8:4104)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:44:37.504Z cpu8:4104)ScsiDeviceIO: 2309: Cmd(0x4124010ad100) 0x2a, CmdSN 0x4e8d7e from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:44:39.501Z cpu15:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512460.
2016-02-07T01:44:39.501Z cpu8:4104)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:44:39.501Z cpu8:4104)ScsiDeviceIO: 2309: Cmd(0x4124010ad100) 0x2a, CmdSN 0x4e8d82 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:44:39.573Z cpu13:4133)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512443.
2016-02-07T01:44:39.574Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x41244160db80) 0x2a, CmdSN 0x80000018 from world 8751385 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:39.574Z cpu13:4133)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512445.
2016-02-07T01:44:39.574Z cpu2:4098)ScsiDeviceIO: 2291: Cmd(0x412401479600) 0x2a, CmdSN 0x80000076 from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:39.574Z cpu13:4133)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512448.
2016-02-07T01:44:39.574Z cpu13:4133)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512451.
2016-02-07T01:44:39.575Z cpu13:4133)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512459.
2016-02-07T01:44:39.575Z cpu8:4104)ScsiDeviceIO: 2291: Cmd(0x412401ba1700) 0x2a, CmdSN 0x4e8d7f from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:39.575Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x412440e5e4c0) 0x2a, CmdSN 0x4e8d81 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:39.575Z cpu8:4104)ScsiDeviceIO: 2291: Cmd(0x41240113ee80) 0x2a, CmdSN 0x4e8d80 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:39.575Z cpu13:4133)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512450.
2016-02-07T01:44:39.575Z cpu8:4104)ScsiDeviceIO: 2291: Cmd(0x412400ece800) 0x2a, CmdSN 0x80000069 from world 8751359 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:39.945Z cpu2:6099)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 121 stampUS 10792038732207 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 457201> drv 14.54] on vol 'xendatastore2'
2016-02-07T01:44:40.505Z cpu12:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512452.
2016-02-07T01:44:40.506Z cpu2:4098)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412401a93340, 4120) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:44:40.506Z cpu2:4098)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:44:40.506Z cpu2:4098)ScsiDeviceIO: 2309: Cmd(0x412401a93340) 0x2a, CmdSN 0x407884 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:44:42.506Z cpu12:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512513.
2016-02-07T01:44:42.506Z cpu6:8755459)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:44:42.506Z cpu6:8755459)ScsiDeviceIO: 2309: Cmd(0x412401a93340) 0x2a, CmdSN 0x407886 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:44:42.578Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512444.
2016-02-07T01:44:42.578Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512449.
2016-02-07T01:44:42.578Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512453.
2016-02-07T01:44:42.578Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512454.
2016-02-07T01:44:42.579Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512455.
2016-02-07T01:44:42.579Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512456.
2016-02-07T01:44:42.579Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512457.
2016-02-07T01:44:42.579Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x412441be3400) 0x2a, CmdSN 0x8000002c from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:42.579Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x4124413dd000) 0x2a, CmdSN 0x8000004b from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:42.579Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x4124403bb3c0) 0x2a, CmdSN 0x80000060 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:42.579Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x412440e5a7c0) 0x2a, CmdSN 0x8000005b from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:42.579Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x4124403b2a40) 0x2a, CmdSN 0x80000011 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:42.579Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512458.
2016-02-07T01:44:42.950Z cpu23:6177)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 129 stampUS 10792041736611 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 402000> drv 14.54] on vol 'xendatastore1'
2016-02-07T01:44:48.108Z cpu13:4133)HBX: 231: Reclaimed heartbeat for volume 4f99169f-12a01f11-cabc-d067e5eee823 (xendatastore2): [Timeout] [HB state abcdef02 offset 3727360 gen 121 stampUS 10792047263125 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <F$
2016-02-07T01:44:48.108Z cpu13:4133)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore2' (held for 8432 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:44:48.215Z cpu8:4132)HBX: 231: Reclaimed heartbeat for volume 4feaec43-24977e2a-c803-d067e5eeee6e (xendatastore1): [Timeout] [HB state abcdef02 offset 3727360 gen 129 stampUS 10792047372038 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB$
2016-02-07T01:44:48.215Z cpu8:4132)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore1' (held for 5533 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:45:01.517Z cpu22:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512647.
2016-02-07T01:45:01.517Z cpu10:4106)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:45:01.517Z cpu10:4106)ScsiDeviceIO: 2309: Cmd(0x412400f04f80) 0x2a, CmdSN 0x4e8d87 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:01.517Z cpu22:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512646.
2016-02-07T01:45:01.517Z cpu4:4100)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124010a1380, 4120) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:45:01.517Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:45:01.517Z cpu4:4100)ScsiDeviceIO: 2309: Cmd(0x4124010a1380) 0x2a, CmdSN 0x40788b from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:03.517Z cpu22:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512666.
2016-02-07T01:45:03.517Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:45:03.517Z cpu16:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512667.
2016-02-07T01:45:03.517Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:45:03.517Z cpu4:4100)ScsiDeviceIO: 2309: Cmd(0x4124010a1380) 0x2a, CmdSN 0x40788c from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:03.589Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512644.
2016-02-07T01:45:03.589Z cpu6:4131)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512642.
2016-02-07T01:45:03.589Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512645.
2016-02-07T01:45:03.589Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512649.
2016-02-07T01:45:03.589Z cpu22:4131)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512654.
2016-02-07T01:45:03.589Z cpu8:4104)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x41240131c8c0, 8755469) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba2:C0:T2:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:45:03.589Z cpu8:4104)ScsiDeviceIO: 2291: Cmd(0x41240131c8c0) 0x2a, CmdSN 0x8000005b from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.589Z cpu8:4104)ScsiDeviceIO: 2291: Cmd(0x412401094cc0) 0x2a, CmdSN 0x80000077 from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.589Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512650.
2016-02-07T01:45:03.589Z cpu22:4131)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512648.
2016-02-07T01:45:03.589Z cpu22:4131)ScsiDeviceIO: 2291: Cmd(0x4124403a2380) 0x2a, CmdSN 0x80000056 from world 8751385 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.589Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512651.
2016-02-07T01:45:03.590Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512652.
2016-02-07T01:45:03.590Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512653.
2016-02-07T01:45:03.590Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512655.
2016-02-07T01:45:03.590Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512663.
2016-02-07T01:45:03.590Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512664.
2016-02-07T01:45:03.803Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512665.
2016-02-07T01:45:03.803Z cpu22:4118)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:45:03.803Z cpu22:4118)ScsiDeviceIO: 2291: Cmd(0x4124403ba4c0) 0x2a, CmdSN 0x8000007f from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.803Z cpu22:4118)ScsiDeviceIO: 2291: Cmd(0x4124413d9300) 0x2a, CmdSN 0x80000052 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.803Z cpu22:4118)ScsiDeviceIO: 2291: Cmd(0x4124412802c0) 0x2a, CmdSN 0x80000006 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.803Z cpu22:4118)ScsiDeviceIO: 2291: Cmd(0x412440e16140) 0x2a, CmdSN 0x80000079 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.803Z cpu22:4118)ScsiDeviceIO: 2291: Cmd(0x41244037c2c0) 0x2a, CmdSN 0x8000003e from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.803Z cpu22:4118)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124413c3e80, 3107733) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:45:03.985Z cpu8:6177)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 121 stampUS 10792062746244 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 457201> drv 14.54] on vol 'xendatastore2'
2016-02-07T01:45:03.985Z cpu13:6095)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 129 stampUS 10792062746290 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 402000> drv 14.54] on vol 'xendatastore1'
2016-02-07T01:45:07.371Z cpu8:4131)HBX: 231: Reclaimed heartbeat for volume 4f99169f-12a01f11-cabc-d067e5eee823 (xendatastore2): [Timeout] [HB state abcdef02 offset 3727360 gen 121 stampUS 10792066528132 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB$
2016-02-07T01:45:07.372Z cpu8:4131)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore2' (held for 3680 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:45:07.469Z cpu5:4133)HBX: 231: Reclaimed heartbeat for volume 4feaec43-24977e2a-c803-d067e5eeee6e (xendatastore1): [Timeout] [HB state abcdef02 offset 3727360 gen 129 stampUS 10792066625772 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB$
2016-02-07T01:45:07.470Z cpu5:4133)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore1' (held for 3567 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:45:40.542Z cpu16:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625513070.
2016-02-07T01:45:40.542Z cpu10:4106)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412401573600, 4120) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba2:C0:T2:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:45:40.542Z cpu10:4106)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:45:40.542Z cpu10:4106)ScsiDeviceIO: 2309: Cmd(0x412401573600) 0x2a, CmdSN 0x4e8d96 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:40.543Z cpu16:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513069.
2016-02-07T01:45:40.543Z cpu1:4097)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124003780c0, 4120) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:45:40.543Z cpu1:4097)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:45:40.543Z cpu1:4097)ScsiDeviceIO: 2309: Cmd(0x4124003780c0) 0x2a, CmdSN 0x407898 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:42.541Z cpu2:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625513099.
2016-02-07T01:45:42.541Z cpu10:4106)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:45:42.541Z cpu10:4106)ScsiDeviceIO: 2309: Cmd(0x412401573600) 0x2a, CmdSN 0x4e8d97 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:42.542Z cpu18:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513100.
2016-02-07T01:45:42.542Z cpu10:4106)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:45:42.542Z cpu10:4106)ScsiDeviceIO: 2309: Cmd(0x4124003780c0) 0x2a, CmdSN 0x407899 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:42.614Z cpu8:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513073.
2016-02-07T01:45:42.614Z cpu3:4132)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625513071.
2016-02-07T01:45:42.614Z cpu22:4118)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:45:42.614Z cpu22:4118)ScsiDeviceIO: 2291: Cmd(0x4124403be2c0) 0x2a, CmdSN 0x80000016 from world 8751385 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.614Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513074.
2016-02-07T01:45:42.614Z cpu0:4132)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625513072.
2016-02-07T01:45:42.614Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513075.
2016-02-07T01:45:42.614Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513076.
2016-02-07T01:45:42.615Z cpu0:4132)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625513082.
2016-02-07T01:45:42.615Z cpu9:4105)ScsiDeviceIO: 2291: Cmd(0x412401439a40) 0x2a, CmdSN 0x8000001a from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.615Z cpu9:4105)ScsiDeviceIO: 2291: Cmd(0x41240103c400) 0x2a, CmdSN 0x80000048 from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.615Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513077.
2016-02-07T01:45:42.615Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513078.
2016-02-07T01:45:42.615Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513079.
2016-02-07T01:45:42.615Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513080.
2016-02-07T01:45:42.616Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513081.
2016-02-07T01:45:42.616Z cpu18:4114)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:45:42.616Z cpu18:4114)ScsiDeviceIO: 2291: Cmd(0x4124414d1c00) 0x2a, CmdSN 0x80000018 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.616Z cpu18:4114)ScsiDeviceIO: 2291: Cmd(0x412440dd5a40) 0x2a, CmdSN 0x80000051 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.616Z cpu18:4114)ScsiDeviceIO: 2291: Cmd(0x412441838f40) 0x2a, CmdSN 0x8000000e from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.616Z cpu18:4114)ScsiDeviceIO: 2291: Cmd(0x41244037bbc0) 0x2a, CmdSN 0x80000035 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.616Z cpu18:4114)ScsiDeviceIO: 2291: Cmd(0x41244037dac0) 0x2a, CmdSN 0x80000071 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.616Z cpu18:4114)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412440383400, 3107733) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:45:43.049Z cpu20:6177)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 121 stampUS 10792101769177 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 457201> drv 14.54] on vol 'xendatastore2'
2016-02-07T01:45:43.049Z cpu9:6175)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 129 stampUS 10792101769224 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 402000> drv 14.54] on vol 'xendatastore1'
2016-02-07T01:45:46.755Z cpu4:4131)HBX: 231: Reclaimed heartbeat for volume 4feaec43-24977e2a-c803-d067e5eeee6e (xendatastore1): [Timeout] [HB state abcdef02 offset 3727360 gen 129 stampUS 10792105909915 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB$
2016-02-07T01:45:46.755Z cpu4:4131)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore1' (held for 4039 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:45:46.756Z cpu0:4132)HBX: 231: Reclaimed heartbeat for volume 4f99169f-12a01f11-cabc-d067e5eee823 (xendatastore2): [Timeout] [HB state abcdef02 offset 3727360 gen 121 stampUS 10792105909403 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB$
2016-02-07T01:45:46.756Z cpu1:4132)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore2' (held for 4040 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:47:20.588Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x4124010937c0) 0x12, CmdSN 0x75382 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T01:52:20.606Z cpu10:9187169)ScsiDeviceIO: 2322: Cmd(0x412400f45400) 0x12, CmdSN 0x7538a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T01:57:20.621Z cpu20:9187437)ScsiDeviceIO: 2322: Cmd(0x412440cec900) 0x12, CmdSN 0x75392 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:02:20.637Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400f206c0) 0x12, CmdSN 0x753bc from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:07:20.654Z cpu18:4149)ScsiDeviceIO: 2322: Cmd(0x412440990400) 0x12, CmdSN 0x753c6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:12:20.668Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x4124419aa600) 0x12, CmdSN 0x753ce from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:17:20.685Z cpu16:4268)ScsiDeviceIO: 2322: Cmd(0x4124413db600) 0x12, CmdSN 0x753d6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:22:20.702Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x412401315000) 0x12, CmdSN 0x753de from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:27:20.716Z cpu18:9186448)ScsiDeviceIO: 2322: Cmd(0x4124403aa900) 0x12, CmdSN 0x753e6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:32:20.733Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412401b9c240) 0x12, CmdSN 0x75406 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:37:20.750Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x412400ef2ec0) 0x12, CmdSN 0x75410 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:42:20.768Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x412401a90840) 0x12, CmdSN 0x75418 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:47:20.785Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x4124003c4d80) 0x12, CmdSN 0x75420 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:52:20.805Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003d6840) 0x12, CmdSN 0x75428 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:57:20.824Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x4124010861c0) 0x12, CmdSN 0x75430 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:02:18.508Z cpu12:3103642)VSCSI: 6226: handle 8338(vscsi0:0):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:02:18.510Z cpu12:3103642)VSCSI: 6226: handle 8339(vscsi0:1):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:02:18.510Z cpu12:3103642)NetPort: 1427: disabled port 0x100000f
2016-02-07T03:02:18.571Z cpu21:3103642)CBT: 692: Disconnecting the cbt device a2cb33f-cbt with filehandle 170701631
2016-02-07T03:02:19.086Z cpu21:3103642)CBT: 692: Disconnecting the cbt device aa7b33c-cbt with filehandle 178762556
2016-02-07T03:02:20.841Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412400e58ec0) 0x12, CmdSN 0x7545e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:02:22.000Z cpu21:3103642)CBT: 1082: Created device b1bb33e-cbt for cbt driver with filehandle 186364734
2016-02-07T03:02:22.001Z cpu21:3103642)VSCSI: 3642: handle 8343(vscsi0:0):Using sync mode due to sparse disks
2016-02-07T03:02:22.001Z cpu21:3103642)VSCSI: 3683: handle 8343(vscsi0:0):Creating Virtual Device for world 3107733 (FSS handle 187020096)
2016-02-07T03:02:22.182Z cpu21:3103642)CBT: 1082: Created device 38b9b343-cbt for cbt driver with filehandle 951694147
2016-02-07T03:02:22.184Z cpu21:3103642)VSCSI: 3642: handle 8344(vscsi0:1):Using sync mode due to sparse disks
2016-02-07T03:02:22.184Z cpu21:3103642)VSCSI: 3683: handle 8344(vscsi0:1):Creating Virtual Device for world 3107733 (FSS handle 580956997)
2016-02-07T03:02:22.184Z cpu21:3103642)NetPort: 1240: enabled port 0x100000f with mac 00:50:56:9f:65:7f
2016-02-07T03:03:44.401Z cpu17:3103642)VSCSI: 6226: handle 8343(vscsi0:0):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:03:44.401Z cpu17:3103642)VSCSI: 6226: handle 8344(vscsi0:1):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:03:44.401Z cpu17:3103642)NetPort: 1427: disabled port 0x100000f
2016-02-07T03:03:44.411Z cpu17:3103642)CBT: 692: Disconnecting the cbt device 38b9b343-cbt with filehandle 951694147
2016-02-07T03:03:44.423Z cpu17:3103642)CBT: 692: Disconnecting the cbt device b1bb33e-cbt with filehandle 186364734
2016-02-07T03:03:48.477Z cpu18:3103642)CBT: 1082: Created device b39b33e-cbt for cbt driver with filehandle 188330814
2016-02-07T03:03:48.539Z cpu18:3103642)CBT: 1082: Created device 1d08b342-cbt for cbt driver with filehandle 487109442
2016-02-07T03:03:48.540Z cpu18:3103642)CBT: 692: Disconnecting the cbt device b39b33e-cbt with filehandle 188330814
2016-02-07T03:03:48.541Z cpu18:3103642)CBT: 692: Disconnecting the cbt device 1d08b342-cbt with filehandle 487109442
2016-02-07T03:03:48.547Z cpu18:3103642)CBT: 1082: Created device 1d0ab342-cbt for cbt driver with filehandle 487240514
2016-02-07T03:03:48.583Z cpu18:3103642)CBT: 1082: Created device 1f76b344-cbt for cbt driver with filehandle 527872836
2016-02-07T03:03:48.584Z cpu18:3103642)CBT: 692: Disconnecting the cbt device 1d0ab342-cbt with filehandle 487240514
2016-02-07T03:03:48.585Z cpu18:3103642)CBT: 692: Disconnecting the cbt device 1f76b344-cbt with filehandle 527872836
2016-02-07T03:03:48.591Z cpu18:3103642)CBT: 1082: Created device 1d0cb342-cbt for cbt driver with filehandle 487371586
2016-02-07T03:03:48.592Z cpu18:3103642)VSCSI: 3642: handle 8345(vscsi0:0):Using sync mode due to sparse disks
2016-02-07T03:03:48.592Z cpu18:3103642)VSCSI: 3683: handle 8345(vscsi0:0):Creating Virtual Device for world 3107733 (FSS handle 952087363)
2016-02-07T03:03:48.617Z cpu18:3103642)CBT: 1082: Created device ff5b348-cbt for cbt driver with filehandle 267760456
2016-02-07T03:03:48.619Z cpu18:3103642)VSCSI: 3642: handle 8346(vscsi0:1):Using sync mode due to sparse disks
2016-02-07T03:03:48.619Z cpu18:3103642)VSCSI: 3683: handle 8346(vscsi0:1):Creating Virtual Device for world 3107733 (FSS handle 394244938)
2016-02-07T03:03:48.619Z cpu18:3103642)NetPort: 1240: enabled port 0x100000f with mac 00:50:56:9f:65:7f
2016-02-07T03:03:50.056Z cpu21:3103642)VSCSI: 6226: handle 8345(vscsi0:0):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:03:50.057Z cpu21:3103642)VSCSI: 6226: handle 8346(vscsi0:1):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:03:50.058Z cpu21:3103642)NetPort: 1427: disabled port 0x100000f
2016-02-07T03:03:50.062Z cpu21:3103642)CBT: 692: Disconnecting the cbt device 1d0cb342-cbt with filehandle 487371586
2016-02-07T03:03:50.072Z cpu21:3103642)CBT: 1082: Created device 1a47b34b-cbt for cbt driver with filehandle 440906571
2016-02-07T03:03:50.073Z cpu21:3103642)CBT: 692: Disconnecting the cbt device 1a47b34b-cbt with filehandle 440906571
2016-02-07T03:03:50.131Z cpu14:3103642)CBT: 692: Disconnecting the cbt device ff5b348-cbt with filehandle 267760456
2016-02-07T03:03:50.188Z cpu14:3103642)CBT: 1082: Created device 1046b349-cbt for cbt driver with filehandle 273068873
2016-02-07T03:03:50.191Z cpu14:3103642)VSCSI: 3642: handle 8347(vscsi0:0):Using sync mode due to sparse disks
2016-02-07T03:03:50.191Z cpu14:3103642)VSCSI: 3683: handle 8347(vscsi0:0):Creating Virtual Device for world 3107733 (FSS handle 440972107)
2016-02-07T03:03:50.198Z cpu14:3103642)CBT: 1082: Created device 20a6b34d-cbt for cbt driver with filehandle 547795789
2016-02-07T03:03:50.210Z cpu14:3103642)CBT: 1082: Created device 1772b351-cbt for cbt driver with filehandle 393392977
2016-02-07T03:03:50.212Z cpu14:3103642)CBT: 692: Disconnecting the cbt device 20a6b34d-cbt with filehandle 547795789
2016-02-07T03:03:50.214Z cpu14:3103642)CBT: 692: Disconnecting the cbt device 1772b351-cbt with filehandle 393392977
2016-02-07T03:03:50.223Z cpu14:3103642)CBT: 1082: Created device 1774b351-cbt for cbt driver with filehandle 393524049
2016-02-07T03:03:50.227Z cpu14:3103642)CBT: 1082: Created device 171eb353-cbt for cbt driver with filehandle 387887955
2016-02-07T03:03:50.229Z cpu14:3103642)CBT: 692: Disconnecting the cbt device 1774b351-cbt with filehandle 393524049
2016-02-07T03:03:50.230Z cpu14:3103642)CBT: 692: Disconnecting the cbt device 171eb353-cbt with filehandle 387887955
2016-02-07T03:03:50.240Z cpu14:3103642)CBT: 1082: Created device 1776b351-cbt for cbt driver with filehandle 393655121
2016-02-07T03:03:50.241Z cpu14:3103642)VSCSI: 3642: handle 8348(vscsi0:1):Using sync mode due to sparse disks
2016-02-07T03:03:50.241Z cpu14:3103642)VSCSI: 3683: handle 8348(vscsi0:1):Creating Virtual Device for world 3107733 (FSS handle 681554770)
2016-02-07T03:03:50.241Z cpu14:3103642)NetPort: 1240: enabled port 0x100000f with mac 00:50:56:9f:65:7f
2016-02-07T03:03:50.906Z cpu12:3103642)VSCSI: 6226: handle 8347(vscsi0:0):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:03:50.907Z cpu12:3103642)VSCSI: 6226: handle 8348(vscsi0:1):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:03:50.907Z cpu12:3103642)NetPort: 1427: disabled port 0x100000f
2016-02-07T03:03:50.911Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 1776b351-cbt with filehandle 393655121
2016-02-07T03:03:50.926Z cpu12:3103642)CBT: 1082: Created device 1940b354-cbt for cbt driver with filehandle 423670612
2016-02-07T03:03:50.928Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 1940b354-cbt with filehandle 423670612
2016-02-07T03:03:50.968Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 1046b349-cbt with filehandle 273068873
2016-02-07T03:03:51.123Z cpu12:3103642)CBT: 1082: Created device 2fb11f5-cbt for cbt driver with filehandle 50008565
2016-02-07T03:03:51.128Z cpu12:3103642)CBT: 1082: Created device 2fd11f9-cbt for cbt driver with filehandle 50139641
2016-02-07T03:03:51.129Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 2fd11f9-cbt with filehandle 50139641
2016-02-07T03:03:51.130Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 2fb11f5-cbt with filehandle 50008565
2016-02-07T03:03:51.135Z cpu12:3103642)CBT: 1082: Created device 2ff11f9-cbt for cbt driver with filehandle 50270713
2016-02-07T03:03:51.421Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 2ff11f9-cbt with filehandle 50270713
2016-02-07T03:03:51.471Z cpu12:3103642)CBT: 1082: Created device 30111f5-cbt for cbt driver with filehandle 50401781
2016-02-07T03:03:51.476Z cpu12:3103642)CBT: 1082: Created device 30111f9-cbt for cbt driver with filehandle 50401785
2016-02-07T03:03:51.478Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 30111f9-cbt with filehandle 50401785
2016-02-07T03:03:51.479Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 30111f5-cbt with filehandle 50401781
2016-02-07T03:03:51.489Z cpu12:3103642)CBT: 1082: Created device 30311f9-cbt for cbt driver with filehandle 50532857
2016-02-07T03:03:51.806Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 30311f9-cbt with filehandle 50532857
2016-02-07T03:03:51.878Z cpu12:3103642)CBT: 1082: Created device 30711f5-cbt for cbt driver with filehandle 50794997
2016-02-07T03:03:51.879Z cpu12:3103642)VSCSI: 3683: handle 8349(vscsi0:0):Creating Virtual Device for world 3107733 (FSS handle 50663927)
2016-02-07T03:03:51.883Z cpu12:3103642)CBT: 1082: Created device 2f911f8-cbt for cbt driver with filehandle 49877496
2016-02-07T03:03:51.884Z cpu12:3103642)VSCSI: 3683: handle 8350(vscsi0:1):Creating Virtual Device for world 3107733 (FSS handle 47256058)
2016-02-07T03:03:51.885Z cpu12:3103642)NetPort: 1240: enabled port 0x100000f with mac 00:50:56:9f:65:7f
2016-02-07T03:07:20.858Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x41240103df00) 0x12, CmdSN 0x754a4 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:12:20.877Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x4124403a8500) 0x12, CmdSN 0x754ac from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:17:20.899Z cpu9:9191416)ScsiDeviceIO: 2322: Cmd(0x412401438740) 0x12, CmdSN 0x754b4 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:22:20.917Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x41244141cb00) 0x12, CmdSN 0x754bc from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:27:20.938Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x412401488680) 0x12, CmdSN 0x754c4 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:32:20.957Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412401486440) 0x12, CmdSN 0x754e4 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:37:20.975Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412441a4d500) 0x12, CmdSN 0x754ee from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:42:20.995Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x412441be5f00) 0x12, CmdSN 0x754f6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:47:21.014Z cpu22:9193469)ScsiDeviceIO: 2322: Cmd(0x412440e5e5c0) 0x12, CmdSN 0x754fe from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:52:21.034Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412401be40c0) 0x12, CmdSN 0x75506 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:57:21.053Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x412441c90d40) 0x12, CmdSN 0x7550e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:02:21.073Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x412440371b80) 0x12, CmdSN 0x75538 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:07:21.094Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400384600) 0x12, CmdSN 0x75542 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:12:21.113Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412440df06c0) 0x12, CmdSN 0x7554a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:17:21.134Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412400e96300) 0x12, CmdSN 0x75552 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:22:21.152Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x412400f44c00) 0x12, CmdSN 0x7555a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:27:21.171Z cpu23:9187544)ScsiDeviceIO: 2322: Cmd(0x4124403d14c0) 0x12, CmdSN 0x75562 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:32:21.192Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x4124003a6b00) 0x12, CmdSN 0x75582 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:37:21.210Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412441be9000) 0x12, CmdSN 0x7558c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:42:21.229Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x412441111140) 0x12, CmdSN 0x75594 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:47:21.249Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124015957c0) 0x12, CmdSN 0x7559c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:52:21.269Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400383d00) 0x12, CmdSN 0x755a4 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:57:21.290Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412441419300) 0x12, CmdSN 0x755ac from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:02:21.307Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x4124412e54c0) 0x12, CmdSN 0x755d6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:07:21.328Z cpu15:9198439)ScsiDeviceIO: 2322: Cmd(0x412441113b40) 0x12, CmdSN 0x755e0 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:12:21.346Z cpu4:4100)ScsiDeviceIO: 2322: Cmd(0x412401576a00) 0x12, CmdSN 0x755e8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:17:21.364Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x41240039cb40) 0x12, CmdSN 0x755f0 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:22:21.385Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400380200) 0x12, CmdSN 0x755f8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:27:21.404Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003a4700) 0x12, CmdSN 0x75600 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:32:21.422Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x4124003bb380) 0x12, CmdSN 0x75620 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:37:21.439Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400382100) 0x12, CmdSN 0x7562a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:42:21.455Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x412440381600) 0x12, CmdSN 0x75632 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:47:21.472Z cpu19:4268)ScsiDeviceIO: 2322: Cmd(0x412441392340) 0x12, CmdSN 0x7563a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:52:21.490Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x412441c90f40) 0x12, CmdSN 0x75642 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:57:21.505Z cpu6:9184891)ScsiDeviceIO: 2322: Cmd(0x41240148ad80) 0x12, CmdSN 0x7564a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:02:21.522Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x412401be06c0) 0x12, CmdSN 0x75674 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:07:21.538Z cpu19:4115)ScsiDeviceIO: 2322: Cmd(0x412440df0bc0) 0x12, CmdSN 0x7567e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:12:21.554Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412441c94b40) 0x12, CmdSN 0x75686 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:17:21.572Z cpu14:4110)ScsiDeviceIO: 2322: Cmd(0x4124414d3600) 0x12, CmdSN 0x7568e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:22:21.588Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x41244160aa80) 0x12, CmdSN 0x75696 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:27:21.604Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x412441be6d00) 0x12, CmdSN 0x7569e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:32:21.623Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x41244171e4c0) 0x12, CmdSN 0x756be from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:37:21.640Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x412441c90a40) 0x12, CmdSN 0x756c8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:42:21.656Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412400ec9f00) 0x12, CmdSN 0x756d0 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:47:21.670Z cpu16:9202831)ScsiDeviceIO: 2322: Cmd(0x4124403c66c0) 0x12, CmdSN 0x756d8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:52:21.686Z cpu15:4111)ScsiDeviceIO: 2322: Cmd(0x412440396e80) 0x12, CmdSN 0x756e0 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:57:21.699Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x4124403cecc0) 0x12, CmdSN 0x756e8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:02:21.714Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x412440e5edc0) 0x12, CmdSN 0x75712 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:07:21.728Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x412441a49200) 0x12, CmdSN 0x7571c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:12:21.744Z cpu4:4100)ScsiDeviceIO: 2322: Cmd(0x412401ba1f00) 0x12, CmdSN 0x75724 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:17:21.758Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412401592ec0) 0x12, CmdSN 0x7572c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:22:21.774Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124011cc900) 0x12, CmdSN 0x75734 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:27:21.789Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x412441c10540) 0x12, CmdSN 0x7573c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:32:21.802Z cpu15:8755483)ScsiDeviceIO: 2322: Cmd(0x412441c9c800) 0x12, CmdSN 0x7575c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:37:21.818Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x4124415c6b40) 0x12, CmdSN 0x75766 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:42:21.831Z cpu14:8751389)ScsiDeviceIO: 2322: Cmd(0x4124413d9100) 0x12, CmdSN 0x7576e from world 0 to dev "naa.600
~ # esxcli storage core adapter stats get
vmhba0
Successful Commands: 3
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 37496
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba1
Successful Commands: 13019647
Blocks Read: 475321286
Blocks Written: 450392695
Read Operations: 7492015
Write Operations: 5352612
Reserve Operations: 18625
Reservation Conflicts: 0
Failed Commands: 9568884
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba2
Successful Commands: 314775389
Blocks Read: 586606724
Blocks Written: 7282562059
Read Operations: 10248659
Write Operations: 303788756
Reserve Operations: 480
Reservation Conflicts: 521
Failed Commands: 1868
Failed Blocks Read: 0
Failed Blocks Written: 52888
Failed Read Operations: 190
Failed Write Operations: 1516
Failed Reserve Operations: 107
Total Splits: 0
PAE Commands: 0
vmhba3
Successful Commands: 314965516
Blocks Read: 589293158
Blocks Written: 7282184408
Read Operations: 10248601
Write Operations: 303980219
Reserve Operations: 369
Reservation Conflicts: 638
Failed Commands: 1782
Failed Blocks Read: 0
Failed Blocks Written: 35248
Failed Read Operations: 182
Failed Write Operations: 1462
Failed Reserve Operations: 101
Total Splits: 0
PAE Commands: 0
vmhba32
Successful Commands: 0
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 0
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba33
Successful Commands: 0
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 0
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba34
Successful Commands: 0
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 0
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba35
Successful Commands: 0
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 0
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
vmhba36
Successful Commands: 0
Blocks Read: 0
Blocks Written: 0
Read Operations: 0
Write Operations: 0
Reserve Operations: 0
Reservation Conflicts: 0
Failed Commands: 256
Failed Blocks Read: 0
Failed Blocks Written: 0
Failed Read Operations: 0
Failed Write Operations: 0
Failed Reserve Operations: 0
Total Splits: 0
PAE Commands: 0
~ # esxcli storage nmp device list
naa.600508e000000000d85b294d14b5f10d
Device Display Name: Dell Serial Attached SCSI Disk (naa.600508e000000000d85b294d14b5f10d)
Storage Array Type: VMW_SATP_LOCAL
Storage Array Type Device Config: SATP VMW_SATP_LOCAL does not support device configuration.
Path Selection Policy: VMW_PSP_FIXED
Path Selection Policy Device Config: {preferred=vmhba1:C1:T0:L0;current=vmhba1:C1:T0:L0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba1:C1:T0:L0
naa.60a9800044313450433f435734567638
Device Display Name: NETAPP Fibre Channel Disk (naa.60a9800044313450433f435734567638)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=2,TPG_state=AO}{TPG_id=3,TPG_state=ANO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba3:C0:T1:L0, vmhba3:C0:T0:L0, vmhba2:C0:T1:L0, vmhba2:C0:T0:L0
naa.60a980004335432d68346a6431745431
Device Display Name: NETAPP Fibre Channel Disk (naa.60a980004335432d68346a6431745431)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=1,TPG_state=ANO}{TPG_id=0,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=6: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T3:L3, vmhba2:C0:T2:L3, vmhba3:C0:T3:L3, vmhba3:C0:T2:L3
naa.60a980004335432d68346b7239454f67
Device Display Name: NETAPP Fibre Channel Disk (naa.60a980004335432d68346b7239454f67)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on;explicit_support=off; explicit_allow=on;alua_followover=on;{TPG_id=1,TPG_state=ANO}{TPG_id=0,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0;lastPathIndex=6: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba2:C0:T3:L2, vmhba2:C0:T2:L2, vmhba3:C0:T3:L2, vmhba3:C0:T2:L2
mpx.vmhba0:C0:T0:L0
Device Display Name: Local PLDS CD-ROM (mpx.vmhba0:C0:T0:L0)
Storage Array Type: VMW_SATP_LOCAL
Storage Array Type Device Config: SATP VMW_SATP_LOCAL does not support device configuration.
Path Selection Policy: VMW_PSP_FIXED
Path Selection Policy Device Config: {preferred=vmhba0:C0:T0:L0;current=vmhba0:C0:T0:L0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba0:C0:T0:L0
VMKernel.log
2016-02-07T01:41:57.827Z cpu13:4133)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore1' (held for 3589 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:42:20.572Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x4124403aa600) 0x12, CmdSN 0x75369 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T01:42:55.440Z cpu13:4290)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511532.
2016-02-07T01:42:55.440Z cpu10:4106)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412401ad6a80, 4120) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba2:C0:T3:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:42:55.440Z cpu10:4106)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:42:55.440Z cpu10:4106)ScsiDeviceIO: 2309: Cmd(0x412401ad6a80) 0x2a, CmdSN 0x4e8d0c from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:42:55.440Z cpu15:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511531.
2016-02-07T01:42:55.440Z cpu4:4100)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124003b0f80, 4120) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:42:55.440Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:42:55.440Z cpu4:4100)ScsiDeviceIO: 2309: Cmd(0x4124003b0f80) 0x2a, CmdSN 0x407805 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:42:55.618Z cpu15:6177)J3: 2611: Aborting txn (0x4100241c9840) callerID: 0xc1d00002 due to failure pre-committing: Optimistic lock acquired by another host
2016-02-07T01:42:55.817Z cpu15:6177)DLX: 3394: vol 'xendatastore1': [Req mode 1] Checking liveness of [type 10c00001 offset 66529280 v 12495, hb offset 3817472
gen 93, mode 1, owner 5611f83b-2d8ebef8-8874-b8ac6f8c98ac mtime 116468 nHld 0 nOvf 0]
2016-02-07T01:43:00.220Z cpu15:6177)DLX: 3394: vol 'xendatastore1': [Req mode 1] Checking liveness of [type 10c00001 offset 66529280 v 12497, hb offset 4173824
gen 157, mode 1, owner 5611ef3c-2a14acf6-cb0c-d067e5eee823 mtime 83644 nHld 0 nOvf 0]
2016-02-07T01:43:13.451Z cpu15:4290)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511727.
2016-02-07T01:43:13.451Z cpu1:4097)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:43:13.451Z cpu1:4097)ScsiDeviceIO: 2309: Cmd(0x41240112a5c0) 0x2a, CmdSN 0x4e8d3d from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:43:13.451Z cpu15:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511726.
2016-02-07T01:43:13.451Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:43:13.451Z cpu4:4100)ScsiDeviceIO: 2309: Cmd(0x4124003ead00) 0x2a, CmdSN 0x407826 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:43:15.451Z cpu15:4290)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511737.
2016-02-07T01:43:15.451Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:43:15.451Z cpu4:4100)ScsiDeviceIO: 2309: Cmd(0x41240112a5c0) 0x2a, CmdSN 0x4e8d3f from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:43:15.452Z cpu15:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511738.
2016-02-07T01:43:15.452Z cpu4:4755)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:43:15.452Z cpu4:4755)ScsiDeviceIO: 2309: Cmd(0x4124003ead00) 0x2a, CmdSN 0x407828 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511723.
2016-02-07T01:43:15.524Z cpu18:4131)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511721.
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511724.
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511733.
2016-02-07T01:43:15.524Z cpu18:4131)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511725.
2016-02-07T01:43:15.524Z cpu10:4106)ScsiDeviceIO: 2291: Cmd(0x412400f7c900) 0x2a, CmdSN 0x4e8d3c from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu12:4756)ScsiDeviceIO: 2291: Cmd(0x41244180e400) 0x2a, CmdSN 0x4e8d3b from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511734.
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511735.
2016-02-07T01:43:15.524Z cpu18:4131)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511722.
2016-02-07T01:43:15.524Z cpu14:4110)ScsiDeviceIO: 2291: Cmd(0x4124403b8ac0) 0x2a, CmdSN 0x80000026 from world 8751385 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511736.
2016-02-07T01:43:15.524Z cpu14:4132)ScsiDeviceIO: 2291: Cmd(0x4124403aa600) 0x2a, CmdSN 0x80000061 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)ScsiDeviceIO: 2291: Cmd(0x412440df29c0) 0x2a, CmdSN 0x8000001c from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)ScsiDeviceIO: 2291: Cmd(0x412440990200) 0x2a, CmdSN 0x80000068 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)ScsiDeviceIO: 2291: Cmd(0x412440d7e9c0) 0x2a, CmdSN 0x8000001f from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)ScsiDeviceIO: 2291: Cmd(0x412441c9e300) 0x2a, CmdSN 0x8000002c from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.524Z cpu14:4132)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x41244171b0c0, 3107733) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:43:15.524Z cpu14:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625511729.
2016-02-07T01:43:15.524Z cpu18:4131)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511730.
2016-02-07T01:43:15.524Z cpu16:4938)NMP: nmp_PathDetermineFailure:2084: SCSI cmd RESERVE failed on path vmhba3:C0:T2:L2, reservation state on device naa.60a980004335432d68346b7239454f67 is unknown.
2016-02-07T01:43:15.525Z cpu13:6177)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 129 stampUS 10791954687028 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 402000> drv 14.54] on vol 'xendatastore1'
2016-02-07T01:43:15.599Z cpu6:6099)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 121 stampUS 10791954686976 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 457201> drv 14.54] on vol 'xendatastore2'
2016-02-07T01:43:15.737Z cpu18:4131)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511732.
2016-02-07T01:43:15.737Z cpu4:4100)ScsiDeviceIO: 2291: Cmd(0x412400d89280) 0x2a, CmdSN 0x80000067 from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.737Z cpu4:4100)ScsiDeviceIO: 2291: Cmd(0x4124003b0f80) 0x2a, CmdSN 0x8000005b from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:15.737Z cpu18:4131)<6>qla2xxx 0000:04:00.0: scsi(7:3:3): Abort command succeeded -- 1 625511741.
2016-02-07T01:43:15.737Z cpu1:4097)ScsiDeviceIO: 2291: Cmd(0x4124003c1f80) 0x2a, CmdSN 0x8000002c from world 8751359 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:43:23.685Z cpu8:4132)HBX: 231: Reclaimed heartbeat for volume 4feaec43-24977e2a-c803-d067e5eeee6e (xendatastore1): [Timeout] [HB state abcdef02 offset 3727360 gen 129 stampUS 10791962846546 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB$
2016-02-07T01:43:23.685Z cpu8:4132)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore1' (held for 8058 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:43:23.820Z cpu20:4131)HBX: 231: Reclaimed heartbeat for volume 4f99169f-12a01f11-cabc-d067e5eee823 (xendatastore2): [Timeout] [HB state abcdef02 offset 3727360 gen 121 stampUS 10791962978862 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <F$
2016-02-07T01:43:23.820Z cpu12:4131)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore2' (held for 7980 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:44:37.504Z cpu15:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512447.
2016-02-07T01:44:37.504Z cpu8:4104)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124010ad100, 4120) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba2:C0:T2:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:44:37.504Z cpu8:4104)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:44:37.504Z cpu8:4104)ScsiDeviceIO: 2309: Cmd(0x4124010ad100) 0x2a, CmdSN 0x4e8d7e from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:44:39.501Z cpu15:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512460.
2016-02-07T01:44:39.501Z cpu8:4104)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:44:39.501Z cpu8:4104)ScsiDeviceIO: 2309: Cmd(0x4124010ad100) 0x2a, CmdSN 0x4e8d82 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:44:39.573Z cpu13:4133)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512443.
2016-02-07T01:44:39.574Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x41244160db80) 0x2a, CmdSN 0x80000018 from world 8751385 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:39.574Z cpu13:4133)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512445.
2016-02-07T01:44:39.574Z cpu2:4098)ScsiDeviceIO: 2291: Cmd(0x412401479600) 0x2a, CmdSN 0x80000076 from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:39.574Z cpu13:4133)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512448.
2016-02-07T01:44:39.574Z cpu13:4133)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512451.
2016-02-07T01:44:39.575Z cpu13:4133)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512459.
2016-02-07T01:44:39.575Z cpu8:4104)ScsiDeviceIO: 2291: Cmd(0x412401ba1700) 0x2a, CmdSN 0x4e8d7f from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:39.575Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x412440e5e4c0) 0x2a, CmdSN 0x4e8d81 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:39.575Z cpu8:4104)ScsiDeviceIO: 2291: Cmd(0x41240113ee80) 0x2a, CmdSN 0x4e8d80 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:39.575Z cpu13:4133)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512450.
2016-02-07T01:44:39.575Z cpu8:4104)ScsiDeviceIO: 2291: Cmd(0x412400ece800) 0x2a, CmdSN 0x80000069 from world 8751359 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:39.945Z cpu2:6099)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 121 stampUS 10792038732207 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 457201> drv 14.54] on vol 'xendatastore2'
2016-02-07T01:44:40.505Z cpu12:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512452.
2016-02-07T01:44:40.506Z cpu2:4098)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412401a93340, 4120) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:44:40.506Z cpu2:4098)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:44:40.506Z cpu2:4098)ScsiDeviceIO: 2309: Cmd(0x412401a93340) 0x2a, CmdSN 0x407884 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:44:42.506Z cpu12:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512513.
2016-02-07T01:44:42.506Z cpu6:8755459)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:44:42.506Z cpu6:8755459)ScsiDeviceIO: 2309: Cmd(0x412401a93340) 0x2a, CmdSN 0x407886 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:44:42.578Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512444.
2016-02-07T01:44:42.578Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512449.
2016-02-07T01:44:42.578Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512453.
2016-02-07T01:44:42.578Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512454.
2016-02-07T01:44:42.579Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512455.
2016-02-07T01:44:42.579Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512456.
2016-02-07T01:44:42.579Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512457.
2016-02-07T01:44:42.579Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x412441be3400) 0x2a, CmdSN 0x8000002c from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:42.579Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x4124413dd000) 0x2a, CmdSN 0x8000004b from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:42.579Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x4124403bb3c0) 0x2a, CmdSN 0x80000060 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:42.579Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x412440e5a7c0) 0x2a, CmdSN 0x8000005b from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:42.579Z cpu16:4112)ScsiDeviceIO: 2291: Cmd(0x4124403b2a40) 0x2a, CmdSN 0x80000011 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:44:42.579Z cpu8:4132)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512458.
2016-02-07T01:44:42.950Z cpu23:6177)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 129 stampUS 10792041736611 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 402000> drv 14.54] on vol 'xendatastore1'
2016-02-07T01:44:48.108Z cpu13:4133)HBX: 231: Reclaimed heartbeat for volume 4f99169f-12a01f11-cabc-d067e5eee823 (xendatastore2): [Timeout] [HB state abcdef02 offset 3727360 gen 121 stampUS 10792047263125 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <F$
2016-02-07T01:44:48.108Z cpu13:4133)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore2' (held for 8432 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:44:48.215Z cpu8:4132)HBX: 231: Reclaimed heartbeat for volume 4feaec43-24977e2a-c803-d067e5eeee6e (xendatastore1): [Timeout] [HB state abcdef02 offset 3727360 gen 129 stampUS 10792047372038 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB$
2016-02-07T01:44:48.215Z cpu8:4132)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore1' (held for 5533 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:45:01.517Z cpu22:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512647.
2016-02-07T01:45:01.517Z cpu10:4106)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:45:01.517Z cpu10:4106)ScsiDeviceIO: 2309: Cmd(0x412400f04f80) 0x2a, CmdSN 0x4e8d87 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:01.517Z cpu22:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512646.
2016-02-07T01:45:01.517Z cpu4:4100)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124010a1380, 4120) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:45:01.517Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:45:01.517Z cpu4:4100)ScsiDeviceIO: 2309: Cmd(0x4124010a1380) 0x2a, CmdSN 0x40788b from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:03.517Z cpu22:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512666.
2016-02-07T01:45:03.517Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:45:03.517Z cpu16:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512667.
2016-02-07T01:45:03.517Z cpu4:4100)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:45:03.517Z cpu4:4100)ScsiDeviceIO: 2309: Cmd(0x4124010a1380) 0x2a, CmdSN 0x40788c from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:03.589Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512644.
2016-02-07T01:45:03.589Z cpu6:4131)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512642.
2016-02-07T01:45:03.589Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512645.
2016-02-07T01:45:03.589Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512649.
2016-02-07T01:45:03.589Z cpu22:4131)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512654.
2016-02-07T01:45:03.589Z cpu8:4104)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x41240131c8c0, 8755469) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba2:C0:T2:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:45:03.589Z cpu8:4104)ScsiDeviceIO: 2291: Cmd(0x41240131c8c0) 0x2a, CmdSN 0x8000005b from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.589Z cpu8:4104)ScsiDeviceIO: 2291: Cmd(0x412401094cc0) 0x2a, CmdSN 0x80000077 from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.589Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512650.
2016-02-07T01:45:03.589Z cpu22:4131)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625512648.
2016-02-07T01:45:03.589Z cpu22:4131)ScsiDeviceIO: 2291: Cmd(0x4124403a2380) 0x2a, CmdSN 0x80000056 from world 8751385 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.589Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512651.
2016-02-07T01:45:03.590Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512652.
2016-02-07T01:45:03.590Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512653.
2016-02-07T01:45:03.590Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512655.
2016-02-07T01:45:03.590Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512663.
2016-02-07T01:45:03.590Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512664.
2016-02-07T01:45:03.803Z cpu13:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625512665.
2016-02-07T01:45:03.803Z cpu22:4118)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:45:03.803Z cpu22:4118)ScsiDeviceIO: 2291: Cmd(0x4124403ba4c0) 0x2a, CmdSN 0x8000007f from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.803Z cpu22:4118)ScsiDeviceIO: 2291: Cmd(0x4124413d9300) 0x2a, CmdSN 0x80000052 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.803Z cpu22:4118)ScsiDeviceIO: 2291: Cmd(0x4124412802c0) 0x2a, CmdSN 0x80000006 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.803Z cpu22:4118)ScsiDeviceIO: 2291: Cmd(0x412440e16140) 0x2a, CmdSN 0x80000079 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.803Z cpu22:4118)ScsiDeviceIO: 2291: Cmd(0x41244037c2c0) 0x2a, CmdSN 0x8000003e from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:03.803Z cpu22:4118)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124413c3e80, 3107733) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:45:03.985Z cpu8:6177)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 121 stampUS 10792062746244 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 457201> drv 14.54] on vol 'xendatastore2'
2016-02-07T01:45:03.985Z cpu13:6095)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 129 stampUS 10792062746290 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 402000> drv 14.54] on vol 'xendatastore1'
2016-02-07T01:45:07.371Z cpu8:4131)HBX: 231: Reclaimed heartbeat for volume 4f99169f-12a01f11-cabc-d067e5eee823 (xendatastore2): [Timeout] [HB state abcdef02 offset 3727360 gen 121 stampUS 10792066528132 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB$
2016-02-07T01:45:07.372Z cpu8:4131)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore2' (held for 3680 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:45:07.469Z cpu5:4133)HBX: 231: Reclaimed heartbeat for volume 4feaec43-24977e2a-c803-d067e5eeee6e (xendatastore1): [Timeout] [HB state abcdef02 offset 3727360 gen 129 stampUS 10792066625772 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB$
2016-02-07T01:45:07.470Z cpu5:4133)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore1' (held for 3567 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:45:40.542Z cpu16:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625513070.
2016-02-07T01:45:40.542Z cpu10:4106)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412401573600, 4120) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba2:C0:T2:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:45:40.542Z cpu10:4106)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:45:40.542Z cpu10:4106)ScsiDeviceIO: 2309: Cmd(0x412401573600) 0x2a, CmdSN 0x4e8d96 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:40.543Z cpu16:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513069.
2016-02-07T01:45:40.543Z cpu1:4097)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124003780c0, 4120) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:45:40.543Z cpu1:4097)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:45:40.543Z cpu1:4097)ScsiDeviceIO: 2309: Cmd(0x4124003780c0) 0x2a, CmdSN 0x407898 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:42.541Z cpu2:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625513099.
2016-02-07T01:45:42.541Z cpu10:4106)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:45:42.541Z cpu10:4106)ScsiDeviceIO: 2309: Cmd(0x412401573600) 0x2a, CmdSN 0x4e8d97 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:42.542Z cpu18:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513100.
2016-02-07T01:45:42.542Z cpu10:4106)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:45:42.542Z cpu10:4106)ScsiDeviceIO: 2309: Cmd(0x4124003780c0) 0x2a, CmdSN 0x407899 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-07T01:45:42.614Z cpu8:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513073.
2016-02-07T01:45:42.614Z cpu3:4132)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625513071.
2016-02-07T01:45:42.614Z cpu22:4118)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-07T01:45:42.614Z cpu22:4118)ScsiDeviceIO: 2291: Cmd(0x4124403be2c0) 0x2a, CmdSN 0x80000016 from world 8751385 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.614Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513074.
2016-02-07T01:45:42.614Z cpu0:4132)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625513072.
2016-02-07T01:45:42.614Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513075.
2016-02-07T01:45:42.614Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513076.
2016-02-07T01:45:42.615Z cpu0:4132)<6>qla2xxx 0000:04:00.0: scsi(7:2:3): Abort command succeeded -- 1 625513082.
2016-02-07T01:45:42.615Z cpu9:4105)ScsiDeviceIO: 2291: Cmd(0x412401439a40) 0x2a, CmdSN 0x8000001a from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.615Z cpu9:4105)ScsiDeviceIO: 2291: Cmd(0x41240103c400) 0x2a, CmdSN 0x80000048 from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.615Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513077.
2016-02-07T01:45:42.615Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513078.
2016-02-07T01:45:42.615Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513079.
2016-02-07T01:45:42.615Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513080.
2016-02-07T01:45:42.616Z cpu0:4131)<6>qla2xxx 0000:05:00.0: scsi(8:2:2): Abort command succeeded -- 1 625513081.
2016-02-07T01:45:42.616Z cpu18:4114)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-07T01:45:42.616Z cpu18:4114)ScsiDeviceIO: 2291: Cmd(0x4124414d1c00) 0x2a, CmdSN 0x80000018 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.616Z cpu18:4114)ScsiDeviceIO: 2291: Cmd(0x412440dd5a40) 0x2a, CmdSN 0x80000051 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.616Z cpu18:4114)ScsiDeviceIO: 2291: Cmd(0x412441838f40) 0x2a, CmdSN 0x8000000e from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.616Z cpu18:4114)ScsiDeviceIO: 2291: Cmd(0x41244037bbc0) 0x2a, CmdSN 0x80000035 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.616Z cpu18:4114)ScsiDeviceIO: 2291: Cmd(0x41244037dac0) 0x2a, CmdSN 0x80000071 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-07T01:45:42.616Z cpu18:4114)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412440383400, 3107733) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-07T01:45:43.049Z cpu20:6177)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 121 stampUS 10792101769177 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 457201> drv 14.54] on vol 'xendatastore2'
2016-02-07T01:45:43.049Z cpu9:6175)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 129 stampUS 10792101769224 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 402000> drv 14.54] on vol 'xendatastore1'
2016-02-07T01:45:46.755Z cpu4:4131)HBX: 231: Reclaimed heartbeat for volume 4feaec43-24977e2a-c803-d067e5eeee6e (xendatastore1): [Timeout] [HB state abcdef02 offset 3727360 gen 129 stampUS 10792105909915 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB$
2016-02-07T01:45:46.755Z cpu4:4131)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore1' (held for 4039 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:45:46.756Z cpu0:4132)HBX: 231: Reclaimed heartbeat for volume 4f99169f-12a01f11-cabc-d067e5eee823 (xendatastore2): [Timeout] [HB state abcdef02 offset 3727360 gen 121 stampUS 10792105909403 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB$
2016-02-07T01:45:46.756Z cpu1:4132)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore2' (held for 4040 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-07T01:47:20.588Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x4124010937c0) 0x12, CmdSN 0x75382 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T01:52:20.606Z cpu10:9187169)ScsiDeviceIO: 2322: Cmd(0x412400f45400) 0x12, CmdSN 0x7538a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T01:57:20.621Z cpu20:9187437)ScsiDeviceIO: 2322: Cmd(0x412440cec900) 0x12, CmdSN 0x75392 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:02:20.637Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400f206c0) 0x12, CmdSN 0x753bc from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:07:20.654Z cpu18:4149)ScsiDeviceIO: 2322: Cmd(0x412440990400) 0x12, CmdSN 0x753c6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:12:20.668Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x4124419aa600) 0x12, CmdSN 0x753ce from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:17:20.685Z cpu16:4268)ScsiDeviceIO: 2322: Cmd(0x4124413db600) 0x12, CmdSN 0x753d6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:22:20.702Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x412401315000) 0x12, CmdSN 0x753de from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:27:20.716Z cpu18:9186448)ScsiDeviceIO: 2322: Cmd(0x4124403aa900) 0x12, CmdSN 0x753e6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:32:20.733Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412401b9c240) 0x12, CmdSN 0x75406 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:37:20.750Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x412400ef2ec0) 0x12, CmdSN 0x75410 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:42:20.768Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x412401a90840) 0x12, CmdSN 0x75418 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:47:20.785Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x4124003c4d80) 0x12, CmdSN 0x75420 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:52:20.805Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003d6840) 0x12, CmdSN 0x75428 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T02:57:20.824Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x4124010861c0) 0x12, CmdSN 0x75430 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:02:18.508Z cpu12:3103642)VSCSI: 6226: handle 8338(vscsi0:0):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:02:18.510Z cpu12:3103642)VSCSI: 6226: handle 8339(vscsi0:1):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:02:18.510Z cpu12:3103642)NetPort: 1427: disabled port 0x100000f
2016-02-07T03:02:18.571Z cpu21:3103642)CBT: 692: Disconnecting the cbt device a2cb33f-cbt with filehandle 170701631
2016-02-07T03:02:19.086Z cpu21:3103642)CBT: 692: Disconnecting the cbt device aa7b33c-cbt with filehandle 178762556
2016-02-07T03:02:20.841Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412400e58ec0) 0x12, CmdSN 0x7545e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:02:22.000Z cpu21:3103642)CBT: 1082: Created device b1bb33e-cbt for cbt driver with filehandle 186364734
2016-02-07T03:02:22.001Z cpu21:3103642)VSCSI: 3642: handle 8343(vscsi0:0):Using sync mode due to sparse disks
2016-02-07T03:02:22.001Z cpu21:3103642)VSCSI: 3683: handle 8343(vscsi0:0):Creating Virtual Device for world 3107733 (FSS handle 187020096)
2016-02-07T03:02:22.182Z cpu21:3103642)CBT: 1082: Created device 38b9b343-cbt for cbt driver with filehandle 951694147
2016-02-07T03:02:22.184Z cpu21:3103642)VSCSI: 3642: handle 8344(vscsi0:1):Using sync mode due to sparse disks
2016-02-07T03:02:22.184Z cpu21:3103642)VSCSI: 3683: handle 8344(vscsi0:1):Creating Virtual Device for world 3107733 (FSS handle 580956997)
2016-02-07T03:02:22.184Z cpu21:3103642)NetPort: 1240: enabled port 0x100000f with mac 00:50:56:9f:65:7f
2016-02-07T03:03:44.401Z cpu17:3103642)VSCSI: 6226: handle 8343(vscsi0:0):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:03:44.401Z cpu17:3103642)VSCSI: 6226: handle 8344(vscsi0:1):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:03:44.401Z cpu17:3103642)NetPort: 1427: disabled port 0x100000f
2016-02-07T03:03:44.411Z cpu17:3103642)CBT: 692: Disconnecting the cbt device 38b9b343-cbt with filehandle 951694147
2016-02-07T03:03:44.423Z cpu17:3103642)CBT: 692: Disconnecting the cbt device b1bb33e-cbt with filehandle 186364734
2016-02-07T03:03:48.477Z cpu18:3103642)CBT: 1082: Created device b39b33e-cbt for cbt driver with filehandle 188330814
2016-02-07T03:03:48.539Z cpu18:3103642)CBT: 1082: Created device 1d08b342-cbt for cbt driver with filehandle 487109442
2016-02-07T03:03:48.540Z cpu18:3103642)CBT: 692: Disconnecting the cbt device b39b33e-cbt with filehandle 188330814
2016-02-07T03:03:48.541Z cpu18:3103642)CBT: 692: Disconnecting the cbt device 1d08b342-cbt with filehandle 487109442
2016-02-07T03:03:48.547Z cpu18:3103642)CBT: 1082: Created device 1d0ab342-cbt for cbt driver with filehandle 487240514
2016-02-07T03:03:48.583Z cpu18:3103642)CBT: 1082: Created device 1f76b344-cbt for cbt driver with filehandle 527872836
2016-02-07T03:03:48.584Z cpu18:3103642)CBT: 692: Disconnecting the cbt device 1d0ab342-cbt with filehandle 487240514
2016-02-07T03:03:48.585Z cpu18:3103642)CBT: 692: Disconnecting the cbt device 1f76b344-cbt with filehandle 527872836
2016-02-07T03:03:48.591Z cpu18:3103642)CBT: 1082: Created device 1d0cb342-cbt for cbt driver with filehandle 487371586
2016-02-07T03:03:48.592Z cpu18:3103642)VSCSI: 3642: handle 8345(vscsi0:0):Using sync mode due to sparse disks
2016-02-07T03:03:48.592Z cpu18:3103642)VSCSI: 3683: handle 8345(vscsi0:0):Creating Virtual Device for world 3107733 (FSS handle 952087363)
2016-02-07T03:03:48.617Z cpu18:3103642)CBT: 1082: Created device ff5b348-cbt for cbt driver with filehandle 267760456
2016-02-07T03:03:48.619Z cpu18:3103642)VSCSI: 3642: handle 8346(vscsi0:1):Using sync mode due to sparse disks
2016-02-07T03:03:48.619Z cpu18:3103642)VSCSI: 3683: handle 8346(vscsi0:1):Creating Virtual Device for world 3107733 (FSS handle 394244938)
2016-02-07T03:03:48.619Z cpu18:3103642)NetPort: 1240: enabled port 0x100000f with mac 00:50:56:9f:65:7f
2016-02-07T03:03:50.056Z cpu21:3103642)VSCSI: 6226: handle 8345(vscsi0:0):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:03:50.057Z cpu21:3103642)VSCSI: 6226: handle 8346(vscsi0:1):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:03:50.058Z cpu21:3103642)NetPort: 1427: disabled port 0x100000f
2016-02-07T03:03:50.062Z cpu21:3103642)CBT: 692: Disconnecting the cbt device 1d0cb342-cbt with filehandle 487371586
2016-02-07T03:03:50.072Z cpu21:3103642)CBT: 1082: Created device 1a47b34b-cbt for cbt driver with filehandle 440906571
2016-02-07T03:03:50.073Z cpu21:3103642)CBT: 692: Disconnecting the cbt device 1a47b34b-cbt with filehandle 440906571
2016-02-07T03:03:50.131Z cpu14:3103642)CBT: 692: Disconnecting the cbt device ff5b348-cbt with filehandle 267760456
2016-02-07T03:03:50.188Z cpu14:3103642)CBT: 1082: Created device 1046b349-cbt for cbt driver with filehandle 273068873
2016-02-07T03:03:50.191Z cpu14:3103642)VSCSI: 3642: handle 8347(vscsi0:0):Using sync mode due to sparse disks
2016-02-07T03:03:50.191Z cpu14:3103642)VSCSI: 3683: handle 8347(vscsi0:0):Creating Virtual Device for world 3107733 (FSS handle 440972107)
2016-02-07T03:03:50.198Z cpu14:3103642)CBT: 1082: Created device 20a6b34d-cbt for cbt driver with filehandle 547795789
2016-02-07T03:03:50.210Z cpu14:3103642)CBT: 1082: Created device 1772b351-cbt for cbt driver with filehandle 393392977
2016-02-07T03:03:50.212Z cpu14:3103642)CBT: 692: Disconnecting the cbt device 20a6b34d-cbt with filehandle 547795789
2016-02-07T03:03:50.214Z cpu14:3103642)CBT: 692: Disconnecting the cbt device 1772b351-cbt with filehandle 393392977
2016-02-07T03:03:50.223Z cpu14:3103642)CBT: 1082: Created device 1774b351-cbt for cbt driver with filehandle 393524049
2016-02-07T03:03:50.227Z cpu14:3103642)CBT: 1082: Created device 171eb353-cbt for cbt driver with filehandle 387887955
2016-02-07T03:03:50.229Z cpu14:3103642)CBT: 692: Disconnecting the cbt device 1774b351-cbt with filehandle 393524049
2016-02-07T03:03:50.230Z cpu14:3103642)CBT: 692: Disconnecting the cbt device 171eb353-cbt with filehandle 387887955
2016-02-07T03:03:50.240Z cpu14:3103642)CBT: 1082: Created device 1776b351-cbt for cbt driver with filehandle 393655121
2016-02-07T03:03:50.241Z cpu14:3103642)VSCSI: 3642: handle 8348(vscsi0:1):Using sync mode due to sparse disks
2016-02-07T03:03:50.241Z cpu14:3103642)VSCSI: 3683: handle 8348(vscsi0:1):Creating Virtual Device for world 3107733 (FSS handle 681554770)
2016-02-07T03:03:50.241Z cpu14:3103642)NetPort: 1240: enabled port 0x100000f with mac 00:50:56:9f:65:7f
2016-02-07T03:03:50.906Z cpu12:3103642)VSCSI: 6226: handle 8347(vscsi0:0):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:03:50.907Z cpu12:3103642)VSCSI: 6226: handle 8348(vscsi0:1):Destroying Device for world 3107733 (pendCom 0)
2016-02-07T03:03:50.907Z cpu12:3103642)NetPort: 1427: disabled port 0x100000f
2016-02-07T03:03:50.911Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 1776b351-cbt with filehandle 393655121
2016-02-07T03:03:50.926Z cpu12:3103642)CBT: 1082: Created device 1940b354-cbt for cbt driver with filehandle 423670612
2016-02-07T03:03:50.928Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 1940b354-cbt with filehandle 423670612
2016-02-07T03:03:50.968Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 1046b349-cbt with filehandle 273068873
2016-02-07T03:03:51.123Z cpu12:3103642)CBT: 1082: Created device 2fb11f5-cbt for cbt driver with filehandle 50008565
2016-02-07T03:03:51.128Z cpu12:3103642)CBT: 1082: Created device 2fd11f9-cbt for cbt driver with filehandle 50139641
2016-02-07T03:03:51.129Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 2fd11f9-cbt with filehandle 50139641
2016-02-07T03:03:51.130Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 2fb11f5-cbt with filehandle 50008565
2016-02-07T03:03:51.135Z cpu12:3103642)CBT: 1082: Created device 2ff11f9-cbt for cbt driver with filehandle 50270713
2016-02-07T03:03:51.421Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 2ff11f9-cbt with filehandle 50270713
2016-02-07T03:03:51.471Z cpu12:3103642)CBT: 1082: Created device 30111f5-cbt for cbt driver with filehandle 50401781
2016-02-07T03:03:51.476Z cpu12:3103642)CBT: 1082: Created device 30111f9-cbt for cbt driver with filehandle 50401785
2016-02-07T03:03:51.478Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 30111f9-cbt with filehandle 50401785
2016-02-07T03:03:51.479Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 30111f5-cbt with filehandle 50401781
2016-02-07T03:03:51.489Z cpu12:3103642)CBT: 1082: Created device 30311f9-cbt for cbt driver with filehandle 50532857
2016-02-07T03:03:51.806Z cpu12:3103642)CBT: 692: Disconnecting the cbt device 30311f9-cbt with filehandle 50532857
2016-02-07T03:03:51.878Z cpu12:3103642)CBT: 1082: Created device 30711f5-cbt for cbt driver with filehandle 50794997
2016-02-07T03:03:51.879Z cpu12:3103642)VSCSI: 3683: handle 8349(vscsi0:0):Creating Virtual Device for world 3107733 (FSS handle 50663927)
2016-02-07T03:03:51.883Z cpu12:3103642)CBT: 1082: Created device 2f911f8-cbt for cbt driver with filehandle 49877496
2016-02-07T03:03:51.884Z cpu12:3103642)VSCSI: 3683: handle 8350(vscsi0:1):Creating Virtual Device for world 3107733 (FSS handle 47256058)
2016-02-07T03:03:51.885Z cpu12:3103642)NetPort: 1240: enabled port 0x100000f with mac 00:50:56:9f:65:7f
2016-02-07T03:07:20.858Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x41240103df00) 0x12, CmdSN 0x754a4 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:12:20.877Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x4124403a8500) 0x12, CmdSN 0x754ac from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:17:20.899Z cpu9:9191416)ScsiDeviceIO: 2322: Cmd(0x412401438740) 0x12, CmdSN 0x754b4 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:22:20.917Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x41244141cb00) 0x12, CmdSN 0x754bc from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:27:20.938Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x412401488680) 0x12, CmdSN 0x754c4 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:32:20.957Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412401486440) 0x12, CmdSN 0x754e4 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:37:20.975Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412441a4d500) 0x12, CmdSN 0x754ee from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:42:20.995Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x412441be5f00) 0x12, CmdSN 0x754f6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:47:21.014Z cpu22:9193469)ScsiDeviceIO: 2322: Cmd(0x412440e5e5c0) 0x12, CmdSN 0x754fe from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:52:21.034Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412401be40c0) 0x12, CmdSN 0x75506 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T03:57:21.053Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x412441c90d40) 0x12, CmdSN 0x7550e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:02:21.073Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x412440371b80) 0x12, CmdSN 0x75538 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:07:21.094Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400384600) 0x12, CmdSN 0x75542 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:12:21.113Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412440df06c0) 0x12, CmdSN 0x7554a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:17:21.134Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412400e96300) 0x12, CmdSN 0x75552 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:22:21.152Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x412400f44c00) 0x12, CmdSN 0x7555a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:27:21.171Z cpu23:9187544)ScsiDeviceIO: 2322: Cmd(0x4124403d14c0) 0x12, CmdSN 0x75562 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:32:21.192Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x4124003a6b00) 0x12, CmdSN 0x75582 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:37:21.210Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412441be9000) 0x12, CmdSN 0x7558c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:42:21.229Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x412441111140) 0x12, CmdSN 0x75594 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:47:21.249Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124015957c0) 0x12, CmdSN 0x7559c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:52:21.269Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400383d00) 0x12, CmdSN 0x755a4 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T04:57:21.290Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412441419300) 0x12, CmdSN 0x755ac from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:02:21.307Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x4124412e54c0) 0x12, CmdSN 0x755d6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:07:21.328Z cpu15:9198439)ScsiDeviceIO: 2322: Cmd(0x412441113b40) 0x12, CmdSN 0x755e0 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:12:21.346Z cpu4:4100)ScsiDeviceIO: 2322: Cmd(0x412401576a00) 0x12, CmdSN 0x755e8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:17:21.364Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x41240039cb40) 0x12, CmdSN 0x755f0 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:22:21.385Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400380200) 0x12, CmdSN 0x755f8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:27:21.404Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003a4700) 0x12, CmdSN 0x75600 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:32:21.422Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x4124003bb380) 0x12, CmdSN 0x75620 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:37:21.439Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400382100) 0x12, CmdSN 0x7562a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:42:21.455Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x412440381600) 0x12, CmdSN 0x75632 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:47:21.472Z cpu19:4268)ScsiDeviceIO: 2322: Cmd(0x412441392340) 0x12, CmdSN 0x7563a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:52:21.490Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x412441c90f40) 0x12, CmdSN 0x75642 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T05:57:21.505Z cpu6:9184891)ScsiDeviceIO: 2322: Cmd(0x41240148ad80) 0x12, CmdSN 0x7564a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:02:21.522Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x412401be06c0) 0x12, CmdSN 0x75674 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:07:21.538Z cpu19:4115)ScsiDeviceIO: 2322: Cmd(0x412440df0bc0) 0x12, CmdSN 0x7567e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:12:21.554Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412441c94b40) 0x12, CmdSN 0x75686 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:17:21.572Z cpu14:4110)ScsiDeviceIO: 2322: Cmd(0x4124414d3600) 0x12, CmdSN 0x7568e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:22:21.588Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x41244160aa80) 0x12, CmdSN 0x75696 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:27:21.604Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x412441be6d00) 0x12, CmdSN 0x7569e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:32:21.623Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x41244171e4c0) 0x12, CmdSN 0x756be from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:37:21.640Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x412441c90a40) 0x12, CmdSN 0x756c8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:42:21.656Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412400ec9f00) 0x12, CmdSN 0x756d0 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:47:21.670Z cpu16:9202831)ScsiDeviceIO: 2322: Cmd(0x4124403c66c0) 0x12, CmdSN 0x756d8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:52:21.686Z cpu15:4111)ScsiDeviceIO: 2322: Cmd(0x412440396e80) 0x12, CmdSN 0x756e0 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T06:57:21.699Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x4124403cecc0) 0x12, CmdSN 0x756e8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:02:21.714Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x412440e5edc0) 0x12, CmdSN 0x75712 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:07:21.728Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x412441a49200) 0x12, CmdSN 0x7571c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:12:21.744Z cpu4:4100)ScsiDeviceIO: 2322: Cmd(0x412401ba1f00) 0x12, CmdSN 0x75724 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:17:21.758Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412401592ec0) 0x12, CmdSN 0x7572c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:22:21.774Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124011cc900) 0x12, CmdSN 0x75734 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:27:21.789Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x412441c10540) 0x12, CmdSN 0x7573c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:32:21.802Z cpu15:8755483)ScsiDeviceIO: 2322: Cmd(0x412441c9c800) 0x12, CmdSN 0x7575c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:37:21.818Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x4124415c6b40) 0x12, CmdSN 0x75766 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-07T07:42:21.831Z cpu14:8751389)ScsiDeviceIO: 2322: Cmd(0x4124413d9100) 0x12, CmdSN 0x7576e from world 0 to dev "naa.600
2016-02-09T00:02:29.738Z cpu4:4148)ScsiDeviceIO: 2322: Cmd(0x412400382000) 0x12, CmdSN 0x7704a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:07:29.754Z cpu12:4108)ScsiDeviceIO: 2322: Cmd(0x4124414d3600) 0x12, CmdSN 0x7706c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:12:29.772Z cpu19:4115)ScsiDeviceIO: 2322: Cmd(0x4124403b9ec0) 0x12, CmdSN 0x77074 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:17:29.789Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412400e5a7c0) 0x12, CmdSN 0x7707c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:22:29.805Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412441c91540) 0x12, CmdSN 0x77084 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:27:29.821Z cpu8:8755471)ScsiDeviceIO: 2322: Cmd(0x412401bec600) 0x12, CmdSN 0x7708c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:32:24.467Z cpu2:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634807327.
2016-02-09T00:32:24.467Z cpu4:8755471)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124010817c0, 4120) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba3:C0:T2:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-09T00:32:24.467Z cpu4:8755471)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-09T00:32:24.467Z cpu4:8755471)ScsiDeviceIO: 2309: Cmd(0x4124010817c0) 0x2a, CmdSN 0x4fc119 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-09T00:32:24.468Z cpu2:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:2): Abort command succeeded -- 1 634807326.
2016-02-09T00:32:24.468Z cpu4:4755)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412400f260c0, 4120) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba2:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-09T00:32:24.468Z cpu4:4755)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-09T00:32:24.468Z cpu4:4755)ScsiDeviceIO: 2309: Cmd(0x412400f260c0) 0x2a, CmdSN 0x4191fc from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-09T00:32:29.840Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400e96900) 0x12, CmdSN 0x77094 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:37:29.857Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412400f62680) 0x12, CmdSN 0x770b6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:42:29.872Z cpu17:9326144)ScsiDeviceIO: 2322: Cmd(0x412440994600) 0x12, CmdSN 0x770be from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:47:29.889Z cpu1:9331862)ScsiDeviceIO: 2322: Cmd(0x412401693a00) 0x12, CmdSN 0x770c6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:52:29.906Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x412401a91340) 0x12, CmdSN 0x770ce from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:57:29.922Z cpu16:9325015)ScsiDeviceIO: 2322: Cmd(0x41244039a0c0) 0x12, CmdSN 0x770d6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:01:28.552Z cpu2:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907599.
2016-02-09T01:01:28.552Z cpu2:4290)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412400f1a240, 4120) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba3:C0:T2:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-09T01:01:28.552Z cpu2:4290)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-09T01:01:28.552Z cpu2:4290)ScsiDeviceIO: 2309: Cmd(0x412400f1a240) 0x2a, CmdSN 0x4fc426 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-09T01:01:28.552Z cpu2:4290)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907598.
2016-02-09T01:01:28.552Z cpu2:4290)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412400f64780, 4120) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T3:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-09T01:01:28.552Z cpu2:4290)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-09T01:01:28.552Z cpu2:4290)ScsiDeviceIO: 2309: Cmd(0x412400f64780) 0x2a, CmdSN 0x419482 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-09T01:01:30.551Z cpu15:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907629.
2016-02-09T01:01:30.551Z cpu9:8747281)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-09T01:01:30.551Z cpu9:8747281)ScsiDeviceIO: 2309: Cmd(0x412400f64780) 0x2a, CmdSN 0x4fc42a from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-09T01:01:30.551Z cpu15:4290)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907630.
2016-02-09T01:01:30.551Z cpu9:8747281)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-09T01:01:30.551Z cpu9:8747281)ScsiDeviceIO: 2309: Cmd(0x412400f1a240) 0x2a, CmdSN 0x419483 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-09T01:01:30.623Z cpu5:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907595.
2016-02-09T01:01:30.624Z cpu12:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907601.
2016-02-09T01:01:30.624Z cpu5:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907596.
2016-02-09T01:01:30.624Z cpu14:4110)ScsiDeviceIO: 2291: Cmd(0x41244114e3c0) 0x2a, CmdSN 0x80000012 from world 8751385 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.624Z cpu14:4110)ScsiDeviceIO: 2291: Cmd(0x41244183a240) 0x2a, CmdSN 0x8000004d from world 8751385 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.624Z cpu12:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907602.
2016-02-09T01:01:30.625Z cpu12:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907605.
2016-02-09T01:01:30.625Z cpu6:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907597.
2016-02-09T01:01:30.626Z cpu12:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907620.
2016-02-09T01:01:30.626Z cpu6:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907621.
2016-02-09T01:01:30.626Z cpu14:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907623.
2016-02-09T01:01:30.627Z cpu14:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907624.
2016-02-09T01:01:30.627Z cpu6:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907628.
2016-02-09T01:01:30.627Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x4124415a9bc0) 0x2a, CmdSN 0x8000005a from world 9206876 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.627Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x412441bec600) 0x2a, CmdSN 0x80000059 from world 9206876 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.627Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x4124403cd6c0) 0x2a, CmdSN 0x8000004b from world 9206876 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.627Z cpu14:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907625.
2016-02-09T01:01:30.628Z cpu6:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907600.
2016-02-09T01:01:30.628Z cpu10:4106)ScsiDeviceIO: 2291: Cmd(0x412401439d40) 0x2a, CmdSN 0x80000014 from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.628Z cpu14:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907626.
2016-02-09T01:01:30.629Z cpu10:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907603.
2016-02-09T01:01:30.629Z cpu14:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907627.
2016-02-09T01:01:30.629Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x412441041440) 0x2a, CmdSN 0x8000004d from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.629Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x41244039b1c0) 0x2a, CmdSN 0x80000063 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.629Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x412440376580) 0x2a, CmdSN 0x8000007b from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.629Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x412441c9e600) 0x2a, CmdSN 0x80000058 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.629Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x412440e12240) 0x2a, CmdSN 0x80000060 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.629Z cpu13:4109)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412441c9de00, 3107733) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T3:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-09T01:01:30.629Z cpu10:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907604.
2016-02-09T01:01:30.630Z cpu10:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907622.
2016-02-09T01:01:30.630Z cpu13:5220)ScsiDeviceIO: 2291: Cmd(0x41244114b7c0) 0x2a, CmdSN 0x4fc427 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.630Z cpu13:5220)ScsiDeviceIO: 2291: Cmd(0x4124403c22c0) 0x2a, CmdSN 0x4fc428 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.630Z cpu10:4133)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412401ae87c0, 4120) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba3:C0:T2:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-09T01:01:30.630Z cpu10:4133)ScsiDeviceIO: 2291: Cmd(0x412401ae87c0) 0x2a, CmdSN 0x4fc429 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.968Z cpu10:4133)HBX: 231: Reclaimed heartbeat for volume 4f99169f-12a01f11-cabc-d067e5eee823 (xendatastore2): [Timeout] [HB state abcdef02 offset 3727360 gen 121 stampUS 10962240765522 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <F$
2016-02-09T01:01:30.968Z cpu10:4133)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore2' (held for 236 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-09T01:01:30.994Z cpu18:6178)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 129 stampUS 10962240421216 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 402000> drv 14.54] on vol 'xendatastore1'
2016-02-09T01:01:31.007Z cpu12:4131)HBX: 231: Reclaimed heartbeat for volume 4feaec43-24977e2a-c803-d067e5eeee6e (xendatastore1): [Timeout] [HB state abcdef02 offset 3727360 gen 129 stampUS 10962240805104 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <F$
2016-02-09T01:01:31.007Z cpu12:4131)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore1' (held for 276 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-09T01:02:29.940Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x412440e10640) 0x12, CmdSN 0x770ec from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:07:29.957Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400381100) 0x12, CmdSN 0x7710e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:12:29.976Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x412440db0240) 0x12, CmdSN 0x77116 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:17:29.993Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003bd380) 0x12, CmdSN 0x7711e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:22:30.009Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124003e3c40) 0x12, CmdSN 0x77126 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:27:30.026Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x41240037bfc0) 0x12, CmdSN 0x7712e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:32:30.045Z cpu14:4110)ScsiDeviceIO: 2322: Cmd(0x4124415ac0c0) 0x12, CmdSN 0x77136 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:37:30.062Z cpu20:9323049)ScsiDeviceIO: 2322: Cmd(0x412441a51ac0) 0x12, CmdSN 0x77158 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:42:30.077Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x412440cea900) 0x12, CmdSN 0x77160 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:47:30.094Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x412400f18640) 0x12, CmdSN 0x77168 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:52:30.113Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412401095ac0) 0x12, CmdSN 0x77170 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:57:30.130Z cpu22:9335606)ScsiDeviceIO: 2322: Cmd(0x4124413d8100) 0x12, CmdSN 0x77178 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:01:29.618Z cpu2:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:32.377Z cpu22:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:01:33.743Z cpu11:8755476)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:33.778Z cpu11:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:33.782Z cpu11:8755476)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:35.692Z cpu18:9215075)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:01:35.725Z cpu18:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:01:35.729Z cpu18:9215075)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:01:35.838Z cpu6:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:35.843Z cpu6:8755476)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:35.893Z cpu6:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:35.897Z cpu6:8755476)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:36.035Z cpu6:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:36.041Z cpu6:8755476)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:36.333Z cpu0:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:36.395Z cpu6:8755476)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:37.785Z cpu18:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:01:37.791Z cpu18:9215075)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:01:37.840Z cpu18:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:01:37.844Z cpu18:9215075)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:01:37.977Z cpu18:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:01:37.983Z cpu18:9215075)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:01:38.252Z cpu23:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:01:38.319Z cpu12:9215075)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:01:48.796Z cpu9:8751383)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:48.796Z cpu9:8751383)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:52.319Z cpu4:8751381)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:52.320Z cpu4:8751381)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:52.326Z cpu2:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:52.339Z cpu9:8751383)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:02:02.657Z cpu22:8755491)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:06.845Z cpu14:8755488)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:06.899Z cpu14:8755488)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:06.903Z cpu14:8755488)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:08.931Z cpu12:8755488)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:08.936Z cpu12:8755488)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:08.986Z cpu12:8755488)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:08.991Z cpu16:8755488)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:09.124Z cpu22:8755488)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:09.129Z cpu16:8755488)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:09.398Z cpu16:8755488)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:09.465Z cpu20:8755488)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:15.891Z cpu16:9210982)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:02:15.892Z cpu16:9210982)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:02:19.338Z cpu13:9210980)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:02:19.339Z cpu13:9210980)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:02:19.341Z cpu16:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:02:19.352Z cpu14:9210982)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:02:30.145Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124011c1c80) 0x12, CmdSN 0x7718f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:02:49.475Z cpu21:8755491)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:49.476Z cpu21:8755491)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:53.045Z cpu16:8755489)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:53.046Z cpu16:8755489)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:53.049Z cpu22:8755488)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:53.063Z cpu12:8755491)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:07:30.162Z cpu6:9331790)ScsiDeviceIO: 2322: Cmd(0x4124003c3480) 0x12, CmdSN 0x771bb from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:12:30.180Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x412441c93340) 0x12, CmdSN 0x771c3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:17:30.198Z cpu21:4769)ScsiDeviceIO: 2322: Cmd(0x412440dd0a40) 0x12, CmdSN 0x771cb from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:22:30.213Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400e5e4c0) 0x12, CmdSN 0x771d3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:27:30.230Z cpu22:9331790)ScsiDeviceIO: 2322: Cmd(0x4124403b2640) 0x12, CmdSN 0x771db from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:32:30.247Z cpu13:4276)ScsiDeviceIO: 2322: Cmd(0x412440398bc0) 0x12, CmdSN 0x771e3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:37:30.263Z cpu12:4108)ScsiDeviceIO: 2322: Cmd(0x412440d7a4c0) 0x12, CmdSN 0x77205 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:42:30.281Z cpu10:9333625)ScsiDeviceIO: 2322: Cmd(0x4124003a0600) 0x12, CmdSN 0x7720d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:47:30.298Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x41240131c6c0) 0x12, CmdSN 0x77215 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:52:30.312Z cpu4:4100)ScsiDeviceIO: 2322: Cmd(0x4124011c9c00) 0x12, CmdSN 0x7721d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:57:30.331Z cpu2:9329988)ScsiDeviceIO: 2322: Cmd(0x412400385a00) 0x12, CmdSN 0x77225 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:02:30.347Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400f44400) 0x12, CmdSN 0x77237 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:07:30.363Z cpu14:9339821)ScsiDeviceIO: 2322: Cmd(0x412440dd4840) 0x12, CmdSN 0x77259 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:12:30.381Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412400f23fc0) 0x12, CmdSN 0x77261 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:17:30.398Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x412440c7e800) 0x12, CmdSN 0x77269 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:22:30.414Z cpu20:4276)ScsiDeviceIO: 2322: Cmd(0x412441609280) 0x12, CmdSN 0x77271 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:27:30.429Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400f1ee40) 0x12, CmdSN 0x77279 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:32:30.445Z cpu4:4100)ScsiDeviceIO: 2322: Cmd(0x412401b98e40) 0x12, CmdSN 0x77281 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:37:30.463Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x4124403dd4c0) 0x12, CmdSN 0x772a3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:42:30.479Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x4124412a4940) 0x12, CmdSN 0x772ab from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:47:30.494Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x41244160e680) 0x12, CmdSN 0x772b3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:52:30.511Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x41244038bd80) 0x12, CmdSN 0x772bb from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:57:30.528Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x41240131a2c0) 0x12, CmdSN 0x772c3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:02:30.542Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400e7c540) 0x12, CmdSN 0x772d5 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:07:30.559Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412400385700) 0x12, CmdSN 0x772f7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:12:30.573Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412401ae4ac0) 0x12, CmdSN 0x772ff from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:17:30.588Z cpu4:4100)ScsiDeviceIO: 2322: Cmd(0x412400f79800) 0x12, CmdSN 0x77307 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:22:30.604Z cpu14:4110)ScsiDeviceIO: 2322: Cmd(0x412441394540) 0x12, CmdSN 0x7730f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:27:30.620Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x412440e5dbc0) 0x12, CmdSN 0x77317 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:32:30.637Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x4124414d4c00) 0x12, CmdSN 0x7731f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:37:30.652Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x4124403d21c0) 0x12, CmdSN 0x77341 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:42:30.667Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412401b9de40) 0x12, CmdSN 0x77349 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:47:30.682Z cpu19:4115)ScsiDeviceIO: 2322: Cmd(0x412441c9c800) 0x12, CmdSN 0x77351 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:52:30.700Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x412441041f40) 0x12, CmdSN 0x77359 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:57:30.716Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x412441a4b700) 0x12, CmdSN 0x77361 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:02:30.730Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x4124003ed700) 0x12, CmdSN 0x77373 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:07:30.745Z cpu5:9336876)ScsiDeviceIO: 2322: Cmd(0x412400392700) 0x12, CmdSN 0x77395 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:12:30.762Z cpu4:9336876)ScsiDeviceIO: 2322: Cmd(0x412400ec8b00) 0x12, CmdSN 0x7739d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:17:30.778Z cpu19:4115)ScsiDeviceIO: 2322: Cmd(0x41244160bf80) 0x12, CmdSN 0x773a5 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:22:30.794Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x4124411494c0) 0x12, CmdSN 0x773ad from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:27:30.808Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x412400389f40) 0x12, CmdSN 0x773b5 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:32:30.825Z cpu14:9341242)ScsiDeviceIO: 2322: Cmd(0x41244180e600) 0x12, CmdSN 0x773bd from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:37:30.839Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400f02b80) 0x12, CmdSN 0x773df from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:42:30.854Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x4124010821c0) 0x12, CmdSN 0x773e7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:47:30.870Z cpu20:9343468)ScsiDeviceIO: 2322: Cmd(0x4124403bb3c0) 0x12, CmdSN 0x773ef from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:52:30.886Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412400390b00) 0x12, CmdSN 0x773f7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:57:30.901Z cpu2:9348375)ScsiDeviceIO: 2322: Cmd(0x412401a96540) 0x12, CmdSN 0x773ff from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:02:30.918Z cpu18:8751389)ScsiDeviceIO: 2322: Cmd(0x4124413d9f00) 0x12, CmdSN 0x77411 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:07:30.933Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x4124003bb980) 0x12, CmdSN 0x77433 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:12:30.948Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400382d00) 0x12, CmdSN 0x7743b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:17:30.962Z cpu22:9215071)ScsiDeviceIO: 2322: Cmd(0x4124403bcec0) 0x12, CmdSN 0x77443 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:22:30.977Z cpu21:9215071)ScsiDeviceIO: 2322: Cmd(0x412440c7c400) 0x12, CmdSN 0x7744b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:27:30.992Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x412400390800) 0x12, CmdSN 0x77453 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:32:31.008Z cpu10:4149)ScsiDeviceIO: 2322: Cmd(0x41240147c800) 0x12, CmdSN 0x7745b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:37:31.024Z cpu5:9349748)ScsiDeviceIO: 2322: Cmd(0x412401574200) 0x12, CmdSN 0x7747d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:42:31.038Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x41244183c340) 0x12, CmdSN 0x77485 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:47:31.053Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412401596bc0) 0x12, CmdSN 0x7748d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:52:31.069Z cpu3:9343467)ScsiDeviceIO: 2322: Cmd(0x4124003d0140) 0x12, CmdSN 0x77495 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:57:31.085Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412401486940) 0x12, CmdSN 0x7749d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:02:31.099Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x412400f44100) 0x12, CmdSN 0x774af from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:07:31.114Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x4124419ae800) 0x12, CmdSN 0x774d1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:12:31.130Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412440995000) 0x12, CmdSN 0x774d9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:17:31.146Z cpu10:4148)ScsiDeviceIO: 2322: Cmd(0x412401439e40) 0x12, CmdSN 0x774e1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:22:31.163Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x412440db5440) 0x12, CmdSN 0x774e9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:27:31.179Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x4124014ca300) 0x12, CmdSN 0x774f1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:32:31.196Z cpu15:9347988)ScsiDeviceIO: 2322: Cmd(0x4124412821c0) 0x12, CmdSN 0x774f9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:37:31.212Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x41240147a300) 0x12, CmdSN 0x7751b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:42:31.230Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x412441c12b40) 0x12, CmdSN 0x77523 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:47:31.247Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x4124011ce300) 0x12, CmdSN 0x7752b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:52:31.262Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412400f02b80) 0x12, CmdSN 0x77533 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:57:31.280Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x4124419ae300) 0x12, CmdSN 0x7753b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:02:31.296Z cpu14:4276)ScsiDeviceIO: 2322: Cmd(0x412441be5c00) 0x12, CmdSN 0x7754d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:07:31.312Z cpu4:9353079)ScsiDeviceIO: 2322: Cmd(0x4124003b1880) 0x12, CmdSN 0x7756f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:12:31.330Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412401aea4c0) 0x12, CmdSN 0x77577 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:17:31.347Z cpu11:9355067)ScsiDeviceIO: 2322: Cmd(0x4124003dab40) 0x12, CmdSN 0x7757f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:22:31.364Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400390100) 0x12, CmdSN 0x77587 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:27:31.380Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412401ad0f80) 0x12, CmdSN 0x7758f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:32:31.397Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x412440cecc00) 0x12, CmdSN 0x77597 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:37:31.414Z cpu12:4108)ScsiDeviceIO: 2322: Cmd(0x4124403999c0) 0x12, CmdSN 0x775b9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:42:31.432Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x412400f78800) 0x12, CmdSN 0x775c1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:47:31.451Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412400eca900) 0x12, CmdSN 0x775c9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:52:31.467Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x4124015956c0) 0x12, CmdSN 0x775d1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:57:31.485Z cpu3:9357325)ScsiDeviceIO: 2322: Cmd(0x412401ae92c0) 0x12, CmdSN 0x775d9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:02:31.503Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x41240102d780) 0x12, CmdSN 0x775eb from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:07:31.519Z cpu9:9350018)ScsiDeviceIO: 2322: Cmd(0x4124014bed80) 0x12, CmdSN 0x7760d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:12:31.538Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x41244139c300) 0x12, CmdSN 0x77615 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:17:31.554Z cpu3:9358667)ScsiDeviceIO: 2322: Cmd(0x412401696b00) 0x12, CmdSN 0x7761d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:22:31.572Z cpu19:4115)ScsiDeviceIO: 2322: Cmd(0x4124403793c0) 0x12, CmdSN 0x77625 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:27:31.591Z cpu8:4149)ScsiDeviceIO: 2322: Cmd(0x4124011c6c80) 0x12, CmdSN 0x7762d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:32:31.608Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x412400378dc0) 0x12, CmdSN 0x77635 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:37:31.626Z cpu2:9359302)ScsiDeviceIO: 2322: Cmd(0x412401ba3f00) 0x12, CmdSN 0x77657 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:42:31.640Z cpu4:8747281)ScsiDeviceIO: 2322: Cmd(0x412400f1d040) 0x12, CmdSN 0x7765f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:47:31.658Z cpu5:4774)ScsiDeviceIO: 2322: Cmd(0x412400e5cec0) 0x12, CmdSN 0x77667 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:52:31.677Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x41240148d980) 0x12, CmdSN 0x7766f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:57:31.694Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400f40b00) 0x12, CmdSN 0x77677 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:02:31.710Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x4124415acac0) 0x12, CmdSN 0x77689 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:07:31.727Z cpu22:4268)ScsiDeviceIO: 2322: Cmd(0x412440384100) 0x12, CmdSN 0x776ab from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:12:31.745Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x41240103af00) 0x12, CmdSN 0x776b3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:17:31.761Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x412400f00580) 0x12, CmdSN 0x776bb from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:22:31.780Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x41240113d980) 0x12, CmdSN 0x776c3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:27:31.796Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x4124010914c0) 0x12, CmdSN 0x776cb from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:32:31.813Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124003b9880) 0x12, CmdSN 0x776d3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:37:31.830Z cpu13:9362236)ScsiDeviceIO: 2322: Cmd(0x4124403cb2c0) 0x12, CmdSN 0x776f5 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:42:31.848Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x41240039cd40) 0x12, CmdSN 0x776fd from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:47:31.864Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x4124014ba580) 0x12, CmdSN 0x77705 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:52:31.882Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x412441c16240) 0x12, CmdSN 0x7770d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:57:31.899Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x412401a93540) 0x12, CmdSN 0x77715 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:02:31.918Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x412401484440) 0x12, CmdSN 0x77727 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:07:31.935Z cpu3:4268)ScsiDeviceIO: 2322: Cmd(0x4124003e6b40) 0x12, CmdSN 0x77749 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:12:31.952Z cpu4:4100)ScsiDeviceIO: 2322: Cmd(0x4124015908c0) 0x12, CmdSN 0x77751 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:17:31.968Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412400383000) 0x12, CmdSN 0x77759 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:22:31.986Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x412400ef2ac0) 0x12, CmdSN 0x77761 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:27:32.002Z cpu11:9356117)ScsiDeviceIO: 2322: Cmd(0x41240103ef00) 0x12, CmdSN 0x77769 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:32:32.020Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x41240147e800) 0x12, CmdSN 0x77771 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:37:32.038Z cpu9:9361716)ScsiDeviceIO: 2322: Cmd(0x412401be65c0) 0x12, CmdSN 0x77793 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:42:32.053Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x412400f24bc0) 0x12, CmdSN 0x7779b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:47:32.071Z cpu7:4268)ScsiDeviceIO: 2322: Cmd(0x4124011c0b80) 0x12, CmdSN 0x777a3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:52:32.090Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x412400d8ea80) 0x12, CmdSN 0x777ab from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:57:32.106Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x4124003da140) 0x12, CmdSN 0x777b3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:02:32.123Z cpu19:4115)ScsiDeviceIO: 2322: Cmd(0x4124403ae800) 0x12, CmdSN 0x777c5 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:07:32.140Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x412400f63680) 0x12, CmdSN 0x777e7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:12:32.158Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x41240038e840) 0x12, CmdSN 0x777ef from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:17:32.176Z cpu8:9363415)ScsiDeviceIO: 2322: Cmd(0x412401596cc0) 0x12, CmdSN 0x777f7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:22:32.191Z cpu10:4276)ScsiDeviceIO: 2322: Cmd(0x412400e5eec0) 0x12, CmdSN 0x777ff from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:27:32.211Z cpu4:4149)ScsiDeviceIO: 2322: Cmd(0x4124003c9480) 0x12, CmdSN 0x77807 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:32:32.225Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x4124415ad9c0) 0x12, CmdSN 0x7780f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:37:32.243Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412401591cc0) 0x12, CmdSN 0x77831 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:42:32.260Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x412401a94440) 0x12, CmdSN 0x77839 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:47:32.279Z cpu9:9362807)ScsiDeviceIO: 2322: Cmd(0x4124003e3040) 0x12, CmdSN 0x77841 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:52:32.296Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124010851c0) 0x12, CmdSN 0x77849 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:57:32.313Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x4124003e4340) 0x12, CmdSN 0x77851 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:02:32.329Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400ef19c0) 0x12, CmdSN 0x77863 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:07:32.345Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x4124015911c0) 0x12, CmdSN 0x77885 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:12:32.362Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x4124003b8980) 0x12, CmdSN 0x7788d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:17:32.379Z cpu4:9369570)ScsiDeviceIO: 2322: Cmd(0x4124003d6640) 0x12, CmdSN 0x77895 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:22:32.395Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124016e69c0) 0x12, CmdSN 0x7789d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:27:32.412Z cpu15:4111)ScsiDeviceIO: 2322: Cmd(0x4124419a9100) 0x12, CmdSN 0x778a5 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:32:32.430Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124003c3480) 0x12, CmdSN 0x778ad from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:37:32.448Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x4124003e8e00) 0x12, CmdSN 0x778cf from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:42:32.463Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x4124013190c0) 0x12, CmdSN 0x778d7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:47:32.479Z cpu2:9368253)ScsiDeviceIO: 2322: Cmd(0x412400f61b80) 0x12, CmdSN 0x778df from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:52:32.496Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412400f23fc0) 0x12, CmdSN 0x778e7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:57:32.512Z cpu3:9371113)ScsiDeviceIO: 2322: Cmd(0x4124010836c0) 0x12, CmdSN 0x778ef from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:02:32.529Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x412401314200) 0x12, CmdSN 0x77901 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:07:32.545Z cpu7:4778)ScsiDeviceIO: 2322: Cmd(0x412400e92c00) 0x12, CmdSN 0x77923 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:12:32.562Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412400392f00) 0x12, CmdSN 0x7792b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:17:32.577Z cpu9:4765)ScsiDeviceIO: 2322: Cmd(0x412400f7a500) 0x12, CmdSN 0x77933 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:22:32.594Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x412440391580) 0x12, CmdSN 0x7793b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:27:32.610Z cpu1:9365600)ScsiDeviceIO: 2322: Cmd(0x412401be11c0) 0x12, CmdSN 0x77943 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:32:32.628Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x412400ef38c0) 0x12, CmdSN 0x7794b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:37:32.642Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400e5bac0) 0x12, CmdSN 0x7796d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:42:32.658Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x4124003dd940) 0x12, CmdSN 0x77975 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:47:32.674Z cpu2:9372885)ScsiDeviceIO: 2322: Cmd(0x412400f7e100) 0x12, CmdSN 0x7797d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:52:32.689Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412401314b00) 0x12, CmdSN 0x77985 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:57:32.708Z cpu0:9375149)ScsiDeviceIO: 2322: Cmd(0x4124016e4cc0) 0x12, CmdSN 0x7798d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:02:32.722Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412400ecb100) 0x12, CmdSN 0x7799f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:07:32.738Z cpu11:4289)ScsiDeviceIO: 2322: Cmd(0x412400f44000) 0x12, CmdSN 0x779c1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:12:32.755Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x4124403b8fc0) 0x12, CmdSN 0x779c9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:17:32.772Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x412400391a00) 0x12, CmdSN 0x779d1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:22:32.788Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x412401ad3380) 0x12, CmdSN 0x779d9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:27:32.802Z cpu11:4268)ScsiDeviceIO: 2322: Cmd(0x4124003aaf40) 0x12, CmdSN 0x779e1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:32:32.819Z cpu20:3107733)ScsiDeviceIO: 2322: Cmd(0x412441718cc0) 0x12, CmdSN 0x779e9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:37:32.835Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x412400d8bb80) 0x12, CmdSN 0x77a0b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:42:32.851Z cpu7:4276)ScsiDeviceIO: 2322: Cmd(0x4124010a9400) 0x12, CmdSN 0x77a13 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:44:36.256Z cpu0:9378615)WARNING: UserObj: 675: Failed to crossdup fd 8, /dev/ptyp0 type CHAR: Busy
2016-02-09T15:44:36.256Z cpu0:9378615)WARNING: UserObj: 675: Failed to crossdup fd 10, /dev/ptyp0 type CHAR: Busy
2016-02-09T15:44:36.256Z cpu13:9378618)WARNING: UserLinux: 1340: unsupported: (void)
2016-02-09T15:47:32.866Z cpu0:9377065)ScsiDeviceIO: 2322: Cmd(0x4124003d5240) 0x12, CmdSN 0x77a2a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:52:32.883Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412401a94640) 0x12, CmdSN 0x77a32 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:57:32.898Z cpu1:9059505)ScsiDeviceIO: 2322: Cmd(0x4124003cbb80) 0x12, CmdSN 0x77a3a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:02:32.913Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x4124003d8240) 0x12, CmdSN 0x77a4c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:07:32.931Z cpu4:4778)ScsiDeviceIO: 2322: Cmd(0x41240143d040) 0x12, CmdSN 0x77a6e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:12:32.944Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x4124016e1fc0) 0x12, CmdSN 0x77a76 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:17:32.963Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x4124403d49c0) 0x12, CmdSN 0x77a7e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:22:32.977Z cpu11:8747281)ScsiDeviceIO: 2322: Cmd(0x4124003c9f80) 0x12, CmdSN 0x77a86 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:27:32.992Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x412441a4da00) 0x12, CmdSN 0x77a8e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:32:33.008Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x412401594bc0) 0x12, CmdSN 0x77a96 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:37:33.026Z cpu17:8755483)ScsiDeviceIO: 2322: Cmd(0x4124411417c0) 0x12, CmdSN 0x77ab8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:42:33.039Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x412440dd4440) 0x12, CmdSN 0x77ac0 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:47:33.056Z cpu9:4268)ScsiDeviceIO: 2322: Cmd(0x412401b99440) 0x12, CmdSN 0x77ac8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:52:33.071Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400f19340) 0x12, CmdSN 0x77ad0 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:57:33.087Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003b0e80) 0x12, CmdSN 0x77ad8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:02:33.103Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x41240038df40) 0x12, CmdSN 0x77aea from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:07:33.121Z cpu23:4268)ScsiDeviceIO: 2322: Cmd(0x4124403794c0) 0x12, CmdSN 0x77b0c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:12:33.136Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400f66a80) 0x12, CmdSN 0x77b14 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:17:33.152Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412401ad4480) 0x12, CmdSN 0x77b1c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:22:33.167Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003e2140) 0x12, CmdSN 0x77b24 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:27:33.182Z cpu14:4110)ScsiDeviceIO: 2322: Cmd(0x4124403d69c0) 0x12, CmdSN 0x77b2c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:32:33.199Z cpu18:3107735)ScsiDeviceIO: 2322: Cmd(0x41244038d780) 0x12, CmdSN 0x77b34 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:37:33.214Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x4124003e2840) 0x12, CmdSN 0x77b56 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:42:33.231Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400f78c00) 0x12, CmdSN 0x77b5e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:47:33.248Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400f1dc40) 0x12, CmdSN 0x77b66 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:52:33.263Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124014acc80) 0x12, CmdSN 0x77b6e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:57:33.280Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400d8c680) 0x12, CmdSN 0x77b76 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T18:02:33.294Z cpu4:8751376)ScsiDeviceIO: 2322: Cmd(0x41240039d740) 0x12, CmdSN 0x77b88 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T18:07:33.310Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x4124014cab00) 0x12, CmdSN 0x77baa from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T18:12:33.326Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x41240172df40) 0x12, CmdSN 0x77bb2 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T18:17:33.344Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x4124003dd440) 0x12, CmdSN 0x77bba from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T18:22:33.359Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x4124003a1000) 0x12, CmdSN 0x77bc2 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T18:27:33.375Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x41240
2016-02-09T00:07:29.754Z cpu12:4108)ScsiDeviceIO: 2322: Cmd(0x4124414d3600) 0x12, CmdSN 0x7706c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:12:29.772Z cpu19:4115)ScsiDeviceIO: 2322: Cmd(0x4124403b9ec0) 0x12, CmdSN 0x77074 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:17:29.789Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412400e5a7c0) 0x12, CmdSN 0x7707c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:22:29.805Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412441c91540) 0x12, CmdSN 0x77084 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:27:29.821Z cpu8:8755471)ScsiDeviceIO: 2322: Cmd(0x412401bec600) 0x12, CmdSN 0x7708c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:32:24.467Z cpu2:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634807327.
2016-02-09T00:32:24.467Z cpu4:8755471)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x4124010817c0, 4120) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba3:C0:T2:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-09T00:32:24.467Z cpu4:8755471)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-09T00:32:24.467Z cpu4:8755471)ScsiDeviceIO: 2309: Cmd(0x4124010817c0) 0x2a, CmdSN 0x4fc119 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-09T00:32:24.468Z cpu2:4290)<6>qla2xxx 0000:04:00.0: scsi(7:2:2): Abort command succeeded -- 1 634807326.
2016-02-09T00:32:24.468Z cpu4:4755)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412400f260c0, 4120) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba2:C0:T2:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-09T00:32:24.468Z cpu4:4755)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-09T00:32:24.468Z cpu4:4755)ScsiDeviceIO: 2309: Cmd(0x412400f260c0) 0x2a, CmdSN 0x4191fc from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-09T00:32:29.840Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400e96900) 0x12, CmdSN 0x77094 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:37:29.857Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412400f62680) 0x12, CmdSN 0x770b6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:42:29.872Z cpu17:9326144)ScsiDeviceIO: 2322: Cmd(0x412440994600) 0x12, CmdSN 0x770be from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:47:29.889Z cpu1:9331862)ScsiDeviceIO: 2322: Cmd(0x412401693a00) 0x12, CmdSN 0x770c6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:52:29.906Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x412401a91340) 0x12, CmdSN 0x770ce from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T00:57:29.922Z cpu16:9325015)ScsiDeviceIO: 2322: Cmd(0x41244039a0c0) 0x12, CmdSN 0x770d6 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:01:28.552Z cpu2:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907599.
2016-02-09T01:01:28.552Z cpu2:4290)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412400f1a240, 4120) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba3:C0:T2:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-09T01:01:28.552Z cpu2:4290)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-09T01:01:28.552Z cpu2:4290)ScsiDeviceIO: 2309: Cmd(0x412400f1a240) 0x2a, CmdSN 0x4fc426 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-09T01:01:28.552Z cpu2:4290)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907598.
2016-02-09T01:01:28.552Z cpu2:4290)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412400f64780, 4120) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T3:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-09T01:01:28.552Z cpu2:4290)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-09T01:01:28.552Z cpu2:4290)ScsiDeviceIO: 2309: Cmd(0x412400f64780) 0x2a, CmdSN 0x419482 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-09T01:01:30.551Z cpu15:4290)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907629.
2016-02-09T01:01:30.551Z cpu9:8747281)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346a6431745431" state in doubt; requested fast path state update...
2016-02-09T01:01:30.551Z cpu9:8747281)ScsiDeviceIO: 2309: Cmd(0x412400f64780) 0x2a, CmdSN 0x4fc42a from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-09T01:01:30.551Z cpu15:4290)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907630.
2016-02-09T01:01:30.551Z cpu9:8747281)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60a980004335432d68346b7239454f67" state in doubt; requested fast path state update...
2016-02-09T01:01:30.551Z cpu9:8747281)ScsiDeviceIO: 2309: Cmd(0x412400f1a240) 0x2a, CmdSN 0x419483 from world 4120 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2016-02-09T01:01:30.623Z cpu5:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907595.
2016-02-09T01:01:30.624Z cpu12:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907601.
2016-02-09T01:01:30.624Z cpu5:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907596.
2016-02-09T01:01:30.624Z cpu14:4110)ScsiDeviceIO: 2291: Cmd(0x41244114e3c0) 0x2a, CmdSN 0x80000012 from world 8751385 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.624Z cpu14:4110)ScsiDeviceIO: 2291: Cmd(0x41244183a240) 0x2a, CmdSN 0x8000004d from world 8751385 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.624Z cpu12:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907602.
2016-02-09T01:01:30.625Z cpu12:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907605.
2016-02-09T01:01:30.625Z cpu6:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907597.
2016-02-09T01:01:30.626Z cpu12:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907620.
2016-02-09T01:01:30.626Z cpu6:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907621.
2016-02-09T01:01:30.626Z cpu14:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907623.
2016-02-09T01:01:30.627Z cpu14:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907624.
2016-02-09T01:01:30.627Z cpu6:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907628.
2016-02-09T01:01:30.627Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x4124415a9bc0) 0x2a, CmdSN 0x8000005a from world 9206876 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.627Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x412441bec600) 0x2a, CmdSN 0x80000059 from world 9206876 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.627Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x4124403cd6c0) 0x2a, CmdSN 0x8000004b from world 9206876 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.627Z cpu14:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907625.
2016-02-09T01:01:30.628Z cpu6:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907600.
2016-02-09T01:01:30.628Z cpu10:4106)ScsiDeviceIO: 2291: Cmd(0x412401439d40) 0x2a, CmdSN 0x80000014 from world 8755469 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.628Z cpu14:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907626.
2016-02-09T01:01:30.629Z cpu10:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907603.
2016-02-09T01:01:30.629Z cpu14:4131)<6>qla2xxx 0000:05:00.0: scsi(8:3:2): Abort command succeeded -- 1 634907627.
2016-02-09T01:01:30.629Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x412441041440) 0x2a, CmdSN 0x8000004d from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.629Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x41244039b1c0) 0x2a, CmdSN 0x80000063 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.629Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x412440376580) 0x2a, CmdSN 0x8000007b from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.629Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x412441c9e600) 0x2a, CmdSN 0x80000058 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.629Z cpu13:4109)ScsiDeviceIO: 2291: Cmd(0x412440e12240) 0x2a, CmdSN 0x80000060 from world 3107733 to dev "naa.60a980004335432d68346b7239454f67" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.629Z cpu13:4109)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412441c9de00, 3107733) to dev "naa.60a980004335432d68346b7239454f67" on path "vmhba3:C0:T3:L2" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-09T01:01:30.629Z cpu10:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907604.
2016-02-09T01:01:30.630Z cpu10:4133)<6>qla2xxx 0000:05:00.0: scsi(8:2:3): Abort command succeeded -- 1 634907622.
2016-02-09T01:01:30.630Z cpu13:5220)ScsiDeviceIO: 2291: Cmd(0x41244114b7c0) 0x2a, CmdSN 0x4fc427 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.630Z cpu13:5220)ScsiDeviceIO: 2291: Cmd(0x4124403c22c0) 0x2a, CmdSN 0x4fc428 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.630Z cpu10:4133)NMP: nmp_ThrottleLogForDevice:2318: Cmd 0x2a (0x412401ae87c0, 4120) to dev "naa.60a980004335432d68346a6431745431" on path "vmhba3:C0:T2:L3" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2016-02-09T01:01:30.630Z cpu10:4133)ScsiDeviceIO: 2291: Cmd(0x412401ae87c0) 0x2a, CmdSN 0x4fc429 from world 4120 to dev "naa.60a980004335432d68346a6431745431" failed H:0x8 D:0x0 P:0x0
2016-02-09T01:01:30.968Z cpu10:4133)HBX: 231: Reclaimed heartbeat for volume 4f99169f-12a01f11-cabc-d067e5eee823 (xendatastore2): [Timeout] [HB state abcdef02 offset 3727360 gen 121 stampUS 10962240765522 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <F$
2016-02-09T01:01:30.968Z cpu10:4133)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore2' (held for 236 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-09T01:01:30.994Z cpu18:6178)HBX: 2313: Waiting for timed out [HB state abcdef02 offset 3727360 gen 129 stampUS 10962240421216 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <FB 402000> drv 14.54] on vol 'xendatastore1'
2016-02-09T01:01:31.007Z cpu12:4131)HBX: 231: Reclaimed heartbeat for volume 4feaec43-24977e2a-c803-d067e5eeee6e (xendatastore1): [Timeout] [HB state abcdef02 offset 3727360 gen 129 stampUS 10962240805104 uuid 5611f2ff-25a21a56-8600-d067e5eeee6e jrnl <F$
2016-02-09T01:01:31.007Z cpu12:4131)FS3Misc: 1440: Long VMFS3 rsv time on 'xendatastore1' (held for 276 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors
2016-02-09T01:02:29.940Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x412440e10640) 0x12, CmdSN 0x770ec from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:07:29.957Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400381100) 0x12, CmdSN 0x7710e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:12:29.976Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x412440db0240) 0x12, CmdSN 0x77116 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:17:29.993Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003bd380) 0x12, CmdSN 0x7711e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:22:30.009Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124003e3c40) 0x12, CmdSN 0x77126 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:27:30.026Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x41240037bfc0) 0x12, CmdSN 0x7712e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:32:30.045Z cpu14:4110)ScsiDeviceIO: 2322: Cmd(0x4124415ac0c0) 0x12, CmdSN 0x77136 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:37:30.062Z cpu20:9323049)ScsiDeviceIO: 2322: Cmd(0x412441a51ac0) 0x12, CmdSN 0x77158 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:42:30.077Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x412440cea900) 0x12, CmdSN 0x77160 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:47:30.094Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x412400f18640) 0x12, CmdSN 0x77168 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:52:30.113Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412401095ac0) 0x12, CmdSN 0x77170 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T01:57:30.130Z cpu22:9335606)ScsiDeviceIO: 2322: Cmd(0x4124413d8100) 0x12, CmdSN 0x77178 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:01:29.618Z cpu2:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:32.377Z cpu22:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:01:33.743Z cpu11:8755476)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:33.778Z cpu11:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:33.782Z cpu11:8755476)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:35.692Z cpu18:9215075)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:01:35.725Z cpu18:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:01:35.729Z cpu18:9215075)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:01:35.838Z cpu6:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:35.843Z cpu6:8755476)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:35.893Z cpu6:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:35.897Z cpu6:8755476)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:36.035Z cpu6:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:36.041Z cpu6:8755476)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:36.333Z cpu0:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:36.395Z cpu6:8755476)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:37.785Z cpu18:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:01:37.791Z cpu18:9215075)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:01:37.840Z cpu18:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:01:37.844Z cpu18:9215075)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:01:37.977Z cpu18:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:01:37.983Z cpu18:9215075)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:01:38.252Z cpu23:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:01:38.319Z cpu12:9215075)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:01:48.796Z cpu9:8751383)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:48.796Z cpu9:8751383)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:52.319Z cpu4:8751381)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:52.320Z cpu4:8751381)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:01:52.326Z cpu2:8755476)NetPort: 1427: disabled port 0x1000015
2016-02-09T02:01:52.339Z cpu9:8751383)NetPort: 1240: enabled port 0x1000015 with mac 00:50:56:9f:65:90
2016-02-09T02:02:02.657Z cpu22:8755491)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:06.845Z cpu14:8755488)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:06.899Z cpu14:8755488)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:06.903Z cpu14:8755488)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:08.931Z cpu12:8755488)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:08.936Z cpu12:8755488)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:08.986Z cpu12:8755488)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:08.991Z cpu16:8755488)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:09.124Z cpu22:8755488)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:09.129Z cpu16:8755488)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:09.398Z cpu16:8755488)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:09.465Z cpu20:8755488)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:15.891Z cpu16:9210982)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:02:15.892Z cpu16:9210982)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:02:19.338Z cpu13:9210980)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:02:19.339Z cpu13:9210980)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:02:19.341Z cpu16:9215075)NetPort: 1427: disabled port 0x1000017
2016-02-09T02:02:19.352Z cpu14:9210982)NetPort: 1240: enabled port 0x1000017 with mac 00:50:56:9f:09:a4
2016-02-09T02:02:30.145Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124011c1c80) 0x12, CmdSN 0x7718f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:02:49.475Z cpu21:8755491)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:49.476Z cpu21:8755491)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:53.045Z cpu16:8755489)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:53.046Z cpu16:8755489)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:02:53.049Z cpu22:8755488)NetPort: 1427: disabled port 0x1000016
2016-02-09T02:02:53.063Z cpu12:8755491)NetPort: 1240: enabled port 0x1000016 with mac 00:50:56:9f:65:8e
2016-02-09T02:07:30.162Z cpu6:9331790)ScsiDeviceIO: 2322: Cmd(0x4124003c3480) 0x12, CmdSN 0x771bb from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:12:30.180Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x412441c93340) 0x12, CmdSN 0x771c3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:17:30.198Z cpu21:4769)ScsiDeviceIO: 2322: Cmd(0x412440dd0a40) 0x12, CmdSN 0x771cb from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:22:30.213Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400e5e4c0) 0x12, CmdSN 0x771d3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:27:30.230Z cpu22:9331790)ScsiDeviceIO: 2322: Cmd(0x4124403b2640) 0x12, CmdSN 0x771db from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:32:30.247Z cpu13:4276)ScsiDeviceIO: 2322: Cmd(0x412440398bc0) 0x12, CmdSN 0x771e3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:37:30.263Z cpu12:4108)ScsiDeviceIO: 2322: Cmd(0x412440d7a4c0) 0x12, CmdSN 0x77205 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:42:30.281Z cpu10:9333625)ScsiDeviceIO: 2322: Cmd(0x4124003a0600) 0x12, CmdSN 0x7720d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:47:30.298Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x41240131c6c0) 0x12, CmdSN 0x77215 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:52:30.312Z cpu4:4100)ScsiDeviceIO: 2322: Cmd(0x4124011c9c00) 0x12, CmdSN 0x7721d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T02:57:30.331Z cpu2:9329988)ScsiDeviceIO: 2322: Cmd(0x412400385a00) 0x12, CmdSN 0x77225 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:02:30.347Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400f44400) 0x12, CmdSN 0x77237 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:07:30.363Z cpu14:9339821)ScsiDeviceIO: 2322: Cmd(0x412440dd4840) 0x12, CmdSN 0x77259 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:12:30.381Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412400f23fc0) 0x12, CmdSN 0x77261 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:17:30.398Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x412440c7e800) 0x12, CmdSN 0x77269 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:22:30.414Z cpu20:4276)ScsiDeviceIO: 2322: Cmd(0x412441609280) 0x12, CmdSN 0x77271 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:27:30.429Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400f1ee40) 0x12, CmdSN 0x77279 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:32:30.445Z cpu4:4100)ScsiDeviceIO: 2322: Cmd(0x412401b98e40) 0x12, CmdSN 0x77281 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:37:30.463Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x4124403dd4c0) 0x12, CmdSN 0x772a3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:42:30.479Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x4124412a4940) 0x12, CmdSN 0x772ab from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:47:30.494Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x41244160e680) 0x12, CmdSN 0x772b3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:52:30.511Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x41244038bd80) 0x12, CmdSN 0x772bb from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T03:57:30.528Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x41240131a2c0) 0x12, CmdSN 0x772c3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:02:30.542Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400e7c540) 0x12, CmdSN 0x772d5 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:07:30.559Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412400385700) 0x12, CmdSN 0x772f7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:12:30.573Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412401ae4ac0) 0x12, CmdSN 0x772ff from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:17:30.588Z cpu4:4100)ScsiDeviceIO: 2322: Cmd(0x412400f79800) 0x12, CmdSN 0x77307 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:22:30.604Z cpu14:4110)ScsiDeviceIO: 2322: Cmd(0x412441394540) 0x12, CmdSN 0x7730f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:27:30.620Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x412440e5dbc0) 0x12, CmdSN 0x77317 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:32:30.637Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x4124414d4c00) 0x12, CmdSN 0x7731f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:37:30.652Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x4124403d21c0) 0x12, CmdSN 0x77341 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:42:30.667Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412401b9de40) 0x12, CmdSN 0x77349 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:47:30.682Z cpu19:4115)ScsiDeviceIO: 2322: Cmd(0x412441c9c800) 0x12, CmdSN 0x77351 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:52:30.700Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x412441041f40) 0x12, CmdSN 0x77359 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T04:57:30.716Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x412441a4b700) 0x12, CmdSN 0x77361 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:02:30.730Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x4124003ed700) 0x12, CmdSN 0x77373 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:07:30.745Z cpu5:9336876)ScsiDeviceIO: 2322: Cmd(0x412400392700) 0x12, CmdSN 0x77395 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:12:30.762Z cpu4:9336876)ScsiDeviceIO: 2322: Cmd(0x412400ec8b00) 0x12, CmdSN 0x7739d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:17:30.778Z cpu19:4115)ScsiDeviceIO: 2322: Cmd(0x41244160bf80) 0x12, CmdSN 0x773a5 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:22:30.794Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x4124411494c0) 0x12, CmdSN 0x773ad from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:27:30.808Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x412400389f40) 0x12, CmdSN 0x773b5 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:32:30.825Z cpu14:9341242)ScsiDeviceIO: 2322: Cmd(0x41244180e600) 0x12, CmdSN 0x773bd from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:37:30.839Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400f02b80) 0x12, CmdSN 0x773df from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:42:30.854Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x4124010821c0) 0x12, CmdSN 0x773e7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:47:30.870Z cpu20:9343468)ScsiDeviceIO: 2322: Cmd(0x4124403bb3c0) 0x12, CmdSN 0x773ef from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:52:30.886Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412400390b00) 0x12, CmdSN 0x773f7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T05:57:30.901Z cpu2:9348375)ScsiDeviceIO: 2322: Cmd(0x412401a96540) 0x12, CmdSN 0x773ff from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:02:30.918Z cpu18:8751389)ScsiDeviceIO: 2322: Cmd(0x4124413d9f00) 0x12, CmdSN 0x77411 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:07:30.933Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x4124003bb980) 0x12, CmdSN 0x77433 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:12:30.948Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400382d00) 0x12, CmdSN 0x7743b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:17:30.962Z cpu22:9215071)ScsiDeviceIO: 2322: Cmd(0x4124403bcec0) 0x12, CmdSN 0x77443 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:22:30.977Z cpu21:9215071)ScsiDeviceIO: 2322: Cmd(0x412440c7c400) 0x12, CmdSN 0x7744b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:27:30.992Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x412400390800) 0x12, CmdSN 0x77453 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:32:31.008Z cpu10:4149)ScsiDeviceIO: 2322: Cmd(0x41240147c800) 0x12, CmdSN 0x7745b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:37:31.024Z cpu5:9349748)ScsiDeviceIO: 2322: Cmd(0x412401574200) 0x12, CmdSN 0x7747d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:42:31.038Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x41244183c340) 0x12, CmdSN 0x77485 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:47:31.053Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412401596bc0) 0x12, CmdSN 0x7748d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:52:31.069Z cpu3:9343467)ScsiDeviceIO: 2322: Cmd(0x4124003d0140) 0x12, CmdSN 0x77495 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T06:57:31.085Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412401486940) 0x12, CmdSN 0x7749d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:02:31.099Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x412400f44100) 0x12, CmdSN 0x774af from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:07:31.114Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x4124419ae800) 0x12, CmdSN 0x774d1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:12:31.130Z cpu18:4114)ScsiDeviceIO: 2322: Cmd(0x412440995000) 0x12, CmdSN 0x774d9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:17:31.146Z cpu10:4148)ScsiDeviceIO: 2322: Cmd(0x412401439e40) 0x12, CmdSN 0x774e1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:22:31.163Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x412440db5440) 0x12, CmdSN 0x774e9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:27:31.179Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x4124014ca300) 0x12, CmdSN 0x774f1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:32:31.196Z cpu15:9347988)ScsiDeviceIO: 2322: Cmd(0x4124412821c0) 0x12, CmdSN 0x774f9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:37:31.212Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x41240147a300) 0x12, CmdSN 0x7751b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:42:31.230Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x412441c12b40) 0x12, CmdSN 0x77523 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:47:31.247Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x4124011ce300) 0x12, CmdSN 0x7752b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:52:31.262Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412400f02b80) 0x12, CmdSN 0x77533 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T07:57:31.280Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x4124419ae300) 0x12, CmdSN 0x7753b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:02:31.296Z cpu14:4276)ScsiDeviceIO: 2322: Cmd(0x412441be5c00) 0x12, CmdSN 0x7754d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:07:31.312Z cpu4:9353079)ScsiDeviceIO: 2322: Cmd(0x4124003b1880) 0x12, CmdSN 0x7756f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:12:31.330Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412401aea4c0) 0x12, CmdSN 0x77577 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:17:31.347Z cpu11:9355067)ScsiDeviceIO: 2322: Cmd(0x4124003dab40) 0x12, CmdSN 0x7757f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:22:31.364Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400390100) 0x12, CmdSN 0x77587 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:27:31.380Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412401ad0f80) 0x12, CmdSN 0x7758f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:32:31.397Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x412440cecc00) 0x12, CmdSN 0x77597 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:37:31.414Z cpu12:4108)ScsiDeviceIO: 2322: Cmd(0x4124403999c0) 0x12, CmdSN 0x775b9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:42:31.432Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x412400f78800) 0x12, CmdSN 0x775c1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:47:31.451Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412400eca900) 0x12, CmdSN 0x775c9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:52:31.467Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x4124015956c0) 0x12, CmdSN 0x775d1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T08:57:31.485Z cpu3:9357325)ScsiDeviceIO: 2322: Cmd(0x412401ae92c0) 0x12, CmdSN 0x775d9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:02:31.503Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x41240102d780) 0x12, CmdSN 0x775eb from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:07:31.519Z cpu9:9350018)ScsiDeviceIO: 2322: Cmd(0x4124014bed80) 0x12, CmdSN 0x7760d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:12:31.538Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x41244139c300) 0x12, CmdSN 0x77615 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:17:31.554Z cpu3:9358667)ScsiDeviceIO: 2322: Cmd(0x412401696b00) 0x12, CmdSN 0x7761d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:22:31.572Z cpu19:4115)ScsiDeviceIO: 2322: Cmd(0x4124403793c0) 0x12, CmdSN 0x77625 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:27:31.591Z cpu8:4149)ScsiDeviceIO: 2322: Cmd(0x4124011c6c80) 0x12, CmdSN 0x7762d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:32:31.608Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x412400378dc0) 0x12, CmdSN 0x77635 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:37:31.626Z cpu2:9359302)ScsiDeviceIO: 2322: Cmd(0x412401ba3f00) 0x12, CmdSN 0x77657 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:42:31.640Z cpu4:8747281)ScsiDeviceIO: 2322: Cmd(0x412400f1d040) 0x12, CmdSN 0x7765f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:47:31.658Z cpu5:4774)ScsiDeviceIO: 2322: Cmd(0x412400e5cec0) 0x12, CmdSN 0x77667 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:52:31.677Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x41240148d980) 0x12, CmdSN 0x7766f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T09:57:31.694Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400f40b00) 0x12, CmdSN 0x77677 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:02:31.710Z cpu20:4116)ScsiDeviceIO: 2322: Cmd(0x4124415acac0) 0x12, CmdSN 0x77689 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:07:31.727Z cpu22:4268)ScsiDeviceIO: 2322: Cmd(0x412440384100) 0x12, CmdSN 0x776ab from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:12:31.745Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x41240103af00) 0x12, CmdSN 0x776b3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:17:31.761Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x412400f00580) 0x12, CmdSN 0x776bb from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:22:31.780Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x41240113d980) 0x12, CmdSN 0x776c3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:27:31.796Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x4124010914c0) 0x12, CmdSN 0x776cb from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:32:31.813Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124003b9880) 0x12, CmdSN 0x776d3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:37:31.830Z cpu13:9362236)ScsiDeviceIO: 2322: Cmd(0x4124403cb2c0) 0x12, CmdSN 0x776f5 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:42:31.848Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x41240039cd40) 0x12, CmdSN 0x776fd from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:47:31.864Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x4124014ba580) 0x12, CmdSN 0x77705 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:52:31.882Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x412441c16240) 0x12, CmdSN 0x7770d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T10:57:31.899Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x412401a93540) 0x12, CmdSN 0x77715 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:02:31.918Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x412401484440) 0x12, CmdSN 0x77727 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:07:31.935Z cpu3:4268)ScsiDeviceIO: 2322: Cmd(0x4124003e6b40) 0x12, CmdSN 0x77749 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:12:31.952Z cpu4:4100)ScsiDeviceIO: 2322: Cmd(0x4124015908c0) 0x12, CmdSN 0x77751 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:17:31.968Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412400383000) 0x12, CmdSN 0x77759 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:22:31.986Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x412400ef2ac0) 0x12, CmdSN 0x77761 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:27:32.002Z cpu11:9356117)ScsiDeviceIO: 2322: Cmd(0x41240103ef00) 0x12, CmdSN 0x77769 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:32:32.020Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x41240147e800) 0x12, CmdSN 0x77771 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:37:32.038Z cpu9:9361716)ScsiDeviceIO: 2322: Cmd(0x412401be65c0) 0x12, CmdSN 0x77793 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:42:32.053Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x412400f24bc0) 0x12, CmdSN 0x7779b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:47:32.071Z cpu7:4268)ScsiDeviceIO: 2322: Cmd(0x4124011c0b80) 0x12, CmdSN 0x777a3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:52:32.090Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x412400d8ea80) 0x12, CmdSN 0x777ab from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T11:57:32.106Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x4124003da140) 0x12, CmdSN 0x777b3 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:02:32.123Z cpu19:4115)ScsiDeviceIO: 2322: Cmd(0x4124403ae800) 0x12, CmdSN 0x777c5 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:07:32.140Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x412400f63680) 0x12, CmdSN 0x777e7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:12:32.158Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x41240038e840) 0x12, CmdSN 0x777ef from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:17:32.176Z cpu8:9363415)ScsiDeviceIO: 2322: Cmd(0x412401596cc0) 0x12, CmdSN 0x777f7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:22:32.191Z cpu10:4276)ScsiDeviceIO: 2322: Cmd(0x412400e5eec0) 0x12, CmdSN 0x777ff from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:27:32.211Z cpu4:4149)ScsiDeviceIO: 2322: Cmd(0x4124003c9480) 0x12, CmdSN 0x77807 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:32:32.225Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x4124415ad9c0) 0x12, CmdSN 0x7780f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:37:32.243Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412401591cc0) 0x12, CmdSN 0x77831 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:42:32.260Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x412401a94440) 0x12, CmdSN 0x77839 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:47:32.279Z cpu9:9362807)ScsiDeviceIO: 2322: Cmd(0x4124003e3040) 0x12, CmdSN 0x77841 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:52:32.296Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124010851c0) 0x12, CmdSN 0x77849 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T12:57:32.313Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x4124003e4340) 0x12, CmdSN 0x77851 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:02:32.329Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400ef19c0) 0x12, CmdSN 0x77863 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:07:32.345Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x4124015911c0) 0x12, CmdSN 0x77885 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:12:32.362Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x4124003b8980) 0x12, CmdSN 0x7788d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:17:32.379Z cpu4:9369570)ScsiDeviceIO: 2322: Cmd(0x4124003d6640) 0x12, CmdSN 0x77895 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:22:32.395Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124016e69c0) 0x12, CmdSN 0x7789d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:27:32.412Z cpu15:4111)ScsiDeviceIO: 2322: Cmd(0x4124419a9100) 0x12, CmdSN 0x778a5 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:32:32.430Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124003c3480) 0x12, CmdSN 0x778ad from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:37:32.448Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x4124003e8e00) 0x12, CmdSN 0x778cf from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:42:32.463Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x4124013190c0) 0x12, CmdSN 0x778d7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:47:32.479Z cpu2:9368253)ScsiDeviceIO: 2322: Cmd(0x412400f61b80) 0x12, CmdSN 0x778df from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:52:32.496Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412400f23fc0) 0x12, CmdSN 0x778e7 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T13:57:32.512Z cpu3:9371113)ScsiDeviceIO: 2322: Cmd(0x4124010836c0) 0x12, CmdSN 0x778ef from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:02:32.529Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x412401314200) 0x12, CmdSN 0x77901 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:07:32.545Z cpu7:4778)ScsiDeviceIO: 2322: Cmd(0x412400e92c00) 0x12, CmdSN 0x77923 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:12:32.562Z cpu3:4099)ScsiDeviceIO: 2322: Cmd(0x412400392f00) 0x12, CmdSN 0x7792b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:17:32.577Z cpu9:4765)ScsiDeviceIO: 2322: Cmd(0x412400f7a500) 0x12, CmdSN 0x77933 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:22:32.594Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x412440391580) 0x12, CmdSN 0x7793b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:27:32.610Z cpu1:9365600)ScsiDeviceIO: 2322: Cmd(0x412401be11c0) 0x12, CmdSN 0x77943 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:32:32.628Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x412400ef38c0) 0x12, CmdSN 0x7794b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:37:32.642Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400e5bac0) 0x12, CmdSN 0x7796d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:42:32.658Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x4124003dd940) 0x12, CmdSN 0x77975 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:47:32.674Z cpu2:9372885)ScsiDeviceIO: 2322: Cmd(0x412400f7e100) 0x12, CmdSN 0x7797d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:52:32.689Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412401314b00) 0x12, CmdSN 0x77985 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T14:57:32.708Z cpu0:9375149)ScsiDeviceIO: 2322: Cmd(0x4124016e4cc0) 0x12, CmdSN 0x7798d from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:02:32.722Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x412400ecb100) 0x12, CmdSN 0x7799f from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:07:32.738Z cpu11:4289)ScsiDeviceIO: 2322: Cmd(0x412400f44000) 0x12, CmdSN 0x779c1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:12:32.755Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x4124403b8fc0) 0x12, CmdSN 0x779c9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:17:32.772Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x412400391a00) 0x12, CmdSN 0x779d1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:22:32.788Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x412401ad3380) 0x12, CmdSN 0x779d9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:27:32.802Z cpu11:4268)ScsiDeviceIO: 2322: Cmd(0x4124003aaf40) 0x12, CmdSN 0x779e1 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:32:32.819Z cpu20:3107733)ScsiDeviceIO: 2322: Cmd(0x412441718cc0) 0x12, CmdSN 0x779e9 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:37:32.835Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x412400d8bb80) 0x12, CmdSN 0x77a0b from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:42:32.851Z cpu7:4276)ScsiDeviceIO: 2322: Cmd(0x4124010a9400) 0x12, CmdSN 0x77a13 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:44:36.256Z cpu0:9378615)WARNING: UserObj: 675: Failed to crossdup fd 8, /dev/ptyp0 type CHAR: Busy
2016-02-09T15:44:36.256Z cpu0:9378615)WARNING: UserObj: 675: Failed to crossdup fd 10, /dev/ptyp0 type CHAR: Busy
2016-02-09T15:44:36.256Z cpu13:9378618)WARNING: UserLinux: 1340: unsupported: (void)
2016-02-09T15:47:32.866Z cpu0:9377065)ScsiDeviceIO: 2322: Cmd(0x4124003d5240) 0x12, CmdSN 0x77a2a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:52:32.883Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x412401a94640) 0x12, CmdSN 0x77a32 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T15:57:32.898Z cpu1:9059505)ScsiDeviceIO: 2322: Cmd(0x4124003cbb80) 0x12, CmdSN 0x77a3a from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:02:32.913Z cpu8:4104)ScsiDeviceIO: 2322: Cmd(0x4124003d8240) 0x12, CmdSN 0x77a4c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:07:32.931Z cpu4:4778)ScsiDeviceIO: 2322: Cmd(0x41240143d040) 0x12, CmdSN 0x77a6e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:12:32.944Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x4124016e1fc0) 0x12, CmdSN 0x77a76 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:17:32.963Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x4124403d49c0) 0x12, CmdSN 0x77a7e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:22:32.977Z cpu11:8747281)ScsiDeviceIO: 2322: Cmd(0x4124003c9f80) 0x12, CmdSN 0x77a86 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:27:32.992Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x412441a4da00) 0x12, CmdSN 0x77a8e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:32:33.008Z cpu7:4103)ScsiDeviceIO: 2322: Cmd(0x412401594bc0) 0x12, CmdSN 0x77a96 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:37:33.026Z cpu17:8755483)ScsiDeviceIO: 2322: Cmd(0x4124411417c0) 0x12, CmdSN 0x77ab8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:42:33.039Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x412440dd4440) 0x12, CmdSN 0x77ac0 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:47:33.056Z cpu9:4268)ScsiDeviceIO: 2322: Cmd(0x412401b99440) 0x12, CmdSN 0x77ac8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:52:33.071Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400f19340) 0x12, CmdSN 0x77ad0 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T16:57:33.087Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003b0e80) 0x12, CmdSN 0x77ad8 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:02:33.103Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x41240038df40) 0x12, CmdSN 0x77aea from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:07:33.121Z cpu23:4268)ScsiDeviceIO: 2322: Cmd(0x4124403794c0) 0x12, CmdSN 0x77b0c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:12:33.136Z cpu1:4097)ScsiDeviceIO: 2322: Cmd(0x412400f66a80) 0x12, CmdSN 0x77b14 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:17:33.152Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412401ad4480) 0x12, CmdSN 0x77b1c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:22:33.167Z cpu2:4098)ScsiDeviceIO: 2322: Cmd(0x4124003e2140) 0x12, CmdSN 0x77b24 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:27:33.182Z cpu14:4110)ScsiDeviceIO: 2322: Cmd(0x4124403d69c0) 0x12, CmdSN 0x77b2c from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:32:33.199Z cpu18:3107735)ScsiDeviceIO: 2322: Cmd(0x41244038d780) 0x12, CmdSN 0x77b34 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:37:33.214Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x4124003e2840) 0x12, CmdSN 0x77b56 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:42:33.231Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x412400f78c00) 0x12, CmdSN 0x77b5e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:47:33.248Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400f1dc40) 0x12, CmdSN 0x77b66 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:52:33.263Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x4124014acc80) 0x12, CmdSN 0x77b6e from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T17:57:33.280Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400d8c680) 0x12, CmdSN 0x77b76 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T18:02:33.294Z cpu4:8751376)ScsiDeviceIO: 2322: Cmd(0x41240039d740) 0x12, CmdSN 0x77b88 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T18:07:33.310Z cpu0:4120)ScsiDeviceIO: 2322: Cmd(0x4124014cab00) 0x12, CmdSN 0x77baa from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T18:12:33.326Z cpu9:4105)ScsiDeviceIO: 2322: Cmd(0x41240172df40) 0x12, CmdSN 0x77bb2 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T18:17:33.344Z cpu5:4101)ScsiDeviceIO: 2322: Cmd(0x4124003dd440) 0x12, CmdSN 0x77bba from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T18:22:33.359Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x4124003a1000) 0x12, CmdSN 0x77bc2 from world 0 to dev "naa.600508e000000000d85b294d14b5f10d" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-09T18:27:33.375Z cpu6:4102)ScsiDeviceIO: 2322: Cmd(0x41240
/var/log # esxcli storage core adapter list
HBA Name Driver Link State UID Description
-------- -------- ---------- ------------------------------------ ------------------------------------------------------------------------
vmhba0 ata_piix link-n/a sata.vmhba0 (0:0:31.2) Intel Corporation 2 port SATA IDE Controller (ICH9)
vmhba1 mpt2sas link-n/a sas.5d4ae52071e3c800 (0:3:0.0) LSI Logic / Symbios Logic Dell PERC H200 Integrated
vmhba2 qla2xxx link-up fc.20000024ff07bc38:21000024ff07bc38 (0:4:0.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA
vmhba3 qla2xxx link-up fc.20000024ff07bc90:21000024ff07bc90 (0:5:0.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA
vmhba32 bnx2i unbound iscsi.vmhba32 Broadcom iSCSI Adapter
vmhba33 bnx2i unbound iscsi.vmhba33 Broadcom iSCSI Adapter
vmhba34 bnx2i unbound iscsi.vmhba34 Broadcom iSCSI Adapter
vmhba35 bnx2i unbound iscsi.vmhba35 Broadcom iSCSI Adapter
vmhba36 ata_piix link-n/a sata.vmhba36 (0:0:31.2) Intel Corporation 2 port SATA IDE Controller (ICH9)
/var/log # lspci | grep -i vmhba
000:000:31.2 Mass storage controller: Intel Corporation 2 port SATA IDE Controller (ICH9) [vmhba0]
000:003:00.0 Mass storage controller: LSI Logic / Symbios Logic Dell PERC H200 Integrated [vmhba1]
000:004:00.0 Serial bus controller: QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA [vmhba2]
000:005:00.0 Serial bus controller: QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA [vmhba3]
HBA Name Driver Link State UID Description
-------- -------- ---------- ------------------------------------ ------------------------------------------------------------------------
vmhba0 ata_piix link-n/a sata.vmhba0 (0:0:31.2) Intel Corporation 2 port SATA IDE Controller (ICH9)
vmhba1 mpt2sas link-n/a sas.5d4ae52071e3c800 (0:3:0.0) LSI Logic / Symbios Logic Dell PERC H200 Integrated
vmhba2 qla2xxx link-up fc.20000024ff07bc38:21000024ff07bc38 (0:4:0.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA
vmhba3 qla2xxx link-up fc.20000024ff07bc90:21000024ff07bc90 (0:5:0.0) QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA
vmhba32 bnx2i unbound iscsi.vmhba32 Broadcom iSCSI Adapter
vmhba33 bnx2i unbound iscsi.vmhba33 Broadcom iSCSI Adapter
vmhba34 bnx2i unbound iscsi.vmhba34 Broadcom iSCSI Adapter
vmhba35 bnx2i unbound iscsi.vmhba35 Broadcom iSCSI Adapter
vmhba36 ata_piix link-n/a sata.vmhba36 (0:0:31.2) Intel Corporation 2 port SATA IDE Controller (ICH9)
/var/log # lspci | grep -i vmhba
000:000:31.2 Mass storage controller: Intel Corporation 2 port SATA IDE Controller (ICH9) [vmhba0]
000:003:00.0 Mass storage controller: LSI Logic / Symbios Logic Dell PERC H200 Integrated [vmhba1]
000:004:00.0 Serial bus controller: QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA [vmhba2]
000:005:00.0 Serial bus controller: QLogic Corp ISP2532-based 8Gb Fibre Channel to PCI Express HBA [vmhba3]
-
Dayworker
- King of the Hill
- Beiträge: 13657
- Registriert: 01.10.2008, 12:54
- Wohnort: laut USV-Log am Ende der Welt...
Unser Forum gestattet schon seit Jahren keine Dateianhänge mehr und wir können auch den Upload-Button nicht abschalten. Der Hinweis, daß man seine Anhänge einfach auf einen Freehoster oder eigenen Webspace verlinkt, steht nun auch in diesem Thread.
Da die Foren-SW nach einer Zeichenanzahl X (irgendwo über 10K Zeichen) rigoros jedes Posting abschneidet, sind selbst deine Ausschnitte durch die Foren-SW noch weiter verkürzt worden. Das macht ein Zusammenfügen mehrerer deiner Postings für einen Gesamtüberblick im Grunde unmöglich.
Da sich unter anderem "kastlr" bei solchen Datenträger-Logs als absoluter Freak in der Analyse gezeigt hat, würde ich eine Verlinkung der vollständigen Logs vorschlagen. Die Analyse dauert dann zwar deutlich länger, es lassen sich damit aber meist in den Ausschnitten nicht sichtbare Wechselwirkungen nachvollziehen.
Da die Foren-SW nach einer Zeichenanzahl X (irgendwo über 10K Zeichen) rigoros jedes Posting abschneidet, sind selbst deine Ausschnitte durch die Foren-SW noch weiter verkürzt worden. Das macht ein Zusammenfügen mehrerer deiner Postings für einen Gesamtüberblick im Grunde unmöglich.
Da sich unter anderem "kastlr" bei solchen Datenträger-Logs als absoluter Freak in der Analyse gezeigt hat, würde ich eine Verlinkung der vollständigen Logs vorschlagen. Die Analyse dauert dann zwar deutlich länger, es lassen sich damit aber meist in den Ausschnitten nicht sichtbare Wechselwirkungen nachvollziehen.
Ok, dann jetzt aber 
Hier der Link zu dem Log File:
https://www.dropbox.com/s/h6zi2p3b2r0g1 ... 2.txt?dl=0
Hier der Link zu dem Log File:
https://www.dropbox.com/s/h6zi2p3b2r0g1 ... 2.txt?dl=0
-
kastlr
- Profi
- Beiträge: 993
- Registriert: 31.03.2008, 17:26
- Wohnort: Einzugsbereich des FC Schalke 04
- Kontaktdaten:
Hallo,
erst einmal danke für die Blumen, aber mit der Analyse und Lösung von VMware/Storage/Performance Problemen verdiene ich mir meine Brötchen.
Nebenbei noch ein paar andere Sachen aus unserem Portfolio, damit läßt sich so ein Tag schon füllen.
Zumal ja alle geradezu tiefenentspannt sind, wenn es im VMware Umfeld mal hängt.
Aber zurück zum Thread, Logfiles sind in jedem Fall sinnvoll.
Collecting diagnostic information for VMware ESX/ESXi (653)
Aus den vorhandenen Logsnippets wird folgendes ersichtlich.
Wenn es zu einem Problem mit den NetApp Volumes kommt, dann ist es immer nur beim Schreiben.
Das kannst du mit folgendem Befehl überprüfen, der liefert dir alle SCSI Fehler gegen deine NetApp Volumes.
In den gefilterten Logeinträgen findest du den SCSI OpCode, 0x2a entspricht einem Write IO.
Hier mal die Liste der OpCodes
Aktuell kann ich nicht erkennen warum der Write fehlschlägt, aber das ist mit so wenigen Informationen auch nicht unerwartet.
Aber du kannst ja damit schon mal dem NetApp Support auf die Füße steigen.
Viel Spaß,
Ralf
erst einmal danke für die Blumen, aber mit der Analyse und Lösung von VMware/Storage/Performance Problemen verdiene ich mir meine Brötchen.
Nebenbei noch ein paar andere Sachen aus unserem Portfolio, damit läßt sich so ein Tag schon füllen.
Zumal ja alle geradezu tiefenentspannt sind, wenn es im VMware Umfeld mal hängt.
Aber zurück zum Thread, Logfiles sind in jedem Fall sinnvoll.
Collecting diagnostic information for VMware ESX/ESXi (653)
Aus den vorhandenen Logsnippets wird folgendes ersichtlich.
Wenn es zu einem Problem mit den NetApp Volumes kommt, dann ist es immer nur beim Schreiben.
Das kannst du mit folgendem Befehl überprüfen, der liefert dir alle SCSI Fehler gegen deine NetApp Volumes.
Code: Alles auswählen
grep H:0x /var/log/vmkernel.log|grep naa.60a98In den gefilterten Logeinträgen findest du den SCSI OpCode, 0x2a entspricht einem Write IO.
Hier mal die Liste der OpCodes
Aktuell kann ich nicht erkennen warum der Write fehlschlägt, aber das ist mit so wenigen Informationen auch nicht unerwartet.
Aber du kannst ja damit schon mal dem NetApp Support auf die Füße steigen.
Viel Spaß,
Ralf
Hallo,
hab noch mal ein Frage zu dem Thread...
Mir ist jetzt aufgefallen, das ich viele Hohe Latenzen und Meldung, wie z.B. sich auf den lokalen SCSI Controller beziehen.
2016-02-23T08:22:22.889Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x4124417c4c80) 0x12, CmdSN 0x84a07 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:27:22.905Z cpu14:4110)ScsiDeviceIO: 2322: Cmd(0x412441a12840) 0x12, CmdSN 0x84a29 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:32:22.921Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x412440d685c0) 0x12, CmdSN 0x84a31 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:37:22.938Z cpu16:4790)ScsiDeviceIO: 2322: Cmd(0x4124403c81c0) 0x12, CmdSN 0x84a39 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:42:22.955Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x41244156a5c0) 0x12, CmdSN 0x84a41 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:47:22.979Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x41244156dcc0) 0x12, CmdSN 0x84a49 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:52:22.986Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x41240149e5c0) 0x12, CmdSN 0x84a51 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:57:23.004Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x41244107ed00) 0x12, CmdSN 0x84a73 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:02:23.019Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x412440d31300) 0x12, CmdSN 0x84a85 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:07:23.035Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x412440fadc80) 0x12, CmdSN 0x84a8d from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:12:23.052Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x4124403d5ec0) 0x12, CmdSN 0x84a95 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:17:23.067Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x412441653bc0) 0x12, CmdSN 0x84a9d from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:22:23.083Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x4124413b0980) 0x12, CmdSN 0x84aa5 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:27:23.099Z cpu23:10399863)ScsiDeviceIO: 2322: Cmd(0x412441a1b6c0) 0x12, CmdSN 0x84ac7 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:32:23.114Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400d84400) 0x12, CmdSN 0x84acf from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:37:23.129Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x4124416519c0) 0x12, CmdSN 0x84ad7 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:42:23.145Z cpu15:4111)ScsiDeviceIO: 2322: Cmd(0x4124415b9ec0) 0x12, CmdSN 0x84adf from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
Diese ......c90c ist mein Dell Perc H200 SCSI Adaper!?
Kann das ein Fehler auf der lokalen Festplatte oder im Controller sein, oder muß ich mich da weiter nicht drum Kümmern?
Zu dem alten Problem, es scheint wohl an der Deduplizierung der Netapp zu liegen, die CPU ist Nachts immer bis 80% Ausgelastet und da kann es passieren, das die Netapp Verbindungen trennt, bzw. nicht rechtzeitig antworten kann. Ich bin gespannt, was Netapp weiter dazu sagt!
hab noch mal ein Frage zu dem Thread...
Mir ist jetzt aufgefallen, das ich viele Hohe Latenzen und Meldung, wie z.B. sich auf den lokalen SCSI Controller beziehen.
2016-02-23T08:22:22.889Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x4124417c4c80) 0x12, CmdSN 0x84a07 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:27:22.905Z cpu14:4110)ScsiDeviceIO: 2322: Cmd(0x412441a12840) 0x12, CmdSN 0x84a29 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:32:22.921Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x412440d685c0) 0x12, CmdSN 0x84a31 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:37:22.938Z cpu16:4790)ScsiDeviceIO: 2322: Cmd(0x4124403c81c0) 0x12, CmdSN 0x84a39 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:42:22.955Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x41244156a5c0) 0x12, CmdSN 0x84a41 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:47:22.979Z cpu17:4113)ScsiDeviceIO: 2322: Cmd(0x41244156dcc0) 0x12, CmdSN 0x84a49 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:52:22.986Z cpu11:4107)ScsiDeviceIO: 2322: Cmd(0x41240149e5c0) 0x12, CmdSN 0x84a51 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T08:57:23.004Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x41244107ed00) 0x12, CmdSN 0x84a73 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:02:23.019Z cpu16:4112)ScsiDeviceIO: 2322: Cmd(0x412440d31300) 0x12, CmdSN 0x84a85 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:07:23.035Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x412440fadc80) 0x12, CmdSN 0x84a8d from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:12:23.052Z cpu23:4119)ScsiDeviceIO: 2322: Cmd(0x4124403d5ec0) 0x12, CmdSN 0x84a95 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:17:23.067Z cpu13:4109)ScsiDeviceIO: 2322: Cmd(0x412441653bc0) 0x12, CmdSN 0x84a9d from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:22:23.083Z cpu22:4118)ScsiDeviceIO: 2322: Cmd(0x4124413b0980) 0x12, CmdSN 0x84aa5 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:27:23.099Z cpu23:10399863)ScsiDeviceIO: 2322: Cmd(0x412441a1b6c0) 0x12, CmdSN 0x84ac7 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:32:23.114Z cpu10:4106)ScsiDeviceIO: 2322: Cmd(0x412400d84400) 0x12, CmdSN 0x84acf from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:37:23.129Z cpu21:4117)ScsiDeviceIO: 2322: Cmd(0x4124416519c0) 0x12, CmdSN 0x84ad7 from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2016-02-23T09:42:23.145Z cpu15:4111)ScsiDeviceIO: 2322: Cmd(0x4124415b9ec0) 0x12, CmdSN 0x84adf from world 0 to dev "naa.600508e0000000002b11767c0700c90c" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
Diese ......c90c ist mein Dell Perc H200 SCSI Adaper!?
Kann das ein Fehler auf der lokalen Festplatte oder im Controller sein, oder muß ich mich da weiter nicht drum Kümmern?
Zu dem alten Problem, es scheint wohl an der Deduplizierung der Netapp zu liegen, die CPU ist Nachts immer bis 80% Ausgelastet und da kann es passieren, das die Netapp Verbindungen trennt, bzw. nicht rechtzeitig antworten kann. Ich bin gespannt, was Netapp weiter dazu sagt!
-
kastlr
- Profi
- Beiträge: 993
- Registriert: 31.03.2008, 17:26
- Wohnort: Einzugsbereich des FC Schalke 04
- Kontaktdaten:
Hallo,
hier die Bedeutung der Fehlermeldungen.Diese Meldung wird von der Disk zurück an den Host gesendet.
Wahrscheinlich unterstützt dein Controller/die Platte das entsprechende Kommando nicht, daher bleibt ihm/ihr gar nichts anderes übrig, dies an den Requestor zu melden.
Ich glaube daher nicht, dass diese Meldungen ursächlich für dein Problem sind.
Gruß,
Ralf
hier die Bedeutung der Fehlermeldungen.
Code: Alles auswählen
SCSI OpCode: 0x12 = Inquiry
SCSI Status Code: 0x02 = check condition
SCSI Sense Code: 0x05 = illegal request
SCSI additional Sense Code: 0x24 0x00 = invalid field in CDBWahrscheinlich unterstützt dein Controller/die Platte das entsprechende Kommando nicht, daher bleibt ihm/ihr gar nichts anderes übrig, dies an den Requestor zu melden.
Ich glaube daher nicht, dass diese Meldungen ursächlich für dein Problem sind.
Gruß,
Ralf
kastlr hat geschrieben:Hi,
neben simplen Hardware Defekten gibt es noch eine ganze Reihe von möglichen Fehlern im Zusammenhang mit Disk IO's.
Und ich bin mir ziemlich sicher, das bei den meisten dieser Fehler deine NetApp paris car service nichts melden wird, selbst wenn Sie Teil oder Verursacher des Problems sein sollte.
Ich würde an deiner Stelle mal in den vmkernel.logs des ESXi Servers nachsehen, ob sich dort Hinweise auf ein Problem finden lassen.
Eventuell wirst du zum Problemzeitraum einige SCSI Fehlermeldungen finden, vielleicht war auch der Kommunikationskanal zur NetApp nicht fehlerfrei.
Aus eigener Erfahrung kann ich dir nur raten, die Umgebung sowohl vom VMware als auch vom NetApp Support überprüfen zu lassen, solange die Logs das Zeitfenster mit dem Problem noch enthalten.
Gruß,
Ralf
Vielen Dank für diese sehr nützliche Tipps
Zurück zu „vSphere 5 / ESXi 5 und 5.1“
Wer ist online?
Mitglieder in diesem Forum: 0 Mitglieder und 34 Gäste