[gelöst] vMotion begrenzen

[JanClaas]

Hallo,
ich betreibe aktuell einen vSphere 5.0 Enterprise Cluster mit 5 Rack und 3 Blade Servern.
Sobald ich einen der Blade-Server in den MaintenanceMode schieben will kriege ich große Probleme mit dem Thema vMotion. VMWare versucht auf den Blades 8 VM's gleichzeitig zu verschieben und läuft dann in Timeout Probleme auf grund fehlender Bandbreite. Bei den Rack-Servern werden immer nur 4 VM's verschoben und dort tauchen dann auch keine Probleme auf.

Netzwerkkonfiguration:

Rack-Server:
2x 1GBit für ESXi Management
1x 1GBit für vMotion mit VLAN tagging
4x 1GBit für VM-Gäste
Dedizierte vSwitches für die 3 Anwendungsfälle.

Blade-Server:
2x 10GBit für ESXi Management, vMotion und VM-Gäst
Auf Grund der Enterprise Lizenz sind keine distributed Switches verwendbar.
Daher wurde ein vSwicht mit mehreren Portgruppen angelegt. Des Weiteren wurde bei der vMotion Portgruppe das TrafficShapping auf 50000Kbit/s für Average Bandwidth und Peak Bandwidth gesetzt, 102400 Kbytes Burst Size, damit vMotion nicht die ganze Bandbreite der beiden Nic's verbraucht. Außerdem wurde beim Management Nic0 auf aktiv und Nic1 auf Standby gesetzt, sowie bei vMotion Nic0 auf standby und Nic1 auf aktiv.

Lauf VMWare Definition kann man über eine 1GBit Leitung bis zu 4 VM's verschieben, über eine 10GBit Verbindung bis zu 8 VM's. Da ich die Bandbreite der vMotion Portgruppe im 10GBit Bereich auf ca. 50% Drosseln wollte passen da natürlich auch keine 8 VM's mehr drauf. Jedoch scheint VMWare die Bandbreiten Reduzierung nicht zu registrieren.

Frage1: Sind meine Traffic Shapping Daten für eine 50% Nutzung der 10GBit Karte korrekt? Gibt es hierzu andere Vorschläge?

Frage2: Wie bringe ich dem ESXi bei, dass er nur 4 VM's beim vMotion gleichzeitig nutzt.

Dell hat mir geraten eine Nic-Partitionierung vorzunehmen und darüber die Bandbreite zu steuern hätte zur Folge, dass ich die 3 Blades komplett neu aufbauen müsste.
Oder was aber unsupported wäre den Wert networkCostPerVmotion von 1 auf 2 anzuheben und somit die Anzahl zu begrenzen.

Bevor ich nun diese beiden Lösungen ausprobiere, die für mich beide sehr experimentell aussehen wollte ich wissen, ob es andere Meinungen / Lösungen zu diesem Problem gibt.

Über Antworten würde ich mich sehr freuen.

[bla!zilla]

Hallo Jan,

die Limits für 4 vMotions bei 1 GbE und 8 vMotions bei 10 GbE sind fest kodiert und abhängig von der erkannten Geschwindigkeit. Bei 2,5 Gb/s gehen also auch nur 4 vMotions gleichzeitig.

Grundsätzlich sollte es nicht zu Timeouts kommen! Die Hypervisor achten darauf und das Verfahren ist sehr auf Sicherheit aus. Statt zu drosseln solltest du eher den Grund für die Timeouts suchen.

Der Einsatz von Traffic Shaping auf der Port Group oder dem vSwitch ist schon sehr verzweifelt. Ich würde das entfernen.

Poste doch bitte mal as vmkernel.log zu dem Zeitpunkt, an dem die vMotion mit einem Timeout auf die Nase fällt.

[JanClaas]

Hallo bla!zilla,

start: 12.04.2014 11:56:46
Abgebrochen: 12.04.2014 12:51:30

Fehlermeldung:
Error message from sluvm11.emea.sensus.net: The migration was canceled because the amount of changing memory for the virtual machine was greater than the available network bandwidth. Attempt the migration again when the virtual machine is not as busy or more network bandwidth is available. Error 12.04.2014 12:51:29

Für mich steht im log irgendwie zu wenig drinnen.

VMKernel.log

2014-04-12T11:24:35.859Z cpu3:4147)<3>ixgbe: vmnic11: ixgbe_free_rx_queue: freed rx queue 1
2014-04-12T11:47:55.919Z cpu8:4147)<3>ixgbe: vmnic11: ixgbe_alloc_rx_queue: allocated rx queue 1
2014-04-12T11:47:55.919Z cpu8:4147)<3>ixgbe: vmnic11: ixgbe_apply_rx_filter: Setting rx queue 1 to 00:50:56:75:1a:69
2014-04-12T11:48:00.919Z cpu20:4147)<3>ixgbe: vmnic11: ixgbe_remove_rx_filter: Clearing 00:50:56:75:1a:69 from rx queue 1
2014-04-12T11:48:00.919Z cpu20:4147)<3>ixgbe: vmnic11: ixgbe_free_rx_queue: freed rx queue 1
2014-04-12T12:22:51.008Z cpu3:4147)<3>ixgbe: vmnic9: ixgbe_alloc_rx_queue: allocated rx queue 1
2014-04-12T12:22:51.008Z cpu3:4147)<3>ixgbe: vmnic9: ixgbe_apply_rx_filter: Setting rx queue 1 to 00:50:56:72:d5:23
2014-04-12T12:22:51.008Z cpu3:4147)<3>ixgbe: vmnic8: ixgbe_alloc_rx_queue: allocated rx queue 1
2014-04-12T12:22:51.008Z cpu3:4147)<3>ixgbe: vmnic8: ixgbe_apply_rx_filter: Setting rx queue 1 to 00:50:56:73:55:a6
2014-04-12T12:23:46.011Z cpu12:4147)<3>ixgbe: vmnic9: ixgbe_remove_rx_filter: Clearing 00:50:56:72:d5:23 from rx queue 1
2014-04-12T12:23:46.011Z cpu12:4147)<3>ixgbe: vmnic9: ixgbe_free_rx_queue: freed rx queue 1
2014-04-12T12:23:46.011Z cpu12:4147)<3>ixgbe: vmnic8: ixgbe_remove_rx_filter: Clearing 00:50:56:73:55:a6 from rx queue 1
2014-04-12T12:23:46.011Z cpu12:4147)<3>ixgbe: vmnic8: ixgbe_free_rx_queue: freed rx queue 1
2014-04-12T12:24:31.013Z cpu16:4147)<3>ixgbe: vmnic11: ixgbe_alloc_rx_queue: allocated rx queue 1
2014-04-12T12:24:31.013Z cpu16:4147)<3>ixgbe: vmnic11: ixgbe_apply_rx_filter: Setting rx queue 1 to 00:50:56:75:1a:69
2014-04-12T12:24:36.013Z cpu16:4147)<3>ixgbe: vmnic11: ixgbe_remove_rx_filter: Clearing 00:50:56:75:1a:69 from rx queue 1
2014-04-12T12:24:36.013Z cpu16:4147)<3>ixgbe: vmnic11: ixgbe_free_rx_queue: freed rx queue 1
2014-04-12T12:36:51.044Z cpu11:4147)<3>ixgbe: vmnic9: ixgbe_alloc_rx_queue: allocated rx queue 1
2014-04-12T12:36:51.044Z cpu11:4147)<3>ixgbe: vmnic9: ixgbe_apply_rx_filter: Setting rx queue 1 to 00:50:56:72:d5:23
2014-04-12T12:36:56.044Z cpu13:4147)<3>ixgbe: vmnic9: ixgbe_remove_rx_filter: Clearing 00:50:56:72:d5:23 from rx queue 1
2014-04-12T12:36:56.044Z cpu13:4147)<3>ixgbe: vmnic9: ixgbe_free_rx_queue: freed rx queue 1
2014-04-12T12:46:20.905Z cpu18:46936)NetPort: 1437: disabled port 0x500000d
2014-04-12T12:46:20.905Z cpu18:47116)VSCSI: 6227: handle 15351(vscsi0:0):Destroying Device for world 46936 (pendCom 0)
2014-04-12T12:46:20.977Z cpu18:47116)VSCSI: 6227: handle 15352(vscsi0:1):Destroying Device for world 46936 (pendCom 0)
2014-04-12T12:46:21.026Z cpu18:47116)CBT: 692: Disconnecting the cbt device 507fb6c-cbt with filehandle 84409196
2014-04-12T12:46:21.194Z cpu18:47116)CBT: 692: Disconnecting the cbt device 5a2fb69-cbt with filehandle 94567273
2014-04-12T12:46:23.291Z cpu22:46936)NetPort: 1250: enabled port 0x500000d with mac 00:50:56:98:5c:4c
2014-04-12T12:46:23.341Z cpu22:47116)CBT: 1082: Created device 521fb6b-cbt for cbt driver with filehandle 86113131
2014-04-12T12:46:23.343Z cpu22:47116)VSCSI: 3642: handle 15353(vscsi0:0):Using sync mode due to sparse disks
2014-04-12T12:46:23.343Z cpu22:47116)VSCSI: 3683: handle 15353(vscsi0:0):Creating Virtual Device for world 46936 (FSS handle 93911917)
2014-04-12T12:46:23.381Z cpu22:47116)CBT: 1082: Created device 649fb70-cbt for cbt driver with filehandle 105511792
2014-04-12T12:46:23.383Z cpu22:47116)VSCSI: 3642: handle 15354(vscsi0:1):Using sync mode due to sparse disks
2014-04-12T12:46:23.383Z cpu22:47116)VSCSI: 3683: handle 15354(vscsi0:1):Creating Virtual Device for world 46936 (FSS handle 90045298)
2014-04-12T12:47:16.071Z cpu23:4147)<3>ixgbe: vmnic9: ixgbe_alloc_rx_queue: allocated rx queue 1
2014-04-12T12:47:16.071Z cpu23:4147)<3>ixgbe: vmnic9: ixgbe_apply_rx_filter: Setting rx queue 1 to 00:50:56:72:d5:23
2014-04-12T12:48:01.073Z cpu18:4147)<3>bnx2 0000:01:00.0: vmnic0: Free NetQ RX Queue (forced: 1): 0
2014-04-12T12:48:01.073Z cpu18:4147)<6>bnx2 0000:01:00.0: vmnic0: RX NetQ allocated on 0
2014-04-12T12:48:01.073Z cpu18:4147)<6>bnx2 0000:01:00.0: vmnic0: NetQ set RX Filter: 0 [bc:30:5b:ea:a3:3f 0]
2014-04-12T12:57:01.096Z cpu15:4147)<3>ixgbe: vmnic8: ixgbe_alloc_rx_queue: allocated rx queue 1

[JanClaas]

Das Traffic Shaping hat der zuständige Dell Techniker empfohlen, damit entsprechend noch Bandbreite für die Netzwerkverbindung der VM-Gäste übrig bleibt.

Ich brauche nur die Sicherheit, dass so ein vMotion mir nicht die ganzen Netzwerkverbindungen killt, wenn das auch ohne Traffic Shaping realisierbar ist, dann bin ich offen für Vorschläge.

[irix]

Hmm... ich meine das ich schon MB/s Angaben in Logs gesehen habe was vMotion angeht. Guck mal ins hostd.log .

Ich kann mich nicht errinneren wann das letzte mal solch eine Fehlermeldung kam. Aktuell vormir sind 10 Hosts im Cluster und vMotion hat 2x10GbE welches es sich teilen muss fuer VM Traffic und co. Ich wuenschte mir er wuerde mehr als 8 gleichzeitig machen

Ich meine mich zu inneren als das damals mit dem MultiNic vMotion losging die Messungen zeigten das er eh nur so 2-3Gbit/s macht.

Gruss
Joerg

[JanClaas]

Hallo Joerg,

hier der hostd.log auszug aus dem Zeitrau: (Der Name der VM um die es geht lautet: "sluexmsv" )

2014-04-12T11:55:00.464Z [3B432B90 info 'TaskManager' opID=3935f440-63] Task Created : haTask-ha-host-vim.HostSystem.acquireCimServicesTicket-178889091
2014-04-12T11:55:00.465Z [3BF93B90 verbose 'Cimsvc' opID=3935f440-63] Ticket issued for CIMOM version 1.0, user vpxuser
2014-04-12T11:55:00.465Z [3BF93B90 info 'TaskManager' opID=3935f440-63] Task Completed : haTask-ha-host-vim.HostSystem.acquireCimServicesTicket-178889091 Status success
2014-04-12T11:55:03.553Z [3BAD5B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:55:23.662Z [FFB79AD0 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:55:29.797Z [FFD81B90 verbose 'DvsTracker'] FetchSwitches: added 0 items
2014-04-12T11:55:29.797Z [FFD81B90 verbose 'DvsTracker'] FetchDVPortgroups: added 0 items
2014-04-12T11:55:29.797Z [FFD81B90 info 'DvsTracker'] FetchUplinkDVPortgroups: added 0 items
2014-04-12T11:55:33.555Z [3B8AFB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:55:53.664Z [3BAD5B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:56:03.557Z [3B9A9B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:56:06.461Z [3B3F1B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T11:56:19.875Z [3BF93B90 verbose 'DvsManager'] PersistAllDvsInfo called
2014-04-12T11:56:23.665Z [FFB79AD0 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:56:33.560Z [3B9A9B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:56:53.668Z [3B82DB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:57:03.564Z [3B3F1B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:57:23.669Z [3B8AFB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:57:33.565Z [3BF93B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:57:37.971Z [3B86EB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T11:57:53.670Z [3B8F0B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:58:03.568Z [3B3F1B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:58:23.672Z [3B82DB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:58:33.570Z [3BF93B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:58:53.673Z [3B3F1B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:59:03.573Z [3BF93B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:59:09.392Z [3B432B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T11:59:22.450Z [3B36FB90 verbose 'Proxysvc Req09983'] New proxy client TCP(local=127.0.0.1:80, peer=127.0.0.1:53128)
2014-04-12T11:59:22.452Z [3B8F0B90 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0x3bd1a4c0] : Invoke done [waitForUpdatesEx] on [vmodl.query.PropertyCollector:ha-property-collector]
2014-04-12T11:59:22.452Z [3B8F0B90 verbose 'Vmomi'] Arg version:
--> "5574"
2014-04-12T11:59:22.452Z [3B8F0B90 verbose 'Vmomi'] Arg options:
--> (vmodl.query.PropertyCollector.WaitOptions) {
--> dynamicType = <unset>,
--> maxWaitSeconds = 600,
--> maxObjectUpdates = 100,
--> }
2014-04-12T11:59:22.452Z [3B8F0B90 info 'Vmomi'] Throw vmodl.fault.RequestCanceled
2014-04-12T11:59:22.452Z [3B8F0B90 info 'Vmomi'] Result:
--> (vmodl.fault.RequestCanceled) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> msg = "",
--> }
2014-04-12T11:59:22.453Z [3B36FB90 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(local=127.0.0.1:0, peer=127.0.0.1:51290) with error N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-04-12T11:59:23.674Z [3B8AFB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:59:33.574Z [3B432B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T11:59:47.146Z [3B86EB90 warning 'HostsvcPlugin' opID=FDAE9F3E-00000B0A-8d] IscsiManagerVmkProvider::CreateUsageMap() called
-->
2014-04-12T11:59:47.158Z [3B9A9B90 verbose 'Proxysvc Req09984'] New proxy client SSL(TCP(local=10.1.14.7:443, peer=10.1.15.31:52510))
2014-04-12T11:59:47.397Z [3B8F0B90 verbose 'Proxysvc Req09985'] New proxy client SSL(TCP(local=10.1.14.7:443, peer=10.1.15.31:52515))
2014-04-12T11:59:47.576Z [3B86EB90 warning 'Default' opID=FDAE9F3E-00000B0A-8d] IscsiManagerVmkProvider::LookupPnicForVnic: false 6
-->
2014-04-12T11:59:47.577Z [3B86EB90 warning 'Default' opID=FDAE9F3E-00000B0A-8d] IscsiManagerVmkProvider::LookupPnicForVnic: false 6
-->
2014-04-12T11:59:48.287Z [3BF93B90 warning 'HostsvcPlugin' opID=FDAE9F3E-00000B0D-86] IscsiManagerVmkProvider::CreateUsageMap() called
-->
2014-04-12T11:59:48.707Z [3BF93B90 warning 'Default' opID=FDAE9F3E-00000B0D-86] IscsiManagerVmkProvider::LookupPnicForVnic: false 6
-->
2014-04-12T11:59:48.707Z [3BF93B90 warning 'Default' opID=FDAE9F3E-00000B0D-86] IscsiManagerVmkProvider::LookupPnicForVnic: false 6
-->
2014-04-12T11:59:48.769Z [3BF93B90 warning 'HostsvcPlugin' opID=FDAE9F3E-00000B0D-86] StorageInfo::CheckVnicCompliance for vmk1 returned 3 error code.
-->
2014-04-12T11:59:48.769Z [FFB79AD0 warning 'HostsvcPlugin' opID=FDAE9F3E-00000B10-d8] IscsiManagerVmkProvider::CreateUsageMap() called
-->
2014-04-12T11:59:49.191Z [FFB79AD0 warning 'Default' opID=FDAE9F3E-00000B10-d8] IscsiManagerVmkProvider::LookupPnicForVnic: false 6
-->
2014-04-12T11:59:49.191Z [FFB79AD0 warning 'Default' opID=FDAE9F3E-00000B10-d8] IscsiManagerVmkProvider::LookupPnicForVnic: false 6
-->
2014-04-12T11:59:49.251Z [FFB79AD0 warning 'HostsvcPlugin' opID=FDAE9F3E-00000B10-d8] StorageInfo::CheckVnicCompliance for vmk1 returned 3 error code.
-->
2014-04-12T11:59:53.676Z [3B8F0B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:00:03.576Z [3B86EB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:00:23.678Z [3BAD5B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:00:29.891Z [3B8F0B90 verbose 'DvsTracker'] FetchSwitches: added 0 items
2014-04-12T12:00:29.891Z [3B8F0B90 verbose 'DvsTracker'] FetchDVPortgroups: added 0 items
2014-04-12T12:00:29.891Z [3B8F0B90 info 'DvsTracker'] FetchUplinkDVPortgroups: added 0 items
2014-04-12T12:00:33.577Z [3B8F0B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:00:40.800Z [3B9A9B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:00:53.680Z [FFD81B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:01:03.580Z [3B3F1B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:01:19.876Z [3B9A9B90 verbose 'DvsManager'] PersistAllDvsInfo called
2014-04-12T12:01:23.682Z [3B36FB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:01:33.582Z [FFB79AD0 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:01:53.683Z [3B3F1B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:02:03.585Z [FFD81B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:02:12.221Z [FFD81B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:02:23.686Z [FFD81B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:02:33.587Z [3B36FB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:02:53.687Z [3B8AFB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:03:03.588Z [3B86EB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:03:23.690Z [3BAD5B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:03:33.589Z [3BF93B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:03:43.631Z [3B36FB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:03:53.692Z [FFD81B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:04:03.592Z [3B82DB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:04:22.904Z [3B3F1B90 verbose 'Proxysvc Req09986'] New proxy client TCP(local=127.0.0.1:80, peer=127.0.0.1:62050)
2014-04-12T12:04:22.905Z [3B3F1B90 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0xffcd5250] : Invoke done [waitForUpdatesEx] on [vmodl.query.PropertyCollector:ha-property-collector]
2014-04-12T12:04:22.905Z [3B3F1B90 verbose 'Vmomi'] Arg version:
--> "5574"
2014-04-12T12:04:22.905Z [3B3F1B90 verbose 'Vmomi'] Arg options:
--> (vmodl.query.PropertyCollector.WaitOptions) {
--> dynamicType = <unset>,
--> maxWaitSeconds = 600,
--> maxObjectUpdates = 100,
--> }
2014-04-12T12:04:22.905Z [3B3F1B90 info 'Vmomi'] Throw vmodl.fault.RequestCanceled
2014-04-12T12:04:22.905Z [3B3F1B90 info 'Vmomi'] Result:
--> (vmodl.fault.RequestCanceled) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> msg = "",
--> }
2014-04-12T12:04:22.906Z [3B82DB90 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(local=127.0.0.1:0, peer=127.0.0.1:53440) with error N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-04-12T12:04:23.694Z [3B8AFB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:04:33.594Z [3B8F0B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:04:53.695Z [3B8AFB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:04:59.878Z [3B9A9B90 verbose 'Locale' opID=SWI-b584eb91-dc] Default resource used for 'counter.virtualDisk.commandsAborted.label' expected in module 'perf'.
2014-04-12T12:04:59.878Z [3B9A9B90 verbose 'Locale' opID=SWI-b584eb91-dc] Default resource used for 'counter.virtualDisk.commandsAborted.summary' expected in module 'perf'.
2014-04-12T12:04:59.878Z [3B9A9B90 verbose 'Locale' opID=SWI-b584eb91-dc] Default resource used for 'counter.virtualDisk.busResets.label' expected in module 'perf'.
2014-04-12T12:04:59.878Z [3B9A9B90 verbose 'Locale' opID=SWI-b584eb91-dc] Default resource used for 'counter.virtualDisk.busResets.summary' expected in module 'perf'.
2014-04-12T12:05:03.596Z [3B8F0B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:05:07.548Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] interactive guest operations are not ready
2014-04-12T12:05:07.549Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Tools are not operations ready
2014-04-12T12:05:07.549Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Tools are not auto-upgrade capable
2014-04-12T12:05:07.550Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Tools guest daemon status changed to: 0
2014-04-12T12:05:07.550Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] guest operations are not ready
2014-04-12T12:05:15.039Z [3B8F0B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:05:23.698Z [3B86EB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:05:29.985Z [3BF93B90 verbose 'DvsTracker'] FetchSwitches: added 0 items
2014-04-12T12:05:29.985Z [3BF93B90 verbose 'DvsTracker'] FetchDVPortgroups: added 0 items
2014-04-12T12:05:29.985Z [3BF93B90 info 'DvsTracker'] FetchUplinkDVPortgroups: added 0 items
2014-04-12T12:05:33.597Z [3B8AFB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:05:36.746Z [3B9A9B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Updating current heartbeatStatus: red
2014-04-12T12:05:36.746Z [3B9A9B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Turning off heartbeat checker
2014-04-12T12:05:38.061Z [3B545B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Disconnect check in progress.
2014-04-12T12:05:38.061Z [3BAD5B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Version status of tools changed to: 2
2014-04-12T12:05:38.065Z [3BAD5B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Tools are not set-display-modes capable
2014-04-12T12:05:38.065Z [3BAD5B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Running status of tools changed to: 0
2014-04-12T12:05:38.066Z [3B8F0B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] VMotionLastStatusCb: Succeeded
2014-04-12T12:05:38.067Z [3B8F0B90 info 'HostsvcPlugin'] Lookupvm: World ID not set for VM 33
2014-04-12T12:05:38.067Z [3B8F0B90 warning 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Unable to get resource settings for a powered on VM
2014-04-12T12:05:38.067Z [3B8F0B90 info 'HostsvcPlugin'] World does not exist worldId 4294967295
2014-04-12T12:05:38.075Z [3B8F0B90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.075Z [3B8F0B90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.075Z [3B8F0B90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.075Z [3B8F0B90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.075Z [3B8F0B90 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:38.075Z [3B8F0B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Time to gather config: 9 (msecs)
2014-04-12T12:05:38.076Z [FFB79AD0 verbose 'Hbrsvc'] Replicator: ReconfigListener triggered for config VM 33
2014-04-12T12:05:38.076Z [3B8F0B90 info 'HostsvcPlugin'] Lookupvm: World ID not set for VM 33
2014-04-12T12:05:38.076Z [3B8F0B90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.076Z [3B8F0B90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.076Z [3B8F0B90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.077Z [3B8F0B90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.077Z [3B8F0B90 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:38.088Z [3B8F0B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] VMotionLastStatusCb:: hostlog state changed from emigrating to success
2014-04-12T12:05:38.088Z [3B8F0B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] VMotionResolveCheck: Operation in progress
2014-04-12T12:05:38.088Z [FFB79AD0 verbose 'Hbrsvc'] Replicator: VmReconfig ignoring VM 33 not configured for replication
2014-04-12T12:05:38.090Z [3BAD5B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] sleepState value is -1
2014-04-12T12:05:38.090Z [3BAD5B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Ignored toolsManifestInfo update of size 0
2014-04-12T12:05:38.090Z [3BAD5B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] MKS ready for connections: false
2014-04-12T12:05:38.090Z [3BAD5B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Tools version changed to: -1
2014-04-12T12:05:38.091Z [3BAD5B90 info 'HostsvcPlugin'] Lookupvm: World ID not set for VM 33
2014-04-12T12:05:38.091Z [3BAD5B90 warning 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Unable to get resource settings for a powered on VM
2014-04-12T12:05:38.091Z [3BAD5B90 info 'HostsvcPlugin'] World does not exist worldId 4294967295
2014-04-12T12:05:38.098Z [3BAD5B90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.098Z [3BAD5B90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.098Z [3BAD5B90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.098Z [3BAD5B90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.098Z [3BAD5B90 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:38.099Z [3BAD5B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Time to gather config: 7 (msecs)
2014-04-12T12:05:38.099Z [3BAD5B90 info 'HostsvcPlugin'] Lookupvm: World ID not set for VM 33
2014-04-12T12:05:38.099Z [3B86EB90 verbose 'Hbrsvc'] Replicator: ReconfigListener triggered for config VM 33
2014-04-12T12:05:38.100Z [3B3F1B90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.100Z [3B3F1B90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.100Z [3B3F1B90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.100Z [3B3F1B90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.100Z [3B3F1B90 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:38.109Z [FFD81B90 info 'HostsvcPlugin'] Lookupvm: World ID not set for VM 33
2014-04-12T12:05:38.109Z [FFD81B90 warning 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Unable to get resource settings for a powered on VM
2014-04-12T12:05:38.110Z [FFD81B90 info 'HostsvcPlugin'] World does not exist worldId 4294967295
2014-04-12T12:05:38.110Z [3B8AFB90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.110Z [3B8AFB90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.110Z [3B8AFB90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.110Z [3B8AFB90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.110Z [3B8AFB90 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:38.117Z [FFD81B90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.117Z [FFD81B90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.117Z [FFD81B90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.117Z [FFD81B90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.117Z [FFD81B90 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:38.117Z [FFD81B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Time to gather config: 7 (msecs)
2014-04-12T12:05:38.117Z [FFD81B90 info 'HostsvcPlugin'] Lookupvm: World ID not set for VM 33
2014-04-12T12:05:38.127Z [3B8F0B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] VMotionStatusCb: Completed
2014-04-12T12:05:38.127Z [3B8F0B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] VMotionResolveCheck: Wrong power state
2014-04-12T12:05:38.128Z [3B86EB90 verbose 'Hbrsvc'] Replicator: VmReconfig ignoring VM 33 not configured for replication
2014-04-12T12:05:38.128Z [3B86EB90 verbose 'Hbrsvc'] Replicator: ReconfigListener triggered for config VM 33
2014-04-12T12:05:38.128Z [3B86EB90 verbose 'Hbrsvc'] Replicator: VmReconfig ignoring VM 33 not configured for replication
2014-04-12T12:05:38.138Z [3BF93B90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.138Z [3BF93B90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.138Z [3BF93B90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.138Z [3BF93B90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.138Z [FFB79AD0 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.139Z [3BF93B90 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:38.139Z [FFB79AD0 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.139Z [FFB79AD0 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.139Z [FFB79AD0 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.139Z [FFB79AD0 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:38.327Z [3B545B90 info 'Libs'] VmdbPipeStreamsOvlError Couldn't read: OVL_STATUS_EOF, (11) Resource temporarily unavailable.
2014-04-12T12:05:38.328Z [3B545B90 info 'Libs'] VmdbCnxDisconnect: Disconnect: closed pipe for sub cnx '/db/connection/#20c61/' (-32)
2014-04-12T12:05:38.328Z [3B545B90 info 'Libs'] VixVigorUpdateVMProperties: Unknown record replay state received over Vigor: -1.
2014-04-12T12:05:38.329Z [3B545B90 info 'Libs'] Vix: [5810 foundryVMPowerOps.c:973]: FoundryVMPowerStateChangeCallback: /vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx, vmx/execState/val = poweredOff.
2014-04-12T12:05:38.329Z [3B545B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Disconnect check in progress.
2014-04-12T12:05:38.329Z [3B545B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Unmounting the vm.
2014-04-12T12:05:38.329Z [3B3F1B90 info 'HostsvcPlugin'] Lookupvm: World ID not set for VM 33
2014-04-12T12:05:38.329Z [3B3F1B90 warning 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Unable to get resource settings for a powered on VM
2014-04-12T12:05:38.329Z [3B3F1B90 info 'HostsvcPlugin'] World does not exist worldId 4294967295
2014-04-12T12:05:38.329Z [3B545B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] VMDB unmount initiated.
2014-04-12T12:05:38.330Z [3B545B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Unmounting VM complete.
2014-04-12T12:05:38.330Z [3B545B90 info 'Libs'] VigorClientCnxDemux unset Vigor path on vm unmount.
2014-04-12T12:05:38.330Z [3B545B90 info 'Libs'] VigorClientCnxDemux VmdbCnx failed: Vigor borrowed connection to running VMX lost
2014-04-12T12:05:38.331Z [3B545B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Mount state values have changed.
2014-04-12T12:05:38.331Z [3B545B90 info 'Libs'] VmdbDbRemoveCnx: Removing Cnx from Db for '/db/connection/#20c61/'
2014-04-12T12:05:38.331Z [3B545B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Mount state values have changed.
2014-04-12T12:05:38.332Z [3B545B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Reloading config state.
2014-04-12T12:05:38.335Z [3B545B90 info 'Libs'] VMHSVMLoadConfig: Not reading VM's configuration due to migration-related poweroff
2014-04-12T12:05:38.336Z [3B545B90 info 'Libs'] VigorClientCnxDemux unset Vigor path on vm unmount.
2014-04-12T12:05:38.336Z [3B545B90 info 'Libs'] VigorClientCnxDemux VmdbCnx failed: Vigor borrowed connection to running VMX lost
2014-04-12T12:05:38.336Z [3B545B90 info 'Libs'] Vix: [5810 foundryVM.c:11400]: FoundryVMMountStateChangeCallback: remount complete, set flag, VM = /vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx
2014-04-12T12:05:38.342Z [3B3F1B90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.342Z [3B3F1B90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.342Z [3B3F1B90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.342Z [3B3F1B90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.342Z [3B3F1B90 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:38.342Z [3B3F1B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Time to gather config: 14 (msecs)
2014-04-12T12:05:38.343Z [FFD81B90 verbose 'Hbrsvc'] Replicator: ReconfigListener triggered for config VM 33
2014-04-12T12:05:38.343Z [3B3F1B90 info 'HostsvcPlugin'] Lookupvm: World ID not set for VM 33
2014-04-12T12:05:38.344Z [3B649B90 info 'Libs'] SOCKET 26 (128)
2014-04-12T12:05:38.344Z [3B649B90 info 'Libs'] recv detected client closed connection
2014-04-12T12:05:38.344Z [3B545B90 info 'Libs'] Detected automation socket close for VM (/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx)
2014-04-12T12:05:38.354Z [FFD81B90 verbose 'Hbrsvc'] Replicator: VmReconfig ignoring VM 33 not configured for replication
2014-04-12T12:05:38.364Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Retrieved current VM state from foundry 5, 2
2014-04-12T12:05:38.364Z [3BAD5B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Retrieved current VM state from foundry 5, 2
2014-04-12T12:05:38.365Z [3B3F1B90 info 'HostsvcPlugin'] Lookupvm: World ID not set for VM 33
2014-04-12T12:05:38.365Z [3B3F1B90 warning 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Unable to get resource settings for a powered on VM
2014-04-12T12:05:38.365Z [3B3F1B90 info 'HostsvcPlugin'] World does not exist worldId 4294967295
2014-04-12T12:05:38.367Z [3B8F0B90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.367Z [3B8F0B90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.367Z [3B8F0B90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.367Z [3B8F0B90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.367Z [3B8F0B90 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:38.373Z [3B82DB90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.373Z [3B82DB90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.373Z [3B82DB90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.373Z [3B82DB90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.373Z [3B82DB90 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:38.383Z [3B3F1B90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.383Z [3B3F1B90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.383Z [3B3F1B90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.383Z [3B3F1B90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.383Z [3B3F1B90 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:38.383Z [3B3F1B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Time to gather config: 18 (msecs)
2014-04-12T12:05:38.384Z [3B3F1B90 info 'HostsvcPlugin'] Lookupvm: World ID not set for VM 33
2014-04-12T12:05:38.384Z [3B86EB90 verbose 'Hbrsvc'] Replicator: ReconfigListener triggered for config VM 33
2014-04-12T12:05:38.393Z [3B86EB90 verbose 'Hbrsvc'] Replicator: VmReconfig ignoring VM 33 not configured for replication
2014-04-12T12:05:38.394Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] VMotionResolveCheck: Firing ResolveCb
2014-04-12T12:05:38.394Z [3BAD5B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] VMotionResolveCheck: Already resolved
2014-04-12T12:05:38.394Z [3BF93B90 info 'VMotionSrc (1397300642587911)'] ResolveCb: VMX reports needsUnregister = true for migrateType MIGRATE_TYPE_VMOTION
2014-04-12T12:05:38.394Z [3B8AFB90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] VMotionResolveCheck: Already resolved
2014-04-12T12:05:38.395Z [3BF93B90 info 'VMotionSrc (1397300642587911)'] ResolveCb: Succeeded
2014-04-12T12:05:38.395Z [3BF93B90 verbose 'VMotionSrc (1397300642587911)'] Migration changed state from MIGRATING to UNREGISTERING
2014-04-12T12:05:38.395Z [3BF93B90 verbose 'VMotionSrc (1397300642587911)'] Finish called
2014-04-12T12:05:38.395Z [3BF93B90 verbose 'VMotionSrc (1397300642587911)'] Set source task result vm downtime to 1006580
2014-04-12T12:05:38.395Z [3BF93B90 verbose 'VMotionSrc (1397300642587911)'] Set source task result precopyBandwidth to 1825556
2014-04-12T12:05:38.395Z [3BF93B90 verbose 'VMotionSrc (1397300642587911)'] Set source task result precopyStunTime to 381650
2014-04-12T12:05:38.395Z [3B3F1B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Tools are not set-resolution capable
2014-04-12T12:05:38.395Z [3B3F1B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Tools are not set-display-topology capable
2014-04-12T12:05:38.395Z [3B3F1B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Running status of tools changed to: 0
2014-04-12T12:05:38.395Z [3BF93B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] State Transition (VM_STATE_EMIGRATING -> VM_STATE_OFF)
2014-04-12T12:05:38.396Z [3BF93B90 verbose 'ha-host'] ModeMgr::End: op = normal, current = normal, count = 3
2014-04-12T12:05:38.396Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Updating current heartbeatStatus: gray
2014-04-12T12:05:38.396Z [3B82DB90 verbose 'HostsvcPlugin'] Received state change for VM '33'
2014-04-12T12:05:38.396Z [3B82DB90 info 'GuestFileTransferImpl'] Entered VmPowerStateListener
2014-04-12T12:05:38.396Z [3B82DB90 info 'GuestFileTransferImpl'] VmPowerStateListener succeeded
2014-04-12T12:05:38.396Z [3B82DB90 info 'Hbrsvc'] Replicator: powerstate change VM: 33 Old: 1 New: 0
2014-04-12T12:05:38.396Z [3B82DB90 info 'Hbrsvc'] Replicator: Poweroff for VM: (id=33)
2014-04-12T12:05:38.396Z [3B82DB90 verbose 'Hbrsvc'] Replicator: Remove group no matching entry for VM (id=33)
2014-04-12T12:05:38.396Z [3BF93B90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.397Z [3BF93B90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.397Z [3BF93B90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.397Z [3BF93B90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.397Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Predicted VM overhead: 277409792 bytes
2014-04-12T12:05:38.397Z [3BF93B90 info 'HostsvcPlugin'] Decremented SIOC Injector Flag2
2014-04-12T12:05:38.397Z [3BF93B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] State Transition (VM_STATE_OFF -> VM_STATE_UNREGISTERING)
2014-04-12T12:05:38.398Z [3B82DB90 verbose 'Statssvc'] Removing vm 33 from poweredOnVms list
2014-04-12T12:05:38.399Z [3BF93B90 verbose 'DatastoreBrowser'] 33-envmgr-datastorebrowser::Destroy
2014-04-12T12:05:38.399Z [3B8AFB90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] VMotionResolveCheck: Already resolved
2014-04-12T12:05:38.399Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] RemoveFromAutoStart
2014-04-12T12:05:38.399Z [3BF93B90 verbose 'Hostsvc::AutoStartManager'] Request spec is (vim.host.AutoStartManager.Config) {
--> dynamicType = <unset>,
--> defaults = (vim.host.AutoStartManager.SystemDefaults) null,
--> powerInfo = (vim.host.AutoStartManager.AutoPowerInfo) [
--> (vim.host.AutoStartManager.AutoPowerInfo) {
--> dynamicType = <unset>,
--> key = 'vim.VirtualMachine:33',
--> startOrder = -1,
--> startDelay = -1,
--> waitForHeartbeat = "no",
--> startAction = "none",
--> stopDelay = -1,
--> stopAction = "none",
--> }
--> ],
--> }
2014-04-12T12:05:38.399Z [3BF93B90 verbose 'Hostsvc::AutoStartManager'] Updated AutoStart sequence.
2014-04-12T12:05:38.400Z [3BF93B90 verbose 'Hostsvc::AutoStartManager'] Autostart info after reconfiguration: (vim.host.AutoStartManager.Config) {
--> dynamicType = <unset>,
--> defaults = (vim.host.AutoStartManager.SystemDefaults) {
--> dynamicType = <unset>,
--> enabled = false,
--> startDelay = 120,
--> stopDelay = 120,
--> waitForHeartbeat = false,
--> stopAction = "PowerOff",
--> },
--> }
2014-04-12T12:05:38.400Z [3BF93B90 verbose 'Vmsvc'] Released Vm Id: 33.
2014-04-12T12:05:38.400Z [3BF93B90 verbose 'HostsvcPlugin'] RemoveEntry '33'
2014-04-12T12:05:38.401Z [3BF93B90 verbose 'HostsvcPlugin'] RemoveEntry succeeded
2014-04-12T12:05:38.401Z [3BF93B90 verbose 'ResourcePool pool11'] Removed child 33 from pool
2014-04-12T12:05:38.401Z [3BF93B90 verbose 'HostsvcPlugin'] Security domain hostd33 not found
2014-04-12T12:05:38.401Z [3BF93B90 verbose 'HostConfigSyncManagerImpl'] Sync called
2014-04-12T12:05:38.401Z [3BF93B90 verbose 'HostConfigSyncManagerImpl'] Scheduling '/sbin/backup.sh' in 5 seconds (backup counter = 2)
2014-04-12T12:05:38.402Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Close Handle called
2014-04-12T12:05:38.402Z [3BAD5B90 info 'GuestFileTransferImpl'] VmOperationListener: unregister notification received for VM: 33
2014-04-12T12:05:38.402Z [3BAD5B90 info 'GuestFileTransferImpl'] VmOperationListener succeeded
2014-04-12T12:05:38.402Z [3BAD5B90 info 'Hbrsvc'] Replicator: UnregisterListener triggered for config VM 33
2014-04-12T12:05:38.402Z [3BAD5B90 verbose 'Statssvc'] EntityRemovedListener: Deleting stats for entity 33
2014-04-12T12:05:38.402Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Shutting down VMDB service...
2014-04-12T12:05:38.402Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Unregistering callback...
2014-04-12T12:05:38.402Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] ...done
2014-04-12T12:05:38.402Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Unsubscribed from events.
2014-04-12T12:05:38.402Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Canceled outstanding Foundry operations.
2014-04-12T12:05:38.402Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Released VM handle.
2014-04-12T12:05:38.403Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Closed VM handle.
2014-04-12T12:05:38.403Z [3BF93B90 warning 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Failed to find activation record, event user unknown.
2014-04-12T12:05:38.403Z [3BF93B90 info 'ha-eventmgr'] Event 437 : Removed sluexmsv on sluvm11.emea.sensus.net from ha-datacenter
2014-04-12T12:05:38.403Z [3BF93B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] State Transition (VM_STATE_UNREGISTERING -> VM_STATE_GONE)
2014-04-12T12:05:38.404Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Unregister completed
2014-04-12T12:05:38.404Z [3BF93B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] VMotion cleanup and unregistration completed
2014-04-12T12:05:38.404Z [3BF93B90 verbose 'VMotionSrc (1397300642587911)'] Migration changed state from UNREGISTERING to DONE
2014-04-12T12:05:38.405Z [3BF93B90 verbose 'VMotionSrc (1397300642587911)'] JrnlEnd: Wrote active=0
2014-04-12T12:05:38.406Z [3BF93B90 verbose 'VMotionSrc (1397300642587911)'] JrnlEnd: Finished journal
2014-04-12T12:05:38.406Z [3BF93B90 verbose 'VMotionSrc (1397300642587911)'] [JrnlFilePersistenceProvider::DeleteJournal] Deleting /var/lib/vmware/hostd/journal/1397300642.31.
2014-04-12T12:05:38.406Z [3BF93B90 verbose 'VMotionSrc (1397300642587911)'] JrnlDelete: Deleted journal
2014-04-12T12:05:38.407Z [3BF93B90 info 'VMotionSrc (1397300642587911)'] CompleteOp: Vmotion task succeeded with result: (vim.host.VMotionManager.SrcVMotionResult) {
--> dynamicType = <unset>,
--> vmDowntime = 1006580,
--> vmPrecopyStunTime = 381650,
--> vmPrecopyBandwidth = 1825556,
--> }
2014-04-12T12:05:38.407Z [FFD81B90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.407Z [3BF93B90 info 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Virtual machine object cleanup
2014-04-12T12:05:38.407Z [FFD81B90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.407Z [3BF93B90 verbose 'vm:/vmfs/volumes/508131d7-503fa877-6a20-d067e5ec35b6/sluexmsv/SLUEXMSV.vmx'] Closed VM handle.
2014-04-12T12:05:38.407Z [FFD81B90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.407Z [FFD81B90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.407Z [FFD81B90 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:38.419Z [FFD81B90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 59632 vmmOvhd.paged: 59168 vmmOvhd.nonpaged: 5203
2014-04-12T12:05:38.419Z [FFD81B90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:05:38.419Z [FFD81B90 info 'ha-host'] Computed from lib overhead: 265564160 for (memPages: 1048576, fbSize: 1024, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 11845632
2014-04-12T12:05:38.420Z [FFD81B90 info 'ha-host'] overhead result (no CBRC): 277409792
2014-04-12T12:05:38.420Z [FFD81B90 info 'ha-host'] Predicted overhead: 277409792 for (memory: 4294967296, video: 4194304, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 4194304, cbrcOverhead: 0
2014-04-12T12:05:39.283Z [3B36FB90 verbose 'Locale' opID=HB-host-47831@14793-a44701da-ae] Default resource used for 'counter.virtualDisk.commandsAborted.label' expected in module 'perf'.
2014-04-12T12:05:39.283Z [3B36FB90 verbose 'Locale' opID=HB-host-47831@14793-a44701da-ae] Default resource used for 'counter.virtualDisk.commandsAborted.summary' expected in module 'perf'.
2014-04-12T12:05:39.283Z [3B36FB90 verbose 'Locale' opID=HB-host-47831@14793-a44701da-ae] Default resource used for 'counter.virtualDisk.busResets.label' expected in module 'perf'.
2014-04-12T12:05:39.283Z [3B36FB90 verbose 'Locale' opID=HB-host-47831@14793-a44701da-ae] Default resource used for 'counter.virtualDisk.busResets.summary' expected in module 'perf'.
2014-04-12T12:05:39.309Z [3BF93B90 verbose 'ha-license-manager' opID=HB-host-47831@14793-a44701da-ae] Load: Loading existing file: /etc/vmware/license.cfg
2014-04-12T12:05:39.321Z [3BF93B90 verbose 'Default' opID=HB-host-47831@14793-a44701da-ae] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)
2014-04-12T12:05:39.870Z [3BAD5B90 info 'VMotion' opID=FDAE9F3E-00000854-6b-ff] CompleteSource [1397300642587911]
2014-04-12T12:05:39.902Z [3BAD5B90 verbose 'Locale' opID=5de272aa] Default resource used for 'counter.virtualDisk.commandsAborted.label' expected in module 'perf'.
2014-04-12T12:05:39.902Z [3BAD5B90 verbose 'Locale' opID=5de272aa] Default resource used for 'counter.virtualDisk.commandsAborted.summary' expected in module 'perf'.
2014-04-12T12:05:39.902Z [3BAD5B90 verbose 'Locale' opID=5de272aa] Default resource used for 'counter.virtualDisk.busResets.label' expected in module 'perf'.
2014-04-12T12:05:39.902Z [3BAD5B90 verbose 'Locale' opID=5de272aa] Default resource used for 'counter.virtualDisk.busResets.summary' expected in module 'perf'.
2014-04-12T12:05:39.923Z [3B82DB90 info 'TaskManager' opID=5de272aa] Task Created : haTask-pool11-vim.ResourcePool.updateConfig-178889188
2014-04-12T12:05:39.923Z [3B8F0B90 verbose 'HostsvcPlugin' opID=5de272aa] SetRsrcSpec pool11 (vim.ResourceConfigSpec) {
--> dynamicType = <unset>,
--> entity = 'vim.ResourcePool:pool11',
--> changeVersion = <unset>,
--> lastModified = <unset>,
--> cpuAllocation = (vim.ResourceAllocationInfo) {
--> dynamicType = <unset>,
--> reservation = 0,
--> expandableReservation = true,
--> limit = -1,
--> shares = (vim.SharesInfo) {
--> dynamicType = <unset>,
--> shares = 264,
--> level = "custom",
--> },
--> overheadLimit = <unset>,
--> },
--> memoryAllocation = (vim.ResourceAllocationInfo) {
--> dynamicType = <unset>,
--> reservation = 848,
--> expandableReservation = true,
--> limit = -1,
--> shares = (vim.SharesInfo) {
--> dynamicType = <unset>,
--> shares = 10528,
--> level = "custom",
--> },
--> overheadLimit = <unset>,
--> },
--> }
2014-04-12T12:05:39.957Z [3B8F0B90 info 'TaskManager' opID=5de272aa] Task Completed : haTask-pool11-vim.ResourcePool.updateConfig-178889188 Status success
2014-04-12T12:05:43.403Z [3BAD5B90 verbose 'HostConfigSyncManagerImpl'] Invoking '/sbin/backup.sh': pending sync requests: 1
2014-04-12T12:05:43.404Z [3BAD5B90 info 'SysCommandPosix'] ForkExec(/sbin/backup.sh) 482417
2014-04-12T12:05:44.143Z [3BAD5B90 info 'HostConfigSyncManagerImpl'] /sbin/backup.sh succeeded
2014-04-12T12:05:49.289Z [3BAD5B90 verbose 'Locale' opID=HB-host-47831@14795-c3327dce-d5] Default resource used for 'counter.virtualDisk.commandsAborted.label' expected in module 'perf'.
2014-04-12T12:05:49.289Z [3BAD5B90 verbose 'Locale' opID=HB-host-47831@14795-c3327dce-d5] Default resource used for 'counter.virtualDisk.commandsAborted.summary' expected in module 'perf'.
2014-04-12T12:05:49.289Z [3BAD5B90 verbose 'Locale' opID=HB-host-47831@14795-c3327dce-d5] Default resource used for 'counter.virtualDisk.busResets.label' expected in module 'perf'.
2014-04-12T12:05:49.289Z [3BAD5B90 verbose 'Locale' opID=HB-host-47831@14795-c3327dce-d5] Default resource used for 'counter.virtualDisk.busResets.summary' expected in module 'perf'.
2014-04-12T12:05:49.312Z [3B432B90 verbose 'ha-license-manager' opID=HB-host-47831@14795-c3327dce-d5] Load: Loading existing file: /etc/vmware/license.cfg
2014-04-12T12:05:49.326Z [3B432B90 verbose 'Default' opID=HB-host-47831@14795-c3327dce-d5] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)
2014-04-12T12:05:53.700Z [3B9A9B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:06:03.600Z [3B432B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:06:19.877Z [3B36FB90 verbose 'DvsManager'] PersistAllDvsInfo called
2014-04-12T12:06:23.701Z [3B9A9B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:06:33.499Z [3B432B90 verbose 'ResourcePool ha-root-pool'] Root pool capacity changed from 24700MHz/189321MB to 24700MHz/189324MB
2014-04-12T12:06:33.601Z [FFD81B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:06:46.559Z [FFD81B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:06:53.703Z [3B9A9B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:07:03.602Z [3B9A9B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:07:23.705Z [3B82DB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:07:33.605Z [3B86EB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:07:53.707Z [3B82DB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:08:03.606Z [3BAD5B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:08:17.969Z [FFD81B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:08:23.710Z [3B8AFB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:08:33.609Z [3B86EB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:08:53.711Z [3B432B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:09:03.610Z [3B86EB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:09:23.714Z [3B36FB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:09:33.613Z [FFD81B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:09:49.377Z [3B86EB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:09:53.717Z [3B9A9B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:10:03.616Z [3BAD5B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:10:23.720Z [3B36FB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:10:30.078Z [3BF93B90 verbose 'DvsTracker'] FetchSwitches: added 0 items
2014-04-12T12:10:30.078Z [3BF93B90 verbose 'DvsTracker'] FetchDVPortgroups: added 0 items
2014-04-12T12:10:30.078Z [3BF93B90 info 'DvsTracker'] FetchUplinkDVPortgroups: added 0 items
2014-04-12T12:10:33.617Z [3B432B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:10:38.878Z [3B9A9B90 verbose 'Proxysvc Req09987'] New proxy client TCP(local=127.0.0.1:80, peer=127.0.0.1:53670)
2014-04-12T12:10:38.880Z [3B8F0B90 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0xc7b42f8] : Invoke done [waitForUpdatesEx] on [vmodl.query.PropertyCollector:ha-property-collector]
2014-04-12T12:10:38.880Z [3B8F0B90 verbose 'Vmomi'] Arg version:
--> "5580"
2014-04-12T12:10:38.880Z [3B8F0B90 verbose 'Vmomi'] Arg options:
--> (vmodl.query.PropertyCollector.WaitOptions) {
--> dynamicType = <unset>,
--> maxWaitSeconds = 600,
--> maxObjectUpdates = 100,
--> }
2014-04-12T12:10:38.880Z [3B8F0B90 info 'Vmomi'] Throw vmodl.fault.RequestCanceled
2014-04-12T12:10:38.880Z [3B8F0B90 info 'Vmomi'] Result:
--> (vmodl.fault.RequestCanceled) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> msg = "",
--> }
2014-04-12T12:10:38.881Z [3B9A9B90 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(local=127.0.0.1:0, peer=127.0.0.1:54015) with error N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-04-12T12:10:53.723Z [FFB79AD0 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:11:03.619Z [3B86EB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:11:19.878Z [3BAD5B90 verbose 'DvsManager'] PersistAllDvsInfo called
2014-04-12T12:11:20.787Z [3B86EB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:11:23.724Z [3B36FB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:11:33.620Z [FFB79AD0 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:11:53.726Z [FFB79AD0 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:12:03.623Z [3B82DB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:12:23.729Z [3B8F0B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:12:33.626Z [3B3F1B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:12:52.197Z [3B3F1B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:12:53.730Z [3B36FB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:13:03.628Z [3B3F1B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:13:23.732Z [3B86EB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:13:33.631Z [3B82DB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:13:53.733Z [3B432B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:14:03.634Z [3B8AFB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:14:23.597Z [FFD81B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:14:23.736Z [3B9A9B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:14:33.636Z [3B8F0B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:14:53.738Z [3B82DB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:15:03.638Z [3B86EB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:15:23.741Z [3BF93B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:15:30.169Z [3BAD5B90 verbose 'DvsTracker'] FetchSwitches: added 0 items
2014-04-12T12:15:30.169Z [3BAD5B90 verbose 'DvsTracker'] FetchDVPortgroups: added 0 items
2014-04-12T12:15:30.169Z [3BAD5B90 info 'DvsTracker'] FetchUplinkDVPortgroups: added 0 items
2014-04-12T12:15:33.641Z [FFD81B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:15:39.362Z [3B86EB90 verbose 'Proxysvc Req09988'] New proxy client TCP(local=127.0.0.1:80, peer=127.0.0.1:60660)
2014-04-12T12:15:39.363Z [FFD81B90 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0xca89aa0] : Invoke done [waitForUpdatesEx] on [vmodl.query.PropertyCollector:ha-property-collector]
2014-04-12T12:15:39.364Z [FFD81B90 verbose 'Vmomi'] Arg version:
--> "5580"
2014-04-12T12:15:39.364Z [FFD81B90 verbose 'Vmomi'] Arg options:
--> (vmodl.query.PropertyCollector.WaitOptions) {
--> dynamicType = <unset>,
--> maxWaitSeconds = 600,
--> maxObjectUpdates = 100,
--> }
2014-04-12T12:15:39.364Z [FFD81B90 info 'Vmomi'] Throw vmodl.fault.RequestCanceled
2014-04-12T12:15:39.364Z [FFD81B90 info 'Vmomi'] Result:
--> (vmodl.fault.RequestCanceled) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> msg = "",
--> }
2014-04-12T12:15:39.364Z [3B86EB90 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(local=127.0.0.1:0, peer=127.0.0.1:51380) with error N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-04-12T12:15:53.743Z [3BF93B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:15:55.005Z [3BAD5B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:16:03.644Z [FFD81B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:16:19.880Z [3BF93B90 verbose 'DvsManager'] PersistAllDvsInfo called
2014-04-12T12:16:23.746Z [3B432B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:16:33.646Z [FFB79AD0 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:16:53.747Z [3BAD5B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:17:03.647Z [3B8AFB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:17:23.748Z [3B86EB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:17:26.416Z [3B8F0B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:17:33.650Z [FFB79AD0 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:17:53.751Z [3BF93B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:18:03.653Z [FFD81B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:18:23.753Z [3B8F0B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:18:33.655Z [3BF93B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:18:53.756Z [3B3F1B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:18:57.835Z [3B9A9B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:19:03.656Z [FFB79AD0 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:19:23.757Z [3BAD5B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:19:33.659Z [3B9A9B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:19:53.760Z [3B432B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:20:03.662Z [FFD81B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:20:23.761Z [3BF93B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:20:29.234Z [3BF93B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:20:30.260Z [3B36FB90 verbose 'DvsTracker'] FetchSwitches: added 0 items
2014-04-12T12:20:30.260Z [3B36FB90 verbose 'DvsTracker'] FetchDVPortgroups: added 0 items
2014-04-12T12:20:30.260Z [3B36FB90 info 'DvsTracker'] FetchUplinkDVPortgroups: added 0 items
2014-04-12T12:20:33.664Z [3B82DB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:20:39.821Z [3B8AFB90 verbose 'Proxysvc Req09989'] New proxy client TCP(local=127.0.0.1:80, peer=127.0.0.1:62168)
2014-04-12T12:20:39.822Z [3B86EB90 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0xc924dc0] : Invoke done [waitForUpdatesEx] on [vmodl.query.PropertyCollector:ha-property-collector]
2014-04-12T12:20:39.822Z [3B86EB90 verbose 'Vmomi'] Arg version:
--> "5580"
2014-04-12T12:20:39.822Z [3B86EB90 verbose 'Vmomi'] Arg options:
--> (vmodl.query.PropertyCollector.WaitOptions) {
--> dynamicType = <unset>,
--> maxWaitSeconds = 600,
--> maxObjectUpdates = 100,
--> }
2014-04-12T12:20:39.822Z [3B86EB90 info 'Vmomi'] Throw vmodl.fault.RequestCanceled
2014-04-12T12:20:39.822Z [3B86EB90 info 'Vmomi'] Result:
--> (vmodl.fault.RequestCanceled) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> msg = "",
--> }
2014-04-12T12:20:39.823Z [3B8AFB90 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(local=127.0.0.1:0, peer=127.0.0.1:60707) with error N7Vmacore15SystemExceptionE(Connection reset by peer)
2014-04-12T12:20:53.763Z [3B36FB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:21:03.666Z [FFD81B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:21:19.881Z [FFD81B90 verbose 'DvsManager'] PersistAllDvsInfo called
2014-04-12T12:21:23.766Z [3BAD5B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:21:33.668Z [3B9A9B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:21:53.768Z [3B86EB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:22:00.644Z [3B82DB90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
2014-04-12T12:22:03.671Z [3B8AFB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:22:23.770Z [3B8AFB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:22:33.673Z [3B8F0B90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:22:53.771Z [3B86EB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:23:03.674Z [3B86EB90 verbose 'SoapAdapter'] Responded to service state request
2014-04-12T12:23:12.743Z [3B545B90 info 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] Disconnect check in progress.
2014-04-12T12:23:12.743Z [3B8AFB90 verbose 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] Handling message _vmx3: The migration was canceled because the amount of changing memory for the virtual machine was greater than the available network bandwidth. Attempt the migration again when the virtual machine is not as busy or more network bandwidth is available.
-->
2014-04-12T12:23:12.743Z [3B545B90 info 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] Disconnect check in progress.
2014-04-12T12:23:12.744Z [3B8AFB90 warning 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] Failed to find activation record, event user unknown.
2014-04-12T12:23:12.744Z [3B8AFB90 info 'ha-eventmgr'] Event 438 : Error message on slusp2010 on sluvm11.emea.sensus.net in ha-datacenter: The migration was canceled because the amount of changing memory for the virtual machine was greater than the available network bandwidth. Attempt the migration again when the virtual machine is not as busy or more network bandwidth is available.
-->
2014-04-12T12:23:12.745Z [3B8F0B90 verbose 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] VMotionLastStatusCb: Failed with error 536871177: The migration was canceled because the amount of changing memory for the VM was greater than the available network bandwidth, meaning the migration was not making forward progress. Please attempt the migration again when the VM is not as busy or more network bandwidth is available.
2014-04-12T12:23:12.745Z [3B545B90 info 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] Disconnect check in progress.
2014-04-12T12:23:12.753Z [3B8F0B90 info 'ha-host'] vmxSwapEnabled = true vmmOvhd.anonymous: 94574 vmmOvhd.paged: 60192 vmmOvhd.nonpaged: 6275
2014-04-12T12:23:12.753Z [3B8F0B90 info 'ha-host'] vmkOvhd.fixd = 303104 vmkOvhd.variable: 2050 vmkOvhd.selfSwap: 3145728
2014-04-12T12:23:12.753Z [3B8F0B90 info 'ha-host'] Computed from lib overhead: 413077504 for (memPages: 2097152, fbSize: 2048, #vcpus: 4, #lsiscsi: 1, #pvscsi: 0, #passthru: 0, #network: 3) vmkOverhead: 20242432
2014-04-12T12:23:12.753Z [3B8F0B90 info 'ha-host'] overhead result (no CBRC): 433319936
2014-04-12T12:23:12.753Z [3B8F0B90 info 'ha-host'] Predicted overhead: 433319936 for (memory: 8589934592, video: 8388608, #vcpus: 4, autodetect: false, enable3D: false, svgaPtFbSize: 8388608, cbrcOverhead: 0
2014-04-12T12:23:12.753Z [3B8F0B90 verbose 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] Time to gather config: 8 (msecs)
2014-04-12T12:23:12.754Z [FFD81B90 verbose 'Hbrsvc'] Replicator: ReconfigListener triggered for config VM 5
2014-04-12T12:23:12.763Z [3B8F0B90 verbose 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] VMotionLastStatusCb:: hostlog state changed from emigrating to failure
2014-04-12T12:23:12.763Z [3B8F0B90 verbose 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] VMotionResolveCheck: Operation in progress
2014-04-12T12:23:12.763Z [3B9A9B90 verbose 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] Retrieved current VM state from foundry 4, 8
2014-04-12T12:23:12.763Z [3B36FB90 verbose 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] Retrieved current VM state from foundry 4, 8
2014-04-12T12:23:12.764Z [FFD81B90 verbose 'Hbrsvc'] Replicator: VmReconfig ignoring VM 5 not configured for replication
2014-04-12T12:23:12.765Z [FFB79AD0 info 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] Answered question _vmx3
2014-04-12T12:23:12.765Z [3B8F0B90 verbose 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] VMotionStatusCb: Completed
2014-04-12T12:23:12.765Z [3B8F0B90 verbose 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] VMotionResolveCheck: Firing ResolveCb
2014-04-12T12:23:12.765Z [3B36FB90 verbose 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] VMotionResolveCheck: Already resolved
2014-04-12T12:23:12.765Z [3B8F0B90 info 'VMotionSrc (1397298097590298)'] ResolveCb: VMX reports needsUnregister = false for migrateType MIGRATE_TYPE_VMOTION
2014-04-12T12:23:12.765Z [3B9A9B90 verbose 'vm:/vmfs/volumes/51dd08dd-94dabe0e-68be-e0db550917a4/slusp2010/slusp2010.vmx'] VMotionResolveCheck: Already resolved
2014-04-12T12:23:12.765Z [3B8F0B90 info 'VMotionSrc (1397298097590298)'] ResolveCb: Failed with fault: (vmodl.fault.SystemError) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = (vmodl.LocalizableMessage) [
--> (vmodl.LocalizableMessage) {
--> dynamicType = <unset>,
--> key = "msg.checkpoint.precopyfailure.noforwardprogress",
--> message = "The migration was canceled because the amount of changing memory for the virtual machine was greater than the available network bandwidth. Attempt the migration again when the virtual machine is not as busy or more network bandwidth is available.
--> ",
--> }
--> ],
--> reason = "The migration was canceled because the amount of ch

[hrhrhrhr]

Hallo,

wenn ich das richtig lese habt ihr 50MBit/s für vMotion eingestellt. Das ist doch arg wenig!
Bei manch aktiver VM hatte ich schon Probleme wenn nur 1GBit/s verfügbar war.

[JanClaas]

Hallo hrhrhrhr,

öhm ja. Das ist mir bisher garnicht aufgeallen.
Für die 50% der 10GBit/s Karte fehlen da tatsächlich noch 2 Nullen bei den Bandbreitenangaben.

Das könnte die Probleme natürlich erklären. Ich werde das am Montag mal umstellen und prüfen, ob der Fehler dadurch verschwindet.

Vielen Dank für den Hinweis.
Manchmal übersieht man die einfachsten Dinge.

[hrhrhrhr]

Hallo JanClaas,

haben unsere vielen Augen etwas gebracht?

[JanClaas]

Hallo hrhrhrhr,

es hat ganz den Anschein. Ich habe nun mehrfach die einzelnen hosts evakuiert und keine Probleme mehr feststellen können und ebenfalls keine Probleme bei der Netzwerkperformaance feststellen können.

Ich habe letztendlich

Damit können wir das Thema als abgeschlossen betrachten.

[hrhrhrhr]

Hallo JanClaas,

sehr schön! Danke für die Rückmeldung.