Description of problem: Migrate VM after Memory hotplug, VM is stuck in migrating state and can't shutdown/reboot. Happened while verify BZ: 1392903 Version-Release number of selected component (if applicable): oVirt Engine Version: 4.1.0-0.0.master.20161119231317.git06d9054.el7.centos Host: OS Version:RHEL - 7.3 - 7.el7 OS Description:Red Hat Enterprise Linux Server 7.3 (Maipo) Kernel Version:3.10.0 - 514.el7.x86_64 KVM Version:2.3.0 - 31.el7_2.23 LIBVIRT Version:libvirt-2.0.0-10.el7 VDSM Version:vdsm-4.18.999-919.git723413e.el7.centos Steps to Reproduce: 1. Create and Start VM 2. Hotplug Memory 3. Migrate VM Actual results: VM is stuck in migrating after the second migration From vdsm log: 2016-11-20 09:12:41,033 ERROR (jsonrpc/5) [virt.vm] (vmId='f8d4c72d-acbb-4c86-8399-c79de41cb2b1') Error fetching vm stats (vm:1288) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 1284, in _getRunningVmStats vm_sample.interval) File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", line 47, in produce balloon(vm, stats, last_sample) File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", line 151, in balloon max_mem = int(vm.conf.get('memSize')) * 1024 TypeError: int() argument must be a string or a number, not 'NoneType' 2016-11-20 09:12:41,033 ERROR (jsonrpc/5) [DynamicBridge] TypeError raised by dispatched function (Bridge:204) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 202, in _dynamicMethod result = fn(*methodArgs) File "/usr/share/vdsm/API.py", line 1389, in getAllVmStats statsList = self._cif.getAllVmStats() File "/usr/share/vdsm/clientIF.py", line 442, in getAllVmStats return [v.getStats() for v in self.vmContainer.values()] File "/usr/share/vdsm/virt/vm.py", line 1217, in getStats stats.update(self._getGuestStats()) File "/usr/share/vdsm/virt/vm.py", line 1399, in _getGuestStats int(self.conf['memSize']) * 100) TypeError: int() argument must be a string or a number, not 'NoneType' 2016-11-20 09:12:41,034 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:552) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 547, in _handle_request res = method(**params) File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 205, in _dynamicMethod raise InvalidCall(fn, methodArgs, e) InvalidCall: Attempt to call function: <bound method Global.getAllVmStats of <API.Global object at 0x28b2f50>> with arguments: () error: int() argument must be a string or a number, not 'NoneType' From engine log: 2016-11-20 16:08:16,140 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [929f2d12-ee89-4c6c-87be-3d3cf1f4ea89] Command 'GetAllVmStatsVDSCommand(HostName = host_mixed_1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='f6a54c7f-fc6c-415e-8931-f27b5ad82109'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetAllVmStatsVDS, error = Attempt to call function: <bound method Global.getAllVmStats of <API.Global object at 0x3b3be50>> with arguments: () error: int() argument must be a string or a number, not 'NoneType', code = -32603
Created attachment 1222205 [details] engine.log
Created attachment 1222206 [details] vdsm log
Created attachment 1222207 [details] qemu vm log
It's not strictly related to this bug, but Francesco, would you re-enable default logging of nonfrequent calls and their arguments? 2016-11-20 09:06:19,122 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.hotplugMemory succeeded in 0.06 seconds (__init__:515)
(In reply to Dan Kenigsberg from comment #4) > It's not strictly related to this bug, but Francesco, would you re-enable > default logging of nonfrequent calls and their arguments? > > 2016-11-20 09:06:19,122 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call > VM.hotplugMemory succeeded in 0.06 seconds (__init__:515) https://gerrit.ovirt.org/#/c/60572/ is addressing this. Pushing it forwar faster would help
The bug is in POST even though the fix is not here - a related logging patch was added.
the logging is indeed a separate issue A proper fix of the issue has been POSTed and will fix this bug which is related to recent etree conversion and memory hotplug
(In reply to Dan Kenigsberg from comment #4) > It's not strictly related to this bug, but Francesco, would you re-enable > default logging of nonfrequent calls and their arguments? > > 2016-11-20 09:06:19,122 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call > VM.hotplugMemory succeeded in 0.06 seconds (__init__:515) On it, we'll figure a way to properly do this.
The fix for this issue should be included in oVirt 4.1.0 beta 1 released on December 1st. If not included please move back to modified.
Verify: Engine: 4.1.0-0.2.master.20161213122836.git2cd5587.el7.centos Host: OS Version:RHEL - 7.3 - 7.el7 OS Description:Red Hat Enterprise Linux Server 7.3 (Maipo) Kernel Version:3.10.0 - 514.el7.x86_64 KVM Version:2.6.0 - 27.1.el7 LIBVIRT Version:libvirt-2.0.0-10.el7 VDSM Version:vdsm-4.18.999-1138.git6c51957.el7.centos Steps: 1. Create and Start VM 2. Hotplug Memory 3. Migrate VM Results: PASS