Bug 1396816 - Internal server error, @ Global.getAllVmStats - argument type error
Summary: Internal server error, @ Global.getAllVmStats - argument type error
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: vdsm
Classification: oVirt
Component: General
Version: 4.18.15.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-4.1.0-alpha
: 4.19.2
Assignee: Martin Polednik
QA Contact: Israel Pinto
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-20 14:17 UTC by Israel Pinto
Modified: 2017-02-01 14:43 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2017-02-01 14:43:12 UTC
oVirt Team: Virt
Embargoed:
rule-engine: ovirt-4.1+
rule-engine: planning_ack+
michal.skrivanek: devel_ack+
mavital: testing_ack+


Attachments (Terms of Use)
engine.log (743.75 KB, application/x-gzip)
2016-11-20 14:21 UTC, Israel Pinto
no flags Details
vdsm log (664.54 KB, application/x-gzip)
2016-11-20 14:22 UTC, Israel Pinto
no flags Details
qemu vm log (1.68 KB, application/x-gzip)
2016-11-20 14:23 UTC, Israel Pinto
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 67141 0 master MERGED domain_descriptor: correctly check if element is present 2020-08-19 14:38:55 UTC

Description Israel Pinto 2016-11-20 14:17:02 UTC
Description of problem:
Migrate VM after Memory hotplug, VM is stuck in migrating state and 
can't shutdown/reboot.
Happened while verify BZ: 1392903

Version-Release number of selected component (if applicable):
oVirt Engine Version: 4.1.0-0.0.master.20161119231317.git06d9054.el7.centos
Host:
OS Version:RHEL - 7.3 - 7.el7
OS Description:Red Hat Enterprise Linux Server 7.3 (Maipo)
Kernel Version:3.10.0 - 514.el7.x86_64
KVM Version:2.3.0 - 31.el7_2.23
LIBVIRT Version:libvirt-2.0.0-10.el7
VDSM Version:vdsm-4.18.999-919.git723413e.el7.centos

Steps to Reproduce:
1. Create and Start VM
2. Hotplug Memory 
3. Migrate VM

Actual results:
VM is stuck in migrating after the second migration

From vdsm log:
2016-11-20 09:12:41,033 ERROR (jsonrpc/5) [virt.vm] (vmId='f8d4c72d-acbb-4c86-8399-c79de41cb2b1') Error fetching vm stats (vm:1288)
Traceback (most recent call last):
  File "/usr/share/vdsm/virt/vm.py", line 1284, in _getRunningVmStats
    vm_sample.interval)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", line 47, in produce
    balloon(vm, stats, last_sample)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", line 151, in balloon
    max_mem = int(vm.conf.get('memSize')) * 1024
TypeError: int() argument must be a string or a number, not 'NoneType'
2016-11-20 09:12:41,033 ERROR (jsonrpc/5) [DynamicBridge] TypeError raised by dispatched function (Bridge:204)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 202, in _dynamicMethod
    result = fn(*methodArgs)
  File "/usr/share/vdsm/API.py", line 1389, in getAllVmStats
    statsList = self._cif.getAllVmStats()
  File "/usr/share/vdsm/clientIF.py", line 442, in getAllVmStats
    return [v.getStats() for v in self.vmContainer.values()]
  File "/usr/share/vdsm/virt/vm.py", line 1217, in getStats
    stats.update(self._getGuestStats())
  File "/usr/share/vdsm/virt/vm.py", line 1399, in _getGuestStats
    int(self.conf['memSize']) * 100)
TypeError: int() argument must be a string or a number, not 'NoneType'
2016-11-20 09:12:41,034 ERROR (jsonrpc/5) [jsonrpc.JsonRpcServer] Internal server error (__init__:552)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 547, in _handle_request
    res = method(**params)
  File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 205, in _dynamicMethod
    raise InvalidCall(fn, methodArgs, e)
InvalidCall: Attempt to call function: <bound method Global.getAllVmStats of <API.Global object at 0x28b2f50>> with arguments: () error: int() argument must be a string or a number, not 'NoneType'


From engine log:
2016-11-20 16:08:16,140 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (DefaultQuartzScheduler4) [929f2d12-ee89-4c6c-87be-3d3cf1f4ea89] Command 'GetAllVmStatsVDSCommand(HostName = host_mixed_1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='f6a54c7f-fc6c-415e-8931-f27b5ad82109'})' execution failed: VDSGenericException: VDSErrorException: Failed to GetAllVmStatsVDS, error = Attempt to call function: <bound method Global.getAllVmStats of <API.Global object at 0x3b3be50>> with arguments: () error: int() argument must be a string or a number, not 'NoneType', code = -32603

Comment 1 Israel Pinto 2016-11-20 14:21:20 UTC
Created attachment 1222205 [details]
engine.log

Comment 2 Israel Pinto 2016-11-20 14:22:00 UTC
Created attachment 1222206 [details]
vdsm log

Comment 3 Israel Pinto 2016-11-20 14:23:31 UTC
Created attachment 1222207 [details]
qemu vm log

Comment 4 Dan Kenigsberg 2016-11-21 06:59:30 UTC
It's not strictly related to this bug, but Francesco, would you re-enable default logging of nonfrequent calls and their arguments?

2016-11-20 09:06:19,122 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call VM.hotplugMemory succeeded in 0.06 seconds (__init__:515)

Comment 5 Michal Skrivanek 2016-11-21 15:32:52 UTC
(In reply to Dan Kenigsberg from comment #4)
> It's not strictly related to this bug, but Francesco, would you re-enable
> default logging of nonfrequent calls and their arguments?
> 
> 2016-11-20 09:06:19,122 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call
> VM.hotplugMemory succeeded in 0.06 seconds (__init__:515)

https://gerrit.ovirt.org/#/c/60572/ is addressing this. Pushing it forwar faster would help

Comment 6 Yaniv Kaul 2016-11-22 07:36:36 UTC
The bug is in POST even though the fix is not here - a related logging patch was added.

Comment 7 Michal Skrivanek 2016-11-22 08:35:04 UTC
the logging is indeed a separate issue
A proper fix of the issue has been POSTed and will fix this bug which is related to recent etree conversion and memory hotplug

Comment 8 Francesco Romani 2016-11-22 08:56:51 UTC
(In reply to Dan Kenigsberg from comment #4)
> It's not strictly related to this bug, but Francesco, would you re-enable
> default logging of nonfrequent calls and their arguments?
> 
> 2016-11-20 09:06:19,122 INFO  (jsonrpc/7) [jsonrpc.JsonRpcServer] RPC call
> VM.hotplugMemory succeeded in 0.06 seconds (__init__:515)

On it, we'll figure a way to properly do this.

Comment 9 Sandro Bonazzola 2016-12-12 14:01:57 UTC
The fix for this issue should be included in oVirt 4.1.0 beta 1 released on December 1st. If not included please move back to modified.

Comment 10 Israel Pinto 2016-12-14 12:19:59 UTC
Verify:
Engine: 4.1.0-0.2.master.20161213122836.git2cd5587.el7.centos
Host: 
OS Version:RHEL - 7.3 - 7.el7
OS Description:Red Hat Enterprise Linux Server 7.3 (Maipo)
Kernel Version:3.10.0 - 514.el7.x86_64
KVM Version:2.6.0 - 27.1.el7
LIBVIRT Version:libvirt-2.0.0-10.el7
VDSM Version:vdsm-4.18.999-1138.git6c51957.el7.centos

Steps:
1. Create and Start VM
2. Hotplug Memory 
3. Migrate VM

Results:
PASS


Note You need to log in before you can comment on or make changes to this bug.