Created attachment 1310159 [details] vdsm log Description of problem: Vdsm shows traceback when HE VM powered-off 2017-08-07 18:09:32,164+0300 ERROR (jsonrpc/0) [virt.vm] (vmId='b58fdeda-45bb-43d2-b336-ef9953171347') Failed to tear down device vnc, device in inconsist ent state (vm:2120) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2117, in _teardown_devices device.teardown() File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/graphics.py", line 75, in teardown self.vmid) File "/usr/lib/python2.7/site-packages/vdsm/virt/libvirtnetwork.py", line 99, in delete_network removeNetwork(netname) File "/usr/lib/python2.7/site-packages/vdsm/virt/libvirtnetwork.py", line 110, in removeNetwork netName = LIBVIRT_NET_PREFIX + network TypeError: cannot concatenate 'str' and 'NoneType' objects Version-Release number of selected component (if applicable): vdsm-4.20.2-25.git7499b81.el7.centos.x86_64 libvirt-client-3.2.0-14.el7_4.2.x86_64 ovirt-engine-4.2.0-0.0.master.20170803140556.git1e7d0dd.el7.centos.noarch How reproducible: Always Steps to Reproduce: 1. Deploy hosted-engine 2. Add master storage domain 3. Wait for auto-import operation 4. Set global maintenance 5. Power off HE VM via HE CLI # hosted-engine --vm-poweroff Actual results: Action succeeds, but the traceback above appears under the vdsm log Expected results: Action succeeds, without any tracebacks under the VDSM log Additional info:
Dan, care to take a look? By default hosted engine is not doing anything specific with the network, so this is new to me.
It smells as if Hosted Engine is relying on an ancient default, which Vdsm has recently dropped. Edy, would you care to look? If my guess is correct, I'd rather we fix it in hosted engine.
If "displayNetwork" is not specified for a VM, we will encounter this problem. When displayNetwork is not specified, the VM console can be reached from all existing interfaces which usually is not that safe. We should fix this in both domains: Not to explode when it is missing and not to define a VM (Engine) with a missing displayNetwork.
Checked on vdsm-4.20.3-88.git6102334.el7.centos.x86_64 Now I have another error Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2180, in _teardown_devices device.teardown() File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/graphics.py", line 74, in teardown display_network = self.specParams['displayNetwork'] KeyError: 'displayNetwork' Change "display_network = self.specParams['displayNetwork']" to "display_network = self.specParams.get('displayNetwork')"
If the displayNetwork key is missing, I would expect to fail on setup already. fromani, any idea what is going on?
(In reply to Edward Haas from comment #5) > If the displayNetwork key is missing, I would expect to fail on setup > already. > > fromani, any idea what is going on? Most likely with the switch to Engine XML, Engine stopped to sent the 'displayNetwork' specParam. Should be easy to check with the Vdsm logs, will do later if noone else can check quickly
(In reply to Artyom from comment #4) > Checked on vdsm-4.20.3-88.git6102334.el7.centos.x86_64 > Now I have another error > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2180, in > _teardown_devices > device.teardown() > File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/graphics.py", > line 74, in teardown > display_network = self.specParams['displayNetwork'] > KeyError: 'displayNetwork' > > Change "display_network = self.specParams['displayNetwork']" to > "display_network = self.specParams.get('displayNetwork')" Please share the Vdsm logs including the VM startup and shutdown.
Created attachment 1333210 [details] vdsm log You can start to check from last traceback
What exactly happened on that box? I took the liberty to login into it and look at the logs, and it seems that Vdsm was upgraded while the VM was running. I'm not sure, but it seems that Vdsm failed to store, or to recover, the specParams in the domain XML, hence the failure on teardown. I need a clean reproducer with all the logs to be sure: The log snippet should include: 1. VM started, dump of the creation parameters 2. VM shutdown, traceback If possible, this will be much appreciated: 3. output of virsh -r dumpxml while the VM is running
Created attachment 1333564 [details] vdsm log and VM dumpxml I added vdsm log that has a start and destroy of HE VM, also you can find dumpxml under archive.
According to fromani, displayNetwork was missing on destroy() because it was not stored in the domxml metadata during startup, and was therefore lost when vdsmd is restarted.
Created attachment 1337634 [details] Another full Vdsm log of the same occurrence
(In reply to Francesco Romani from comment #12) > Created attachment 1337634 [details] > Another full Vdsm log of the same occurrence Still can't reproduce the bug.
There is a high chance this bug was fixed by commit 70daa1566 in master branch.
What is preventing this from moving to MODIFIED?
(In reply to Francesco Romani from comment #14) > There is a high chance this bug was fixed by commit 70daa1566 in master > branch. I'm confident this is the fix, moving to MODIFIED
Verified on vdsm-4.20.3-173.gitae7a215.el7.centos.x86_64
This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017. Since the problem described in this bug report should be resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.