Created attachment 1405380 [details] Logs Description of problem: Cannot plug vNIC after was unplugged Another new regression - after VM's vNIC was unplugged it's can't be plugged back - 2018-03-07 16:39:19,951+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugNicVDSCommand] (default task-16) [41d2aaa4] NIC hot-set: <?xml version="1.0" encoding="UTF-8"?><hotplug> <devices> <interface type="bridge"> <model type="virtio"/> <link state="up"/> <source bridge="ovirtmgmt"/> <alias name="ua-a17b8639-477b-43fd-8621-ea4c1f80745b"/> <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"/> <mac address="00:00:00:00:00:50"/> <filterref filter="vdsm-no-mac-spoofing"/> <bandwidth/> </interface> </devices> <metadata xmlns:ovirt-vm="http://ovirt.org/vm/1.0"> <ovirt-vm:vm> <ovirt-vm:device mac_address="00:00:00:00:00:50"> <ovirt-vm:custom/> </ovirt-vm:device> </ovirt-vm:vm> </metadata> </hotplug> 2018-03-07 16:39:21,325+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugNicVDSCommand] (default task-16) [41d2aaa4] Failed in 'HotPlugNicVDS' method 2018-03-07 16:39:21,333+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-16) [41d2aaa4] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM camel-vdsa.qa.lab.tlv.redhat.com command HotPlugNicVDS failed: XML error: non unique alias detected: ua-a17b8639-477b-43fd-8621-ea4c1f80745b 2018-03-07 16:39:21,233+0200 INFO (jsonrpc/1) [virt.vm] (vmId='f8abc451-ad7d-4916-ae0b-fca5cfa643bb') Hotplug NIC xml: <?xml version='1.0' encoding='utf-8'?> <interface type="bridge"> <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci" /> <mac address="00:00:00:00:00:50" /> <model type="virtio" /> <source bridge="ovirtmgmt" /> <filterref filter="vdsm-no-mac-spoofing" /> <alias name="ua-a17b8639-477b-43fd-8621-ea4c1f80745b" /> </interface> (vm:2947) 2018-03-07 16:39:21,321+0200 ERROR (jsonrpc/1) [virt.vm] (vmId='f8abc451-ad7d-4916-ae0b-fca5cfa643bb') Hotplug failed (vm:2953) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2951, in hotplugNic self._dom.attachDevice(nicXml) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 98, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 570, in attachDevice if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self) libvirtError: XML error: non unique alias detected: ua-a17b8639-477b-43fd-8621-ea4c1f80745b Version-Release number of selected component (if applicable): 4.2.2.2-0.1.el7 vdsm-4.20.20-1.el7ev.x86_64 kernel-3.10.0-858.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. Start VM with 1 vNIC(ovirtmgmt) 2. Unplug vNIC 3. Try to plug it back Actual results: Error while executing action Edit VM Interface properties: Failed to activate VM Network Interface. Expected results: Must work
When did this last run successfully? Does this affect 4.1-on-7.5 as well? Would you try to reproduce it directly with virsh, and pass the buck to libvirt? I'd suspect the error lies there.
It would be nice to see what version of libvirt is installed on the host. And the domain XML before the attempt to attach the interface would help too, since we need to make sure the interface was really removed before we try to plug it back in. BTW, I just tried this with upstream libvirt and it works as expected.
(In reply to Dan Kenigsberg from comment #1) > When did this last run successfully? Does this affect 4.1-on-7.5 as well? > Would you try to reproduce it directly with virsh, and pass the buck to > libvirt? I'd suspect the error lies there. Dan, 4.1 + 7.5 working as expected. so i guess it's on our side in 4.2 4.2.2-1 was the last successful. Failed on 4.2.2.2
(In reply to Jiri Denemark from comment #2) > It would be nice to see what version of libvirt is installed on the host. > And the domain XML before the attempt to attach the interface would help > too, since we need to make sure the interface was really removed before we > try to plug it back in. > > BTW, I just tried this with upstream libvirt and it works as expected. Libvirt version is libvirt-daemon-3.9.0-13.el7.x86_64 libvirt-client-3.9.0-13.el7.x86_64 domxml prior the attempt to plug(attaching now)
Created attachment 1405769 [details] domxml before the attempt to plug back
I guess it's because 4.1 did not use user-defined aliases, which is a new libvirt feature in 7.5. In other words, it's a libvirt bug and I was able to reproduce it. See bug 1553075.
Michael, this issue blocks a very specific scenario: unplugging and then plugging a device from/to a running VM. To be clear, hot-plugging a new device into the VM should work. Also, hot-unplugging a device, shutting down the VM, and then plugging it back when the VM is down or when it is started again (hot-plug) should work. I see two options here: 1. To disable user-aliases until libvirt is fixed. 2. To disable/change the automation test according to the above. I strongly prefer #2. We know that we need user-aliases in 4.2 to support host-level hooks that change a property of the devices that we use for their identification (mac addresses in case of NICs, paths in case of disks). Disabling user-aliases will prevent us from evaluating this change (for example, we now also know we have an issue with usb devices set with user-aliases in libvirt). From the user point of view, that's an annoying regression but I'm not convinced it's that critical. So can we change the automation test?
(In reply to Arik from comment #7) > Michael, this issue blocks a very specific scenario: unplugging and then > plugging a device from/to a running VM. > > To be clear, hot-plugging a new device into the VM should work. > Also, hot-unplugging a device, shutting down the VM, and then plugging it > back when the VM is down or when it is started again (hot-plug) should work. > > I see two options here: > 1. To disable user-aliases until libvirt is fixed. > 2. To disable/change the automation test according to the above. > > I strongly prefer #2. We know that we need user-aliases in 4.2 to support > host-level hooks that change a property of the devices that we use for their > identification (mac addresses in case of NICs, paths in case of disks). > Disabling user-aliases will prevent us from evaluating this change (for > example, we now also know we have an issue with usb devices set with > user-aliases in libvirt). > From the user point of view, that's an annoying regression but I'm not > convinced it's that critical. > > So can we change the automation test? We will skip this tests until the libvirt bug will get fixed.
This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
libvirt-3.9.0-14.el7_5.1 should fix this, please test.
Will test with the new d/s build
QE now only have officially libvirt-client-3.9.0-14.el7.x86_64 libvirt-daemon-3.9.0-14.el7.x86_64 and it failedQA with this build. I'm not going to verify this report with a brew version libvrti, only when we get. So moving to NEW until this version will be available for QE. FailedQA with libvirt-client-3.9.0-14.el7.x86_64 libvirt-daemon-3.9.0-14.el7.x86_64 vdsm-4.20.22-1.el7ev.x86_64 4.2.2.4-0.1.el7
(In reply to Michael Burman from comment #14) > QE now only have officially libvirt-client-3.9.0-14.el7.x86_64 > libvirt-daemon-3.9.0-14.el7.x86_64 and it failedQA with this build. What do you mean by "officially"? > I'm not going to verify this report with a brew version libvrti, only when > we get. libvirt-3.9.0-14.el7_5.1 is available to QE > > So moving to NEW until this version will be available for QE. retest with version where the issue is fixed, please.
(In reply to Michal Skrivanek from comment #15) > (In reply to Michael Burman from comment #14) > > QE now only have officially libvirt-client-3.9.0-14.el7.x86_64 > > libvirt-daemon-3.9.0-14.el7.x86_64 and it failedQA with this build. > > What do you mean by "officially"? > > > I'm not going to verify this report with a brew version libvrti, only when > > we get. > > libvirt-3.9.0-14.el7_5.1 is available to QE > > > > > So moving to NEW until this version will be available for QE. > > retest with version where the issue is fixed, please. No it's not available, we have libvirt-client-3.9.0-14.el7.x86_64 only. Will test when we get it. Not going to test until then, if you want to keep it ON_QA i don't mind. Thanks,
Verified on - libvirt-daemon-3.9.0-14.el7_5.2.x86_64 and vdsm-4.20.23-1.el7ev.x86_64 with kernel-3.10.0-862.el7.x86_64 libvirt-daemon-driver-storage-iscsi-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-driver-interface-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-config-nwfilter-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-driver-storage-mpath-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-kvm-3.9.0-14.el7_5.2.x86_64 libvirt-python-3.9.0-1.el7.x86_64 libvirt-daemon-driver-storage-core-3.9.0-14.el7_5.2.x86_64 libvirt-client-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-driver-qemu-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-driver-storage-rbd-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-driver-storage-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-driver-secret-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-driver-nwfilter-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-driver-storage-scsi-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.2.x86_64 libvirt-lock-sanlock-3.9.0-14.el7_5.2.x86_64 libvirt-libs-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-config-network-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-driver-storage-logical-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-driver-network-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-driver-storage-disk-3.9.0-14.el7_5.2.x86_64 libvirt-daemon-driver-nodedev-3.9.0-14.el7_5.2.x86_64
This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018. Since the problem described in this bug report should be resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.