Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1552693

Summary: Cannot plug vNIC after was unplugged [libvirt bug 1554962 ]
Product: [oVirt] vdsm Reporter: Michael Burman <mburman>
Component: CoreAssignee: Michal Skrivanek <michal.skrivanek>
Status: CLOSED CURRENTRELEASE QA Contact: Michael Burman <mburman>
Severity: high Docs Contact:
Priority: high    
Version: ---CC: ahadas, bugs, danken, jdenemar, mburman, michal.skrivanek
Target Milestone: ovirt-4.2.2Keywords: Automation, Regression
Target Release: ---Flags: rule-engine: ovirt-4.2+
rule-engine: blocker+
danken: ci_coverage_complete?
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-29 11:17:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1554962    
Bug Blocks:    
Attachments:
Description Flags
Logs
none
domxml before the attempt to plug back none

Description Michael Burman 2018-03-07 14:43:49 UTC 
Created attachment 1405380 [details]
Logs

Description of problem:
Cannot plug vNIC after was unplugged 

Another new regression - after VM's vNIC was unplugged it's can't be plugged back - 
2018-03-07 16:39:19,951+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugNicVDSCommand] (default task-16) [41d2aaa4] NIC hot-set: <?xml version="1.0" encoding="UTF-8"?><hotplug>
  <devices>
    <interface type="bridge">
      <model type="virtio"/>
      <link state="up"/>
      <source bridge="ovirtmgmt"/>
      <alias name="ua-a17b8639-477b-43fd-8621-ea4c1f80745b"/>
      <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci"/>
      <mac address="00:00:00:00:00:50"/>
      <filterref filter="vdsm-no-mac-spoofing"/>
      <bandwidth/>
    </interface>
  </devices>
  <metadata xmlns:ovirt-vm="http://ovirt.org/vm/1.0">
    <ovirt-vm:vm>
      <ovirt-vm:device mac_address="00:00:00:00:00:50">
        <ovirt-vm:custom/>
      </ovirt-vm:device>
    </ovirt-vm:vm>
  </metadata>
</hotplug>

2018-03-07 16:39:21,325+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugNicVDSCommand] (default task-16) [41d2aaa4] Failed in 'HotPlugNicVDS' method
2018-03-07 16:39:21,333+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-16) [41d2aaa4] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM camel-vdsa.qa.lab.tlv.redhat.com command HotPlugNicVDS failed: XML error: non unique alias detected: ua-a17b8639-477b-43fd-8621-ea4c1f80745b


2018-03-07 16:39:21,233+0200 INFO  (jsonrpc/1) [virt.vm] (vmId='f8abc451-ad7d-4916-ae0b-fca5cfa643bb') Hotplug NIC xml: <?xml version='1.0' encoding='utf-8'?>
<interface type="bridge">
    <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci" />
    <mac address="00:00:00:00:00:50" />
    <model type="virtio" />
    <source bridge="ovirtmgmt" />
    <filterref filter="vdsm-no-mac-spoofing" />
    <alias name="ua-a17b8639-477b-43fd-8621-ea4c1f80745b" />
</interface>
 (vm:2947)
2018-03-07 16:39:21,321+0200 ERROR (jsonrpc/1) [virt.vm] (vmId='f8abc451-ad7d-4916-ae0b-fca5cfa643bb') Hotplug failed (vm:2953)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2951, in hotplugNic
    self._dom.attachDevice(nicXml)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 98, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 570, in attachDevice
    if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self)
libvirtError: XML error: non unique alias detected: ua-a17b8639-477b-43fd-8621-ea4c1f80745b

Version-Release number of selected component (if applicable):
4.2.2.2-0.1.el7
vdsm-4.20.20-1.el7ev.x86_64
kernel-3.10.0-858.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Start VM with 1 vNIC(ovirtmgmt)
2. Unplug vNIC
3. Try to plug it back

Actual results:
Error while executing action Edit VM Interface properties: Failed to activate VM Network Interface.

Expected results:
Must work
Comment 1 Dan Kenigsberg 2018-03-07 17:18:15 UTC 
When did this last run successfully? Does this affect 4.1-on-7.5 as well?
Would you try to reproduce it directly with virsh, and pass the buck to libvirt? I'd suspect the error lies there.
Comment 2 Jiri Denemark 2018-03-08 08:59:40 UTC 
It would be nice to see what version of libvirt is installed on the host. And the domain XML before the attempt to attach the interface would help too, since we need to make sure the interface was really removed before we try to plug it back in.

BTW, I just tried this with upstream libvirt and it works as expected.
Comment 3 Michael Burman 2018-03-08 09:20:09 UTC 
(In reply to Dan Kenigsberg from comment #1)
> When did this last run successfully? Does this affect 4.1-on-7.5 as well?
> Would you try to reproduce it directly with virsh, and pass the buck to
> libvirt? I'd suspect the error lies there.

Dan, 4.1 + 7.5 working as expected. so i guess it's on our side in 4.2
4.2.2-1 was the last successful.
Failed on 4.2.2.2
Comment 4 Michael Burman 2018-03-08 09:22:23 UTC 
(In reply to Jiri Denemark from comment #2)
> It would be nice to see what version of libvirt is installed on the host.
> And the domain XML before the attempt to attach the interface would help
> too, since we need to make sure the interface was really removed before we
> try to plug it back in.
> 
> BTW, I just tried this with upstream libvirt and it works as expected.

Libvirt version is 
libvirt-daemon-3.9.0-13.el7.x86_64
libvirt-client-3.9.0-13.el7.x86_64

domxml prior the attempt to plug(attaching now)
Comment 5 Michael Burman 2018-03-08 09:23:00 UTC 
Created attachment 1405769 [details]
domxml before the attempt to plug back
Comment 6 Jiri Denemark 2018-03-08 10:10:48 UTC 
I guess it's because 4.1 did not use user-defined aliases, which is a new libvirt feature in 7.5. In other words, it's a libvirt bug and I was able to reproduce it. See bug 1553075.
Comment 7 Arik 2018-03-08 10:26:57 UTC 
Michael, this issue blocks a very specific scenario: unplugging and then plugging a device from/to a running VM.

To be clear, hot-plugging a new device into the VM should work.
Also, hot-unplugging a device, shutting down the VM, and then plugging it back when the VM is down or when it is started again (hot-plug) should work.

I see two options here:
1. To disable user-aliases until libvirt is fixed.
2. To disable/change the automation test according to the above.

I strongly prefer #2. We know that we need user-aliases in 4.2 to support host-level hooks that change a property of the devices that we use for their identification (mac addresses in case of NICs, paths in case of disks).
Disabling user-aliases will prevent us from evaluating this change (for example, we now also know we have an issue with usb devices set with user-aliases in libvirt).
From the user point of view, that's an annoying regression but I'm not convinced it's that critical.

So can we change the automation test?
Comment 8 Michael Burman 2018-03-08 11:45:24 UTC 
(In reply to Arik from comment #7)
> Michael, this issue blocks a very specific scenario: unplugging and then
> plugging a device from/to a running VM.
> 
> To be clear, hot-plugging a new device into the VM should work.
> Also, hot-unplugging a device, shutting down the VM, and then plugging it
> back when the VM is down or when it is started again (hot-plug) should work.
> 
> I see two options here:
> 1. To disable user-aliases until libvirt is fixed.
> 2. To disable/change the automation test according to the above.
> 
> I strongly prefer #2. We know that we need user-aliases in 4.2 to support
> host-level hooks that change a property of the devices that we use for their
> identification (mac addresses in case of NICs, paths in case of disks).
> Disabling user-aliases will prevent us from evaluating this change (for
> example, we now also know we have an issue with usb devices set with
> user-aliases in libvirt).
> From the user point of view, that's an annoying regression but I'm not
> convinced it's that critical.
> 
> So can we change the automation test?

We will skip this tests until the libvirt bug will get fixed.
Comment 11 Red Hat Bugzilla Rules Engine 2018-03-09 21:41:41 UTC 
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
Comment 12 Dan Kenigsberg 2018-03-15 11:38:54 UTC 
libvirt-3.9.0-14.el7_5.1 should fix this, please test.
Comment 13 Michael Burman 2018-03-15 11:48:22 UTC 
Will test with the new d/s build
Comment 14 Michael Burman 2018-03-18 14:12:03 UTC 
QE now only have officially libvirt-client-3.9.0-14.el7.x86_64
libvirt-daemon-3.9.0-14.el7.x86_64 and it failedQA with this build.
I'm not going to verify this report with a brew version libvrti, only when we get.

So moving to NEW until this version will be available for QE.

FailedQA with 
libvirt-client-3.9.0-14.el7.x86_64
libvirt-daemon-3.9.0-14.el7.x86_64
vdsm-4.20.22-1.el7ev.x86_64
4.2.2.4-0.1.el7
Comment 15 Michal Skrivanek 2018-03-19 11:18:04 UTC 
(In reply to Michael Burman from comment #14)
> QE now only have officially libvirt-client-3.9.0-14.el7.x86_64
> libvirt-daemon-3.9.0-14.el7.x86_64 and it failedQA with this build.

What do you mean by "officially"?

> I'm not going to verify this report with a brew version libvrti, only when
> we get.

libvirt-3.9.0-14.el7_5.1 is available to QE

> 
> So moving to NEW until this version will be available for QE.

retest with version where the issue is fixed, please.
Comment 16 Michael Burman 2018-03-19 12:22:25 UTC 
(In reply to Michal Skrivanek from comment #15)
> (In reply to Michael Burman from comment #14)
> > QE now only have officially libvirt-client-3.9.0-14.el7.x86_64
> > libvirt-daemon-3.9.0-14.el7.x86_64 and it failedQA with this build.
> 
> What do you mean by "officially"?
> 
> > I'm not going to verify this report with a brew version libvrti, only when
> > we get.
> 
> libvirt-3.9.0-14.el7_5.1 is available to QE
> 
> > 
> > So moving to NEW until this version will be available for QE.
> 
> retest with version where the issue is fixed, please.

No it's not available, we have libvirt-client-3.9.0-14.el7.x86_64 only.
Will test when we get it. Not going to test until then, if you want to keep it ON_QA i don't mind. 
Thanks,
Comment 22 Michael Burman 2018-03-26 07:51:39 UTC 
Verified on - libvirt-daemon-3.9.0-14.el7_5.2.x86_64 and vdsm-4.20.23-1.el7ev.x86_64 with 

kernel-3.10.0-862.el7.x86_64

libvirt-daemon-driver-storage-iscsi-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-driver-interface-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-config-nwfilter-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-driver-storage-mpath-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-kvm-3.9.0-14.el7_5.2.x86_64
libvirt-python-3.9.0-1.el7.x86_64
libvirt-daemon-driver-storage-core-3.9.0-14.el7_5.2.x86_64
libvirt-client-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-driver-qemu-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-driver-storage-rbd-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-driver-storage-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-driver-secret-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-driver-nwfilter-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-driver-storage-scsi-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.2.x86_64
libvirt-lock-sanlock-3.9.0-14.el7_5.2.x86_64
libvirt-libs-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-config-network-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-driver-storage-logical-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-driver-network-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-driver-storage-disk-3.9.0-14.el7_5.2.x86_64
libvirt-daemon-driver-nodedev-3.9.0-14.el7_5.2.x86_64
Comment 23 Sandro Bonazzola 2018-03-29 11:17:17 UTC 
This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.