Bug 1553075 - User-aliases not cleared on detach-device [NEEDINFO]
Summary: User-aliases not cleared on detach-device
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.6
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: Han Han
URL:
Whiteboard:
Depends On: 1543775
Blocks: 1554962
TreeView+ depends on / blocked
 
Reported: 2018-03-08 09:00 UTC by Arik
Modified: 2018-10-30 09:55 UTC (History)
13 users (show)

Fixed In Version: libvirt-4.3.0-1.el7
Doc Type: Bug Fix
Doc Text:
Previously, the libvirt service used the incorrect XML configuration when checking for a duplicate user alias. As a consequence, hot-plugging a device with a unique alias to a guest virtual machine in some cases failed with a "non unique alias" error message. With this update, libvirt uses the correct XML files for verifying the device alias, which prevents the problem from occurring.
Clone Of:
: 1554962 (view as bug list)
Environment:
Last Closed: 2018-10-30 09:53:14 UTC
Target Upstream Version:
michal.skrivanek: needinfo? (spower)


Attachments (Terms of Use)
domain xml after starting the vm (9.64 KB, text/plain)
2018-03-08 09:17 UTC, Arik
no flags Details
domain xml after unplugging the nic (9.09 KB, text/plain)
2018-03-08 09:17 UTC, Arik
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:3113 None None None 2018-10-30 09:55:20 UTC
Red Hat Bugzilla 1553162 None CLOSED Attaching device to running domain fails with "non unique alias detected: usb" 2019-09-15 12:35:52 UTC

Internal Links: 1553162

Description Arik 2018-03-08 09:00:51 UTC
Description of problem:
Devices that were set with user-aliases cannot be plugged (attached) after being unplugged (detached) from a VM.

Version-Release number of selected component (if applicable):
I tested it with libvirt 4.0.0.2.fc27 but it also happens on rhel 7.5.

How reproducible:
100%

Steps to Reproduce:
1. Create a VM with network interface/disk set with user-alias (e.g., ua-e3e109ca-a89a-45af-b240-503efeb497ef)
2. Unplug the device (virsh detach-device myvm nic.xml)
3. Try to plug the device (virsh attach-device myvm nic.xml)

Actual results:
The following error:
XML Error: non unique alias detected

Expected results:
The device should be plugged back.

Additional info:
my nic.xml:
<interface type='bridge'>
      <mac address='00:1a:4a:16:01:00'/>
      <source bridge='ovirtmgmt'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <filterref filter='vdsm-no-mac-spoofing'/>
      <alias name='ua-e3e109ca-a89a-45af-b240-503efeb497ef'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

Setting this to severity to urgent because it is required in rhel 7.5 in order to keep supporting host-level hooks in oVirt that change mac addresses or paths of disks in the upcoming 4.2 version of oVirt.

Comment 2 Jiri Denemark 2018-03-08 09:05:27 UTC
Could you attach the domain XML after step 2?

Comment 3 Arik 2018-03-08 09:17:05 UTC
Created attachment 1405767 [details]
domain xml after starting the vm

Comment 4 Arik 2018-03-08 09:17:31 UTC
Created attachment 1405768 [details]
domain xml after unplugging the nic

Comment 5 Arik 2018-03-08 09:19:06 UTC
(In reply to Jiri Denemark from comment #2)
> Could you attach the domain XML after step 2?

Done.
And the error on step 3 is:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2907, in hotplugNic
    self._dom.attachDevice(nicXml)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 98, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 570, in attachDevice
    if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self)
libvirtError: XML error: non unique alias detected: ua-e3e109ca-a89a-45af-b240-503efeb497ef

Comment 6 Jiri Denemark 2018-03-08 09:59:39 UTC
I was able to reproduce this. The important part is that the device must be present in the persistent config and detached from the live one. Then virsh attach-device *--live* fails because it finds a conflict with the persistent definition. This makes no sense.

Comment 9 Peter Krempa 2018-03-08 11:16:02 UTC
It looks like the following function is the problem ...

int
virDomainDeviceValidateAliasForHotplug(virDomainObjPtr vm,
                                       virDomainDeviceDefPtr dev,
                                       unsigned int flags)
{
    virDomainDefPtr persDef = NULL;
    virDomainDefPtr liveDef = NULL;

    if (virDomainObjGetDefs(vm, flags, &liveDef, &persDef) < 0)
        return -1;

    if (persDef &&
        virDomainDeviceValidateAliasImpl(vm->def, dev) < 0)
        return -1;

As vm->def is in fact the live definition if the VM is running ...


    if (liveDef &&
        virDomainDeviceValidateAliasImpl(vm->newDef, dev) < 0)

vm->newDef is the persistent definition here. It also may be NULL for transient VMs, so this will also crash in certain configs.

        return -1;

    return 0;
}

Comment 10 Michal Privoznik 2018-03-09 11:57:20 UTC
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2018-March/msg00466.html

Comment 11 Michal Privoznik 2018-03-12 12:29:14 UTC
I've just merged the patch upstream:

commit e5673ed44465edcfccdd906158136ad8dcb94b9b
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Fri Mar 9 08:31:44 2018 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Mon Mar 12 13:27:24 2018 +0100

    virDomainDeviceValidateAliasForHotplug: Use correct domain defintion
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1553075
    
    For some weird reason this function is getting live and
    persistent def for domain but then accesses vm->def and
    vm->newDef directly. This is rather unsafe as we can be
    accessing NULL pointer.
    
    Signed-off-by: Michal Privoznik <mprivozn@redhat.com>


v4.1.0-91-ge5673ed44

Comment 13 Jiri Denemark 2018-03-12 12:45:14 UTC
Bug 1543775 needs to be fixed too, otherwise this patch will break device hotplug even more.

Comment 14 jiyan 2018-03-13 02:42:54 UTC
Paste the reproducing steps in pure libvirt here.

Version:
kernel-3.10.0-860.el7.x86_64
qemu-kvm-rhev-2.10.0-21.el7.x86_64
libvirt-3.9.0-14.el7.x86_64

Steps to reproduce:
1. Prepare a shutdown VM with the following conf:
# virsh domstate test1
shut off

# virsh dumpxml test1 --inactive |grep "<interface" -A10
    <interface type='bridge'>
      <mac address='00:1a:4a:16:01:00'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <filterref filter='vdsm-no-mac-spoofing'/>
      <alias name='ua-e3e109ca-a89a-45af-b240-503efeb497ef'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

2. Start VM and hot-unplug the interface device
# cat nic.xml 
<interface type='bridge'>
      <mac address='00:1a:4a:16:01:00'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <filterref filter='vdsm-no-mac-spoofing'>
      </filterref>
      <alias name='ua-e3e109ca-a89a-45af-b240-503efeb497ef'/>
</interface>

# virsh start test1
Domain test1 started

# virsh detach-device test1 nic.xml 
Device detached successfully

# virsh dumpxml test1 |grep "<interface" -A10
    <interface type='bridge'>
      <mac address='00:1a:4a:16:01:00'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <filterref filter='vdsm-no-mac-spoofing'/>
      <alias name='ua-e3e109ca-a89a-45af-b240-503efeb497ef'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

3. Hot-plug the interface again
# virsh attach-device test1 nic.xml 
error: Failed to attach device from nic.xml
error: XML error: non unique alias detected: ua-e3e109ca-a89a-45af-b240-503efeb497ef

Comment 16 jiyan 2018-03-13 13:42:37 UTC
Paste another scenario related to this issue here, which can be considered when verifying this bug.

Description:
Cold-plugging disk with same alias that has existed in another 'optional' disk in dumpxml of VM do not raise error

Version:
kernel-3.10.0-860.el7.x86_64
qemu-kvm-rhev-2.10.0-21.el7.x86_64
libvirt-3.9.0-14.virtcov.el7.x86_64

Steps:
1. Prepare a shutdown VM with the following VM, and start VM to confirm there is no error info raised.
# ll /var/lib/libvirt/images/
total 7054660
-rw-r--r--. 1 root root 1442119680 Feb 27 04:01 optionalRHEL-7.5-x86_64-latest.qcow2
-rw-------. 1 root root 1428029440 Mar 11 23:16 test1RHEL-7.5-x86_64-latest-clone.qcow2

# virsh domstate test1
shut off

# virsh dumpxml test1 --inactive grep "<disk" -A7
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test1RHEL-7.5-x86_64-latest.qcow2'/>
      <target dev='hda' bus='usb'/>
      <alias name='ua-disk1'/>
      <address type='usb' bus='0' port='1.3'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/0.qcow2' startupPolicy='optional'/>
      <target dev='hdc' bus='usb'/>
      <alias name='ua-disk2'/>
      <address type='usb' bus='0' port='1.4'/>
    </disk>

# virsh start test1
Domain test1 started

# virsh dumpxml test1 |grep "<disk" -A7
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test1RHEL-7.5-x86_64-latest.qcow2'/>
      <backingStore/>
      <target dev='hda' bus='usb'/>
      <alias name='ua-disk1'/>
      <address type='usb' bus='0' port='1.3'/>
    </disk>

2. Hot-plug / Cold-plug a virtual disk with alias that is same with 'ua-disk2', hot-plugging failed while cold-plugging succeeded
# cat disk.xml 
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/optionalRHEL-7.5-x86_64-latest.qcow2'/>
      <target dev='hdb' bus='usb'/>
      <alias name='ua-disk2'/>
    </disk>

# virsh attach-device test1 disk.xml 
error: Failed to attach device from disk.xml
error: XML error: non unique alias detected: ua-disk2

# virsh attach-device test1 disk.xml --config
Device attached successfully

3. Restart VM 
# virsh destroy test1
Domain test1 destroyed

# virsh start test1
error: Failed to start domain test1
error: XML error: non unique alias detected: ua-disk2

# virsh dumpxml test1 --inactive|grep "<disk" -A7
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test1RHEL-7.5-x86_64-latest.qcow2'/>
      <target dev='hda' bus='usb'/>
      <alias name='ua-disk1'/>
      <address type='usb' bus='0' port='1.3'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/optionalRHEL-7.5-x86_64-latest.qcow2'/>
      <target dev='hdb' bus='usb'/>
      <alias name='ua-disk2'/>
      <address type='usb' bus='0' port='1.5'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/0.qcow2' startupPolicy='optional'/>
      <target dev='hdc' bus='usb'/>
      <alias name='ua-disk2'/>
      <address type='usb' bus='0' port='1.4'/>
    </disk>

Comment 20 Han Han 2018-07-10 02:34:43 UTC
Verified on libvirt-4.5.0-2.virtcov.el7.x86_64 qemu-kvm-rhev-2.12.0-7.el7.x86_64:
Prepare a running VM and nic xml with alias:
# virsh list --all 
 Id    Name                           State
----------------------------------------------------
 63    nest7.6-1                      running

# cat /tmp/nic.xml 
    <interface type='network'>
      <mac address='52:54:00:72:7a:34'/>
      <source network='default'/>
      <model type='virtio'/>
      <alias name='ua-xxx'/>
    </interface>

I. Live attach and detach twice:

Comment 21 Han Han 2018-07-10 02:50:26 UTC
Sorry for the former incomplete message.

Verified on libvirt-4.5.0-2.virtcov.el7.x86_64 qemu-kvm-rhev-2.12.0-7.el7.x86_64:
Prepare a running VM and nic xml with alias:
# virsh list --all 
 Id    Name                           State
----------------------------------------------------
 63    nest7.6-1                      running

# cat /tmp/nic.xml 
    <interface type='network'>
      <mac address='52:54:00:72:7a:34'/>
      <source network='default'/>
      <model type='virtio'/>
      <alias name='ua-09f558d3-1223-4e82-9334-87b1eeb26925'/>
    </interface>

I. Live attach twice, then detach once, attach once and detach twice at last:
+ virsh attach-device nest7.6-1 /tmp/nic.xml
Device attached successfully

+ virsh attach-device nest7.6-1 /tmp/nic.xml
error: Failed to attach device from /tmp/nic.xml
error: XML error: non unique alias detected: ua-09f558d3-1223-4e82-9334-87b1eeb26925

+ virsh detach-device nest7.6-1 /tmp/nic.xml
Device detached successfully

+ virsh attach-device nest7.6-1 /tmp/nic.xml
Device attached successfully

+ virsh detach-device nest7.6-1 /tmp/nic.xml
Device detached successfully

+ virsh detach-device nest7.6-1 /tmp/nic.xml
error: Failed to detach device from /tmp/nic.xml
error: device not found: no device matching MAC address 52:54:00:72:7a:34 found

II. Test as scenario I with --config:
+ virsh attach-device nest7.6-1 /tmp/nic.xml --config
Device attached successfully

+ virsh attach-device nest7.6-1 /tmp/nic.xml --config
error: Failed to attach device from /tmp/nic.xml
error: XML error: non unique alias detected: ua-09f558d3-1223-4e82-9334-87b1eeb26925

+ virsh detach-device nest7.6-1 /tmp/nic.xml --config
Device detached successfully

+ virsh attach-device nest7.6-1 /tmp/nic.xml --config
Device attached successfully

+ virsh detach-device nest7.6-1 /tmp/nic.xml --config
Device detached successfully

+ virsh detach-device nest7.6-1 /tmp/nic.xml --config
error: Failed to detach device from /tmp/nic.xml
error: device not found: no device matching MAC address 52:54:00:72:7a:34 found


III. Test as scenario I with --persistent:
+ virsh attach-device nest7.6-1 /tmp/nic.xml --persistent
Device attached successfully

+ virsh attach-device nest7.6-1 /tmp/nic.xml --persistent
error: Failed to attach device from /tmp/nic.xml
error: XML error: non unique alias detected: ua-09f558d3-1223-4e82-9334-87b1eeb26925

+ virsh detach-device nest7.6-1 /tmp/nic.xml --persistent
Device detached successfully

+ virsh attach-device nest7.6-1 /tmp/nic.xml --persistent
Device attached successfully

+ virsh detach-device nest7.6-1 /tmp/nic.xml --persistent
Device detached successfully

+ virsh detach-device nest7.6-1 /tmp/nic.xml --persistent
error: Failed to detach device from /tmp/nic.xml
error: device not found: no device matching MAC address 52:54:00:72:7a:34 found

Works as expected. Verified.

Comment 23 errata-xmlrpc 2018-10-30 09:53:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113


Note You need to log in before you can comment on or make changes to this bug.