Hide Forgot
Description of problem: When plugging virtual interfaces to guests OpenStack Compute (Nova), for some specific VIF types, has for some time created Libvirt XML that looks like this to represent the network device: <interface type='ethernet'> <mac address='02:b9:38:d2:ef:bf'/> <script path=''/> <target dev='tapb938d2ef-bf'/> <model type='virtio'/> <driver name='qemu'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> At some point between the versions of Libvirt shipped in RHEL 7.2, and the version shipped in RHEL 7.3 (2.0.0) Libvirt started rejecting the empty path attribute to the script tag: <script path=''/> When OpenStack operators upgraded to RHEL 7.3 (or the aligned CentOS update) they started hitting this: http://lists.openstack.org/pipermail/openstack-operators/2016-December/012357.html Work is being done to stop Nova generating the empty path attribute and script element if it is not needed for the given interface: https://bugs.launchpad.net/nova/+bug/1649527 ...but the problem we have is what to do with existing guests that were created on earlier versions of Nova and Libvirt which allowed this without complaint (even though technically it sounds like it was never truly valid Libvirt XML). According to operator reports the mere presence of such a guest on a host is enough to make basic functionality like virsh list barf. Michal suggested it might be possible to make Libvirt behave more gracefully here for existing VMs: """Libvirt differentiates between XMLs loaded from an older libvirt (e.g. during rpm update) and new instances being defined. Thus we can put different accepting rules onto these two sources of XMLs. While in define (=for newly created VMs) we can deny empty script path, we can allow that if the XML was an outcome from an older libvirt. """ This would be ideal, then eventually as these VMs are rebuilt for whatever reason over time Nova will generate new XML for them. Version-Release number of selected component (if applicable): 2.0.0
I can reproduce # rpm -q libvirt qemu-kvm-rhev libvirt-1.2.17-13.el7.x86_64 qemu-kvm-rhev-2.3.0-31.el7_2.24.x86_64 1. Configure the qemu.conf by https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/App_Generic_Ethernet.html 2. # virsh dumpxml rhel7.2 | grep /interface -B6 ...... <interface type='ethernet'> <mac address='02:b9:38:d2:ef:bf'/> <script path=''/> <model type='virtio'/> <driver name='qemu'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </interface> # virsh start rhel7.2 Domain rhel7.2 started update the qemu-kvm-rhev, libvirt 1. # rpm -q libvirt qemu-kvm-rhev libvirt-2.5.0-1.el7.x86_64 qemu-kvm-rhev-2.8.0-2.el7.x86_64 2. # virsh dumpxml rhel7.3 | grep /interface -B7 ...... <interface type='ethernet'> <mac address='02:b9:38:d2:ef:bf'/> <script path=''/> <target dev='tapb938d2ef-bf'/> <model type='virtio'/> <driver name='qemu'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </interface> # virsh start rhel7.3 error: Failed to start domain rhel7.3 error: Cannot find '' in path: No such file or directory
Michal is there anything posted on the upstream libvirt list we can point to here in terms of WIP?
Patch proposed upstream: https://www.redhat.com/archives/libvir-list/2017-February/msg00041.html
Per an upstream discussion I am closing this as notabug.
(In reply to Jaroslav Suchanek from comment #5) > Per an upstream discussion I am closing this as notabug. The reason RHEL and CentOS users and partners were only hit by this now is that the versions in 7.2 and 7.3 respectively were: 7.2: libvirt-1.2.17-13.el7.x86_64.rpm 7.3: libvirt-2.0.0-10.el7.x86_64.rpm So yes, this was introduced in v1.3.2 but it only hit folks when v2.0.0 came out with 7.3. OpenStack Nova *has* been patched to omit the errant XML when creating new guests the problem is we have existing customer environments using impacted SDN solutions with running guests that already have such XML. In such production environments if they update to 7.3 they can not even virsh list and we can not force a workload outage to rebuild the guests on a `yum update`.
Moving to POST: commit 1d9ab0f04af310e52f80b4281751655bb3bb7601 Author: Michal Privoznik <mprivozn@redhat.com> AuthorDate: Thu Feb 2 14:16:20 2017 +0100 Commit: Michal Privoznik <mprivozn@redhat.com> CommitDate: Thu Feb 16 17:39:34 2017 +0100 qemu: Allow empty script path to <interface/> Before 9c17d665fdc5f (v1.3.2 - I know, right?) it was possible to have the following interface configuration: <interface type='ethernet'/> <script path=''/> </interface> This resulted in -netdev tap,script=,.. Fortunately, qemu helped us to get away with this as it just ignored the empty script path. However, after the commit mentioned above it's libvirtd who is executing the script. Unfortunately without special case-ing empty script path. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> v3.0.0-202-g1d9ab0f04
Verified on upstream, the result is as expected. The guest can start successfully with <script path=''/>. # rpm -q libvirt qemu-kvm-rhev libvirt-3.1.0-1.el7.x86_64 qemu-kvm-rhev-2.8.0-4.el7.x86_64 1. Edit the guest's xml <interface type='ethernet'> <mac address='02:b9:38:d2:ef:bf'/> <script path=''/> <target dev='tapb938d2ef-bf'/> <model type='virtio'/> <driver name='qemu'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> # virsh edit rhel7.3 error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng Extra element devices in interleave Element domain failed to validate content Failed. Try again? [y,n,i,f,?]: =========> press 'i' Domain rhel7.3 XML configuration edited. 2. # virsh start rhel7.3 Domain rhel7.3 started # virsh dumpxml rhel7.3 | grep /interface -B8 <interface type='ethernet'> <mac address='02:b9:38:d2:ef:bf'/> <script path=''/> <target dev='tapb938d2ef-bf'/> <model type='virtio'/> <driver name='qemu'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
Test on libvirt-3.1.0-1.el7.x86_64 by the steps in #c14, no difference from #c14. Set this bug as verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1846