Bug 1412834

Summary: Handle empty <script path=""/> coming from guests generated on older Libvirt versions
Product: Red Hat Enterprise Linux 7 Reporter: Stephen Gordon <sgordon>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: yalzhang <yalzhang>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: berrange, dyuan, jsuchane, knoel, mprivozn, rbalakri, snagar, xuzhang
Target Milestone: rcKeywords: Reopened, ZStream
Target Release: 7.4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-3.1.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1429510 (view as bug list) Environment:
Last Closed: 2017-08-01 17:21:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1408464, 1416881, 1416884, 1416886, 1416887, 1429510, 1431252    

Description Stephen Gordon 2017-01-12 22:52:40 UTC
Description of problem:

When plugging virtual interfaces to guests OpenStack Compute (Nova), for some specific VIF types, has for some time created Libvirt XML that looks like this to represent the network device:

    <interface type='ethernet'>
      <mac address='02:b9:38:d2:ef:bf'/>
      <script path=''/>
      <target dev='tapb938d2ef-bf'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

At some point between the versions of Libvirt shipped in RHEL 7.2, and the version shipped in RHEL 7.3 (2.0.0) Libvirt started rejecting the empty path attribute to the script tag:

    <script path=''/>

When OpenStack operators upgraded to RHEL 7.3 (or the aligned CentOS update) they started hitting this:

    http://lists.openstack.org/pipermail/openstack-operators/2016-December/012357.html

Work is being done to stop Nova generating the empty path attribute and script element if it is not needed for the given interface:

    https://bugs.launchpad.net/nova/+bug/1649527

...but the problem we have is what to do with existing guests that were created on earlier versions of Nova and Libvirt which allowed this without complaint (even though technically it sounds like it was never truly valid Libvirt XML).

According to operator reports the mere presence of such a guest on a host is enough to make basic functionality like virsh list barf. Michal suggested it might be possible to make Libvirt behave more gracefully here for existing VMs:

"""Libvirt differentiates between XMLs loaded from an older libvirt (e.g.
during rpm update) and new instances being defined. Thus we can put
different accepting rules onto these two sources of XMLs. While in
define (=for newly created VMs) we can deny empty script path, we can
allow that if the XML was an outcome from an older libvirt.
"""

This would be ideal, then eventually as these VMs are rebuilt for whatever reason over time Nova will generate new XML for them.

Version-Release number of selected component (if applicable):

2.0.0

Comment 1 yalzhang@redhat.com 2017-01-17 07:59:11 UTC
I can reproduce
# rpm -q libvirt qemu-kvm-rhev
libvirt-1.2.17-13.el7.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.24.x86_64

1. Configure the qemu.conf by
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/App_Generic_Ethernet.html

2. # virsh dumpxml rhel7.2 | grep /interface -B6
......
    <interface type='ethernet'>
      <mac address='02:b9:38:d2:ef:bf'/>
      <script path=''/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>

# virsh start rhel7.2
Domain rhel7.2 started

update the qemu-kvm-rhev, libvirt
1. # rpm -q libvirt qemu-kvm-rhev
libvirt-2.5.0-1.el7.x86_64
qemu-kvm-rhev-2.8.0-2.el7.x86_64

2. # virsh dumpxml rhel7.3 | grep /interface -B7
......
    <interface type='ethernet'>
      <mac address='02:b9:38:d2:ef:bf'/>
      <script path=''/>
      <target dev='tapb938d2ef-bf'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
# virsh start rhel7.3
error: Failed to start domain rhel7.3
error: Cannot find '' in path: No such file or directory

Comment 3 Stephen Gordon 2017-01-26 17:00:58 UTC
Michal is there anything posted on the upstream libvirt list we can point to here in terms of WIP?

Comment 4 Michal Privoznik 2017-02-02 13:51:40 UTC
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2017-February/msg00041.html

Comment 5 Jaroslav Suchanek 2017-02-09 14:31:02 UTC
Per an upstream discussion I am closing this as notabug.

Comment 6 Stephen Gordon 2017-02-09 15:04:56 UTC
(In reply to Jaroslav Suchanek from comment #5)
> Per an upstream discussion I am closing this as notabug.

The reason RHEL and CentOS users and partners were only hit by this now is that the versions in 7.2 and 7.3 respectively were:

7.2: libvirt-1.2.17-13.el7.x86_64.rpm  
7.3: libvirt-2.0.0-10.el7.x86_64.rpm 

So yes, this was introduced in v1.3.2 but it only hit folks when v2.0.0 came out with 7.3.

OpenStack Nova *has* been patched to omit the errant XML when creating new guests the problem is we have existing customer environments using impacted SDN solutions with running guests that already have such XML. In such production environments if they update to 7.3 they can not even virsh list and we can not force a workload outage to rebuild the guests on a `yum update`.

Comment 13 Michal Privoznik 2017-02-16 16:52:50 UTC
Moving to POST:

commit 1d9ab0f04af310e52f80b4281751655bb3bb7601
Author:     Michal Privoznik <mprivozn>
AuthorDate: Thu Feb 2 14:16:20 2017 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Thu Feb 16 17:39:34 2017 +0100

    qemu: Allow empty script path to <interface/>
    
    Before 9c17d665fdc5f (v1.3.2 - I know, right?) it was possible to
    have the following interface configuration:
    
      <interface type='ethernet'/>
        <script path=''/>
      </interface>
    
    This resulted in -netdev tap,script=,.. Fortunately, qemu helped
    us to get away with this as it just ignored the empty script
    path. However, after the commit mentioned above it's libvirtd
    who is executing the script. Unfortunately without special
    case-ing empty script path.
    
    Signed-off-by: Michal Privoznik <mprivozn>

v3.0.0-202-g1d9ab0f04

Comment 14 yalzhang@redhat.com 2017-02-17 10:03:04 UTC
Verified on upstream, the result is as expected. The guest can start successfully with <script path=''/>.

# rpm -q libvirt qemu-kvm-rhev
libvirt-3.1.0-1.el7.x86_64
qemu-kvm-rhev-2.8.0-4.el7.x86_64

1. Edit the guest's xml
   <interface type='ethernet'>
      <mac address='02:b9:38:d2:ef:bf'/>
      <script path=''/>
      <target dev='tapb938d2ef-bf'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

# virsh edit rhel7.3
error: XML document failed to validate against schema: Unable to validate doc against /usr/share/libvirt/schemas/domain.rng
Extra element devices in interleave
Element domain failed to validate content

Failed. Try again? [y,n,i,f,?]:  =========> press 'i'
Domain rhel7.3 XML configuration edited.

2. 
# virsh start rhel7.3
Domain rhel7.3 started

# virsh dumpxml rhel7.3 | grep /interface -B8
    <interface type='ethernet'>
      <mac address='02:b9:38:d2:ef:bf'/>
      <script path=''/>
      <target dev='tapb938d2ef-bf'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

Comment 17 yalzhang@redhat.com 2017-03-06 08:19:13 UTC
Test on libvirt-3.1.0-1.el7.x86_64 by the steps in #c14, no difference from #c14.
Set this bug as verified.

Comment 19 errata-xmlrpc 2017-08-01 17:21:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 20 errata-xmlrpc 2017-08-02 00:01:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846