Bug 1502754

Summary:	log an "unsupported" error when an interface connected to an OVS bridge has a nwfilter rule
Product:	Red Hat Enterprise Linux 7	Reporter:	Dominik Holler <dholler>
Component:	libvirt	Assignee:	Laine Stump <laine>
Status:	CLOSED ERRATA	QA Contact:	yalzhang <yalzhang>
Severity:	unspecified	Docs Contact:
Priority:	medium
Version:	7.4	CC:	dyuan, jdenemar, jinqi, jsuchane, laine, xuzhang, yalzhang
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-4.5.0-26.el7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-31 19:58:29 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Dominik Holler 2017-10-16 15:18:49 UTC

Description of problem: 
libvirt's network filtering is applied by ebtables/iptables, which does not work for OpenVSwitch bridges. For this reason, libvirt should not accept network filters for openvswitch ports if the filters are not applied.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create a VM with a network interface with virtualport of type openvswitch with clean-traffic filter, e.g. like this

    <interface type='bridge'>
      <mac address='00:1a:4a:16:01:53'/>
      <source bridge='br-int'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='5d26b673-debd-442f-ba20-e0998253e1f9'/>
      </virtualport>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <filterref filter='clean-traffic'/>
      <link state='up'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>

A second interface is useful to get the correct behavior:

    <interface type='bridge'>
      <mac address='00:1a:4a:16:01:51'/>
      <source bridge='ovirtmgmt'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <filterref filter='clean-traffic'/>
      <link state='up'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
2. Check if network filtering works, e.g. by assign two IP addresses to each interface and ping them from outside.
3.

Actual results:
On the second network interface the clean-traffic network filter is applied, so only one IP address replies the ping. On the openvswitch network interface, the network filter is not applied and both IP addresses reply to the ping.

Expected results:
Either the network filter is applied or libvirt refuses to start the configuration.

Additional info:

Comment 2 Laine Stump 2017-10-27 10:20:30 UTC

This is another of those items that can't always be properly validated when a domain is defined, because the connection details of an <interface type='network'> aren't known until runtime. We can of course validate *some* types of interfaces at definition time, and for the rest we'll need to rely on a check at runtime once the actual interface type has been determined.

Note that this will result in some previously-working domains to fail to start, which could upset some users. But then I guess that's the whole point of this exercise :-)

Comment 3 Laine Stump 2017-12-13 19:27:21 UTC

Proposed fix posted upstream:

https://www.redhat.com/archives/libvir-list/2017-December/msg00482.html

A warning: This fix would prevent existing guests with both <virtualport> and <filterref> from starting, which could be disruptive in layered products if <filterref> is being unconditionally added  even when the interface is connected directly via OVS.

Comment 4 Laine Stump 2017-12-14 19:43:24 UTC

Pushed upstream (will be in 4.0.0):

commit 16a9a28129993d87639a22e4799557e15c22ac1c
Author: Laine Stump <laine>
Date:   Tue Dec 12 16:18:07 2017 -0500

    qemu: log error on attempts to set filterref on an OVS-connected interface

Comment 7 yalzhang@redhat.com 2018-05-30 08:33:44 UTC

Test below scenarios, no check when update-device and hotplug
1. start vm ---> PASS
2. update-device to add the filterref ---> FAIL
3. hotplug device with such config ---> FAIL

Test on libvirt-4.3.0-1.el7.x86_64:
1. vm with interface connected to ovs bridge
1). Start vm:
# virsh dumpxml rhel | grep /interface -B10
    <interface type='bridge'>
      <mac address='52:54:00:be:d8:82'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='71db5b4e-50ab-4db0-bc4a-49be2f2e5dca'/>
      </virtualport>
      <model type='rtl8139'/>
      <filterref filter='no-mac-broadcast'/>
      <alias name='ua-88ee9909-3d84-4414-9ca3-51f4c00370d1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>

# virsh start rhel
error: Failed to start domain rhel
error: unsupported configuration: filterref is not supported for network interfaces with virtualport type openvswitch

2). Hotplug device with such unsupported config:
Start a vm without any interface, prepare xml with the config and attach-device 
# virsh start rhel
Domain rhel started

# cat interface.xml
<interface type='bridge'>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'/>
     <filterref filter='no-mac-broadcast'/>
    </interface>

# virsh attach-device rhel interface.xml
Device attached successfully

# virsh dumpxml rhel | grep /interface -B11
    <interface type='bridge'>
      <mac address='52:54:00:52:db:f2'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='a4988d0f-1ff4-451e-b132-b3e6e013751d'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='rtl8139'/>
      <filterref filter='no-mac-broadcast'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>

3). Update-device to add filterref:
 a. Start a vm with bridge type interface connected to ovs bridge;
 b. dump the interface xml into a file, then add the <filterref/>:
# cat interface.xml
<interface type='bridge'>
      <mac address='52:54:00:52:db:f2'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='a4988d0f-1ff4-451e-b132-b3e6e013751d'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <filterref filter='no-mac-broadcast'/>
      <alias name='ua-b5134819-12c9-41d8-b175-1fac353d4238'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>

# virsh update-device rhel interface.xml
Device updated successfully

# virsh dumpxml rhel | grep /interface -B9
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='a4988d0f-1ff4-451e-b132-b3e6e013751d'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <filterref filter='no-mac-broadcast'/>


2. vm with network type interface connected to ovs bridge:
1) start the vm
# virsh net-dumpxml ovs-bridge
<network>
  <name>ovs-bridge</name>
  <uuid>06fa1b11-2c74-484b-9ce2-bc6375930d25</uuid>
  <forward mode='bridge'/>
  <bridge name='ovsbr0'/>
  <virtualport type='openvswitch'/>
</network>
# virsh dumpxml rhel |grep /interface -B6
    <interface type='network'>
      <mac address='52:54:00:17:32:53'/>
      <source network='ovs-bridge'/>
      <model type='rtl8139'/>
      <filterref filter='no-mac-broadcast'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>

# virsh start rhel
error: Failed to start domain rhel
error: unsupported configuration: filterref is not supported for network interfaces with virtualport type openvswitch

2)update-device
delete the filterref setting, start the vm and do update-devie 
# virsh dumpxml rhel | grep /interface -B10
    <interface type='bridge'>
      <mac address='52:54:00:17:32:53'/>
      <source network='ovs-bridge' bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='c1e9bb71-6b03-40ef-916b-8d91eb606f5f'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>

# cat interface.xml
<interface type='bridge'>
      <mac address='52:54:00:17:32:53'/>
      <source network='ovs-bridge' bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='c1e9bb71-6b03-40ef-916b-8d91eb606f5f'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <filterref filter='clean-traffic'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>

# virsh update-device rhel interface.xml
error: Failed to update device from interface.xml
error: Operation not supported: unable to change config on 'bridge' network type

3) hotplug
start vm without any interface
prepare xml as below, then hotplug
# cat interface2.xml
<interface type='network'>
    <source network='ovs-bridge'/>
    <filterref filter='clean-traffic'/>
    </interface>

# virsh attach-device rhel interface2.xml
Device attached successfully

# virsh dumpxml rhel | grep /interface -B11
    <interface type='bridge'>
      <mac address='52:54:00:5a:9b:60'/>
      <source network='ovs-bridge' bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='7510e8ed-f04e-4c83-aa39-98cb38fbe432'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='rtl8139'/>
      <filterref filter='clean-traffic'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>

Comment 8 Laine Stump 2018-07-06 16:02:31 UTC

Yes, I think you are correct - The validation appears to not be done for update-device and hotplug. I'm moving it back to ASSIGNED and will look at it when I return from PTO.

Comment 10 Laine Stump 2019-03-04 02:22:43 UTC

The current check is in the wrong place - it should be lower in the call chain, in a place that is common not just to qemu commandline and hotplug and qemu netdev update, but also a place common to other hypervisor drivers. 

Ideally this would happen in the nwfilter driver itself (in the nwfilter driver's virNWFilterInstantiateFilter() maybe). The problem is that by the time we get in the nwfilter driver, we only have the filter binding, which just contains the name of the interface and the name of the filter, but no information about the type of that interface, or what it is connected to. This could either be solved by adding info to the binding, or by using virNetDev*() functions in the nwfilter driver to learn the necessary information.

Failing that, the other place common to all of these is virDomainConfNWFilterInstantiate(). This function *does* get the entire virDomainNetDefPtr, but I'm not really comfortable with the idea of putting code in a function in the conf directory that knows technical details about interface types, etc. (Possibly I'm being too strict about the restrictions; I need to think about it a bit).

Comment 12 Laine Stump 2019-09-16 00:22:46 UTC

These three commits were pushed upstream to fix the problem for hotplug and device update:

commit e3fb443dce69a28554387936faca7977a05aa739
Author: Laine Stump <laine>
Date:   Thu Sep 12 14:56:41 2019 -0400

    conf: make arg to virDomainNetGetActualVirtPortProfile() a const

commit 5366cd83bb7b1718d6a609d4a50044337988b69e
Author: Laine Stump <laine>
Date:   Thu Sep 12 18:25:21 2019 -0400

    qemu: move runtime netdev validation into a separate function
    
commit ac71a071c8c495339b94e3465c707ba5843ad156
Author: Laine Stump <laine>
Date:   Thu Sep 12 21:22:30 2019 -0400

    qemu: call common NetDef validation for hotplug and device update

Comment 15 yalzhang@redhat.com 2019-09-20 08:00:43 UTC

Test on libvirt-4.5.0-26.el7.x86_64, the result is as expected, set the bug to be verified.
1. start vm:
# virsh dumpxml rhel7 | grep /interface -B10
    <interface type='bridge'>
      <mac address='52:54:00:be:d8:82'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='a211923a-0a5f-443b-ad9c-074c15722c7d'/>
      </virtualport>
      <model type='virtio'/>
      <filterref filter='no-mac-broadcast'/>
      <alias name='ua-88ee9909-3d84-4414-9ca3-51f4c00370d1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
# virsh start rhel7
error: Failed to start domain rhel7
error: unsupported configuration: filterref is not supported for network interfaces with virtualport type openvswitch

# grep error /var/log/libvirt/libvirtd.log
2019-09-20 03:38:08.709+0000: 31686: error : qemuDomainValidateActualNetDef:4417 : unsupported configuration: filterref is not supported for network interfaces with virtualport type openvswitch
2019-09-20 03:38:08.721+0000: 31686: error : virCommandWait:2623 : internal error: Child process (ovs-vsctl --timeout=5 -- --if-exists del-port) unexpected exit status 1: ovs-vsctl: 'del-port' command requires at least 1 arguments
2019-09-20 03:38:08.721+0000: 31686: error : virNetDevOpenvswitchRemovePort:239 : internal error: Unable to delete port (null) from OVS

# ovs-vsctl show
3b3cd2a0-807e-4d7d-8412-a655471e36cc
    Bridge "ovsbr0"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
    ovs_version: "2.9.0"

2. hotplug:
# cat interface.xml
 <interface type='bridge'>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
      </virtualport>
      <model type='virtio'/>
      <filterref filter='no-mac-broadcast'/>
      <alias name='ua-88ee9909-3d84-4414-9ca3-51f4c00370d1'/>
    </interface>

# virsh attach-device rhel7 interface.xml
error: Failed to attach device from interface.xml
error: unsupported configuration: filterref is not supported for network interfaces with virtualport type openvswitch

3. live update interface:
# virsh dumpxml rhel7 | grep /interface -B10
    <interface type='bridge'>
      <mac address='52:54:00:22:1e:42'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='b8aaa0f6-2fc5-4d85-9277-3dd6b10e1d40'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='ua-88ee9909-3d84-4414-9ca3-51f4c00370d1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
# cat interface.xml
    <interface type='bridge'>
      <mac address='52:54:00:22:1e:42'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='b8aaa0f6-2fc5-4d85-9277-3dd6b10e1d40'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='ua-88ee9909-3d84-4414-9ca3-51f4c00370d1'/>
<filterref filter='clean-traffic'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
# virsh update-device rhel7 interface.xml
error: Failed to update device from interface.xml
error: unsupported configuration: filterref is not supported for network interfaces with virtualport type openvswitch


with networks:
1. start vm
# virsh net-dumpxml ovs-net
<network>
  <name>ovs-net</name>
  <uuid>9638858b-4762-4000-9f73-303a2360b15a</uuid>
  <forward mode='bridge'/>
  <bridge name='ovsbr0'/>
  <vlan trunk='yes'>
    <tag id='42' nativeMode='untagged'/>
    <tag id='47'/>
  </vlan>
  <virtualport type='openvswitch'>
    <parameters interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
  </virtualport>
  <portgroup name='dontpanic'>
    <vlan>
      <tag id='42'/>
    </vlan>
  </portgroup>
</network>
# virsh dumpxml rhel7 | grep /interface -B6
    <interface type='network'>
      <mac address='52:54:00:c8:7d:79'/>
      <source network='ovs-net'/>
      <model type='virtio'/>
      <filterref filter='clean-traffic'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

2. hotplug:
# cat interface.xml
<interface type='network'>
  <source network='ovs-net'/>
  <model type='virtio'/>
<filterref filter='clean-traffic'/>
</interface>

# virsh attach-device rhel7 interface.xml 
error: Failed to attach device from interface.xml
error: unsupported configuration: filterref is not supported for network interfaces with virtualport type openvswitch

3. update-device:
# cat inter.xml
    <interface type='bridge'>
      <mac address='52:54:00:c8:7d:79'/>
      <source network='ovs-net' bridge='ovsbr0'/>
      <vlan trunk='yes'>
        <tag id='42' nativeMode='untagged'/>
        <tag id='47'/>
      </vlan>
      <virtualport type='openvswitch'>
        <parameters interfaceid='09b11c53-8b5c-4eeb-8f00-d84eaa0aaa4f'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
<filterref filter='clean-traffic'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
# virsh update-device rhel7 inter.xml
error: Failed to update device from inter.xml
error: unsupported configuration: filterref is not supported for network interfaces with virtualport type openvswitch

4. migration from old libvirt libvirt-3.9.0-14.el7.x86_64 with such invalid setting:
# virsh migrate rhel7 --live --verbose qemu+ssh://10.*.*.*/system  
root@10.*.*.*'s password: 
error: unsupported configuration: filterref is not supported for network interfaces with virtualport type openvswitch

Comment 16 Jing Qi 2020-02-05 03:30:24 UTC

Verified with libvirt-4.5.0-32.virtcov.el7.x86_64 & qemu-kvm-rhev-2.12.0-43.el7.x86_64

# virsh start avocado-vt-vm 
 <interface type='bridge'>
      <mac address='52:54:00:be:d8:82'/>
      <source bridge='ovsbr0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='a211923a-0a5f-443b-ad9c-074c15722c7d'/>
      </virtualport>
      <model type='virtio'/>
      <filterref filter='no-mac-broadcast'/>
      <alias name='ua-88ee9909-3d84-4414-9ca3-51f4c00370d1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>


# virsh update-device avocado-vt-vm interface.xml
error: Failed to update device from interface.xml
error: unsupported configuration: filterref is not supported for network interfaces with virtualport type openvswitch

# virsh attach-device avocado-vt-vm interface.xml
error: Failed to attach device from interface.xml
error: unsupported configuration: filterref is not supported for network interfaces with virtualport type openvswitch

Comment 18 errata-xmlrpc 2020-03-31 19:58:29 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1094

Comment 19 Dominik Holler 2020-05-13 07:31:18 UTC

In which RHEL8 version is this bug fixed?

Comment 20 yalzhang@redhat.com 2020-05-13 08:44:24 UTC

As the patch is included in libvirt-4.5.0-26, and the version for RHEL8 different releases:
RHEL 8.0.0 --> libvirt-4.5.0-23， lastest RHEL 8.0.0.z --> libvirt-4.5.0-24
RHEL 8.1.0 --> libvirt-4.5.0-31
RHEL-AV 8.0.0---> libvirt-5.0.0-7

I think it fixed till RHEL 8.1.0 and RHEL-AV 8.0.0

Comment 21 Dominik Holler 2020-05-13 09:19:31 UTC

I checked libvirt downstream 6.0.0-17 , and it seems to contain the fix, while upstream 5.6.0-10 from ovirt-master-advanced-virtualization-testing seems not to include the fix.
I was just surprised that the fix is available on upstream on CentOS 7 before CentOS 8. This might indicate that there is something mixed in downstream, too.

Comment 22 Laine Stump 2020-05-14 23:53:49 UTC

Although these patches were backported to the RHEL7 libvirt-4.5.0-26.el7, and RHEL8 libvirt is also based of libvirt-4.5.0, neither of these is based off the other - they are not downstream or upstream of each other in either direction. It's merely a (kind of) coincidence that they are both based on the same libvirt upstream version.

The patches in question here went into libvirt-5.8.0 upstream, and in order to make it into any downstream, whether it be RHEL/CentOS 7/8 or Fedora $whatever, either that downstream needs to rebase libvirt to 5.8.0+, or there must be an explicit BZ requesting and tracking a backport. This bug was filed against RHEL7, so the fix was backported to RHEL7, but not cloned to RHEL8. Since I don't think of bugs who have a potential workaround of "don't do that" to be critical, I generally won't clone them to other downstreams myself, but will just rely on the periodic rebasing of libvirt to eventually take care of the issue (and of course if someone else considers the bug more important, they can always clone the BZ to a different downstream release :-)

Unfortunately (in this case), the policy for base RHEL8 libvirt (i.e. *not* the libvirt that is included in the "Advanced Virtualization" stream) has become "rebase very infrequently), so it's been stuck at 4.5.0 for a long time now.

libvirt in RHEL-8.3.0 will be rebased to at least libvirt-6.0.0, so this bug will be fixed in that release. If you need it sooner, it is also in RHEL-AV 8.2.1 (which is based on libvirt-6.0.0). If you're using CentOS rather than RHEL, I don't recall the name it's been given, but there is a CentOS version of the Advanced Virtualization stream that will have that version of libvirt (and newer-than-standard CentOS versions of the other virt packages, e.g. qemu-kvm).

(BTW, while looking into this I noticed that in Comment 12 I had mistakenly pasted the commit logs of a local branch, *not* the actual commits on upstream master. The correct commit ids are: 57de1988c4c2b015a5f459a5504a814833ffa194, 3cff23f7f16dc2b74d54f14b77790dc37df01ded, and 70a29b378a397f6f4571c95b9d3cbba711a90859 (just in case anyone in the future needs to further research, or backport, these patches)).