RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1555001 - ifup ovs-bridge enables OVS internal port which leaks BUM traffic to the kernel
Summary: ifup ovs-bridge enables OVS internal port which leaks BUM traffic to the kernel
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: initscripts
Version: 8.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.4
Assignee: Jan Macku
QA Contact: Daniel Rusek
URL:
Whiteboard:
Depends On: 1628900
Blocks: 1628227 1916632
TreeView+ depends on / blocked
 
Reported: 2018-03-13 18:29 UTC by Andreas Karis
Modified: 2021-06-10 15:18 UTC (History)
17 users (show)

Fixed In Version: initscripts-10.00.11-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-18 14:56:07 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github fedora-sysv initscripts pull 355 0 None closed network: add option to keep the link down 2021-02-18 16:16:06 UTC
Red Hat Knowledge Base (Solution) 3383521 0 None None None 2018-03-15 22:16:30 UTC
Red Hat Product Errata RHBA-2021:1624 0 None None None 2021-05-18 14:56:21 UTC

Description Andreas Karis 2018-03-13 18:29:31 UTC
Description of problem:
ifup ovs-bridge enabled OVS internal port which leaks BUM traffic to the kernel

OVS bridge internal ports (https://access.redhat.com/solutions/2530481) should be switched off by default, or they will cause leaks of BUM traffic against the kernel, which is particularly painful with DPDK.

This either needs a fix in initscripts, or in openvswitch

[root@overcloud-compute-0 network-scripts]# rpm -qf ifup-ovs
openvswitch-2.6.1-16.git20161206.el7ost.x86_64
[root@overcloud-compute-0 network-scripts]# rpm -qf ifup-eth
initscripts-9.49.39-1.el7_4.1.x86_64

OVS bridges with IP addresses should of course be enabled with `ip link set dev <br> up`. But OVS bridges without IP address should remain disabled.

Additional info:

Some further research shows that this is the default behavior of our ifcfg scripts:
~~~
root@overcloud-compute-0 ~]# ovs-vsctl add-br br-test
[root@overcloud-compute-0 ~]# ip link ls dev br-test
20: br-test: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether ce:07:67:7d:89:48 brd ff:ff:ff:ff:ff:ff
[root@overcloud-compute-0 ~]# ovs-vsctl del-br br-test
~~~
Note: a manually created bridge's kernel interface is down by default.

~~~
[root@overcloud-compute-0 ~]# cd /etc/sysconfig/network-scripts/
[root@overcloud-compute-0 network-scripts]# cp ifcfg-br-link ifcfg-br-test
[root@overcloud-compute-0 network-scripts]# vi !$
vi ifcfg-br-test
[root@overcloud-compute-0 network-scripts]# ifup br-test
[root@overcloud-compute-0 network-scripts]# ip link ls dev br-test
21: br-test: <BROADCAST,PROMISC,UP,LOWER_UP> mtu 2000 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 1000
    link/ether 0e:ac:8c:9e:67:4e brd ff:ff:ff:ff:ff:ff
[root@overcloud-compute-0 network-scripts]# cat ifcfg-br-test
# This file is autogenerated by os-net-config
DEVICE=br-test
ONBOOT=yes
HOTPLUG=no
NM_CONTROLLED=no
PEERDNS=no
DEVICETYPE=ovs
TYPE=OVSUserBridge
MTU=2000
OVS_EXTRA="set bridge br-test fail_mode=standalone"
[root@overcloud-compute-0 network-scripts]# 
~~~
Note: a bridge that was started with ifup will have a kernel space interface which is up by default

I switched `/etc/sysconfig/network-scripts/ifup-ovs` into debug mode:
~~~
#!/bin/bash -x
~~~

And ran ifdown / ifup br-test

And the culprit lies here:
~~~
+ '[' '' '!=' yes ']'
+ /etc/sysconfig/network-scripts/ifup-eth ifcfg-br-test
+ exit 0
~~~

In `/etc/sysconfig/network-scripts/ifup-ovs`
~~~
     31 OTHERSCRIPT="/etc/sysconfig/network-scripts/ifup-${REAL_DEVICETYPE}"
     32 
     33 if [ ! -x ${OTHERSCRIPT} ]; then
     34         OTHERSCRIPT="/etc/sysconfig/network-scripts/ifup-eth"
     35 fi
~~~

I then switched ifup-eth into debug mode:
~~~
#!/bin/bash -x
~~~
and added some extra verbosity to the file:
~~~
    216 else
    217     if [ -z "${IPADDR}" -a -z "${IPADDR0}" -a -z "${IPADDR1}" -a -z "${IPADDR2}" ]; then
    218         echo "debug start"
    219         # enable device without IP, useful for e.g. PPPoE
    220         ip link set dev ${REALDEVICE} up
    221         ethtool_set
    222         echo "debug end"
    223         [ -n "${LINKDELAY}" ] && /bin/sleep ${LINKDELAY}
    224     else
~~~

and ran ifdown / ifup br-test.
~~~
+ '[' -z '' -a -z '' -a -z '' -a -z '' ']'
+ echo 'debug start'
debug start
+ ip link set dev br-test up
+ ethtool_set
+ oldifs=' 	
'
+ IFS=';'
+ '[' -n '' ']'
+ IFS=' 	
'
+ echo 'debug end'
debug end
+ '[' -n '' ']'
+ '[' -z yes -a no = no -a br-test '!=' lo ']'
+ '[' -x /usr/bin/firewall-cmd -a br-test '!=' lo ']'
+ /usr/bin/firewall-cmd --zone= --change-interface=br-test
+ '[' OVSUserBridge = Bridge ']'
+ /etc/sysconfig/network-scripts/ifup-ipv6 ifcfg-br-test
+ is_true ''
+ case "$1" in
+ return 1
+ exec /etc/sysconfig/network-scripts/ifup-post ifcfg-br-test
[root@overcloud-compute-0 network-scripts]# 
~~~

As the comment states: "# enable device without IP, useful for e.g. PPPoE", but in case of OVS, this is definitely *not* useful at all, see: https://bugzilla.redhat.com/show_bug.cgi?id=1359854

OVS bridge internal ports (https://access.redhat.com/solutions/2530481) should be switched off by default, or they will cause leaks of BUM traffic against the kernel, which is particularly painful with DPDK.

~~~
OTHERSCRIPT="/etc/sysconfig/network-scripts/ifup-${REAL_DEVICETYPE}"

if [ ! -x ${OTHERSCRIPT} ]; then
        OTHERSCRIPT="/etc/sysconfig/network-scripts/ifup-eth"
fi
~~~

Looking at debug output, we can hack our way around it:
~~~
+ OTHERSCRIPT=/etc/sysconfig/network-scripts/ifup-
+ '[' '!' -x /etc/sysconfig/network-scripts/ifup- ']'
+ OTHERSCRIPT=/etc/sysconfig/network-scripts/ifup-eth
~~~

~~~
[root@overcloud-compute-0 network-scripts]# cat ifup-
#!/bin/bash -x
# NOOP
echo "here"
[root@overcloud-compute-0 network-scripts]# ifdown br-test
ifup [root@overcloud-compute-0 network-scripts]# ifup br-test
+ echo here
here
[root@overcloud-compute-0 network-scripts]# ip link ls dev br-test
29: br-test: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether ae:e2:3c:05:b0:4f brd ff:ff:ff:ff:ff:ff
[root@overcloud-compute-0 network-scripts]# 
~~~

This of course should not be a workaround for production. Also, I'm not sure if the contents of ifup-eth don't set other useful things, as this script executes lots of actions.

Comment 2 Andreas Karis 2018-03-13 18:41:10 UTC
We may also make a bad assumption here:
~~~
    113                 # When dhcp is not enabled, it is possible that someone may want
    114                 # a standalone bridge (i.e it may not have any ports). Configure it.
    115                 if [ "${OVSBOOTPROTO}" != "dhcp" ] && [ -z "${OVSINTF}" ] && \
    116                         [ "${OVSBRIDGECONFIGURED}" != "yes" ]; then
    117                         ${OTHERSCRIPT} ${CONFIG}
    118                 fi
    119                 exit 0
    120                 ;;
~~~

By the way, another workaround - set OVSBRIDGECONFIGURED="yes".
~~~
[root@overcloud-compute-0 network-scripts]# tail -1 ifcfg-br-test
OVSBRIDGECONFIGURED="yes"
[root@overcloud-compute-0 network-scripts]# ifdown br-test
if[root@overcloud-compute-0 network-scripts]# ifup br-test
[root@overcloud-compute-0 network-scripts]# ip link ls dev br-test
32: br-test: <BROADCAST,PROMISC> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 66:a9:d8:73:d7:49 brd ff:ff:ff:ff:ff:ff
[root@overcloud-compute-0 network-scripts]# 
~~~

Comment 3 Dan Sneddon 2018-03-13 19:28:03 UTC
(In reply to Andreas Karis from comment #2)

> By the way, another workaround - set OVSBRIDGECONFIGURED="yes".

Andreas, what do you think about the possibility of making changes to os-net-config to add OVSBRIDGECONFIGURED="yes" to OVS bridge ifcfg files? Is there a better way to go about fixing this, or is that a satisfactory solution?

Comment 4 Andreas Karis 2018-03-13 19:48:49 UTC
I'm not aware of the implications of disabling the execution of /etc/sysconfig/network-scripts/ifup-eth altogether.
~~~
[root@overcloud-compute-0 network-scripts]# wc -l /etc/sysconfig/network-scripts/ifup-eth
350 /etc/sysconfig/network-scripts/ifup-eth
~~~

That's a rather long script. If it does nothing in our case, fine. But I think it did some steps related to Networkmanager, it also sets the MTU (yes, the inteface should be down, but somebody could configure an interface as down with a given MTU, sets some things for firewall-cmd. Added `-x` to ifup-eth and ran `ifup br-test` again:
~~~
[root@overcloud-compute-0 network-scripts]# ifup br-test
+ . /etc/init.d/functions
++ TEXTDOMAIN=initscripts
++ umask 022
++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
++ export PATH
++ '[' 90058 -ne 1 -a -z '' ']'
++ '[' -d /run/systemd/system ']'
++ case "$0" in
++ '[' -z '' ']'
++ COLUMNS=80
++ '[' -z '' ']'
++ '[' -c /dev/stderr -a -r /dev/stderr ']'
+++ /sbin/consoletype
++ CONSOLETYPE=pty
++ '[' -z '' ']'
++ '[' -z '' ']'
++ '[' -f /etc/sysconfig/i18n -o -f /etc/locale.conf ']'
++ . /etc/profile.d/lang.sh
++ unset LANGSH_SOURCED
++ '[' -z '' ']'
++ '[' -f /etc/sysconfig/init ']'
++ . /etc/sysconfig/init
+++ BOOTUP=color
+++ RES_COL=60
+++ MOVE_TO_COL='echo -en \033[60G'
+++ SETCOLOR_SUCCESS='echo -en \033[0;32m'
+++ SETCOLOR_FAILURE='echo -en \033[0;31m'
+++ SETCOLOR_WARNING='echo -en \033[0;33m'
+++ SETCOLOR_NORMAL='echo -en \033[0;39m'
++ '[' pty = serial ']'
++ __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d'
++ '[' '' = 1 ']'
+++ cat /proc/cmdline
++ strstr 'BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.17.1.el7.x86_64 root=UUID=fa414390-f78d-49d4-a164-54615a32977b ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 rcu_nocbs=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 tuned.non_isolcpus=00300003 intel_pstate=disable nosoftlockup' rc.debug
++ '[' 'BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.17.1.el7.x86_64 root=UUID=fa414390-f78d-49d4-a164-54615a32977b ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 rcu_nocbs=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 tuned.non_isolcpus=00300003 intel_pstate=disable nosoftlockup' = 'BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.17.1.el7.x86_64 root=UUID=fa414390-f78d-49d4-a164-54615a32977b ro console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on isolcpus=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 nohz=on nohz_full=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 rcu_nocbs=2,4,6,8,10,12,14,16,18,22,24,26,28,30,32,34,36,38,3,5,7,9,11,13,15,17,19,23,25,27,29,31,33,35,37,39 tuned.non_isolcpus=00300003 intel_pstate=disable nosoftlockup' ']'
++ return 1
++ return 0
+ cd /etc/sysconfig/network-scripts
+ . ./network-functions
++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
++ export PATH
+++ hostname
++ HOSTNAME=overcloud-compute-0
++ '[' -z '/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d' ']'
+ '[' -f ../network ']'
+ . ../network
++ NOZEROCONF=yes
+ CONFIG=ifcfg-br-test
+ need_config ifcfg-br-test
+ local nconfig
+ CONFIG=ifcfg-ifcfg-br-test
+ '[' -f ifcfg-ifcfg-br-test ']'
+ CONFIG=ifcfg-br-test
+ '[' -f ifcfg-br-test ']'
+ return
+ source_config
+ CONFIG=ifcfg-br-test
+ DEVNAME=br-test
+ . /etc/sysconfig/network-scripts/ifcfg-br-test
++ DEVICE=br-test
++ ONBOOT=yes
++ HOTPLUG=no
++ NM_CONTROLLED=no
++ PEERDNS=no
++ DEVICETYPE=ovs
++ TYPE=OVSUserBridge
++ MTU=2000
++ OVS_EXTRA='set bridge br-test fail_mode=standalone'
+ '[' -r keys-br-test ']'
+ case "$TYPE" in
+ DEVICETYPE=ovs
+ '[' -n '' ']'
+ '[' -n '' ']'
+ '[' -z br-test -a -n '' ']'
+ '[' -z ovs ']'
+ '[' -z '' -a -n '' ']'
+ '[' -z '' ']'
+ REALDEVICE=br-test
+ '[' -z '' ']'
+ SYSCTLDEVICE=br-test
+ '[' br-test '!=' br-test ']'
+ ISALIAS=no
+ is_nm_running
+ dbus-send --system --print-reply --dest=org.freedesktop.DBus /org/freedesktop/DBus org.freedesktop.DBus.GetNameOwner string:org.freedesktop.NetworkManager
+ '[' br-test '!=' lo ']'
+ nm_con_load ifcfg-br-test
+ dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager /org/freedesktop/NetworkManager/Settings org.freedesktop.NetworkManager.Settings.LoadConnections array:string:/etc/sysconfig/network-scripts/ifcfg-br-test
+ is_false no
+ case "$1" in
+ return 0
+ '[' '' = bootp -o '' = dhcp ']'
+ is_available br-test
+ '[' -z br-test ']'
+ '[' -d /sys/class/net/br-test ']'
+ return 0
+ '[' -n '' ']'
+ '[' OVSUserBridge = Bridge ']'
+ '[' OVSUserBridge = Tap ']'
+ '[' -n '' ']'
+ is_available_wait br-test
+ '[' -z br-test ']'
+ local retry=
+ is_available br-test
+ '[' -z br-test ']'
+ '[' -d /sys/class/net/br-test ']'
+ return 0
+ return 0
+ '[' -n '' ']'
+ '[' -n 2000 ']'
+ ip link set dev br-test mtu 2000
+ is_wireless_device br-test
+ '[' -x /usr/sbin/iw ']'
+ return 1
+ '[' -n '' ']'
+ '[' '' = yes -a no = no -a '' '!=' '' ']'
+ '[' no = no ']'
+ is_bonding_device br-test
+ '[' -f /sys/class/net/br-test/bonding/slaves ']'
+ '[' -n '' ']'
+ '[' -n '' ']'
+ '[' -z '' -a -z '' -a -z '' -a -z '' ']'
+ ip link set dev br-test up
+ ethtool_set
+ oldifs=' 	
'
+ IFS=';'
+ '[' -n '' ']'
+ IFS=' 	
'
+ '[' -n '' ']'
+ '[' -z yes -a no = no -a br-test '!=' lo ']'
+ '[' -x /usr/bin/firewall-cmd -a br-test '!=' lo ']'
+ /usr/bin/firewall-cmd --zone= --change-interface=br-test
+ '[' OVSUserBridge = Bridge ']'
+ /etc/sysconfig/network-scripts/ifup-ipv6 ifcfg-br-test
+ is_true ''
+ case "$1" in
+ return 1
+ exec /etc/sysconfig/network-scripts/ifup-post ifcfg-br-test
~~~

Otherwise, yes, we could push that with os-net-config. I guess the final solution, though, would best be a change to the init scripts (adding a flag) and then with os-net-config enabling or disabling the flag as desired. But I'm really unsure.

Comment 6 Andreas Karis 2018-03-14 15:57:22 UTC
Hi,

I think by far the best workaround is to use a hook:

https://access.redhat.com/solutions/8694
~~~
cat <<'EOF'>/sbin/ifup-local
#!/bin/bash 

if=$1

if `/bin/ovs-vsctl show | grep Bridge | egrep -q " ${if}\$"`; then
  shutdown_interface=true
  if [ `/sbin/ip a ls dev $if | grep 'inet ' | wc -l` -ge 1 ]; then
    shutdown_interface=false
  elif [ `/sbin/ip a ls dev $if | grep 'inet6 ' | wc -l` -ge 2 ]; then
    shutdown_interface=false
  fi

  if $shutdown_interface ; then
    /sbin/ip link set dev $if down
  fi
fi
EOF
chmod +x /sbin/ifup-local
~~~

I'll have to test the above script a bit more, but quick and dirty and with some minimal testing, this seems to work and to be the way to go.

Comment 8 Andreas Karis 2018-03-15 22:16:50 UTC
Workaround for the time being: https://access.redhat.com/solutions/3383521

Comment 12 Lukáš Nykrýn 2019-02-15 14:15:43 UTC
Just a small update, I need to sync with NM guys to find out, how they are solving this issue.

Comment 16 Till Maas 2020-08-19 14:28:53 UTC
This bug depends on bug 1628900 for NetworkManager and bug 1650221 provides an alternative solution. Can you please confirm whether that would work for you and update the blocks field if it does. Thanks.

Comment 18 Chris Williams 2020-11-11 21:47:52 UTC
Red Hat Enterprise Linux 7 shipped it's final minor release on September 29th, 2020. 7.9 was the last minor releases scheduled for RHEL 7.
From intial triage it does not appear the remaining Bugzillas meet the inclusion criteria for Maintenance Phase 2 and will now be closed. 

From the RHEL life cycle page:
https://access.redhat.com/support/policy/updates/errata#Maintenance_Support_2_Phase
"During Maintenance Support 2 Phase for Red Hat Enterprise Linux version 7,Red Hat defined Critical and Important impact Security Advisories (RHSAs) and selected (at Red Hat discretion) Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available."

If this BZ was closed in error and meets the above criteria please re-open it flag for 7.9.z, provide suitable business and technical justifications, and follow the process for Accelerated Fixes:
https://source.redhat.com/groups/public/pnt-cxno/pnt_customer_experience_and_operations_wiki/support_delivery_accelerated_fix_release_handbook  

Feature Requests can re-opened and moved to RHEL 8 if the desired functionality is not already present in the product. 

Please reach out to the applicable Product Experience Engineer[0] if you have any questions or concerns.  

[0] https://bugzilla.redhat.com/page.cgi?id=agile_component_mapping.html&product=Red+Hat+Enterprise+Linux+7

Comment 31 errata-xmlrpc 2021-05-18 14:56:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (initscripts bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1624


Note You need to log in before you can comment on or make changes to this bug.