Description of problem: OVS internal ports with veth create infinite port add loop Version-Release number of selected component (if applicable): openvswitch-2.5.0-14.git20160727.el7fdp.x86_64 How reproducible: A customer ran into this one. Obviously, an internal port is the wrong choice, and this needs to be a normal OVS port. But why is this behavior / loop / recursion happening? How to reproduce this issue (note: TYPE=OVSIntPort and the veth part): ~~~ [root@overcloud-compute-0 network-scripts]# cat ifcfg-eth3 NAME=eth3 ONBOOT=yes NETBOOT=yes BOOTPROTO=static DEVICE=eth3 DEVICETYPE=ovs TYPE=OVSPort OVS_OPTIONS="vlan_mode=native-untagged tag=1" OVS_BRIDGE=br-eth3 MTU=9000 [root@overcloud-compute-0 network-scripts]# cat ifcfg-br-eth3 DEVICE=br-eth3 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSBridge HOTPLUG=no USERCTL=no [root@overcloud-compute-0 network-scripts]# cat ifcfg-mgmt0-lldp DEVICE=eth3_ll1 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSIntPort OVS_BRIDGE=br-eth3 HOTPLUG=no USERCTL=no BOOTPROTO=static MTU=9000 OVS_EXTRA="set bridge br-eth3 other-config:forward-bpdu=true" ~~~ Run manually ~~~ ip link add name eth3_ll1 type veth peer name eth3_ll2 ~~~ Run ~~~ ifup br-eth3 ifup eth3 ifup mgmt0-lldp ip link set dev eth3_ll2 up ~~~ The moment eth3_ll2 comes up, the following happens - this happens as soon as I bring up the opposite end of the veth port: ~~~ [root@overcloud-compute-0 ~]# tail -f /var/log/openvswitch/ovs-vswitchd.log & [1] 929479 [root@overcloud-compute-0 ~]# 2017-05-15T19:00:16.021Z|17950|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17374 2017-05-15T19:00:16.027Z|17951|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17375 2017-05-15T19:00:16.034Z|17952|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17376 2017-05-15T19:00:16.042Z|17953|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17377 2017-05-15T19:00:16.052Z|17954|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17378 2017-05-15T19:00:16.060Z|17955|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17379 2017-05-15T19:00:16.068Z|17956|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17380 2017-05-15T19:00:16.074Z|17957|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17381 2017-05-15T19:00:16.083Z|17958|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17382 2017-05-15T19:00:16.092Z|17959|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17383 ~~~ ~~~ [root@overcloud-compute-0 ~]# (reverse-i-search)`id': journalctl | egrep 'eth3|mgmt0' | grep br^Cge [root@overcloud-compute-0 ~]# ^C [root@overcloud-compute-0 ~]# ifup mgmt0-lldp 2017-05-15T19:00:40.662Z|17960|dpif|WARN|Dropped 32 log messages in last 25 seconds (most recently, 25 seconds ago) due to excessive rate 2017-05-15T19:00:40.662Z|17961|dpif|WARN|system@ovs-system: failed to add eth3_ll1 as port: File exists ovs-vsctl: Error detected while setting up 'eth3_ll1'. See ovs-vswitchd log for details. 2017-05-15T19:00:40.922Z|17962|dpif|WARN|system@ovs-system: failed to add eth3_ll1 as port: File exists [root@overcloud-compute-0 ~]# [root@overcloud-compute-0 ~]# [root@overcloud-compute-0 ~]# [root@overcloud-compute-0 ~]# ip link ls | grep ll 28: eth3_ll2@eth3_ll1: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN mode DEFAULT qlen 1000 29: eth3_ll1@eth3_ll2: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 9000 qdisc noqueue state LOWERLAYERDOWN mode DEFAULT qlen 1000 [root@overcloud-compute-0 ~]# ip link set dev eth3_ll2 up [root@overcloud-compute-0 ~]# 2017-05-15T19:00:56.849Z|17963|dpif|WARN|system@ovs-system: failed to add eth3_ll1 as port: File exists 2017-05-15T19:00:56.851Z|17964|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17384 2017-05-15T19:00:56.854Z|17965|dpif|WARN|system@ovs-system: failed to add eth3_ll1 as port: File exists 2017-05-15T19:00:56.856Z|17966|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17385 2017-05-15T19:00:56.859Z|17967|dpif|WARN|system@ovs-system: failed to add eth3_ll1 as port: File exists 2017-05-15T19:00:56.861Z|17968|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17386 2017-05-15T19:00:56.864Z|17969|dpif|WARN|system@ovs-system: failed to add eth3_ll1 as port: File exists 2017-05-15T19:00:56.866Z|17970|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17387 2017-05-15T19:00:56.869Z|17971|dpif|WARN|system@ovs-system: failed to add eth3_ll1 as port: File exists 2017-05-15T19:00:56.873Z|17972|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17388 2017-05-15T19:00:56.877Z|17973|dpif|WARN|system@ovs-system: failed to add eth3_ll1 as port: File exists 2017-05-15T19:00:56.880Z|17974|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 17389 ~~~ Note that the issue does not occur when I switch the port to a non-internal port type TYPE=OVSPort ~~~ [root@overcloud-compute-0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-mgmt0-lldp DEVICE=eth3_ll1 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=br-eth3 HOTPLUG=no USERCTL=no BOOTPROTO=static MTU=9000 OVS_EXTRA="set bridge br-eth3 other-config:forward-bpdu=true" ~~~ ~~~ [root@overcloud-compute-0 ~]# ovs-vsctl show | grep br-eth3 -A7 Bridge "br-eth3" Port "eth3" tag: 1 Interface "eth3" Port "eth3_ll1" Interface "eth3_ll1" Port "br-eth3" Interface "br-eth3" type: internal Bridge br-ex Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "vlan902" tag: 902 ~~~ ~~~ [root@overcloud-compute-0 ~]# ifup br-eth3 ifu2017-05-15T19:03:49.036Z|30270|bridge|INFO|bridge br-eth3: added interface br-eth3 on port 65534 2017-05-15T19:03:49.036Z|30271|bridge|INFO|bridge br-eth3: using datapath ID 0000c2b7b41b0947 2017-05-15T19:03:49.036Z|30272|connmgr|INFO|br-eth3: added service controller "punix:/var/run/openvswitch/br-eth3.mgmt" p br-eth[root@overcloud-compute-0 ~]# ifup eth3 ~~~ ~~~ 2017-05-15T19:03:54.990Z|30273|bridge|INFO|bridge br-eth3: added interface eth3 on port 1 [root@overcloud-compute-0 ~]# [root@overcloud-compute-0 ~]# [root@overcloud-compute-0 ~]# [root@overcloud-compute-0 ~]# ifup mgmt0-lldp 2017-05-15T19:04:02.071Z|30274|bridge|INFO|bridge br-eth3: added interface eth3_ll1 on port 2 [root@overcloud-compute-0 ~]# [root@overcloud-compute-0 ~]# [root@overcloud-compute-0 ~]# ip link ls | grep ll 28: eth3_ll2@eth3_ll1: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN mode DEFAULT qlen 1000 29: eth3_ll1@eth3_ll2: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 9000 qdisc noqueue master ovs-system state LOWERLAYERDOWN mode DEFAULT qlen 1000 [root@overcloud-compute-0 ~]# ip link set dev eth3_ll2 up [root@overcloud-compute-0 ~]# ip link ls | grep ll 28: eth3_ll2@eth3_ll1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000 29: eth3_ll1@eth3_ll2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master ovs-system state UP mode DEFAULT qlen 1000 ~~~ bout the difference between OVSPort and OVSIntPort ~~~ OVSPort) ifup_ovs_bridge ${OTHERSCRIPT} ${CONFIG} ${2} # The port might be already in the database but not yet # in the datapath. So, remove the stale interface first. ovs-vsctl -t ${TIMEOUT} \ -- --if-exists del-port "$OVS_BRIDGE" "$DEVICE" \ -- add-port "$OVS_BRIDGE" "$DEVICE" $OVS_OPTIONS ${OVS_EXTRA+-- $OVS_EXTRA} OVSINTF=${DEVICE} /sbin/ifup "$OVS_BRIDGE" ;; OVSIntPort) ifup_ovs_bridge ovs-vsctl -t ${TIMEOUT} \ -- --if-exists del-port "$OVS_BRIDGE" "$DEVICE" \ -- add-port "$OVS_BRIDGE" "$DEVICE" $OVS_OPTIONS \ -- set Interface "$DEVICE" type=internal ${OVS_EXTRA+-- $OVS_EXTRA} if [ -n "${OVSDHCPINTERFACES}" ]; then for _iface in ${OVSDHCPINTERFACES}; do /sbin/ifup ${_iface} done fi BOOTPROTO="${OVSBOOTPROTO}" ${OTHERSCRIPT} ${CONFIG} ${2} ;; ~~~
Should there be some check that prohibits to set an attached "external" interface (if veth or physical) as type=internal? IMO it doesn't really make sense that somebody can create a veth or tap port and then attach it as type=internal. Or even take a physical interface and attach it as type=internal.
saw the same behavior with ovs 2.5 and 2.6, but didn't try 2.7 or master versions
The above configuration is not valid, as you should not use an system interface as an internal one. Due to the fact that the config is applied async no error message is displayed. On 2.7 you can see it when doing the ovs-vsctl show command: $ ovs-vsctl show da273bd9-bdbc-41c8-a81a-41c16ef5bdbd Bridge bzb Port "eth3_ll1" Interface "eth3_ll1" type: internal error: "could not add network device eth3_ll1 to ofproto (File exists)" Port bzb Interface bzb type: internal ovs_version: "2.7.90" For the loop issue I have send a patch upstream to OVS: https://mail.openvswitch.org/pipermail/ovs-dev/2017-June/333294.html
Hi, Awesome, thanks for the bug fix. " as you should not use an system interface as an internal one" Yes, that's what I guessed. Isn't there a way to tell the user that he's doing something bad here, or prohibit such kind of action? - Andreas
Currently this miss configuration is not detected when configuring, as the change is applied asynchronously. You will get a error in the log message, and you will see it with ovs-vsctl show. Patch got applied; https://github.com/openvswitch/ovs/commit/67ac844b55d4c5f6bbfa01773c82b3d6d8b62131