Bug 1841194

The fix for relevant upstream bug was backported up to queens and added to OSP 13, updating status and version

verified on 2020-05-19.2

According to our records, this should be resolved by openstack-neutron-12.1.1-18.el7ost.  This build is available now.

Hi,

This issue is still reproducible. Customer had updated their env from 13z11 to 13z12 and sees that the required neutron package i.e. openstack-neutron-12.1.1-18.el7ost is there but we still see that br-int is flooded

When sending continuous packets from VM-1 towards VM-3 the age timer is not reset for the destination mac of VM-3 fa:16:3e:26:b6:d1.
Instead the age timer keeps increasing until it reaches 300 seconds and then the entry is removed from fdb and you have again port flooding and so also VM-2 receives the packets.


Can this be verified further and worked upon for the possible fix?

[root@oscar23com014 ~]# neutron-server --version
neutron-server 12.1.1
[root@oscar23com014 ~]#

[root@oscar23com014 ~]# rpm -qv openstack-neutron
openstack-neutron-12.1.1-18.el7ost.noarch
[root@oscar23com014 ~]# ovs-appctl fdb/show br-int
 port  VLAN  MAC                Age
    3     2  00:22:bd:f8:19:ff  161
    3     1  fa:16:3e:26:b6:d1   38
    5     2  fa:16:3e:7f:62:5b    0
    7     2  fa:16:3e:8b:8d:74    0
    8     1  fa:16:3e:31:37:a2    0
[root@oscar23com014 ~]# ovs-appctl fdb/show br-int
 port  VLAN  MAC                Age
    3     2  00:22:bd:f8:19:ff  168
    3     1  fa:16:3e:26:b6:d1   45
    5     2  fa:16:3e:7f:62:5b    1
    7     2  fa:16:3e:8b:8d:74    0
    8     1  fa:16:3e:31:37:a2    0
[root@oscar23com014 ~]# ovs-appctl fdb/show br-int
 port  VLAN  MAC                Age
    3     2  00:22:bd:f8:19:ff  170
    3     1  fa:16:3e:26:b6:d1   47
    5     2  fa:16:3e:7f:62:5b    1
    7     2  fa:16:3e:8b:8d:74    0
    8     1  fa:16:3e:31:37:a2    0

Recapping an IRC chat about this, this upstream fix uses a new configuration option: in ovs agent configuration [agent] section:
# When set to True, the accepted egress unicast traffic will not use action
# NORMAL. The accepted egress packets will be taken care of in the final egress
# tables direct output flows for unicast traffic. (boolean value)
#explicitly_egress_direct = false

And related upstream release notes:
https://review.opendev.org/#/c/709410/3/releasenotes/notes/accepted_egress_direct-cc23873e213c6919.yaml

Hi,

Customer has verified the issue after
- updating env from 13z11 to 13z12
- also updating explicitly_egress_direct to 'true' in /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/openvswitch_agent.ini in all controllers node.


The port flooding issue remains after the change, the behavior is the same as before.

Summary of setup and observation:
----------------------------------------
setup:
VM1 running on hypervisor A
VM2 running on hypervisor A
VM3 running on hypervisor B
All 3 VM are in the same external provider network

When the destination mac is in the fdb table of br-int on the compute the age timer keeps increasing here even though sending continuous packets and when the mac is then deleted due to age timer expiry you have again port flooding.
 The patch doesn't seem to fix the underlying issue - ingress MACs (coming from br-ex in our case as we use provider networks here) are not learned by br-int, neither when the mac entry is there the aging timer is not reset in case of sending continuously packets.
As indicated in the problem description
1. Only an ARP sent out by VM3 for resolving of the IP of VM1 triggers that the mac of VM3 comes into the FDB table of br-int on the compute node where VM1 is running. When VM3 is just sending packets towards VM1 without first sending out ARP as it already knows the mac, then in this case the mac of VM3 does not come into the FDB table of br-int.
2. Secondly, when the mac of VM3 is listed in FDB table of br-int after ARP, the mac aging timer is not being reset when VM3 continuously is sending packets towards VM1. The mac aging timer expires after 5 minutes causing that the mac of VM3 is removed from the FDB and there is again port flooding on br-int of compute node A when VM1 sends traffic to VM3. On br-ex this does happen correctly. The mac aging timer is reset by incoming packets from this mac of VM3 and not only an ARP triggers the mac/port mapping to become listed in FDB, also when VM3 just starts sending packets towards VM-1 the mac becomes listed in FDB of br-ex on the compute node A where VM1 is running.
Problem seems that incoming packets on br-int do not trigger an update in FDB, neither create the entry in FDB neither reset the mac aging timer of an existing entry.



Thanks,
Shravan

Also, as upstream bug https://bugs.launchpad.net/neutron/+bug/1732067 can get confusing, confirming this reported issue here is with ovs firewall (which should be fixed by the explicitly_egress_direct option).
Linked to this, an in-progress LP https://review.opendev.org/#/c/738397/ mentions flooding issue on *ingress* packets with this option with potential fix https://review.opendev.org/#/c/738397/

For other cases, note there are still patches in progress for iptables_hybrid firewall https://review.opendev.org/#/c/712640/ and for DVR use case https://review.opendev.org/#/c/715419

Addressing my mistake after reading comment 15, the explicitly_egress_direct parameter should be set on *compute* nodes, the hypervisors where we want to change behaviour (modifying the L2 agent configuration on another node like controller will not have an effect there).

After additional checks, with correct packages versions it was confirmed fix is working properly. We can close this bug back then, and keep tracking of tripleo integration in bug #1883214