Bug 1957025

()[root@controller-0 /]# ovn-sbctl find port_binding logical_port="cr-lrp-d796ca6d-dd92-49c2-9f4b-e94413f2fb8d"
_uuid               : 42d7a1c3-8f7c-4a03-bedd-bda5b6a9b209
chassis             : cdda0ab3-ec8f-4932-b3db-a6c4c573d06d
datapath            : 46ef0b93-ca0b-488b-99be-34617fdcb9ca
encap               : []
external_ids        : {}
gateway_chassis     : []
ha_chassis_group    : 69af9edb-0e1f-4b1f-a982-abfb2b029bdd
logical_port        : cr-lrp-d796ca6d-dd92-49c2-9f4b-e94413f2fb8d
mac                 : ["fa:16:3e:61:a1:98 10.2.113.250/24 2620:52:4:230d::3d9/64"]
nat_addresses       : []
options             : {distributed-port=lrp-d796ca6d-dd92-49c2-9f4b-e94413f2fb8d}
parent_port         : []
tag                 : []
tunnel_key          : 3
type                : chassisredirect
up                  : true
virtual_parent      : []

2021-05-04T16:38:33.338Z|00034|binding|INFO|cr-lrp-d796ca6d-dd92-49c2-9f4b-e94413f2fb8d: Claiming fa:16:3e:61:a1:98 10.2.113.250/24 2620:52:4:230d::3d9/64
2021-05-04T16:38:33.338Z|00035|binding|INFO|Changing chassis for lport cr-lrp-8732c441-8155-4fe6-bf64-da68e871c211 from 466e1018-1a42-4d16-907a-d1fc62910ea3 to 651b4ea3-f4ea-42b3-bc11-1ec03154aebb.


()[root@controller-0 /]# ovn-nbctl lrp-get-gateway-chassis lrp-d796ca6d-dd92-49c2-9f4b-e94413f2fb8d
lrp-d796ca6d-dd92-49c2-9f4b-e94413f2fb8d_466e1018-1a42-4d16-907a-d1fc62910ea3     5  <-- controller-2
lrp-d796ca6d-dd92-49c2-9f4b-e94413f2fb8d_66758056-d693-4a33-89af-dd15d08b05fa     4
lrp-d796ca6d-dd92-49c2-9f4b-e94413f2fb8d_98cc5a8e-c116-4f82-8c67-b1242888cb09     3
lrp-d796ca6d-dd92-49c2-9f4b-e94413f2fb8d_651b4ea3-f4ea-42b3-bc11-1ec03154aebb     2  <-- networker-1
lrp-d796ca6d-dd92-49c2-9f4b-e94413f2fb8d_4f2021dd-ee45-4677-aa17-854aa0d228ca     1


()[root@controller-0 /]# ovn-sbctl --column _uuid,hostname list chassis 466e1018-1a42-4d16-907a-d1fc62910ea3
_uuid               : cdda0ab3-ec8f-4932-b3db-a6c4c573d06d
hostname            : controller-2.osp-002.prod.iad2.dc.redhat.com

()[root@controller-0 /]# ovn-sbctl --column _uuid,hostname list chassis 651b4ea3-f4ea-42b3-bc11-1ec03154aebb
_uuid               : 45ebb384-c348-425a-b3c4-9369e54e11a1
hostname            : networker-1.osp-002.prod.iad2.dc.redhat.com
()[root@controller-0 /]#

**Important** follow up on this issue: 

After consulting with one of my OVN gurus, I learned that the Openflow rules should not be present in the node upon power cycle.

That was definitely the case, BUT it turned out that the br-int bridge did not have fail-mode set to secure [0]. Because of that,
the default 'NORMAL' action rule was present and that is what caused the "chaos" as default bridge operations were being performed
on all of its ports:

  [root@networker-0 openvswitch]# ovs-ofctl dump-flows br-int
   cookie=0x0, duration=1611.902s, table=0, n_packets=857734338, n_bytes=55326381337, priority=0 actions=NORMAL

Further inspection is needed to understand why/how the br-int bridge in some of the nodes were not set with fail-mode secure.

This workaround fixed the problem in the cluster:

[stack@director1 ~]$ for i in `OS_CLOUD=undercloud openstack server list| grep control | awk '{print $8}' | sed 's/ctlplane=//'`; do echo $i ; \
  ssh $i sudo ovs-vsctl set-fail-mode br-int secure ; done




[0]: 
http://www.openvswitch.org/support/dist-docs/ovs-vsctl.8.txt  ==>   Controller Failure Settings

If OVN creates br-int (default) then it sets fail-mode=secure:

static const struct ovsrec_bridge *
create_br_int(struct ovsdb_idl_txn *ovs_idl_txn,
              const struct ovsrec_open_vswitch_table *ovs_table)
{
    <snip>

    struct ovsrec_bridge *bridge;
    bridge = ovsrec_bridge_insert(ovs_idl_txn);
    ovsrec_bridge_set_name(bridge, bridge_name);
>>> ovsrec_bridge_set_fail_mode(bridge, "secure");
    ovsrec_bridge_set_ports(bridge, &port, 1);

but if that bridge already exists when ovn-controller starts, I don't think it will enforce fail-mode=secure on the existing bridge.

Is there any way that br-int would have been created by something (even ovn-controller) and then fail-mode got reset, then when ovn-controller restarts it doesn't touch the bridge?

(In reply to Dan Williams from comment #4)
> If OVN creates br-int (default) then it sets fail-mode=secure:
> 
> static const struct ovsrec_bridge *
> create_br_int(struct ovsdb_idl_txn *ovs_idl_txn,
>               const struct ovsrec_open_vswitch_table *ovs_table)
> {
>     <snip>
> 
>     struct ovsrec_bridge *bridge;
>     bridge = ovsrec_bridge_insert(ovs_idl_txn);
>     ovsrec_bridge_set_name(bridge, bridge_name);
> >>> ovsrec_bridge_set_fail_mode(bridge, "secure");
>     ovsrec_bridge_set_ports(bridge, &port, 1);
> 
> but if that bridge already exists when ovn-controller starts, I don't think
> it will enforce fail-mode=secure on the existing bridge.
> 
> Is there any way that br-int would have been created by something (even
> ovn-controller) and then fail-mode got reset, then when ovn-controller
> restarts it doesn't touch the bridge?

Thanks, Dan. I still think that just as ovn creates br-int with secure fail mode when needed,
I am thinking that that should be checked and set, regardless. Is that an okay approach?

static const struct ovsrec_bridge *
process_br_int(struct ovsdb_idl_txn *ovs_idl_txn,
               const struct ovsrec_bridge_table *bridge_table,
               const struct ovsrec_open_vswitch_table *ovs_table)
{
    const struct ovsrec_bridge *br_int = get_br_int(bridge_table,
                                                    ovs_table);
    if (!br_int) {
        br_int = create_br_int(ovs_idl_txn, ovs_table);
    } <=== else { ... ovsrec_bridge_set_fail_mode(bridge, "secure");  ... }  <---- br_int for ovn must be always used with fail mode "secure" 

    ...
    return br_int;
}

(In reply to ffernand from comment #5)
> (In reply to Dan Williams from comment #4)
> > If OVN creates br-int (default) then it sets fail-mode=secure:
> > 
> > static const struct ovsrec_bridge *
> > create_br_int(struct ovsdb_idl_txn *ovs_idl_txn,
> >               const struct ovsrec_open_vswitch_table *ovs_table)
> > {
> >     <snip>
> > 
> >     struct ovsrec_bridge *bridge;
> >     bridge = ovsrec_bridge_insert(ovs_idl_txn);
> >     ovsrec_bridge_set_name(bridge, bridge_name);
> > >>> ovsrec_bridge_set_fail_mode(bridge, "secure");
> >     ovsrec_bridge_set_ports(bridge, &port, 1);
> > 
> > but if that bridge already exists when ovn-controller starts, I don't think
> > it will enforce fail-mode=secure on the existing bridge.
> > 
> > Is there any way that br-int would have been created by something (even
> > ovn-controller) and then fail-mode got reset, then when ovn-controller
> > restarts it doesn't touch the bridge?
> 
> Thanks, Dan. I still think that just as ovn creates br-int with secure fail
> mode when needed,
> I am thinking that that should be checked and set, regardless. Is that an
> okay approach?

I would agree; if OVN needs fail-mode=secure, then it should probably enforce that on the integration bridge.

(In reply to Dan Williams from comment #6)
> (In reply to ffernand from comment #5)
> > (In reply to Dan Williams from comment #4)
> > > If OVN creates br-int (default) then it sets fail-mode=secure:
> > > 
> > > static const struct ovsrec_bridge *
> > > create_br_int(struct ovsdb_idl_txn *ovs_idl_txn,
> > >               const struct ovsrec_open_vswitch_table *ovs_table)
> > > {
> > >     <snip>
> > > 
> > >     struct ovsrec_bridge *bridge;
> > >     bridge = ovsrec_bridge_insert(ovs_idl_txn);
> > >     ovsrec_bridge_set_name(bridge, bridge_name);
> > > >>> ovsrec_bridge_set_fail_mode(bridge, "secure");
> > >     ovsrec_bridge_set_ports(bridge, &port, 1);
> > > 
> > > but if that bridge already exists when ovn-controller starts, I don't think
> > > it will enforce fail-mode=secure on the existing bridge.
> > > 
> > > Is there any way that br-int would have been created by something (even
> > > ovn-controller) and then fail-mode got reset, then when ovn-controller
> > > restarts it doesn't touch the bridge?
> > 
> > Thanks, Dan. I still think that just as ovn creates br-int with secure fail
> > mode when needed,
> > I am thinking that that should be checked and set, regardless. Is that an
> > okay approach?
> 
> I would agree; if OVN needs fail-mode=secure, then it should probably
> enforce that on the integration bridge.

And perhaps warn if it isn't set, since this should be an exceptional condition that the admin should look into?

Changes posted upstream at: https://patchwork.ozlabs.org/project/ovn/patch/20210507012226.1504699-1-flavio@flaviof.com/

Changes v2 posted upstream at: https://patchwork.ozlabs.org/project/ovn/patch/20210507174947.1879798-1-flavio@flaviof.com/

Fix is merged upstream.
I'm assigning bz to Numan, so he can help me out creating a d/s rpm for ovn 2.13 that includes the fix.

https://github.com/ovn-org/ovn/commit/9cc334bc1a036a93cc1a541513d48f4df6933e9b
https://github.com/ovn-org/ovn/commit/be65a461ce134c1e874b65add402f7c2744f29f5

I talked to numan,this bug is addressed  by setting the fail_mode to secure in
ovn-controller  if this is not set.

since it is hard to set the env like osp, numan said we can simply reproduce by deleting the fail_mode after ovn-controller started.

on version 
# rpm -qa|grep ovn
ovn-2021-21.03.0-21.el8fdp.x86_64
ovn-2021-central-21.03.0-21.el8fdp.x86_64
ovn-2021-host-21.03.0-21.el8fdp.x86_64

# ovs-vsctl show
aa1481e5-933e-4276-be67-f56d95703c54
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.15.1"
[root@dell-per740-54 load_balance]# ovs-vsctl clear  bridge br-int fail_mode      --------delete
[root@dell-per740-54 load_balance]# ovs-vsctl show
aa1481e5-933e-4276-be67-f56d95703c54
    Bridge br-int                  ----------------------none
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "2.15.1"
[root@dell-per740-54 load_balance]# ovs-vsctl get bridge br-int fail_mode  --------------none
[]


verified on version:

# rpm -qa|grep ovn
ovn2.13-host-20.12.0-118.el8fdp.x86_64
ovn2.13-20.12.0-118.el8fdp.x86_64
ovn2.13-central-20.12.0-118.el8fdp.x86_64

ovn-2021-21.03.0-40.el8fdp.x86_64
ovn-2021-central-21.03.0-40.el8fdp.x86_64
ovn-2021-host-21.03.0-40.el8fdp.x86_64

# ovs-vsctl get bridge br-int fail_mode
secure
[root@dell-per730-19 ~]# ovs-vsctl clear  bridge br-int fail_mode
[root@dell-per730-19 ~]# ovs-vsctl get bridge br-int fail_mode      --------------fail-mode is not empty even we clear it
secure
[root@dell-per730-19 ~]# systemctl restart ovn-controller
[root@dell-per730-19 ~]# ovs-vsctl get bridge br-int fail_mode
secure
[root@dell-per730-19 ~]# ovs-vsctl show
e9fe1808-a871-435a-8b77-7d31b8a9c00d
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
    Bridge br-provider
        Port veth0_c0_p
            Interface veth0_c0_p
        Port br-provider
            Interface br-provider
                type: internal
        Port enp4s0d1
            Interface enp4s0d1
    ovs_version: "2.15.1"
[root@dell-per730-19 ~]# ovs-vsctl clear  bridge br-int fail_mode
[root@dell-per730-19 ~]# 
[root@dell-per730-19 ~]# ovs-vsctl show
e9fe1808-a871-435a-8b77-7d31b8a9c00d
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
    Bridge br-provider
        Port veth0_c0_p
            Interface veth0_c0_p
        Port br-provider
            Interface br-provider
                type: internal
        Port enp4s0d1
            Interface enp4s0d1
    ovs_version: "2.15.1"

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ovn2.13 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2507

*** Bug 1942085 has been marked as a duplicate of this bug. ***