Description of problem:

This bug is related to https://bugzilla.redhat.com/show_bug.cgi?id=1672606 (ovsdbapp queue size).

As an optimization, networking-ovn could reduce the number of transactions to the SBDB.

After the patch to check for the agents health status was introduced, networking-ovn started to bump the nb_cfg for every transaction in NB DB.

That behavior ended up increasing the number of transactions because every time nb_cfg is increased that would be propagated to the SB DB and the metadata agent would get an event which would result in it starting yet another transaction to update its Chassis external_ids column (SB DB) to indicate that it's still alive.

How reproducible:

Always

Steps to Reproduce (used DevStack for this one):

1. Listen to the events to the SBDB

$ tailf /opt/stack/data/ovs/ovnsb_db.db

2. Start a transaction, in this example I will clean the port_bind table which will trigger the port to go down and up again:

$ ovn-sbctl clear port_bind ae2db6eb-26d1-401e-aa0f-c866ec9974f3 chassis

3. Now looking at the transactions we will see that we have 8 operations in the SBDB:

OVSDB JSON 198 979a9514e40bec7e4fb21c134b4d2d417b1463b3
{"_date":1549290530763,"Port_Binding":{"96837e20-2535-4e3e-8361-49e25107adaf":{"chassis":["set",[]]}},"_comment":"ovs-sbctl: ovn-sbctl clear port_bind 96837e20-2535-4e3e-8361-49e25107adaf chassis"}
OVSDB JSON 140 ecb2b62ffd3c83ecbd1ce5bc914f3678b7968b81
{"_date":1549290530769,"Port_Binding":{"96837e20-2535-4e3e-8361-49e25107adaf":{"chassis":["uuid","d3cbc656-2e88-4e51-a412-44e2aa327c1a"]}}}
OVSDB JSON 93 b1675d63e5f4ff2ce24a33247dafe9bd345e8537
{"SB_Global":{"21cbcd3d-2dca-47e5-b964-b4195aabee75":{"nb_cfg":9593}},"_date":1549290531413}
OVSDB JSON 91 b2b716ba0b16ae2e4a4f86cc2404022e39469c5d
{"Chassis":{"d3cbc656-2e88-4e51-a412-44e2aa327c1a":{"nb_cfg":9593}},"_date":1549290531421}
OVSDB JSON 405 0a7ea709527baa2a2defe1a9e1adbd6c719eb68c
{"Chassis":{"d3cbc656-2e88-4e51-a412-44e2aa327c1a":{"external_ids":["map",[["datapath-type",""],["iface-types","erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan"],["neutron:ovn-metadata-id","f6c9cc58-47c7-49b9-bd6e-94f7aba12e9d"],["neutron:ovn-metadata-sb-cfg","9593"],["ovn-bridge-mappings","public:br-ex"],["ovn-cms-options","enable-chassis-as-gw"]]]}},"_date":1549290531428}
OVSDB JSON 531 35e95217c6ede96f56602ef18d19fb9f3ea1fe56
{"_date":1549290531579,"SB_Global":{"21cbcd3d-2dca-47e5-b964-b4195aabee75":{"nb_cfg":9594}},"Port_Binding":{"dafe6ad6-98b4-4691-b963-3c026d817367":{"external_ids":["map",[["neutron:cidrs","172.24.4.228/24 2001:db8::171/64"],["neutron:device_id","3e24f3fb-87c1-4e33-a132-f63ad76039f5"],["neutron:device_owner","network:router_gateway"],["neutron:network_name","neutron-25bbb5aa-7e59-415f-b12f-9ed7a6049a76"],["neutron:port_name",""],["neutron:project_id",""],["neutron:revision_number","210"],["neutron:security_group_ids",""]]]}}}
OVSDB JSON 91 dd6b725b6a875318d7207c4565d09f2acd66308e
{"Chassis":{"d3cbc656-2e88-4e51-a412-44e2aa327c1a":{"nb_cfg":9594}},"_date":1549290531586}
OVSDB JSON 405 139e6404aec4701d62757479ce1e8c7b940ead2c
{"Chassis":{"d3cbc656-2e88-4e51-a412-44e2aa327c1a":{"external_ids":["map",[["datapath-type",""],["iface-types","erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan"],["neutron:ovn-metadata-id","f6c9cc58-47c7-49b9-bd6e-94f7aba12e9d"],["neutron:ovn-metadata-sb-cfg","9594"],["ovn-bridge-mappings","public:br-ex"],["ovn-cms-options","enable-chassis-as-gw"]]]}},"_date":1549290531596}

Now, when we disable the increase of nb_cfg (by commenting the code, there's no config for it) this number of transactions is reduced to 3 (same command above):

OVSDB JSON 198 c588ca88fbbcdbacbedbe6b6b1ddcf897f84606d
{"_date":1549289886733,"Port_Binding":{"96837e20-2535-4e3e-8361-49e25107adaf":{"chassis":["set",[]]}},"_comment":"ovs-sbctl: ovn-sbctl clear port_bind 96837e20-2535-4e3e-8361-49e25107adaf chassis"}
OVSDB JSON 140 26497f926d2b3c0f906f9fc0030cd56233b6014a
{"_date":1549289886740,"Port_Binding":{"96837e20-2535-4e3e-8361-49e25107adaf":{"chassis":["uuid","d3cbc656-2e88-4e51-a412-44e2aa327c1a"]}}}
OVSDB JSON 462 a69244cf4999af5fcda3145a95f830c6d17ffb12
{"_date":1549289887593,"Port_Binding":{"dafe6ad6-98b4-4691-b963-3c026d817367":{"external_ids":["map",[["neutron:cidrs","172.24.4.228/24 2001:db8::171/64"],["neutron:device_id","3e24f3fb-87c1-4e33-a132-f63ad76039f5"],["neutron:device_owner","network:router_gateway"],["neutron:network_name","neutron-25bbb5aa-7e59-415f-b12f-9ed7a6049a76"],["neutron:port_name",""],["neutron:project_id",""],["neutron:revision_number","208"],["neutron:security_group_ids",""]]]}}}