This bug was initially created as a copy of Bug #1992705

I am copying this bug because: 

We need a tracking bug in OSP and may need to implement changes in OSP to work around this issue.

Description of problem:
OVN CI jobs started to fail - https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/upgrades/view/ffu/

Root cause is that during the upgrade controller gets to latest version where OVN container is running on host with OVS2.13 but the computes need "hybrid" mode where only OVN container is switched to the latest version and the host stays the same with OVS2.11. This worked well up until new version of OVN.


Error in logs:
2021-08-11T14:44:54.158Z|00038|reconnect|INFO|tcp:172.17.1.46:6642: continuing to reconnect in the background but suppressing further logging                                                                                                
2021-08-11T14:46:26.727Z|00039|fatal_signal|WARN|terminating with signal 15 (Terminated)
2021-08-11T14:46:31.546Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log
2021-08-11T14:46:31.548Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting...
2021-08-11T14:46:31.548Z|00003|reconnect|INFO|unix:/run/openvswitch/db.sock: connected
2021-08-11T14:46:31.550Z|00004|ovsdb_idl|WARN|Open_vSwitch database lacks Datapath table (database needs upgrade?)
2021-08-11T14:46:31.550Z|00005|ovsdb_idl|WARN|Open_vSwitch table in Open_vSwitch database lacks datapaths column (database needs upgrade?)                                                                                                   
2021-08-11T14:46:31.550Z|00006|ovsdb_idl|WARN|Open_vSwitch database lacks Datapath table (database needs upgrade?)
2021-08-11T14:46:31.551Z|00007|ovsdb_idl|WARN|Open_vSwitch table in Open_vSwitch database lacks datapaths column (database needs upgrade?)                                                                                                   
2021-08-11T14:46:31.553Z|00008|main|INFO|OVN internal version is : [21.06.0-20.17.0-56.0]
2021-08-11T14:46:31.553Z|00009|main|INFO|OVS IDL reconnected, force recompute.
2021-08-11T14:46:31.553Z|00010|reconnect|INFO|tcp:172.17.1.46:6642: connecting...
2021-08-11T14:46:31.553Z|00011|main|INFO|OVNSB IDL reconnected, force recompute.
2021-08-11T14:46:31.553Z|00012|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"}                                                                        
2021-08-11T14:46:31.554Z|00013|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"}                                                                        
2021-08-11T14:46:31.554Z|00014|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"}                                                                        
2021-08-11T14:46:31.555Z|00015|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"}                                                                        
2021-08-11T14:46:31.555Z|00016|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"}                                                                        
2021-08-11T14:46:32.215Z|00017|reconnect|INFO|tcp:172.17.1.60:6642: connecting...
2021-08-11T14:46:33.215Z|00018|reconnect|INFO|tcp:172.17.1.60:6642: connection attempt timed out
2021-08-11T14:46:34.215Z|00019|reconnect|INFO|tcp:172.17.1.60:6642: connecting...
2021-08-11T14:46:35.215Z|00020|reconnect|INFO|tcp:172.17.1.60:6642: connection attempt timed out
2021-08-11T14:46:35.215Z|00021|reconnect|INFO|tcp:172.17.1.60:6642: waiting 2 seconds before reconnect
2021-08-11T14:46:37.215Z|00022|reconnect|INFO|tcp:172.17.1.60:6642: connecting...
2021-08-11T14:46:39.057Z|00023|reconnect|INFO|tcp:172.17.1.60:6642: connection attempt failed (No route to host)
2021-08-11T14:46:39.057Z|00024|reconnect|INFO|tcp:172.17.1.60:6642: waiting 4 seconds before reconnect
2021-08-11T14:46:41.545Z|00025|memory|INFO|4488 kB peak resident set size after 10.0 seconds
2021-08-11T14:46:43.057Z|00026|reconnect|INFO|tcp:172.17.1.60:6642: connecting...
2021-08-11T14:46:45.069Z|00027|reconnect|INFO|tcp:172.17.1.60:6642: connection attempt failed (No route to host)
2021-08-11T14:46:45.069Z|00028|reconnect|INFO|tcp:172.17.1.60:6642: continuing to reconnect in the background but suppressing further logging
2021-08-11T14:47:31.553Z|00029|ovsdb_idl|WARN|Dropped 269083 log messages in last 60 seconds (most recently, 0 seconds ago) due to excessive rate
2021-08-11T14:47:31.553Z|00030|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"}
2021-08-11T14:48:31.553Z|00031|ovsdb_idl|WARN|Dropped 271389 log messages in last 60 seconds (most recently, 0 seconds ago) due to excessive rate
2021-08-11T14:48:31.553Z|00032|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"}
2021-08-11T14:49:31.553Z|00033|ovsdb_idl|WARN|Dropped 266958 log messages in last 60 seconds (most recently, 0 seconds ago) due to excessive rate
2021-08-11T14:49:31.553Z|00034|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"}
2021-08-11T14:50:31.553Z|00035|ovsdb_idl|WARN|Dropped 270291 log messages in last 60 seconds (most recently, 0 seconds ago) due to excessive rate
2021-08-11T14:50:31.553Z|00036|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"}
2021-08-11T14:51:31.553Z|00037|ovsdb_idl|WARN|Dropped 267861 log messages in last 60 seconds (most recently, 0 seconds ago) due to excessive rate
2021-08-11T14:51:31.553Z|00038|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"}



Version-Release number of selected component (if applicable):

docker ps | grep ovn_controller
c9aa688033bf        undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ovn-controller:16.2_20210804.1   "dumb-init --singl..."   45 minutes ago      Up 45 minutes                                   ovn_controller

[root@compute-0 ~]# docker exec -u root -it ovn_controller rpm -qa | grep openv
network-scripts-openvswitch2.15-2.15.0-26.el8fdp.x86_64
openvswitch2.15-2.15.0-26.el8fdp.x86_64
python3-openvswitch2.15-2.15.0-26.el8fdp.x86_64
rhosp-network-scripts-openvswitch-2.15-4.el8ost.1.noarch
openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch
rhosp-openvswitch-2.15-4.el8ost.1.noarch
python3-rhosp-openvswitch-2.15-4.el8ost.1.noarch
[root@compute-0 ~]# docker exec -u root -it ovn_controller rpm -qa | grep ovn
ovn-2021-21.06.0-17.el8fdp.x86_64
rhosp-ovn-2021-4.el8ost.1.noarch
ovn-2021-host-21.06.0-17.el8fdp.x86_64
rhosp-ovn-host-2021-4.el8ost.1.noarch

[root@compute-0 ~]# rpm -qa | grep openvswitch
rhosp-openvswitch-2.11-0.7.el7ost.noarch
rhosp-openvswitch-ovn-central-2.11-0.7.el7ost.noarch
python-openvswitch2.11-2.11.3-86.el7fdp.x86_64
rhosp-openvswitch-ovn-host-2.11-0.7.el7ost.noarch
openvswitch-selinux-extra-policy-1.0-17.el7fdp.noarch
python-rhosp-openvswitch-2.11-0.7.el7ost.noarch
openstack-neutron-openvswitch-12.1.1-42.el7ost.noarch
openvswitch2.11-2.11.3-86.el7fdp.x86_64