This bug was initially created as a copy of Bug #1992705 I am copying this bug because: The functionality fixed in the previous release regressed. See https://bugzilla.redhat.com/show_bug.cgi?id=1992705#c13 Description of problem: OVN CI jobs started to fail - https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/upgrades/view/ffu/ Root cause is that during the upgrade controller gets to latest version where OVN container is running on host with OVS2.13 but the computes need "hybrid" mode where only OVN container is switched to the latest version and the host stays the same with OVS2.11. This worked well up until new version of OVN. Error in logs: 2021-08-11T14:44:54.158Z|00038|reconnect|INFO|tcp:172.17.1.46:6642: continuing to reconnect in the background but suppressing further logging 2021-08-11T14:46:26.727Z|00039|fatal_signal|WARN|terminating with signal 15 (Terminated) 2021-08-11T14:46:31.546Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovn-controller.log 2021-08-11T14:46:31.548Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting... 2021-08-11T14:46:31.548Z|00003|reconnect|INFO|unix:/run/openvswitch/db.sock: connected 2021-08-11T14:46:31.550Z|00004|ovsdb_idl|WARN|Open_vSwitch database lacks Datapath table (database needs upgrade?) 2021-08-11T14:46:31.550Z|00005|ovsdb_idl|WARN|Open_vSwitch table in Open_vSwitch database lacks datapaths column (database needs upgrade?) 2021-08-11T14:46:31.550Z|00006|ovsdb_idl|WARN|Open_vSwitch database lacks Datapath table (database needs upgrade?) 2021-08-11T14:46:31.551Z|00007|ovsdb_idl|WARN|Open_vSwitch table in Open_vSwitch database lacks datapaths column (database needs upgrade?) 2021-08-11T14:46:31.553Z|00008|main|INFO|OVN internal version is : [21.06.0-20.17.0-56.0] 2021-08-11T14:46:31.553Z|00009|main|INFO|OVS IDL reconnected, force recompute. 2021-08-11T14:46:31.553Z|00010|reconnect|INFO|tcp:172.17.1.46:6642: connecting... 2021-08-11T14:46:31.553Z|00011|main|INFO|OVNSB IDL reconnected, force recompute. 2021-08-11T14:46:31.553Z|00012|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"} 2021-08-11T14:46:31.554Z|00013|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"} 2021-08-11T14:46:31.554Z|00014|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"} 2021-08-11T14:46:31.555Z|00015|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"} 2021-08-11T14:46:31.555Z|00016|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"} 2021-08-11T14:46:32.215Z|00017|reconnect|INFO|tcp:172.17.1.60:6642: connecting... 2021-08-11T14:46:33.215Z|00018|reconnect|INFO|tcp:172.17.1.60:6642: connection attempt timed out 2021-08-11T14:46:34.215Z|00019|reconnect|INFO|tcp:172.17.1.60:6642: connecting... 2021-08-11T14:46:35.215Z|00020|reconnect|INFO|tcp:172.17.1.60:6642: connection attempt timed out 2021-08-11T14:46:35.215Z|00021|reconnect|INFO|tcp:172.17.1.60:6642: waiting 2 seconds before reconnect 2021-08-11T14:46:37.215Z|00022|reconnect|INFO|tcp:172.17.1.60:6642: connecting... 2021-08-11T14:46:39.057Z|00023|reconnect|INFO|tcp:172.17.1.60:6642: connection attempt failed (No route to host) 2021-08-11T14:46:39.057Z|00024|reconnect|INFO|tcp:172.17.1.60:6642: waiting 4 seconds before reconnect 2021-08-11T14:46:41.545Z|00025|memory|INFO|4488 kB peak resident set size after 10.0 seconds 2021-08-11T14:46:43.057Z|00026|reconnect|INFO|tcp:172.17.1.60:6642: connecting... 2021-08-11T14:46:45.069Z|00027|reconnect|INFO|tcp:172.17.1.60:6642: connection attempt failed (No route to host) 2021-08-11T14:46:45.069Z|00028|reconnect|INFO|tcp:172.17.1.60:6642: continuing to reconnect in the background but suppressing further logging 2021-08-11T14:47:31.553Z|00029|ovsdb_idl|WARN|Dropped 269083 log messages in last 60 seconds (most recently, 0 seconds ago) due to excessive rate 2021-08-11T14:47:31.553Z|00030|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"} 2021-08-11T14:48:31.553Z|00031|ovsdb_idl|WARN|Dropped 271389 log messages in last 60 seconds (most recently, 0 seconds ago) due to excessive rate 2021-08-11T14:48:31.553Z|00032|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"} 2021-08-11T14:49:31.553Z|00033|ovsdb_idl|WARN|Dropped 266958 log messages in last 60 seconds (most recently, 0 seconds ago) due to excessive rate 2021-08-11T14:49:31.553Z|00034|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"} 2021-08-11T14:50:31.553Z|00035|ovsdb_idl|WARN|Dropped 270291 log messages in last 60 seconds (most recently, 0 seconds ago) due to excessive rate 2021-08-11T14:50:31.553Z|00036|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"} 2021-08-11T14:51:31.553Z|00037|ovsdb_idl|WARN|Dropped 267861 log messages in last 60 seconds (most recently, 0 seconds ago) due to excessive rate 2021-08-11T14:51:31.553Z|00038|ovsdb_idl|WARN|transaction error: {"details":"datapaths is not a valid column name","error":"syntax error","syntax":"[\"datapaths\"]"} Version-Release number of selected component (if applicable): docker ps | grep ovn_controller c9aa688033bf undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-ovn-controller:16.2_20210804.1 "dumb-init --singl..." 45 minutes ago Up 45 minutes ovn_controller [root@compute-0 ~]# docker exec -u root -it ovn_controller rpm -qa | grep openv network-scripts-openvswitch2.15-2.15.0-26.el8fdp.x86_64 openvswitch2.15-2.15.0-26.el8fdp.x86_64 python3-openvswitch2.15-2.15.0-26.el8fdp.x86_64 rhosp-network-scripts-openvswitch-2.15-4.el8ost.1.noarch openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch rhosp-openvswitch-2.15-4.el8ost.1.noarch python3-rhosp-openvswitch-2.15-4.el8ost.1.noarch [root@compute-0 ~]# docker exec -u root -it ovn_controller rpm -qa | grep ovn ovn-2021-21.06.0-17.el8fdp.x86_64 rhosp-ovn-2021-4.el8ost.1.noarch ovn-2021-host-21.06.0-17.el8fdp.x86_64 rhosp-ovn-host-2021-4.el8ost.1.noarch [root@compute-0 ~]# rpm -qa | grep openvswitch rhosp-openvswitch-2.11-0.7.el7ost.noarch rhosp-openvswitch-ovn-central-2.11-0.7.el7ost.noarch python-openvswitch2.11-2.11.3-86.el7fdp.x86_64 rhosp-openvswitch-ovn-host-2.11-0.7.el7ost.noarch openvswitch-selinux-extra-policy-1.0-17.el7fdp.noarch python-rhosp-openvswitch-2.11-0.7.el7ost.noarch openstack-neutron-openvswitch-12.1.1-42.el7ost.noarch openvswitch2.11-2.11.3-86.el7fdp.x86_64