Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1771854

Summary: [ovsdb-server] Allow replicating from older schema servers
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Numan Siddique <nusiddiq>
Component: openvswitch2.12Assignee: Numan Siddique <nusiddiq>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: medium Docs Contact:
Priority: unspecified    
Version: FDP 20.ACC: ctrautma, dalvarez, fleitner, jhsiao, jishi, kfida, ovs-qe, ralongi, sathlang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1760763
: 1775795 (view as bug list) Environment:
Last Closed: 2019-12-11 12:14:59 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1760763    
Bug Blocks: 1759974, 1766586, 1775795    

Description Numan Siddique 2019-11-13 07:07:21 UTC 
+++ This bug was initially created as a clone of Bug #1760763 +++

When we have a pacemaker based cluster of ovsdb-servers and we try to upgrade them one by one, there can be situations where the DB schema is updated.

Right now, ovsdb-server backup instances connecting to a master ovsdb-server with an older schema will refuse to replicate and this may have issues to get the pacemaker cluster back to a stable state.

The purpose of this BZ is to allow ovsdb-server to replicate from a master instance which is on an older schema. Since backwards compatibility of the schema is guaranteed, this should have no side effect and ensures consistency.

--- Additional comment from Numan Siddique on 2019-11-04 13:15:05 UTC ---

The fix is available in openvswitch2.11-2.11.0-27-el8fdn for now.
It should be available in next FDP release.
Comment 2 Jianlin Shi 2019-11-25 03:15:44 UTC 
on which version is the bug fixed?
Comment 3 Jianlin Shi 2019-11-25 03:31:50 UTC 
succeed to reproduce the issue on openvswitch2.11-2.11.0-26:

master 11.1.1.2 installed:
[root@dell-per740-12 bz1771854]# rpm -qa | grep -E "openvswitch|ovn"
ovn2.11-central-2.11.0-6.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-19.el8fdp.noarch
ovn2.11-2.11.0-6.el8fdp.x86_64
ovn2.11-host-2.11.0-6.el8fdp.x86_64
openvswitch2.11-2.11.0-26.el8fdp.x86_64
[root@dell-per740-12 bz1771854]# ovn-nbctl --version
ovn-nbctl (Open vSwitch) 2.11.0
DB Schema 5.14.0

and start pcs:
iptables -F
setenforce 0
(sleep 2;echo "hacluster"; sleep 2; echo "redhat" ) | pcs host auth 11.1.1.2 11.1.1.11
pcs cluster setup my_cluster --force --start 11.1.1.2 11.1.1.11
pcs cluster enable --all
pcs property set stonith-enabled=false
pcs property set no-quorum-policy=ignore
pcs cluster cib tmp-cib.xml    
cp tmp-cib.xml tmp-cib.deltasrc
pcs status    
pcs -f tmp-cib.xml resource create ip-11.1.1.50 ocf:heartbeat:IPaddr2 ip=11.1.1.50   op monitor interval=30s
pcs -f tmp-cib.xml resource create ovndb_servers  ocf:ovn:ovndb-servers manage_northd=yes master_ip=11.1.1.50 nb_master_port=6641 sb_master_port=6642 promotable
pcs -f tmp-cib.xml resource meta ovndb_servers-clone notify=true
pcs -f tmp-cib.xml constraint order start ip-11.1.1.50 then promote ovndb_servers-clone
pcs -f tmp-cib.xml constraint colocation add ip-11.1.1.50 with master ovndb_servers-clone
pcs -f tmp-cib.xml constraint location ip-11.1.1.50 prefers 11.1.1.2=1000
pcs -f tmp-cib.xml constraint location ovndb_servers-clone prefers 11.1.1.2=1000
pcs -f tmp-cib.xml constraint location ip-11.1.1.50 prefers 11.1.1.11=500
pcs -f tmp-cib.xml constraint location ovndb_servers-clone prefers 11.1.1.11=500
pcs cluster cib-push tmp-cib.xml diff-against=tmp-cib.deltasrc
pcs status

slave 11.1.1.11 installed:

[root@ibm-x3650m5-03 bz1771854]# rpm -qa | grep -E "openvswitch|ovn"                                  
openvswitch-selinux-extra-policy-1.0-19.el8fdp.noarch
ovn2.11-host-2.11.1-20.el8fdp.x86_64
openvswitch2.11-2.11.0-26.el8fdp.x86_64
ovn2.11-central-2.11.1-20.el8fdp.x86_64
ovn2.11-2.11.1-20.el8fdp.x86_64
[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl --version
ovn-nbctl (Open vSwitch) 2.11.1
DB Schema 5.16.0

steps:

after start pcs on master, then run 

[root@dell-per740-12 bz1771854]# ovn-nbctl ls-add ls1                                                 
[root@dell-per740-12 bz1771854]# ovn-nbctl show                                                       
switch 9278dfa5-cc86-4ed1-b17f-bf6791f2340c (ls1

then
pcs cluster stop 11.1.1.2

the ovn would change to 11.1.1.11:
[root@ibm-x3650m5-03 bz1771854]# pcs status
Cluster name: my_cluster                                                                              
Cluster Summary:                                                                                      
  * Stack: corosync                                                                                   
  * Current DC: 11.1.1.11 (version 2.0.3-2.el8-2c9cea563e) - partition with quorum                    
  * Last updated: Sun Nov 24 22:29:39 2019                                                            
  * Last change:  Sun Nov 24 22:12:52 2019 by root via crm_attribute on 11.1.1.11                     
  * 2 nodes configured                                                                                
  * 3 resource instances configured                                                                   
                                                                                                      
Node List:                                                                                            
  * Online: [ 11.1.1.11 ]                                                                             
  * OFFLINE: [ 11.1.1.2 ]                                                                             
                                                                                                      
Full List of Resources:                                                                               
  * ip-11.1.1.50        (ocf::heartbeat:IPaddr2):        Started 11.1.1.11                            
  * Clone Set: ovndb_servers-clone [ovndb_servers] (promotable):                                      
    * Masters: [ 11.1.1.11 ]                                                                          
    * Stopped: [ 11.1.1.2 ]                                                                           
                                                                                                      
Failed Resource Actions:                                                                              
  * ovndb_servers_monitor_30000 on 11.1.1.11 'not running' (7): call=15, status='complete', exitreason='', last-rc-change='2019-11-24 22:11:57 -05:00', queued=0ms, exec=57ms
                                                                                                      
Daemon Status:                                                                                        
  corosync: active/enabled                                                                            
  pacemaker: active/enabled                                                                           
  pcsd: active/disabled

but no ls showed:
[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl show

from /var/log/openvswitch/ovsdb-server-nb.log I get:
2019-11-25T03:11:57.758Z|00017|replication|INFO|Schema version mismatch, OVN_Northbound not replicateed
Comment 4 Numan Siddique 2019-11-26 16:31:44 UTC 
Hi Jianlin Shi,

Please note the fix for this BZ is available in openvswitch2.11-2.11.0-27 and openvswitch2.12- 2.12.0-6
Comment 5 Jianlin Shi 2019-11-28 04:35:36 UTC 
reproduced on openvswitch2.12-2.12.0-4.el7fdp.x86_64:

setup on server 11.1.1.2:
iptables -F
setenforce 0
(sleep 2;echo "hacluster"; sleep 2; echo "redhat" ) | pcs cluster auth 11.1.1.2 11.1.1.11
pcs cluster setup --force --start --name my_cluster 11.1.1.2 11.1.1.11
pcs cluster enable --all
pcs property set stonith-enabled=false
pcs property set no-quorum-policy=ignore
pcs cluster cib tmp-cib.xml    
cp tmp-cib.xml tmp-cib.deltasrc
pcs status    
pcs -f tmp-cib.xml resource create ip-11.1.1.50 ocf:heartbeat:IPaddr2 ip=11.1.1.50   op monitor interval=30s
pcs -f tmp-cib.xml resource create ovndb_servers  ocf:ovn:ovndb-servers manage_northd=yes master_ip=11.1.1.50 nb_master_port=6641 sb_master_port=6642 master
pcs -f tmp-cib.xml resource meta ovndb_servers-master notify=true
pcs -f tmp-cib.xml constraint order start ip-11.1.1.50 then promote ovndb_servers-master
pcs -f tmp-cib.xml constraint colocation add ip-11.1.1.50 with master ovndb_servers-master
pcs -f tmp-cib.xml constraint location ip-11.1.1.50 prefers 11.1.1.2=1000
pcs -f tmp-cib.xml constraint location ovndb_servers-master prefers 11.1.1.2=1000
pcs -f tmp-cib.xml constraint location ip-11.1.1.50 prefers 11.1.1.11=500
pcs -f tmp-cib.xml constraint location ovndb_servers-master prefers 11.1.1.11=500
pcs cluster cib-push tmp-cib.xml diff-against=tmp-cib.deltasrc
pcs status

ovs on server with old ovn db schema:
[root@dell-per740-12 openvswitch]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch
oovn2.11-2.11.1-20.el7fdp.x86_64                                                                      
ovn2.11-central-2.11.1-20.el7fdp.x86_64                                                               
ovn2.11-host-2.11.1-20.el7fdp.x86_64                                                                  
vopenvswitch2.12-2.12.0-4.el7fdp.x86_64                                                               
n[root@dell-per740-12 openvswitch]# ovn-nbctl --version
ovn-nbctl (Open vSwitch) 2.11.1                                                                       
DB Schema 5.16.0 

ovs on client 11.1.1.11 with new ovn db schema:
[root@ibm-x3650m5-03 ovs2.12.0-4]# rpm -qa | grep -E "openvswitch|ovn"                                
ovn2.11-2.11.0-6.el7fdp.x86_64                                                                        
openvswitch2.12-2.12.0-4.el7fdp.x86_64
ovn2.11-central-2.11.0-6.el7fdp.x86_64                                                                
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch
ovn2.11-host-2.11.0-6.el7fdp.x86_64
[root@ibm-x3650m5-03 ovs2.12.0-4]# ovn-nbctl --version
ovn-nbctl (Open vSwitch) 2.11.0
DB Schema 5.14.0

stop pcs cluster 11.1.1.2 to let it be on client and create ls:

[root@ibm-x3650m5-03 ovs2.12.0-4]# pcs status                                                         
Cluster name: my_cluster                                                                              
                                                                                                      
WARNINGS:                                                                                             
Corosync and pacemaker node names do not match (IPs used in setup?)                                   
                                                                                                      
Stack: corosync                                                                                       
Current DC: ibm-x3650m5-03.rhts.eng.pek2.redhat.com (version 1.1.21-3.el7-f14e36fd43) - partition with quorum
Last updated: Wed Nov 27 22:51:06 2019                                                                
Last change: Wed Nov 27 22:50:42 2019 by root via crm_attribute on ibm-x3650m5-03.rhts.eng.pek2.redhat.com
                                                                                                      
2 nodes configured                                                                                    
3 resources configured                                                                                
                                                                                                      
Online: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]                                                   
OFFLINE: [ dell-per740-12.rhts.eng.pek2.redhat.com ]                                                  
                                                                                                      
Full list of resources:                                                                               
                                                                                                      
 ip-11.1.1.50   (ocf::heartbeat:IPaddr2):       Started ibm-x3650m5-03.rhts.eng.pek2.redhat.com       
 Master/Slave Set: ovndb_servers-master [ovndb_servers]
     Masters: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]
     Stopped: [ dell-per740-12.rhts.eng.pek2.redhat.com ]
                                                                                                      
Failed Resource Actions:
* ovndb_servers_monitor_30000 on ibm-x3650m5-03.rhts.eng.pek2.redhat.com 'not running' (7): call=15, status=complete, exitreason='',
    last-rc-change='Wed Nov 27 22:49:38 2019', queued=0ms, exec=63ms                                  
                                                                                                      
Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/disabled
[root@ibm-x3650m5-03 ovs2.12.0-4]# ovn-nbctl ls-add ls1                                               
[root@ibm-x3650m5-03 ovs2.12.0-4]# ovn-nbctl show                                                     
switch 29ac621f-1dcb-4638-a9dc-b897155910be (ls1)

then start 11.1.1.2 with pcs cluster start 11.1.1.2, master would change to 11.1.1.2 and pcs cluster stop 11.1.1.11:

[root@dell-per740-12 openvswitch]# pcs status
Cluster name: my_cluster

WARNINGS:
Corosync and pacemaker node names do not match (IPs used in setup?)

Stack: corosync
Current DC: dell-per740-12.rhts.eng.pek2.redhat.com (version 1.1.21-3.el7-f14e36fd43) - partition with quorum
Last updated: Wed Nov 27 22:57:28 2019
Last change: Wed Nov 27 22:57:24 2019 by root via crm_attribute on dell-per740-12.rhts.eng.pek2.redhat.com

2 nodes configured
3 resources configured

Online: [ dell-per740-12.rhts.eng.pek2.redhat.com ]
OFFLINE: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]

Full list of resources:

 ip-11.1.1.50   (ocf::heartbeat:IPaddr2):       Started dell-per740-12.rhts.eng.pek2.redhat.com
 Master/Slave Set: ovndb_servers-master [ovndb_servers]
     Masters: [ dell-per740-12.rhts.eng.pek2.redhat.com ]
     Stopped: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/disabled
[root@dell-per740-12 openvswitch]# ovn-nbctl show

<=== no ls showed

[root@dell-per740-12 openvswitch]# grep mismatch -i /var/log/openvswitch/ovsdb-server-nb.log 
2019-11-28T03:51:33.401Z|00015|replication|INFO|Schema version mismatch, OVN_Northbound not replicated

<=== get error message
Comment 6 Jianlin Shi 2019-11-28 04:38:33 UTC 
no openvswitch2.12- 2.12.0-6 fdp package, installed openvswitch2.12- 2.12.0-8, and get the following result:

2019-11-28T04:33:47.841Z|00015|replication|INFO|Schema version mismatch, checking if OVN_Northbound can still be replicated or not.
2019-11-28T04:33:47.842Z|00016|replication|INFO|Column load_balancer mismatch in local db schema table Logical_Switch.
2019-11-28T04:33:47.842Z|00017|replication|INFO|OVN_Northbound cannot be replicated.                  
2019-11-28T04:33:47.842Z|00018|replication|WARN|Nothing to replicate.

<==== showed can't be replicated.

the master version is:
[root@dell-per740-12 bz1771854]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch
ovn2.11-host-2.11.0-6.el7fdp.x86_64                                                                   
openvswitch2.12-2.12.0-8.el7fdp.x86_64                                                                
ovn2.11-2.11.0-6.el7fdp.x86_64                                                                        
ovn2.11-central-2.11.0-6.el7fdp.x86_64

and the other node:
[root@ibm-x3650m5-03 openvswitch]# rpm -qa | grep -E "openvswitch|ovn"
ovn2.11-2.11.1-20.el7fdp.x86_64
openvswitch2.12-2.12.0-8.el7fdp.x86_64                                                                
ovn2.11-central-2.11.1-20.el7fdp.x86_64                                                               
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch                                                 
ovn2.11-host-2.11.1-20.el7fdp.x86_64


Numan, please have a look
Comment 7 Jianlin Shi 2019-11-29 04:29:35 UTC 
with reproducer in https://bugzilla.redhat.com/show_bug.cgi?id=1775795#c2,

script to start pcs:
iptables -F
setenforce 0
(sleep 2;echo "hacluster"; sleep 2; echo "redhat" ) | pcs cluster auth 11.1.1.2 11.1.1.11
pcs cluster setup --force --start --name my_cluster 11.1.1.2 11.1.1.11
pcs cluster enable --all
pcs property set stonith-enabled=false
pcs property set no-quorum-policy=ignore
pcs cluster cib tmp-cib.xml    
cp tmp-cib.xml tmp-cib.deltasrc
pcs status    
pcs -f tmp-cib.xml resource create ip-11.1.1.50 ocf:heartbeat:IPaddr2 ip=11.1.1.50   op monitor interval=30s
pcs -f tmp-cib.xml resource create ovndb_servers  ocf:ovn:ovndb-servers manage_northd=yes master_ip=11.1.1.50 nb_master_port=6641 sb_master_port=6642 master
pcs -f tmp-cib.xml resource meta ovndb_servers-master notify=true
pcs -f tmp-cib.xml constraint order start ip-11.1.1.50 then promote ovndb_servers-master
pcs -f tmp-cib.xml constraint colocation add ip-11.1.1.50 with master ovndb_servers-master
pcs -f tmp-cib.xml constraint location ip-11.1.1.50 prefers 11.1.1.2=1000
pcs -f tmp-cib.xml constraint location ovndb_servers-master prefers 11.1.1.2=1000
pcs -f tmp-cib.xml constraint location ip-11.1.1.50 prefers 11.1.1.11=500
pcs -f tmp-cib.xml constraint location ovndb_servers-master prefers 11.1.1.11=500
pcs cluster cib-push tmp-cib.xml diff-against=tmp-cib.deltasrc
pcs status



reproduced on openvswitch2.12.0-4:

[root@ibm-x3650m5-03 bz1771854]# pcs status
Cluster name: my_cluster

WARNINGS:
Corosync and pacemaker node names do not match (IPs used in setup?)

Stack: corosync
Current DC: ibm-x3650m5-03.rhts.eng.pek2.redhat.com (version 1.1.21-3.el7-f14e36fd43) - partition with quorum
Last updated: Thu Nov 28 22:59:45 2019
Last change: Thu Nov 28 22:58:52 2019 by root via crm_attribute on ibm-x3650m5-03.rhts.eng.pek2.redhat.com

2 nodes configured
3 resources configured

Online: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]                                                   
OFFLINE: [ dell-per740-12.rhts.eng.pek2.redhat.com ]                                                  

Full list of resources:

 ip-11.1.1.50   (ocf::heartbeat:IPaddr2):       Started ibm-x3650m5-03.rhts.eng.pek2.redhat.com
 Master/Slave Set: ovndb_servers-master [ovndb_servers]                                               
     Masters: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]                                             
     Stopped: [ dell-per740-12.rhts.eng.pek2.redhat.com ]                                             

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/disabled                                                                               
[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl show                                                       
2019-11-29T03:59:49Z|00001|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)
2019-11-29T03:59:49Z|00002|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)
2019-11-29T03:59:49Z|00003|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
switch d3dd369a-d127-40cc-8d5a-32a5047efd6a (ls1)                                                     
[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl ls-add ls2                                                 
2019-11-29T03:59:53Z|00002|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)
2019-11-29T03:59:53Z|00003|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)
2019-11-29T03:59:53Z|00004|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl show                                                       
2019-11-29T03:59:54Z|00001|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)
2019-11-29T03:59:54Z|00002|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)
2019-11-29T03:59:54Z|00003|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
switch a43b0570-f10f-46fa-8edc-a5eb910dcbd0 (ls2)                                                     
switch d3dd369a-d127-40cc-8d5a-32a5047efd6a (ls1)

<==== create ls2 on node B

[root@dell-per740-12 bz1771854]# cp 3.16.0/* /usr/share/openvswitch/                                      
cp: overwrite ‘/usr/share/openvswitch/ovn-nb.ovsschema’? y                                            
cp: overwrite ‘/usr/share/openvswitch/ovn-sb.ovsschema’? y                                            
[root@dell-per740-12 bz1771854]# pcs cluster start 11.1.1.2                                           
11.1.1.2: Starting Cluster (corosync)...                                                              
11.1.1.2: Starting Cluster (pacemaker)...                                                             
[root@dell-per740-12 bz1771854]# ovn-nbctl show                                                       
switch d3dd369a-d127-40cc-8d5a-32a5047efd6a (ls1)

<===== ls2 can't be showed on node A                                                     
[root@dell-per740-12 bz1771854]# rpm -qa | grep -E "openvswitch|ovn"                                  
ovn2.11-2.11.1-20.el7fdp.x86_64                                                                       
openvswitch2.12-2.12.0-4.el7fdp.x86_64                                                                
ovn2.11-central-2.11.1-20.el7fdp.x86_64                                                               
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch                                                 
ovn2.11-host-2.11.1-20.el7fdp.x86_64

2019-11-29T04:00:22.377Z|00015|replication|INFO|Schema version mismatch, OVN_Northbound not replicated
2019-11-29T04:00:22.378Z|00016|replication|WARN|Nothing to replicate.

<=== log for ovsdb-nb

Verified on openvswitch2.12.0-8:

[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl ls-add ls2                                                 
2019-11-29T04:24:46Z|00002|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)                                                                  
2019-11-29T04:24:46Z|00003|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)                                                                              
2019-11-29T04:24:46Z|00004|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)                                                     
[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl show
2019-11-29T04:24:48Z|00001|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)                                                                  
2019-11-29T04:24:48Z|00002|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)                                                                              
2019-11-29T04:24:48Z|00003|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)                                                     
switch 198dbea0-6526-4268-8fea-be5b69c53f7d (ls1)                                                     
switch 060f10c1-7533-4d40-afd6-dda7b63c560f (ls2)

<==== add ls2 on node B

[root@ibm-x3650m5-03 bz1771854]# pcs status                                                           
Cluster name: my_cluster                                                                                                      
      
WARNINGS:        
Corosync and pacemaker node names do not match (IPs used in setup?)                                   
                                
Stack: corosync                    
Current DC: ibm-x3650m5-03.rhts.eng.pek2.redhat.com (version 1.1.21-3.el7-f14e36fd43) - partition with quorum                                                                                              
Last updated: Thu Nov 28 23:25:20 2019                               
Last change: Thu Nov 28 23:24:36 2019 by root via crm_attribute on ibm-x3650m5-03.rhts.eng.pek2.redhat.com                                                                                                 
                                               
2 nodes configured                                                                                                                       
3 resources configured                                                                                                       
                                                                                                                                                      
Online: [ dell-per740-12.rhts.eng.pek2.redhat.com ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]           
                                                                                                      
Full list of resources:                                                                                                                  
                                                                                                                             
 ip-11.1.1.50   (ocf::heartbeat:IPaddr2):       Started ibm-x3650m5-03.rhts.eng.pek2.redhat.com                                                       
 Master/Slave Set: ovndb_servers-master [ovndb_servers]                                               
     Masters: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]                                             
     Slaves: [ dell-per740-12.rhts.eng.pek2.redhat.com ]                                                                                 
                                                                                                                             
Daemon Status:                                                                                                                                        
  corosync: active/enabled                                                                            
  pacemaker: active/enabled                                                                                                              
  pcsd: active/disabled

[root@dell-per740-12 bz1771854]# cp 3.16.0/ovn-* /usr/share/openvswitch/
cp: overwrite ‘/usr/share/openvswitch/ovn-nb.ovsschema’? y
cp: overwrite ‘/usr/share/openvswitch/ovn-sb.ovsschema’? y
[root@dell-per740-12 bz1771854]#
[root@dell-per740-12 bz1771854]#
[root@dell-per740-12 bz1771854]# pcs cluster start 11.1.1.2
11.1.1.2: Starting Cluster (corosync)...       
11.1.1.2: Starting Cluster (pacemaker)...

<==== copy 3.16.0 schema and start on node A

[root@dell-per740-12 bz1771854]# ovn-nbctl show
switch 198dbea0-6526-4268-8fea-be5b69c53f7d (ls1)
switch 060f10c1-7533-4d40-afd6-dda7b63c560f (ls2)

<==== ls2 can be shown on node A

[root@dell-per740-12 bz1771854]# rpm -qa | grep -E "openvswitch|ovn"
ovn2.11-2.11.1-20.el7fdp.x86_64
openvswitch2.12-2.12.0-8.el7fdp.x86_64
ovn2.11-central-2.11.1-20.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch
ovn2.11-host-2.11.1-20.el7fdp.x86_64

2019-11-29T04:25:16.931Z|00015|replication|INFO|Schema version mismatch, checking if OVN_Northbound caa
n still be replicated or not.
2019-11-29T04:25:16.932Z|00016|replication|INFO|OVN_Northbound can be replicated.
2019-11-29T04:25:16.933Z|00017|replication|INFO|Monitor request received. Resetting the database

<=== message in /var/log/openvswitch/ovsdb-server-nb.log

set VERIFIED
Comment 9 errata-xmlrpc 2019-12-11 12:14:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4206