Bug 1771854

Summary: [ovsdb-server] Allow replicating from older schema servers
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Numan Siddique <nusiddiq>
Component: openvswitch2.12Assignee: Numan Siddique <nusiddiq>
Status: CLOSED ERRATA QA Contact: Jianlin Shi <jishi>
Severity: medium Docs Contact:
Priority: unspecified    
Version: FDP 20.ACC: ctrautma, dalvarez, fleitner, jhsiao, jishi, kfida, ovs-qe, ralongi, sathlang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1760763
: 1775795 (view as bug list) Environment:
Last Closed: 2019-12-11 12:14:59 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1760763    
Bug Blocks: 1759974, 1766586, 1775795    

Description Numan Siddique 2019-11-13 07:07:21 UTC 
+++ This bug was initially created as a clone of Bug #1760763 +++

When we have a pacemaker based cluster of ovsdb-servers and we try to upgrade them one by one, there can be situations where the DB schema is updated.

Right now, ovsdb-server backup instances connecting to a master ovsdb-server with an older schema will refuse to replicate and this may have issues to get the pacemaker cluster back to a stable state.

The purpose of this BZ is to allow ovsdb-server to replicate from a master instance which is on an older schema. Since backwards compatibility of the schema is guaranteed, this should have no side effect and ensures consistency.

--- Additional comment from Numan Siddique on 2019-11-04 13:15:05 UTC ---

The fix is available in openvswitch2.11-2.11.0-27-el8fdn for now.
It should be available in next FDP release.
Comment 2 Jianlin Shi 2019-11-25 03:15:44 UTC 
on which version is the bug fixed?
Comment 3 Jianlin Shi 2019-11-25 03:31:50 UTC 
succeed to reproduce the issue on openvswitch2.11-2.11.0-26:

master 11.1.1.2 installed:
[root@dell-per740-12 bz1771854]# rpm -qa | grep -E "openvswitch|ovn"
ovn2.11-central-2.11.0-6.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-19.el8fdp.noarch
ovn2.11-2.11.0-6.el8fdp.x86_64
ovn2.11-host-2.11.0-6.el8fdp.x86_64
openvswitch2.11-2.11.0-26.el8fdp.x86_64
[root@dell-per740-12 bz1771854]# ovn-nbctl --version
ovn-nbctl (Open vSwitch) 2.11.0
DB Schema 5.14.0

and start pcs:
iptables -F
setenforce 0
(sleep 2;echo "hacluster"; sleep 2; echo "redhat" ) | pcs host auth 11.1.1.2 11.1.1.11
pcs cluster setup my_cluster --force --start 11.1.1.2 11.1.1.11
pcs cluster enable --all
pcs property set stonith-enabled=false
pcs property set no-quorum-policy=ignore
pcs cluster cib tmp-cib.xml    
cp tmp-cib.xml tmp-cib.deltasrc
pcs status    
pcs -f tmp-cib.xml resource create ip-11.1.1.50 ocf:heartbeat:IPaddr2 ip=11.1.1.50   op monitor interval=30s
pcs -f tmp-cib.xml resource create ovndb_servers  ocf:ovn:ovndb-servers manage_northd=yes master_ip=11.1.1.50 nb_master_port=6641 sb_master_port=6642 promotable
pcs -f tmp-cib.xml resource meta ovndb_servers-clone notify=true
pcs -f tmp-cib.xml constraint order start ip-11.1.1.50 then promote ovndb_servers-clone
pcs -f tmp-cib.xml constraint colocation add ip-11.1.1.50 with master ovndb_servers-clone
pcs -f tmp-cib.xml constraint location ip-11.1.1.50 prefers 11.1.1.2=1000
pcs -f tmp-cib.xml constraint location ovndb_servers-clone prefers 11.1.1.2=1000
pcs -f tmp-cib.xml constraint location ip-11.1.1.50 prefers 11.1.1.11=500
pcs -f tmp-cib.xml constraint location ovndb_servers-clone prefers 11.1.1.11=500
pcs cluster cib-push tmp-cib.xml diff-against=tmp-cib.deltasrc
pcs status

slave 11.1.1.11 installed:

[root@ibm-x3650m5-03 bz1771854]# rpm -qa | grep -E "openvswitch|ovn"                                  
openvswitch-selinux-extra-policy-1.0-19.el8fdp.noarch
ovn2.11-host-2.11.1-20.el8fdp.x86_64
openvswitch2.11-2.11.0-26.el8fdp.x86_64
ovn2.11-central-2.11.1-20.el8fdp.x86_64
ovn2.11-2.11.1-20.el8fdp.x86_64
[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl --version
ovn-nbctl (Open vSwitch) 2.11.1
DB Schema 5.16.0

steps:

after start pcs on master, then run 

[root@dell-per740-12 bz1771854]# ovn-nbctl ls-add ls1                                                 
[root@dell-per740-12 bz1771854]# ovn-nbctl show                                                       
switch 9278dfa5-cc86-4ed1-b17f-bf6791f2340c (ls1

then
pcs cluster stop 11.1.1.2

the ovn would change to 11.1.1.11:
[root@ibm-x3650m5-03 bz1771854]# pcs status
Cluster name: my_cluster                                                                              
Cluster Summary:                                                                                      
  * Stack: corosync                                                                                   
  * Current DC: 11.1.1.11 (version 2.0.3-2.el8-2c9cea563e) - partition with quorum                    
  * Last updated: Sun Nov 24 22:29:39 2019                                                            
  * Last change:  Sun Nov 24 22:12:52 2019 by root via crm_attribute on 11.1.1.11                     
  * 2 nodes configured                                                                                
  * 3 resource instances configured                                                                   
                                                                                                      
Node List:                                                                                            
  * Online: [ 11.1.1.11 ]                                                                             
  * OFFLINE: [ 11.1.1.2 ]                                                                             
                                                                                                      
Full List of Resources:                                                                               
  * ip-11.1.1.50        (ocf::heartbeat:IPaddr2):        Started 11.1.1.11                            
  * Clone Set: ovndb_servers-clone [ovndb_servers] (promotable):                                      
    * Masters: [ 11.1.1.11 ]                                                                          
    * Stopped: [ 11.1.1.2 ]                                                                           
                                                                                                      
Failed Resource Actions:                                                                              
  * ovndb_servers_monitor_30000 on 11.1.1.11 'not running' (7): call=15, status='complete', exitreason='', last-rc-change='2019-11-24 22:11:57 -05:00', queued=0ms, exec=57ms
                                                                                                      
Daemon Status:                                                                                        
  corosync: active/enabled                                                                            
  pacemaker: active/enabled                                                                           
  pcsd: active/disabled

but no ls showed:
[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl show

from /var/log/openvswitch/ovsdb-server-nb.log I get:
2019-11-25T03:11:57.758Z|00017|replication|INFO|Schema version mismatch, OVN_Northbound not replicateed
Comment 4 Numan Siddique 2019-11-26 16:31:44 UTC 
Hi Jianlin Shi,

Please note the fix for this BZ is available in openvswitch2.11-2.11.0-27 and openvswitch2.12- 2.12.0-6
Comment 5 Jianlin Shi 2019-11-28 04:35:36 UTC 
reproduced on openvswitch2.12-2.12.0-4.el7fdp.x86_64:

setup on server 11.1.1.2:
iptables -F
setenforce 0
(sleep 2;echo "hacluster"; sleep 2; echo "redhat" ) | pcs cluster auth 11.1.1.2 11.1.1.11
pcs cluster setup --force --start --name my_cluster 11.1.1.2 11.1.1.11
pcs cluster enable --all
pcs property set stonith-enabled=false
pcs property set no-quorum-policy=ignore
pcs cluster cib tmp-cib.xml    
cp tmp-cib.xml tmp-cib.deltasrc
pcs status    
pcs -f tmp-cib.xml resource create ip-11.1.1.50 ocf:heartbeat:IPaddr2 ip=11.1.1.50   op monitor interval=30s
pcs -f tmp-cib.xml resource create ovndb_servers  ocf:ovn:ovndb-servers manage_northd=yes master_ip=11.1.1.50 nb_master_port=6641 sb_master_port=6642 master
pcs -f tmp-cib.xml resource meta ovndb_servers-master notify=true
pcs -f tmp-cib.xml constraint order start ip-11.1.1.50 then promote ovndb_servers-master
pcs -f tmp-cib.xml constraint colocation add ip-11.1.1.50 with master ovndb_servers-master
pcs -f tmp-cib.xml constraint location ip-11.1.1.50 prefers 11.1.1.2=1000
pcs -f tmp-cib.xml constraint location ovndb_servers-master prefers 11.1.1.2=1000
pcs -f tmp-cib.xml constraint location ip-11.1.1.50 prefers 11.1.1.11=500
pcs -f tmp-cib.xml constraint location ovndb_servers-master prefers 11.1.1.11=500
pcs cluster cib-push tmp-cib.xml diff-against=tmp-cib.deltasrc
pcs status

ovs on server with old ovn db schema:
[root@dell-per740-12 openvswitch]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch
oovn2.11-2.11.1-20.el7fdp.x86_64                                                                      
ovn2.11-central-2.11.1-20.el7fdp.x86_64                                                               
ovn2.11-host-2.11.1-20.el7fdp.x86_64                                                                  
vopenvswitch2.12-2.12.0-4.el7fdp.x86_64                                                               
n[root@dell-per740-12 openvswitch]# ovn-nbctl --version
ovn-nbctl (Open vSwitch) 2.11.1                                                                       
DB Schema 5.16.0 

ovs on client 11.1.1.11 with new ovn db schema:
[root@ibm-x3650m5-03 ovs2.12.0-4]# rpm -qa | grep -E "openvswitch|ovn"                                
ovn2.11-2.11.0-6.el7fdp.x86_64                                                                        
openvswitch2.12-2.12.0-4.el7fdp.x86_64
ovn2.11-central-2.11.0-6.el7fdp.x86_64                                                                
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch
ovn2.11-host-2.11.0-6.el7fdp.x86_64
[root@ibm-x3650m5-03 ovs2.12.0-4]# ovn-nbctl --version
ovn-nbctl (Open vSwitch) 2.11.0
DB Schema 5.14.0

stop pcs cluster 11.1.1.2 to let it be on client and create ls:

[root@ibm-x3650m5-03 ovs2.12.0-4]# pcs status                                                         
Cluster name: my_cluster                                                                              
                                                                                                      
WARNINGS:                                                                                             
Corosync and pacemaker node names do not match (IPs used in setup?)                                   
                                                                                                      
Stack: corosync                                                                                       
Current DC: ibm-x3650m5-03.rhts.eng.pek2.redhat.com (version 1.1.21-3.el7-f14e36fd43) - partition with quorum
Last updated: Wed Nov 27 22:51:06 2019                                                                
Last change: Wed Nov 27 22:50:42 2019 by root via crm_attribute on ibm-x3650m5-03.rhts.eng.pek2.redhat.com
                                                                                                      
2 nodes configured                                                                                    
3 resources configured                                                                                
                                                                                                      
Online: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]                                                   
OFFLINE: [ dell-per740-12.rhts.eng.pek2.redhat.com ]                                                  
                                                                                                      
Full list of resources:                                                                               
                                                                                                      
 ip-11.1.1.50   (ocf::heartbeat:IPaddr2):       Started ibm-x3650m5-03.rhts.eng.pek2.redhat.com       
 Master/Slave Set: ovndb_servers-master [ovndb_servers]
     Masters: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]
     Stopped: [ dell-per740-12.rhts.eng.pek2.redhat.com ]
                                                                                                      
Failed Resource Actions:
* ovndb_servers_monitor_30000 on ibm-x3650m5-03.rhts.eng.pek2.redhat.com 'not running' (7): call=15, status=complete, exitreason='',
    last-rc-change='Wed Nov 27 22:49:38 2019', queued=0ms, exec=63ms                                  
                                                                                                      
Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/disabled
[root@ibm-x3650m5-03 ovs2.12.0-4]# ovn-nbctl ls-add ls1                                               
[root@ibm-x3650m5-03 ovs2.12.0-4]# ovn-nbctl show                                                     
switch 29ac621f-1dcb-4638-a9dc-b897155910be (ls1)

then start 11.1.1.2 with pcs cluster start 11.1.1.2, master would change to 11.1.1.2 and pcs cluster stop 11.1.1.11:

[root@dell-per740-12 openvswitch]# pcs status
Cluster name: my_cluster

WARNINGS:
Corosync and pacemaker node names do not match (IPs used in setup?)

Stack: corosync
Current DC: dell-per740-12.rhts.eng.pek2.redhat.com (version 1.1.21-3.el7-f14e36fd43) - partition with quorum
Last updated: Wed Nov 27 22:57:28 2019
Last change: Wed Nov 27 22:57:24 2019 by root via crm_attribute on dell-per740-12.rhts.eng.pek2.redhat.com

2 nodes configured
3 resources configured

Online: [ dell-per740-12.rhts.eng.pek2.redhat.com ]
OFFLINE: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]

Full list of resources:

 ip-11.1.1.50   (ocf::heartbeat:IPaddr2):       Started dell-per740-12.rhts.eng.pek2.redhat.com
 Master/Slave Set: ovndb_servers-master [ovndb_servers]
     Masters: [ dell-per740-12.rhts.eng.pek2.redhat.com ]
     Stopped: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/disabled
[root@dell-per740-12 openvswitch]# ovn-nbctl show

<=== no ls showed

[root@dell-per740-12 openvswitch]# grep mismatch -i /var/log/openvswitch/ovsdb-server-nb.log 
2019-11-28T03:51:33.401Z|00015|replication|INFO|Schema version mismatch, OVN_Northbound not replicated

<=== get error message
Comment 6 Jianlin Shi 2019-11-28 04:38:33 UTC 
no openvswitch2.12- 2.12.0-6 fdp package, installed openvswitch2.12- 2.12.0-8, and get the following result:

2019-11-28T04:33:47.841Z|00015|replication|INFO|Schema version mismatch, checking if OVN_Northbound can still be replicated or not.
2019-11-28T04:33:47.842Z|00016|replication|INFO|Column load_balancer mismatch in local db schema table Logical_Switch.
2019-11-28T04:33:47.842Z|00017|replication|INFO|OVN_Northbound cannot be replicated.                  
2019-11-28T04:33:47.842Z|00018|replication|WARN|Nothing to replicate.

<==== showed can't be replicated.

the master version is:
[root@dell-per740-12 bz1771854]# rpm -qa | grep -E "openvswitch|ovn"
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch
ovn2.11-host-2.11.0-6.el7fdp.x86_64                                                                   
openvswitch2.12-2.12.0-8.el7fdp.x86_64                                                                
ovn2.11-2.11.0-6.el7fdp.x86_64                                                                        
ovn2.11-central-2.11.0-6.el7fdp.x86_64

and the other node:
[root@ibm-x3650m5-03 openvswitch]# rpm -qa | grep -E "openvswitch|ovn"
ovn2.11-2.11.1-20.el7fdp.x86_64
openvswitch2.12-2.12.0-8.el7fdp.x86_64                                                                
ovn2.11-central-2.11.1-20.el7fdp.x86_64                                                               
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch                                                 
ovn2.11-host-2.11.1-20.el7fdp.x86_64


Numan, please have a look
Comment 7 Jianlin Shi 2019-11-29 04:29:35 UTC 
with reproducer in https://bugzilla.redhat.com/show_bug.cgi?id=1775795#c2,

script to start pcs:
iptables -F
setenforce 0
(sleep 2;echo "hacluster"; sleep 2; echo "redhat" ) | pcs cluster auth 11.1.1.2 11.1.1.11
pcs cluster setup --force --start --name my_cluster 11.1.1.2 11.1.1.11
pcs cluster enable --all
pcs property set stonith-enabled=false
pcs property set no-quorum-policy=ignore
pcs cluster cib tmp-cib.xml    
cp tmp-cib.xml tmp-cib.deltasrc
pcs status    
pcs -f tmp-cib.xml resource create ip-11.1.1.50 ocf:heartbeat:IPaddr2 ip=11.1.1.50   op monitor interval=30s
pcs -f tmp-cib.xml resource create ovndb_servers  ocf:ovn:ovndb-servers manage_northd=yes master_ip=11.1.1.50 nb_master_port=6641 sb_master_port=6642 master
pcs -f tmp-cib.xml resource meta ovndb_servers-master notify=true
pcs -f tmp-cib.xml constraint order start ip-11.1.1.50 then promote ovndb_servers-master
pcs -f tmp-cib.xml constraint colocation add ip-11.1.1.50 with master ovndb_servers-master
pcs -f tmp-cib.xml constraint location ip-11.1.1.50 prefers 11.1.1.2=1000
pcs -f tmp-cib.xml constraint location ovndb_servers-master prefers 11.1.1.2=1000
pcs -f tmp-cib.xml constraint location ip-11.1.1.50 prefers 11.1.1.11=500
pcs -f tmp-cib.xml constraint location ovndb_servers-master prefers 11.1.1.11=500
pcs cluster cib-push tmp-cib.xml diff-against=tmp-cib.deltasrc
pcs status



reproduced on openvswitch2.12.0-4:

[root@ibm-x3650m5-03 bz1771854]# pcs status
Cluster name: my_cluster

WARNINGS:
Corosync and pacemaker node names do not match (IPs used in setup?)

Stack: corosync
Current DC: ibm-x3650m5-03.rhts.eng.pek2.redhat.com (version 1.1.21-3.el7-f14e36fd43) - partition with quorum
Last updated: Thu Nov 28 22:59:45 2019
Last change: Thu Nov 28 22:58:52 2019 by root via crm_attribute on ibm-x3650m5-03.rhts.eng.pek2.redhat.com

2 nodes configured
3 resources configured

Online: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]                                                   
OFFLINE: [ dell-per740-12.rhts.eng.pek2.redhat.com ]                                                  

Full list of resources:

 ip-11.1.1.50   (ocf::heartbeat:IPaddr2):       Started ibm-x3650m5-03.rhts.eng.pek2.redhat.com
 Master/Slave Set: ovndb_servers-master [ovndb_servers]                                               
     Masters: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]                                             
     Stopped: [ dell-per740-12.rhts.eng.pek2.redhat.com ]                                             

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/disabled                                                                               
[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl show                                                       
2019-11-29T03:59:49Z|00001|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)
2019-11-29T03:59:49Z|00002|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)
2019-11-29T03:59:49Z|00003|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
switch d3dd369a-d127-40cc-8d5a-32a5047efd6a (ls1)                                                     
[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl ls-add ls2                                                 
2019-11-29T03:59:53Z|00002|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)
2019-11-29T03:59:53Z|00003|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)
2019-11-29T03:59:53Z|00004|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl show                                                       
2019-11-29T03:59:54Z|00001|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)
2019-11-29T03:59:54Z|00002|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)
2019-11-29T03:59:54Z|00003|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)
switch a43b0570-f10f-46fa-8edc-a5eb910dcbd0 (ls2)                                                     
switch d3dd369a-d127-40cc-8d5a-32a5047efd6a (ls1)

<==== create ls2 on node B

[root@dell-per740-12 bz1771854]# cp 3.16.0/* /usr/share/openvswitch/                                      
cp: overwrite ‘/usr/share/openvswitch/ovn-nb.ovsschema’? y                                            
cp: overwrite ‘/usr/share/openvswitch/ovn-sb.ovsschema’? y                                            
[root@dell-per740-12 bz1771854]# pcs cluster start 11.1.1.2                                           
11.1.1.2: Starting Cluster (corosync)...                                                              
11.1.1.2: Starting Cluster (pacemaker)...                                                             
[root@dell-per740-12 bz1771854]# ovn-nbctl show                                                       
switch d3dd369a-d127-40cc-8d5a-32a5047efd6a (ls1)

<===== ls2 can't be showed on node A                                                     
[root@dell-per740-12 bz1771854]# rpm -qa | grep -E "openvswitch|ovn"                                  
ovn2.11-2.11.1-20.el7fdp.x86_64                                                                       
openvswitch2.12-2.12.0-4.el7fdp.x86_64                                                                
ovn2.11-central-2.11.1-20.el7fdp.x86_64                                                               
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch                                                 
ovn2.11-host-2.11.1-20.el7fdp.x86_64

2019-11-29T04:00:22.377Z|00015|replication|INFO|Schema version mismatch, OVN_Northbound not replicated
2019-11-29T04:00:22.378Z|00016|replication|WARN|Nothing to replicate.

<=== log for ovsdb-nb

Verified on openvswitch2.12.0-8:

[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl ls-add ls2                                                 
2019-11-29T04:24:46Z|00002|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)                                                                  
2019-11-29T04:24:46Z|00003|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)                                                                              
2019-11-29T04:24:46Z|00004|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)                                                     
[root@ibm-x3650m5-03 bz1771854]# ovn-nbctl show
2019-11-29T04:24:48Z|00001|ovsdb_idl|WARN|Logical_Router table in OVN_Northbound database lacks policies column (database needs upgrade?)                                                                  
2019-11-29T04:24:48Z|00002|ovsdb_idl|WARN|OVN_Northbound database lacks Logical_Router_Policy table (database needs upgrade?)                                                                              
2019-11-29T04:24:48Z|00003|ovsdb_idl|WARN|Logical_Switch_Port table in OVN_Northbound database lacks ha_chassis_group column (database needs upgrade?)                                                     
switch 198dbea0-6526-4268-8fea-be5b69c53f7d (ls1)                                                     
switch 060f10c1-7533-4d40-afd6-dda7b63c560f (ls2)

<==== add ls2 on node B

[root@ibm-x3650m5-03 bz1771854]# pcs status                                                           
Cluster name: my_cluster                                                                                                      
      
WARNINGS:        
Corosync and pacemaker node names do not match (IPs used in setup?)                                   
                                
Stack: corosync                    
Current DC: ibm-x3650m5-03.rhts.eng.pek2.redhat.com (version 1.1.21-3.el7-f14e36fd43) - partition with quorum                                                                                              
Last updated: Thu Nov 28 23:25:20 2019                               
Last change: Thu Nov 28 23:24:36 2019 by root via crm_attribute on ibm-x3650m5-03.rhts.eng.pek2.redhat.com                                                                                                 
                                               
2 nodes configured                                                                                                                       
3 resources configured                                                                                                       
                                                                                                                                                      
Online: [ dell-per740-12.rhts.eng.pek2.redhat.com ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]           
                                                                                                      
Full list of resources:                                                                                                                  
                                                                                                                             
 ip-11.1.1.50   (ocf::heartbeat:IPaddr2):       Started ibm-x3650m5-03.rhts.eng.pek2.redhat.com                                                       
 Master/Slave Set: ovndb_servers-master [ovndb_servers]                                               
     Masters: [ ibm-x3650m5-03.rhts.eng.pek2.redhat.com ]                                             
     Slaves: [ dell-per740-12.rhts.eng.pek2.redhat.com ]                                                                                 
                                                                                                                             
Daemon Status:                                                                                                                                        
  corosync: active/enabled                                                                            
  pacemaker: active/enabled                                                                                                              
  pcsd: active/disabled

[root@dell-per740-12 bz1771854]# cp 3.16.0/ovn-* /usr/share/openvswitch/
cp: overwrite ‘/usr/share/openvswitch/ovn-nb.ovsschema’? y
cp: overwrite ‘/usr/share/openvswitch/ovn-sb.ovsschema’? y
[root@dell-per740-12 bz1771854]#
[root@dell-per740-12 bz1771854]#
[root@dell-per740-12 bz1771854]# pcs cluster start 11.1.1.2
11.1.1.2: Starting Cluster (corosync)...       
11.1.1.2: Starting Cluster (pacemaker)...

<==== copy 3.16.0 schema and start on node A

[root@dell-per740-12 bz1771854]# ovn-nbctl show
switch 198dbea0-6526-4268-8fea-be5b69c53f7d (ls1)
switch 060f10c1-7533-4d40-afd6-dda7b63c560f (ls2)

<==== ls2 can be shown on node A

[root@dell-per740-12 bz1771854]# rpm -qa | grep -E "openvswitch|ovn"
ovn2.11-2.11.1-20.el7fdp.x86_64
openvswitch2.12-2.12.0-8.el7fdp.x86_64
ovn2.11-central-2.11.1-20.el7fdp.x86_64
openvswitch-selinux-extra-policy-1.0-14.el7fdp.noarch
ovn2.11-host-2.11.1-20.el7fdp.x86_64

2019-11-29T04:25:16.931Z|00015|replication|INFO|Schema version mismatch, checking if OVN_Northbound caa
n still be replicated or not.
2019-11-29T04:25:16.932Z|00016|replication|INFO|OVN_Northbound can be replicated.
2019-11-29T04:25:16.933Z|00017|replication|INFO|Monitor request received. Resetting the database

<=== message in /var/log/openvswitch/ovsdb-server-nb.log

set VERIFIED
Comment 9 errata-xmlrpc 2019-12-11 12:14:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4206