I'm not sure if this bug is valid for 13. I deployed OSP13 z8 and ran a minor update to a puddle with OVS 2.11. After the converge step, I can see the following:


 Docker container set: ovn-dbs-bundle [192.168.24.1:8787/rhosp13/openstack-ovn-northd:pcmklatest]
   ovn-dbs-bundle-0     (ocf::ovn:ovndb-servers):       Slave controller-0
   ovn-dbs-bundle-1     (ocf::ovn:ovndb-servers):       Master controller-1
   ovn-dbs-bundle-2     (ocf::ovn:ovndb-servers):       Slave controller-2


[root@controller-0 ~]# docker exec -it ovn-dbs-bundle-docker-0 ovn-nbctl -V
ovn-nbctl (Open vSwitch) 2.11.1
DB Schema 5.16.0

[root@controller-0 ~]# docker exec -it ovn-dbs-bundle-docker-0 ovn-nbctl --db="tcp:172.17.1.12:6641" ls-add sw0
[root@controller-0 ~]# docker exec -it ovn-dbs-bundle-docker-0 ovn-nbctl --db="tcp:172.17.1.12:6641" ls-list
ae42865e-0adf-451f-bab1-ca0756d87c99 (neutron-91f3a27c-2550-4a8f-bc53-9708549efa59)
210de217-083b-4d4b-b13a-1f123e6311ee (sw0)


Looks like all is in a good shape and I didn't have to cleanup the resource.
Shall we close this BZ or you think I missed something?

Thanks,
Daniel

Hi,

I don't have enough details to answer here.  As you put it the output looks good.  We can see that the ovndb moved in a expected way during update. Either, you're test is missing something, or the problem doesn't happen on osp13 for some reason to be investigated.

I'm on pto, so you could check with somebody else in upgrade dfg if your test procedure is correct, or with pidone maybe to check why that one is working.

Thanks,

Hi,

I've been able to test that one on an update of osp13 GA to latest.

The job run there[1], it fails on last reboot on an unrelated ci related issue[2].  To check I:

 - ensure the ovn tasks added by that review were actually run during the controller update[3] : checked
 - ensure the cluster was functional at the end of the update[4] : checked

[1] http://staging-jenkins2-qe-playground.usersys.redhat.com/job/DFG-upgrades-updates-13-from-ga-HA-ipv4/8/
[2] ovn reboot not yet supported by rhos-qe-jenkins https://code.engineering.redhat.com/gerrit/#/c/187789/
[3] log are from the undercloud-0/home/stack/overcloud_update_run_Controller.log file:

2019-12-13 19:16:32 | TASK [Clear ovndb cluster pacemaker error] *************************************
2019-12-13 19:16:32 | changed: [controller-2] => {"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.371630", "end": "2019-12-14 00:16:04.477486", "rc": 0, "start": "2019-12-14 00:16:
04.105856", "stderr": "", "stderr_lines": [], "stdout": "Cleaned up ovn-dbs-bundle-docker-0 on controller-2\nCleaned up ovn-dbs-bundle-docker-0 on controller-1\nCleaned up ovn-dbs-bundle-docker-0 on controller-0\n
Cleaned up ovn-dbs-bundle-0 on controller-2\nCleaned up ovn-dbs-bundle-0 on controller-1\nCleaned up ovn-dbs-bundle-0 on controller-0\nCleaned up ovn-dbs-bundle-docker-1 on controller-2\nCleaned up ovn-dbs-bundle-
docker-1 on controller-1\nCleaned up ovn-dbs-bundle-docker-1 on controller-0\nCleaned up ovn-dbs-bundle-1 on controller-2\nCleaned up ovn-dbs-bundle-1 on controller-1\nCleaned up ovn-dbs-bundle-1 on controller-0\n
Cleaned up ovn-dbs-bundle-docker-2 on controller-2\nCleaned up ovn-dbs-bundle-docker-2 on controller-1\nCleaned up ovn-dbs-bundle-docker-2 on controller-0\nCleaned up ovn-dbs-bundle-2 on controller-2\nCleaned up o
vn-dbs-bundle-2 on controller-1\nCleaned up ovn-dbs-bundle-2 on controller-0\nCleaned up ovndb_servers:0 on ovn-dbs-bundle-0\nCleaned up ovndb_servers:1 on ovn-dbs-bundle-1\nCleaned up ovndb_servers:2 on ovn-dbs-b
undle-2", "stdout_lines": ["Cleaned up ovn-dbs-bundle-docker-0 on controller-2", "Cleaned up ovn-dbs-bundle-docker-0 on controller-1", "Cleaned up ovn-dbs-bundle-docker-0 on controller-0", "Cleaned up ovn-dbs-bund
le-0 on controller-2", "Cleaned up ovn-dbs-bundle-0 on controller-1", "Cleaned up ovn-dbs-bundle-0 on controller-0", "Cleaned up ovn-dbs-bundle-docker-1 on controller-2", "Cleaned up ovn-dbs-bundle-docker-1 on con
troller-1", "Cleaned up ovn-dbs-bundle-docker-1 on controller-0", "Cleaned up ovn-dbs-bundle-1 on controller-2", "Cleaned up ovn-dbs-bundle-1 on controller-1", "Cleaned up ovn-dbs-bundle-1 on controller-0", "Clean
ed up ovn-dbs-bundle-docker-2 on controller-2", "Cleaned up ovn-dbs-bundle-docker-2 on controller-1", "Cleaned up ovn-dbs-bundle-docker-2 on controller-0", "Cleaned up ovn-dbs-bundle-2 on controller-2", "Cleaned u
p ovn-dbs-bundle-2 on controller-1", "Cleaned up ovn-dbs-bundle-2 on controller-0", "Cleaned up ovndb_servers:0 on ovn-dbs-bundle-0", "Cleaned up ovndb_servers:1 on ovn-dbs-bundle-1", "Cleaned up ovndb_servers:2 o
n ovn-dbs-bundle-2"]}
2019-12-13 19:16:32 | TASK [Ban ovndb resource on the current node.] *********************************
2019-12-13 19:16:32 | changed: [controller-2] => {"changed": true, "cmd": "pcs resource ban ovn-dbs-bundle $(hostname | cut -d. -f1)", "delta": "0:00:00.549494", "end": "2019-12-14 00:16:05.390602", "rc": 0, "star
t": "2019-12-14 00:16:04.841108", "stderr": "", "stderr_lines": [], "stdout": "Warning: Creating location constraint cli-ban-ovn-dbs-bundle-on-controller-2 with a score of -INFINITY for resource ovn-dbs-bundle on
node controller-2.\nThis will prevent ovn-dbs-bundle from running on controller-2 until the constraint is removed. This will be the case even if controller-2 is the last node in the cluster.", "stdout_lines": ["Wa
rning: Creating location constraint cli-ban-ovn-dbs-bundle-on-controller-2 with a score of -INFINITY for resource ovn-dbs-bundle on node controller-2.", "This will prevent ovn-dbs-bundle from running on controller
-2 until the constraint is removed. This will be the case even if controller-2 is the last node in the cluster."]}
2019-12-13 19:32:48 | TASK [Remove the ban] **********************************************************
2019-12-13 19:33:18 | changed: [controller-2] => {"changed": true, "cmd": "pcs resource clear ovn-dbs-bundle", "delta": "0:00:00.885100", "end": "2019-12-14 00:32:47.250469", "rc": 0, "start": "2019-12-14 00:32:46
.365369", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
2019-12-13 19:50:46 | TASK [Clear ovndb cluster pacemaker error] *************************************
2019-12-13 19:50:46 | changed: [controller-1] => {"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.400108", "end": "2019-12-14 00:50:23.613883", "rc": 0, "start": "2019-12-14 00:50:
23.213775", "stderr": "", "stderr_lines": [], "stdout": "Cleaned up ovn-dbs-bundle-docker-0 on controller-2\nCleaned up ovn-dbs-bundle-docker-0 on controller-1\nCleaned up ovn-dbs-bundle-docker-0 on controller-0\n
Cleaned up ovn-dbs-bundle-0 on controller-2\nCleaned up ovn-dbs-bundle-0 on controller-1\nCleaned up ovn-dbs-bundle-0 on controller-0\nCleaned up ovn-dbs-bundle-docker-1 on controller-2\nCleaned up ovn-dbs-bundle-
docker-1 on controller-1\nCleaned up ovn-dbs-bundle-docker-1 on controller-0\nCleaned up ovn-dbs-bundle-1 on controller-2\nCleaned up ovn-dbs-bundle-1 on controller-1\nCleaned up ovn-dbs-bundle-1 on controller-0\n
2019-12-13 19:16:32 | TASK [Clear ovndb cluster pacemaker error] *************************************
2019-12-13 19:16:32 | changed: [controller-2] => {"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.371630", "end": "2019-12-14 00:16:04.477486", "rc": 0, "start": "2019-12-14 00:16:
04.105856", "stderr": "", "stderr_lines": [], "stdout": "Cleaned up ovn-dbs-bundle-docker-0 on controller-2\nCleaned up ovn-dbs-bundle-docker-0 on controller-1\nCleaned up ovn-dbs-bundle-docker-0 on controller-0\n
Cleaned up ovn-dbs-bundle-0 on controller-2\nCleaned up ovn-dbs-bundle-0 on controller-1\nCleaned up ovn-dbs-bundle-0 on controller-0\nCleaned up ovn-dbs-bundle-docker-1 on controller-2\nCleaned up ovn-dbs-bundle-
docker-1 on controller-1\nCleaned up ovn-dbs-bundle-docker-1 on controller-0\nCleaned up ovn-dbs-bundle-1 on controller-2\nCleaned up ovn-dbs-bundle-1 on controller-1\nCleaned up ovn-dbs-bundle-1 on controller-0\n
Cleaned up ovn-dbs-bundle-docker-2 on controller-2\nCleaned up ovn-dbs-bundle-docker-2 on controller-1\nCleaned up ovn-dbs-bundle-docker-2 on controller-0\nCleaned up ovn-dbs-bundle-2 on controller-2\nCleaned up o
vn-dbs-bundle-2 on controller-1\nCleaned up ovn-dbs-bundle-2 on controller-0\nCleaned up ovndb_servers:0 on ovn-dbs-bundle-0\nCleaned up ovndb_servers:1 on ovn-dbs-bundle-1\nCleaned up ovndb_servers:2 on ovn-dbs-b
undle-2", "stdout_lines": ["Cleaned up ovn-dbs-bundle-docker-0 on controller-2", "Cleaned up ovn-dbs-bundle-docker-0 on controller-1", "Cleaned up ovn-dbs-bundle-docker-0 on controller-0", "Cleaned up ovn-dbs-bund
le-0 on controller-2", "Cleaned up ovn-dbs-bundle-0 on controller-1", "Cleaned up ovn-dbs-bundle-0 on controller-0", "Cleaned up ovn-dbs-bundle-docker-1 on controller-2", "Cleaned up ovn-dbs-bundle-docker-1 on con
troller-1", "Cleaned up ovn-dbs-bundle-docker-1 on controller-0", "Cleaned up ovn-dbs-bundle-1 on controller-2", "Cleaned up ovn-dbs-bundle-1 on controller-1", "Cleaned up ovn-dbs-bundle-1 on controller-0", "Clean
ed up ovn-dbs-bundle-docker-2 on controller-2", "Cleaned up ovn-dbs-bundle-docker-2 on controller-1", "Cleaned up ovn-dbs-bundle-docker-2 on controller-0", "Cleaned up ovn-dbs-bundle-2 on controller-2", "Cleaned u
p ovn-dbs-bundle-2 on controller-1", "Cleaned up ovn-dbs-bundle-2 on controller-0", "Cleaned up ovndb_servers:0 on ovn-dbs-bundle-0", "Cleaned up ovndb_servers:1 on ovn-dbs-bundle-1", "Cleaned up ovndb_servers:2 o
n ovn-dbs-bundle-2"]}
2019-12-13 19:16:32 | TASK [Ban ovndb resource on the current node.] *********************************
2019-12-13 19:16:32 | changed: [controller-2] => {"changed": true, "cmd": "pcs resource ban ovn-dbs-bundle $(hostname | cut -d. -f1)", "delta": "0:00:00.549494", "end": "2019-12-14 00:16:05.390602", "rc": 0, "star
t": "2019-12-14 00:16:04.841108", "stderr": "", "stderr_lines": [], "stdout": "Warning: Creating location constraint cli-ban-ovn-dbs-bundle-on-controller-2 with a score of -INFINITY for resource ovn-dbs-bundle on
node controller-2.\nThis will prevent ovn-dbs-bundle from running on controller-2 until the constraint is removed. This will be the case even if controller-2 is the last node in the cluster.", "stdout_lines": ["Wa
rning: Creating location constraint cli-ban-ovn-dbs-bundle-on-controller-2 with a score of -INFINITY for resource ovn-dbs-bundle on node controller-2.", "This will prevent ovn-dbs-bundle from running on controller
-2 until the constraint is removed. This will be the case even if controller-2 is the last node in the cluster."]}
2019-12-13 19:32:48 | TASK [Remove the ban] **********************************************************
2019-12-13 19:33:18 | changed: [controller-2] => {"changed": true, "cmd": "pcs resource clear ovn-dbs-bundle", "delta": "0:00:00.885100", "end": "2019-12-14 00:32:47.250469", "rc": 0, "start": "2019-12-14 00:32:46
.365369", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
2019-12-13 19:50:46 | TASK [Clear ovndb cluster pacemaker error] *************************************
2019-12-13 19:50:46 | changed: [controller-1] => {"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.400108", "end": "2019-12-14 00:50:23.613883", "rc": 0, "start": "2019-12-14 00:50:
23.213775", "stderr": "", "stderr_lines": [], "stdout": "Cleaned up ovn-dbs-bundle-docker-0 on controller-2\nCleaned up ovn-dbs-bundle-docker-0 on controller-1\nCleaned up ovn-dbs-bundle-docker-0 on controller-0\n
Cleaned up ovn-dbs-bundle-0 on controller-2\nCleaned up ovn-dbs-bundle-0 on controller-1\nCleaned up ovn-dbs-bundle-0 on controller-0\nCleaned up ovn-dbs-bundle-docker-1 on controller-2\nCleaned up ovn-dbs-bundle-
docker-1 on controller-1\nCleaned up ovn-dbs-bundle-docker-1 on controller-0\nCleaned up ovn-dbs-bundle-1 on controller-2\nCleaned up ovn-dbs-bundle-1 on controller-1\nCleaned up ovn-dbs-bundle-1 on controller-0\n
Cleaned up ovn-dbs-bundle-docker-2 on controller-2\nCleaned up ovn-dbs-bundle-docker-2 on controller-1\nCleaned up ovn-dbs-bundle-docker-2 on controller-0\nCleaned up ovn-dbs-bundle-2 on controller-2\nCleaned up o
vn-dbs-bundle-2 on controller-1\nCleaned up ovn-dbs-bundle-2 on controller-0\nCleaned up ovndb_servers:0 on ovn-dbs-bundle-0\nCleaned up ovndb_servers:1 on ovn-dbs-bundle-1\nCleaned up ovndb_servers:2 on ovn-dbs-b
undle-2", "stdout_lines": ["Cleaned up ovn-dbs-bundle-docker-0 on controller-2", "Cleaned up ovn-dbs-bundle-docker-0 on controller-1", "Cleaned up ovn-dbs-bundle-docker-0 on controller-0", "Cleaned up ovn-dbs-bund
le-0 on controller-2", "Cleaned up ovn-dbs-bundle-0 on controller-1", "Cleaned up ovn-dbs-bundle-0 on controller-0", "Cleaned up ovn-dbs-bundle-docker-1 on controller-2", "Cleaned up ovn-dbs-bundle-docker-1 on con
troller-1", "Cleaned up ovn-dbs-bundle-docker-1 on controller-0", "Cleaned up ovn-dbs-bundle-1 on controller-2", "Cleaned up ovn-dbs-bundle-1 on controller-1", "Cleaned up ovn-dbs-bundle-1 on controller-0", "Clean
ed up ovn-dbs-bundle-docker-2 on controller-2", "Cleaned up ovn-dbs-bundle-docker-2 on controller-1", "Cleaned up ovn-dbs-bundle-docker-2 on controller-0", "Cleaned up ovn-dbs-bundle-2 on controller-2", "Cleaned u
p ovn-dbs-bundle-2 on controller-1", "Cleaned up ovn-dbs-bundle-2 on controller-0", "Cleaned up ovndb_servers:0 on ovn-dbs-bundle-0", "Cleaned up ovndb_servers:1 on ovn-dbs-bundle-1", "Cleaned up ovndb_servers:2 o
n ovn-dbs-bundle-2"]}
2019-12-13 19:50:46 | TASK [Ban ovndb resource on the current node.] *********************************
2019-12-13 19:50:46 | changed: [controller-1] => {"changed": true, "cmd": "pcs resource ban ovn-dbs-bundle $(hostname | cut -d. -f1)", "delta": "0:00:00.568389", "end": "2019-12-14 00:50:24.536692", "rc": 0, "star
t": "2019-12-14 00:50:23.968303", "stderr": "", "stderr_lines": [], "stdout": "Warning: Creating location constraint cli-ban-ovn-dbs-bundle-on-controller-1 with a score of -INFINITY for resource ovn-dbs-bundle on
node controller-1.\nThis will prevent ovn-dbs-bundle from running on controller-1 until the constraint is removed. This will be the case even if controller-1 is the last node in the cluster.", "stdout_lines": ["Wa
rning: Creating location constraint cli-ban-ovn-dbs-bundle-on-controller-1 with a score of -INFINITY for resource ovn-dbs-bundle on node controller-1.", "This will prevent ovn-dbs-bundle from running on controller
-1 until the constraint is removed. This will be the case even if controller-1 is the last node in the cluster."]}
2019-12-13 20:08:18 | TASK [Remove the ban] **********************************************************
2019-12-13 20:08:18 | changed: [controller-1] => {"changed": true, "cmd": "pcs resource clear ovn-dbs-bundle", "delta": "0:00:00.963508", "end": "2019-12-14 01:07:49.254015", "rc": 0, "start": "2019-12-14 01:07:48
.290507", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
2019-12-13 20:36:02 | TASK [Clear ovndb cluster pacemaker error] *************************************
2019-12-13 20:36:02 | changed: [controller-0] => {"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.458971", "end": "2019-12-14 01:25:53.602726", "rc": 0, "start": "2019-12-14 01:25:
53.143755", "stderr": "", "stderr_lines": [], "stdout": "Cleaned up ovn-dbs-bundle-docker-0 on controller-2\nCleaned up ovn-dbs-bundle-docker-0 on controller-1\nCleaned up ovn-dbs-bundle-docker-0 on controller-0\n
Cleaned up ovn-dbs-bundle-0 on controller-2\nCleaned up ovn-dbs-bundle-0 on controller-1\nCleaned up ovn-dbs-bundle-0 on controller-0\nCleaned up ovn-dbs-bundle-docker-1 on controller-2\nCleaned up ovn-dbs-bundle-
docker-1 on controller-1\nCleaned up ovn-dbs-bundle-docker-1 on controller-0\nCleaned up ovn-dbs-bundle-1 on controller-2\nCleaned up ovn-dbs-bundle-1 on controller-1\nCleaned up ovn-dbs-bundle-1 on controller-0\n
Cleaned up ovn-dbs-bundle-docker-2 on controller-2\nCleaned up ovn-dbs-bundle-docker-2 on controller-1\nCleaned up ovn-dbs-bundle-docker-2 on controller-0\nCleaned up ovn-dbs-bundle-2 on controller-2\nCleaned up o
vn-dbs-bundle-2 on controller-1\nCleaned up ovn-dbs-bundle-2 on controller-0\nCleaned up ovndb_servers:0 on ovn-dbs-bundle-0\nCleaned up ovndb_servers:1 on ovn-dbs-bundle-1\nCleaned up ovndb_servers:2 on ovn-dbs-b


[4] from pcslog on the controller-0:

Cluster name: tripleo_cluster
Stack: corosync
Current DC: controller-2 (version 1.1.20-5.el7_7.2-3c4c782f70) - partition with quorum
Last updated: Sat Dec 14 07:55:36 2019
Last change: Sat Dec 14 07:26:30 2019 by redis-bundle-1 via crm_attribute on controller-1

15 nodes configured
47 resources configured

Online: [ controller-0 controller-1 controller-2 ]
GuestOnline: [ galera-bundle-0@controller-0 galera-bundle-1@controller-1 galera-bundle-2@controller-2 ovn-dbs-bundle-0@controller-0 ovn-dbs-bundle-1@controller-2 ovn-dbs-bundle-2@controller-1 rabbitmq-bundle-0@controller-0 rabbitmq-bundle-1@controller-1 rabbitmq-bundle-2@controller-2 redis-bundle-0@controller-0 redis-bundle-1@controller-1 redis-bundle-2@controller-2 ]

Full list of resources:

 Docker container set: rabbitmq-bundle [192.168.24.1:8787/rhosp13/openstack-rabbitmq:pcmklatest]
   rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Started controller-0
   rabbitmq-bundle-1    (ocf::heartbeat:rabbitmq-cluster):      Started controller-1
   rabbitmq-bundle-2    (ocf::heartbeat:rabbitmq-cluster):      Started controller-2
 Docker container set: galera-bundle [192.168.24.1:8787/rhosp13/openstack-mariadb:pcmklatest]
   galera-bundle-0      (ocf::heartbeat:galera):        Master controller-0
   galera-bundle-1      (ocf::heartbeat:galera):        Master controller-1
   galera-bundle-2      (ocf::heartbeat:galera):        Master controller-2
 Docker container set: redis-bundle [192.168.24.1:8787/rhosp13/openstack-redis:pcmklatest]
   redis-bundle-0       (ocf::heartbeat:redis): Slave controller-0
   redis-bundle-1       (ocf::heartbeat:redis): Master controller-1
   redis-bundle-2       (ocf::heartbeat:redis): Slave controller-2
 ip-192.168.24.7        (ocf::heartbeat:IPaddr2):       Started controller-2
 ip-10.0.0.101  (ocf::heartbeat:IPaddr2):       Started controller-1
 ip-172.17.1.17 (ocf::heartbeat:IPaddr2):       Started controller-2
 ip-172.17.1.11 (ocf::heartbeat:IPaddr2):       Started controller-2
 ip-172.17.3.18 (ocf::heartbeat:IPaddr2):       Started controller-2
 ip-172.17.4.18 (ocf::heartbeat:IPaddr2):       Started controller-1
 Docker container set: haproxy-bundle [192.168.24.1:8787/rhosp13/openstack-haproxy:pcmklatest]
   haproxy-bundle-docker-0      (ocf::heartbeat:docker):        Started controller-0
   haproxy-bundle-docker-1      (ocf::heartbeat:docker):        Started controller-1
   haproxy-bundle-docker-2      (ocf::heartbeat:docker):        Started controller-2
 Docker container set: ovn-dbs-bundle [192.168.24.1:8787/rhosp13/openstack-ovn-northd:pcmklatest]
   ovn-dbs-bundle-0     (ocf::ovn:ovndb-servers):       Slave controller-0
   ovn-dbs-bundle-1     (ocf::ovn:ovndb-servers):       Master controller-2
   ovn-dbs-bundle-2     (ocf::ovn:ovndb-servers):       Slave controller-1
 Docker container: openstack-cinder-volume [192.168.24.1:8787/rhosp13/openstack-cinder-volume:pcmklatest]
   openstack-cinder-volume-docker-0     (ocf::heartbeat:docker):        Started controller-2
 Docker container: openstack-cinder-backup [192.168.24.1:8787/rhosp13/openstack-cinder-backup:pcmklatest]
   openstack-cinder-backup-docker-0     (ocf::heartbeat:docker):        Started controller-2

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4335