Bug 1740100

Summary:	OSP 14->15: after staged upgrade some pacemaker-managed services run just 1 instance
Product:	Red Hat OpenStack	Reporter:	Jiri Stransky <jstransk>
Component:	openstack-tripleo-heat-templates	Assignee:	Jiri Stransky <jstransk>
Status:	CLOSED ERRATA	QA Contact:	Sasha Smolyak <ssmolyak>
Severity:	medium	Docs Contact:
Priority:	high
Version:	15.0 (Stein)	CC:	lmiccini, mburns, michele
Target Milestone:	---	Keywords:	Triaged, ZStream
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	openstack-tripleo-heat-templates-10.6.1-0.20190819180520.6a38682.el8ost.noarch	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-05 11:59:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1727807

Description Jiri Stransky 2019-08-12 09:56:38 UTC

Description of problem:

Upgrade of controllers went without errors, but from the pacemaker-managed services, only Galera scaled up as expected:

[root@controller-0 ~]# pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: controller-0 (version 2.0.1-4.el8_0.3-0eb7991564) - partition with quorum
Last updated: Mon Aug 12 09:55:41 2019
Last change: Fri Aug  9 17:38:19 2019 by root via cibadmin on controller-0

8 nodes configured
25 resources configured

Online: [ controller-0 controller-1 controller-2 ]
GuestOnline: [ galera-bundle-0@controller-0 galera-bundle-1@controller-1 galera-bundle-2@controller-2 rabbitmq-bundle-0@controller-0 redis-bundle-0@controller-0 ]

Full list of resources:

 podman container set: galera-bundle [192.168.24.1:8787/rhosp15/openstack-mariadb:pcmklatest]
   galera-bundle-0      (ocf::heartbeat:galera):        Master controller-0
   galera-bundle-1      (ocf::heartbeat:galera):        Master controller-1
   galera-bundle-2      (ocf::heartbeat:galera):        Master controller-2
 podman container: rabbitmq-bundle [192.168.24.1:8787/rhosp15/openstack-rabbitmq:pcmklatest]
   rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Started controller-0
 podman container: redis-bundle [192.168.24.1:8787/rhosp15/openstack-redis:pcmklatest]
   redis-bundle-0       (ocf::heartbeat:redis): Master controller-0
 ip-192.168.24.8        (ocf::heartbeat:IPaddr2):       Started controller-0
 ip-10.0.0.106  (ocf::heartbeat:IPaddr2):       Started controller-0
 ip-172.17.1.16 (ocf::heartbeat:IPaddr2):       Started controller-0
 ip-172.17.1.23 (ocf::heartbeat:IPaddr2):       Started controller-0
 ip-172.17.3.11 (ocf::heartbeat:IPaddr2):       Started controller-0
 ip-172.17.4.25 (ocf::heartbeat:IPaddr2):       Started controller-0
 podman container set: haproxy-bundle [192.168.24.1:8787/rhosp15/openstack-haproxy:pcmklatest]
   haproxy-bundle-podman-0      (ocf::heartbeat:podman):        Started controller-0
   haproxy-bundle-podman-1      (ocf::heartbeat:podman):        Stopped
   haproxy-bundle-podman-2      (ocf::heartbeat:podman):        Stopped
 podman container: openstack-cinder-volume [192.168.24.1:8787/rhosp15/openstack-cinder-volume:pcmklatest]
   openstack-cinder-volume-podman-0     (ocf::heartbeat:podman):        Started controller-0

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled



Version-Release number of selected component (if applicable):

openstack-tripleo-heat-templates-10.6.1-0.20190801110459.7fbedf0.el8ost.noarch

+ patches applied according to current WIP upgrade guide.


Steps to Reproduce:

https://gitlab.cee.redhat.com/osp15/osp-upgrade-el8/blob/de2c507dcefb0854cc53289bbe3c512af137112e/README.md

Comment 3 Jiri Stransky 2019-08-13 12:25:01 UTC

Testing https://review.opendev.org/#/c/676173/

Comment 5 Jiri Stransky 2019-08-19 15:33:56 UTC

Merged to stable/stein.

Comment 13 Alex McLeod 2020-02-19 12:44:14 UTC

If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.

Comment 16 errata-xmlrpc 2020-03-05 11:59:15 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0643