Bug 1740100 - OSP 14->15: after staged upgrade some pacemaker-managed services run just 1 instance
Summary: OSP 14->15: after staged upgrade some pacemaker-managed services run just 1 i...
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 15.0 (Stein)
Hardware: x86_64
OS: Linux
Target Milestone: ---
: ---
Assignee: Jiri Stransky
QA Contact: Sasha Smolyak
Depends On:
Blocks: 1727807
TreeView+ depends on / blocked
Reported: 2019-08-12 09:56 UTC by Jiri Stransky
Modified: 2020-03-05 12:00 UTC (History)
3 users (show)

Fixed In Version: openstack-tripleo-heat-templates-10.6.1-0.20190819180520.6a38682.el8ost.noarch
Doc Type: No Doc Update
Doc Text:
Clone Of:
Last Closed: 2020-03-05 11:59:15 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
OpenStack gerrit 677057 0 'None' MERGED Force re-run of pacemaker bundle init containers during upgrade-scaleup 2020-02-20 07:33:48 UTC
Red Hat Product Errata RHBA-2020:0643 0 None None None 2020-03-05 12:00:08 UTC

Description Jiri Stransky 2019-08-12 09:56:38 UTC
Description of problem:

Upgrade of controllers went without errors, but from the pacemaker-managed services, only Galera scaled up as expected:

[root@controller-0 ~]# pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: controller-0 (version 2.0.1-4.el8_0.3-0eb7991564) - partition with quorum
Last updated: Mon Aug 12 09:55:41 2019
Last change: Fri Aug  9 17:38:19 2019 by root via cibadmin on controller-0

8 nodes configured
25 resources configured

Online: [ controller-0 controller-1 controller-2 ]
GuestOnline: [ galera-bundle-0@controller-0 galera-bundle-1@controller-1 galera-bundle-2@controller-2 rabbitmq-bundle-0@controller-0 redis-bundle-0@controller-0 ]

Full list of resources:

 podman container set: galera-bundle []
   galera-bundle-0      (ocf::heartbeat:galera):        Master controller-0
   galera-bundle-1      (ocf::heartbeat:galera):        Master controller-1
   galera-bundle-2      (ocf::heartbeat:galera):        Master controller-2
 podman container: rabbitmq-bundle []
   rabbitmq-bundle-0    (ocf::heartbeat:rabbitmq-cluster):      Started controller-0
 podman container: redis-bundle []
   redis-bundle-0       (ocf::heartbeat:redis): Master controller-0
 ip-        (ocf::heartbeat:IPaddr2):       Started controller-0
 ip-  (ocf::heartbeat:IPaddr2):       Started controller-0
 ip- (ocf::heartbeat:IPaddr2):       Started controller-0
 ip- (ocf::heartbeat:IPaddr2):       Started controller-0
 ip- (ocf::heartbeat:IPaddr2):       Started controller-0
 ip- (ocf::heartbeat:IPaddr2):       Started controller-0
 podman container set: haproxy-bundle []
   haproxy-bundle-podman-0      (ocf::heartbeat:podman):        Started controller-0
   haproxy-bundle-podman-1      (ocf::heartbeat:podman):        Stopped
   haproxy-bundle-podman-2      (ocf::heartbeat:podman):        Stopped
 podman container: openstack-cinder-volume []
   openstack-cinder-volume-podman-0     (ocf::heartbeat:podman):        Started controller-0

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Version-Release number of selected component (if applicable):


+ patches applied according to current WIP upgrade guide.

Steps to Reproduce:


Comment 3 Jiri Stransky 2019-08-13 12:25:01 UTC
Testing https://review.opendev.org/#/c/676173/

Comment 5 Jiri Stransky 2019-08-19 15:33:56 UTC
Merged to stable/stein.

Comment 13 Alex McLeod 2020-02-19 12:44:14 UTC
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.

Comment 16 errata-xmlrpc 2020-03-05 11:59:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.