Bug 1547581

Summary: FFU: openstack-cinder-volume pcs resource is not deleted after running upgrade-steps_playbook.yaml
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: openstack-tripleo-heat-templatesAssignee: Marios Andreou <mandreou>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 13.0 (Queens)CC: dbecker, jfrancoa, jschluet, lbezdick, mandreou, mbracho, mbultel, mburns, morazi, rhel-osp-director-maint
Target Milestone: betaKeywords: Reopened, Triaged
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: openstack-tripleo-heat-templates-8.0.2-0.20180416194362.29a5ad5.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-27 13:45:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Marius Cornea 2018-02-21 15:01:19 UTC
Description of problem:

FFU: openstack-cinder-volume pcs resource is not deleted after running upgrade-steps_playbook.yaml. We should probably delete the pcs resource and start the service inside a container during deploy steps.

PCS resources:

root@controller-0 heat-admin]# pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: controller-0 (version 1.1.16-12.el7_4.7-94ff4df) - partition with quorum
Last updated: Wed Feb 21 02:20:05 2018
Last change: Wed Feb 21 02:19:57 2018 by root via cibadmin on controller-0

4 nodes configured
17 resources configured

Online: [ controller-0 ]
GuestOnline: [ galera-bundle-0@controller-0 rabbitmq-bundle-0@controller-0 redis-bundle-0@controller-0 ]

Full list of resources:

 ip-	(ocf::heartbeat:IPaddr2):	Started controller-0
 ip-	(ocf::heartbeat:IPaddr2):	Started controller-0
 ip-	(ocf::heartbeat:IPaddr2):	Started controller-0
 ip-	(ocf::heartbeat:IPaddr2):	Started controller-0
 ip-	(ocf::heartbeat:IPaddr2):	Started controller-0
 ip-	(ocf::heartbeat:IPaddr2):	Started controller-0
 openstack-cinder-volume	(systemd:openstack-cinder-volume):	Started controller-0
 Docker container: rabbitmq-bundle [trunk.registry.rdoproject.org/master/centos-binary-rabbitmq:pcmklatest]
   rabbitmq-bundle-0	(ocf::heartbeat:rabbitmq-cluster):	Started controller-0
 Docker container: galera-bundle [trunk.registry.rdoproject.org/master/centos-binary-mariadb:pcmklatest]
   galera-bundle-0	(ocf::heartbeat:galera):	Master controller-0
 Docker container: redis-bundle [trunk.registry.rdoproject.org/master/centos-binary-redis:pcmklatest]
   redis-bundle-0	(ocf::heartbeat:redis):	Master controller-0
 Docker container: haproxy-bundle [trunk.registry.rdoproject.org/master/centos-binary-haproxy:pcmklatest]
   haproxy-bundle-docker-0	(ocf::heartbeat:docker):	Started controller-0

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Comment 2 Marios Andreou 2018-03-07 14:57:03 UTC
please see comment at https://bugzilla.redhat.com/show_bug.cgi?id=1547570#c3

Comment 3 Marios Andreou 2018-03-07 14:57:28 UTC

*** This bug has been marked as a duplicate of bug 1547570 ***

Comment 4 Marius Cornea 2018-03-07 15:25:21 UTC
As we synced on IRC based on my tests with the latest set of FFU patches I can see that the rabbitmq and redis pcs resources get deleted and only the cinder-volume remains created so I am re-opening this bug  to keep track of it.

Comment 7 Marios Andreou 2018-03-21 16:53:54 UTC
started poking at this today, WRT defaults the resource registry has it like [0] i.e. dockerized. In my setup, I have docker-ha so cinder-volume instead points to [1] - the block on line 80 should take care of removing the existing cinder-volume cluster resource (and i have damien's patch applied from [2])

After a completed run of the upgrade_steps_playbook + deploy_steps_playbook the pcs resource for cinder-volume (systemd) is still around on my environment too, sounds like the same issue as reported by mcornea (still investigating). 

[0] https://github.com/openstack/tripleo-heat-templates/blob/51537c0e4bac520e440f1bc06373d62fc9b8189f/overcloud-resource-registry-puppet.j2.yaml#L115 
[1] https://github.com/openstack/tripleo-heat-templates/blob/51537c0e4bac520e440f1bc06373d62fc9b8189f/docker/services/pacemaker/cinder-volume.yaml#L267-L280
[2] https://review.openstack.org/#/c/535859/18/doc/source/install/post_deployment/upgrade.rst@639

Comment 8 Marius Cornea 2018-04-10 18:33:29 UTC
FWIW this is fixed for FFU so I think we can close this bug unless we want to track this issue for regular upgrade.

Comment 12 Scott Lewis 2018-04-30 14:59:49 UTC
This item has been properly Triaged and planned for the OSP13 release, and is being tagged for tracking. For details, see https://url.corp.redhat.com/1851efd

Comment 14 errata-xmlrpc 2018-06-27 13:45:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.