Bug 1647474

Summary: [OSP9 UPDATES] RMQ fails to start after minor update and reboot
Product: Red Hat OpenStack Reporter: John Eckersberg <jeckersb>
Component: openstack-tripleo-heat-templatesAssignee: John Eckersberg <jeckersb>
Status: CLOSED ERRATA QA Contact: pkomarov
Severity: urgent Docs Contact:
Priority: urgent    
Version: 9.0 (Mitaka)CC: achernet, apevec, augol, jamsmith, jeckersb, jjoyce, joflynn, jschluet, lhh, mbultel, mburns, michele, nlevinki, pkomarov, psedlak, rbartal, slinaber, srevivo, tvignaud, yprokule
Target Milestone: zstreamKeywords: Triaged, ZStream
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-2.0.0-69.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1557519
: 1647587 (view as bug list) Environment:
Last Closed: 2018-11-27 21:19:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1533406, 1536064, 1557513, 1557519, 1647593, 1654041, 1654042    
Bug Blocks: 1647587    

Comment 1 John Eckersberg 2018-11-07 14:57:40 UTC
https://code.engineering.redhat.com/gerrit/#/c/133313/

Comment 2 John Eckersberg 2018-11-15 16:12:31 UTC
*** Bug 1644710 has been marked as a duplicate of this bug. ***

Comment 8 pkomarov 2018-11-19 16:38:48 UTC
verified, 

[stack@undercloud-0 ~]$ rhos-release -L
Installed repositories (rhel-7.6):
  9-director
  9
  ceph-1.3
  ceph-osd-1.3
  rhel-7.6
[stack@undercloud-0 ~]$ openstack stack list
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| ID                                   | Stack Name | Stack Status    | Creation Time       | Updated Time        |
+--------------------------------------+------------+-----------------+---------------------+---------------------+
| 6b8726c8-63ab-4675-b0a4-da0808931fb4 | overcloud  | UPDATE_COMPLETE | 2018-11-19T13:18:34 | 2018-11-19T14:27:39 |
+--------------------------------------+------------+-----------------+---------------------+---------------------+

[stack@undercloud-0 ~]$ cat core_puddle_version 
2018-11-15.1[stack@undercloud-0 ~]$ rpm -qa|grep openstack-tripleo-heat-templates
openstack-tripleo-heat-templates-liberty-2.0.0-69.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-69.el7ost.noarch

[stack@undercloud-0 ~]$ rpm -qa|grep openstack-puppet-modules
openstack-puppet-modules-8.1.13-8.el7ost.noarch

[stack@undercloud-0 ~]$ ansible controller -mshell -b -a'pcs cluster stop --request-timeout=300'
controller-0 | SUCCESS | rc=0 >>
Stopping Cluster (pacemaker)...
Stopping Cluster (corosync)...

controller-1 | SUCCESS | rc=0 >>
Stopping Cluster (pacemaker)...
Stopping Cluster (corosync)...

controller-2 | SUCCESS | rc=0 >>
Stopping Cluster (pacemaker)...
Stopping Cluster (corosync)...

[stack@undercloud-0 ~]$ ansible controller -mshell -b -a'reboot'


[stack@undercloud-0 ~]$ ansible controller -mshell -b -a'pcs status|grep -A1 rabbitmq-clone'
 [WARNING]: Found both group and host with same name: undercloud

controller-0 | SUCCESS | rc=0 >>
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ controller-0 controller-1 controller-2 ]

controller-2 | SUCCESS | rc=0 >>
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ controller-0 controller-1 controller-2 ]

controller-1 | SUCCESS | rc=0 >>
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ controller-0 controller-1 controller-2 ]



[stack@undercloud-0 ~]$ ansible controller -mshell -b -a' rabbitmqctl eval "rabbit_mnesia:cluster_status_from_mnesia()."'
 [WARNING]: Found both group and host with same name: undercloud

controller-0 | SUCCESS | rc=0 >>
{ok,{['rabbit@controller-1','rabbit@controller-2','rabbit@controller-0'],
     ['rabbit@controller-0','rabbit@controller-1','rabbit@controller-2'],
     ['rabbit@controller-1','rabbit@controller-2','rabbit@controller-0']}}

controller-2 | SUCCESS | rc=0 >>
{ok,{['rabbit@controller-1','rabbit@controller-2','rabbit@controller-0'],
     ['rabbit@controller-0','rabbit@controller-1','rabbit@controller-2'],
     ['rabbit@controller-1','rabbit@controller-0','rabbit@controller-2']}}

controller-1 | SUCCESS | rc=0 >>
{ok,{['rabbit@controller-1','rabbit@controller-2','rabbit@controller-0'],
     ['rabbit@controller-0','rabbit@controller-1','rabbit@controller-2'],
     ['rabbit@controller-0','rabbit@controller-2','rabbit@controller-1']}}

Comment 9 pkomarov 2018-11-19 17:09:43 UTC
more checks are needed

Comment 10 Joanne O'Flynn 2018-11-22 11:25:47 UTC
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory, please provide draft documentation text for this bug as soon as possible.

If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-".


To add draft documentation text:

1. Select the documentation type from the Doc Type drop-down field.

2. A template is provided in the Doc Text field based on the Doc Type value selected. Enter the draft text in the Doc Text field.

Comment 13 errata-xmlrpc 2018-11-27 21:19:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3693