Bug 1380815

Summary: rhel-osp-director: overcloud minor update fails with :Error: Could not find data item mysql_clustercheck_password in any Hiera data file
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED NOTABUG QA Contact: Omri Hochman <ohochman>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.0 (Liberty)CC: arkady_kanevsky, aschultz, david_paterson, dbecker, John_walsh, jslagle, mandreou, mburns, mcornea, morazi, randy_perryman, rhel-osp-director-maint, sasha, wayne_allen, yprokule
Target Milestone: asyncKeywords: Triaged
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-10 11:17:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
versionlock list none

Description Alexander Chuzhoy 2016-09-30 16:19:51 UTC
rhel-osp-director:   overcloud minor update fails with :Error: Could not find data item mysql_clustercheck_password in any Hiera data file

Environment:
openstack-puppet-modules-7.1.3-1.el7ost.noarch
openstack-tripleo-heat-templates-0.8.14-18.el7ost.noarch
openstack-tripleo-heat-templates-kilo-0.8.14-18.el7ost.noarch
instack-undercloud-2.2.7-7.el7ost.noarch


Steps to reproduce:
1. Deploy 8.0 undercloud + overcloud with yum versionlock (see attached):

Deployment command:
openstack overcloud deploy --log-file ~/pilot/overcloud_deployment.log -t 600 --stack overcloud --templates ~/pilot/templates/overcloud -e ~/pilot/templates/overcloud/environments/network-isolation.yaml -e ~/pilot/templates/network-environment.yaml -e ~/pilot/templates/overcloud/environments/storage-environment.yaml -e ~/pilot/templates/dell-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e ~/pilot/templates/dell-cinder-backends.yaml --control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage --swift-storage-flavor swift-storage --block-storage-flavor block-storage --neutron-public-interface bond1 --neutron-network-type vlan --neutron-disable-tunneling --os-auth-url http://192.168.120.202:5000/v2.0 --os-project-name admin --os-user-id admin --os-password d4948cb4d2c3d98ff0236a19ebc10043dc90a46f --control-scale 3 --compute-scale 3 --ceph-storage-scale 3 --ntp-server 0.centos.pool.ntp.org --neutron-network-vlan-ranges physint:201:250,physext --neutron-bridge-mappings physint:br-tenant,physext:br-ex


2. Update the undercloud, make sure the overcloud nodes are subscribed to cdn and run the overcloud update:
yes ""|openstack overcloud update stack overcloud -i --templates ~/pilot/templates/overcloud -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e ~/pilot/templates/overcloud/environments/network-isolation.yaml -e ~/pilot/templates/network-environment.yaml -e ~/pilot/templates/overcloud/environments/storage-environment.yaml -e ~/pilot/templates/dell-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e ~/pilot/templates/dell-cinder-backends.yaml


Result:
The overcloud update fails.
Some heat debugging shows:
[stack@director ~]$ echo -e `heat deployment-show af7fa86b-986f-4989-a130-33bb6d931214`
{ "status": "FAILED", "server_id": "15a9afcf-7e42-4542-a1b9-3df18324c712", "config_id": "cb7ac8e6-e444-4a1d-be25-b386a6ceb75b", "output_values": { "deploy_stdout": "", "deploy_stderr": "Could not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass
Could not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass
Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
Error: Could not find data item mysql_clustercheck_password in any Hiera data file and no default supplied at /var/lib/heat-config/heat-config-puppet/cb7ac8e6-e444-4a1d-be25-b386a6ceb75b.pp:482 on node overcloud-controller-2.localdomain
Error: Could not find data item mysql_clustercheck_password in any Hiera data file and no default supplied at /var/lib/heat-config/heat-config-puppet/cb7ac8e6-e444-4a1d-be25-b386a6ceb75b.pp:482 on node overcloud-controller-2.localdomain
", "deploy_status_code": 1 }, "creation_time": "2016-09-29T22:49:16", "updated_time": "2016-09-30T15:26:13", "input_values": {}, "action": "UPDATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", "id": "af7fa86b-986f-4989-a130-33bb6d931214" }



I noticed that pcs cluster was in maintenance mode. Tried to unset the maintenance mode and restart the cluster:
Note: it lists computes as members:
[root@overcloud-controller-2 ~]# pcs status               
Cluster name: tripleo_cluster                             
Last updated: Fri Sep 30 16:16:45 2016          Last change: Fri Sep 30 15:56:43 2016 by root via cibadmin on overcloud-controller-0
Stack: corosync                                                                                                                     
Current DC: overcloud-controller-1 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum                                      
6 nodes and 258 resources configured                                                                                                

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
RemoteOnline: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]

Full list of resources:

 ip-192.168.120.127     (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0
 ip-192.168.120.126     (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1
 Clone Set: haproxy-clone [haproxy]                                                   
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 ip-192.168.190.5       (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2                           
 ip-192.168.170.120     (ocf::heartbeat:IPaddr2):       Started overcloud-controller-0                           
 ip-192.168.140.121     (ocf::heartbeat:IPaddr2):       Started overcloud-controller-1                           
 ip-192.168.140.120     (ocf::heartbeat:IPaddr2):       Started overcloud-controller-2                           
 Master/Slave Set: redis-master [redis]                                                                          
     Masters: [ overcloud-controller-1 ]                                                                         
     Slaves: [ overcloud-controller-0 overcloud-controller-2 ]                                                   
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Master/Slave Set: galera-master [galera]                                                                        
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: mongod-clone [mongod]                                                                                
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: rabbitmq-clone [rabbitmq]                                                                            
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: memcached-clone [memcached]                                                                          
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]                                            
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]                                                            
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-ceilometer-alarm-notifier-clone [openstack-ceilometer-alarm-notifier]                      
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]                                                  
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]                                            
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]                                                
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]                                                      
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]                                                  
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-heat-api-clone [openstack-heat-api]                                                        
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]                                        
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-nova-api-clone [openstack-nova-api]                                                        
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]                                  
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]                                
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-keystone-clone [openstack-keystone]                                                        
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]                                        
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]                                          
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]                          
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]                                                    
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]                                                        
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]                                                    
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]                                          
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]                                          
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: delay-clone [delay]                                                                                  
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: neutron-server-clone [neutron-server]                                                                
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]                           
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: httpd-clone [httpd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-ceilometer-alarm-evaluator-clone [openstack-ceilometer-alarm-evaluator]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started overcloud-controller-0
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
 overcloud-controller-1-ipmi    (stonith:fence_ipmilan):        Started overcloud-controller-2
 overcloud-controller-0-ipmi    (stonith:fence_ipmilan):        Started overcloud-controller-1
 overcloud-controller-2-ipmi    (stonith:fence_ipmilan):        Started overcloud-controller-0
 nova-evacuate  (ocf::openstack:NovaEvacuate):  Started overcloud-controller-1
 Clone Set: neutron-openvswitch-agent-compute-clone [neutron-openvswitch-agent-compute]
     Started: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: libvirtd-compute-clone [libvirtd-compute]
     Started: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: ceilometer-compute-clone [ceilometer-compute]
     Started: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: nova-compute-checkevacuate-clone [nova-compute-checkevacuate]
     Started: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: nova-compute-clone [nova-compute]
     Started: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ]
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ipmilan-overcloud-compute-0    (stonith:fence_ipmilan):        Started overcloud-controller-2
 ipmilan-overcloud-compute-1    (stonith:fence_ipmilan):        Started overcloud-controller-0
 ipmilan-overcloud-compute-2    (stonith:fence_ipmilan):        Started overcloud-controller-1
 fence-nova     (stonith:fence_compute):        Started overcloud-controller-2
 overcloud-compute-0.localdomain        (ocf::pacemaker:remote):        Started overcloud-controller-0
 overcloud-compute-1.localdomain        (ocf::pacemaker:remote):        Started overcloud-controller-1
 overcloud-compute-2.localdomain        (ocf::pacemaker:remote):        Started overcloud-controller-2

Failed Actions:
* nova-compute_monitor_0 on overcloud-compute-0.localdomain 'OCF_PENDING' (196): call=26, status=complete, exitreason='none',
    last-rc-change='Fri Sep 30 16:06:18 2016', queued=0ms, exec=6ms


PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled


Expected result:
Successful minor update.

Comment 1 Alexander Chuzhoy 2016-09-30 16:21:44 UTC
Created attachment 1206319 [details]
versionlock list

Comment 3 Alexander Chuzhoy 2016-10-01 03:11:07 UTC
I managed to update successfully.
Two differences/fixes from the previous run:
1. Added "OVERCLOUD_MYSQL_CLUSTERCHECK_PASSWORD=A6uQ33sWarHDm3WuqCGrdJbVY
" line to /home/stack/tripleo-overcloud-passwords file
2. Patched the /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml file the custom parts used to deploy.

Not sure this is a bug.
Need to see the update passing without "fix" #1 above.

Comment 4 Marios Andreou 2016-10-03 06:51:40 UTC
(In reply to Alexander Chuzhoy from comment #3)
> I managed to update successfully.
> Two differences/fixes from the previous run:
> 1. Added "OVERCLOUD_MYSQL_CLUSTERCHECK_PASSWORD=A6uQ33sWarHDm3WuqCGrdJbVY
> " line to /home/stack/tripleo-overcloud-passwords file
> 2. Patched the
> /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.
> yaml file the custom parts used to deploy.
> 
> Not sure this is a bug.
> Need to see the update passing without "fix" #1 above.


Hi Sasha, I knew this sounded familiar so I went digging... yes there is a bug for this upstream @ [1] and downstream @ [2] 

The fix at [3] is apparently in openstack-tripleo-heat-templates-0.8.14-16.el7ost.noarch [4]

Looking at the version lock list you attached here sounds like the env didn't have the fixup (tht @ openstack-tripleo-heat-templates-0.8.14-11.el7ost )

thanks,

[1] https://bugs.launchpad.net/tripleo/+bug/1598890
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1358779  
[3] https://review.openstack.org/#/c/337304/ 
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1358779#c3

Comment 5 Randy Perryman 2016-10-03 13:22:30 UTC
Question:
I do not see an upgrade procedure from a deployed Cloud with openstack-tripleo-heat-templates-0.8.14-11.el7ost to the openstack-tripleo-heat-templates-0.8.14-16.el7ost.noarch.  

As the 8.14.11 is GA are there instructions on how to update?

Comment 6 James Slagle 2017-03-01 19:18:22 UTC
(In reply to Randy Perryman from comment #5)
> Question:
> I do not see an upgrade procedure from a deployed Cloud with
> openstack-tripleo-heat-templates-0.8.14-11.el7ost to the
> openstack-tripleo-heat-templates-0.8.14-16.el7ost.noarch.  
> 
> As the 8.14.11 is GA are there instructions on how to update?

you would follow the minor version update procedure:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/8/html/upgrading_red_hat_openstack_platform/sect-updating_the_environment

sasha, is this still an issue based on comment #4?