Hide Forgot
rhel-osp-director: overcloud minor update fails with :Error: Could not find data item mysql_clustercheck_password in any Hiera data file Environment: openstack-puppet-modules-7.1.3-1.el7ost.noarch openstack-tripleo-heat-templates-0.8.14-18.el7ost.noarch openstack-tripleo-heat-templates-kilo-0.8.14-18.el7ost.noarch instack-undercloud-2.2.7-7.el7ost.noarch Steps to reproduce: 1. Deploy 8.0 undercloud + overcloud with yum versionlock (see attached): Deployment command: openstack overcloud deploy --log-file ~/pilot/overcloud_deployment.log -t 600 --stack overcloud --templates ~/pilot/templates/overcloud -e ~/pilot/templates/overcloud/environments/network-isolation.yaml -e ~/pilot/templates/network-environment.yaml -e ~/pilot/templates/overcloud/environments/storage-environment.yaml -e ~/pilot/templates/dell-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e ~/pilot/templates/dell-cinder-backends.yaml --control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage --swift-storage-flavor swift-storage --block-storage-flavor block-storage --neutron-public-interface bond1 --neutron-network-type vlan --neutron-disable-tunneling --os-auth-url http://192.168.120.202:5000/v2.0 --os-project-name admin --os-user-id admin --os-password d4948cb4d2c3d98ff0236a19ebc10043dc90a46f --control-scale 3 --compute-scale 3 --ceph-storage-scale 3 --ntp-server 0.centos.pool.ntp.org --neutron-network-vlan-ranges physint:201:250,physext --neutron-bridge-mappings physint:br-tenant,physext:br-ex 2. Update the undercloud, make sure the overcloud nodes are subscribed to cdn and run the overcloud update: yes ""|openstack overcloud update stack overcloud -i --templates ~/pilot/templates/overcloud -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e ~/pilot/templates/overcloud/environments/network-isolation.yaml -e ~/pilot/templates/network-environment.yaml -e ~/pilot/templates/overcloud/environments/storage-environment.yaml -e ~/pilot/templates/dell-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e ~/pilot/templates/dell-cinder-backends.yaml Result: The overcloud update fails. Some heat debugging shows: [stack@director ~]$ echo -e `heat deployment-show af7fa86b-986f-4989-a130-33bb6d931214` { "status": "FAILED", "server_id": "15a9afcf-7e42-4542-a1b9-3df18324c712", "config_id": "cb7ac8e6-e444-4a1d-be25-b386a6ceb75b", "output_values": { "deploy_stdout": "", "deploy_stderr": "Could not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass Could not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided. Error: Could not find data item mysql_clustercheck_password in any Hiera data file and no default supplied at /var/lib/heat-config/heat-config-puppet/cb7ac8e6-e444-4a1d-be25-b386a6ceb75b.pp:482 on node overcloud-controller-2.localdomain Error: Could not find data item mysql_clustercheck_password in any Hiera data file and no default supplied at /var/lib/heat-config/heat-config-puppet/cb7ac8e6-e444-4a1d-be25-b386a6ceb75b.pp:482 on node overcloud-controller-2.localdomain ", "deploy_status_code": 1 }, "creation_time": "2016-09-29T22:49:16", "updated_time": "2016-09-30T15:26:13", "input_values": {}, "action": "UPDATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", "id": "af7fa86b-986f-4989-a130-33bb6d931214" } I noticed that pcs cluster was in maintenance mode. Tried to unset the maintenance mode and restart the cluster: Note: it lists computes as members: [root@overcloud-controller-2 ~]# pcs status Cluster name: tripleo_cluster Last updated: Fri Sep 30 16:16:45 2016 Last change: Fri Sep 30 15:56:43 2016 by root via cibadmin on overcloud-controller-0 Stack: corosync Current DC: overcloud-controller-1 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum 6 nodes and 258 resources configured Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] RemoteOnline: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Full list of resources: ip-192.168.120.127 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-192.168.120.126 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 Clone Set: haproxy-clone [haproxy] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] ip-192.168.190.5 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 ip-192.168.170.120 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 ip-192.168.140.121 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 ip-192.168.140.120 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 Master/Slave Set: redis-master [redis] Masters: [ overcloud-controller-1 ] Slaves: [ overcloud-controller-0 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Master/Slave Set: galera-master [galera] Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: mongod-clone [mongod] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: rabbitmq-clone [rabbitmq] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: memcached-clone [memcached] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: neutron-l3-agent-clone [neutron-l3-agent] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-ceilometer-alarm-notifier-clone [openstack-ceilometer-alarm-notifier] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-heat-engine-clone [openstack-heat-engine] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-heat-api-clone [openstack-heat-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-nova-api-clone [openstack-nova-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-keystone-clone [openstack-keystone] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-glance-registry-clone [openstack-glance-registry] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-cinder-api-clone [openstack-cinder-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-glance-api-clone [openstack-glance-api] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: delay-clone [delay] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: neutron-server-clone [neutron-server] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: httpd-clone [httpd] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-ceilometer-alarm-evaluator-clone [openstack-ceilometer-alarm-evaluator] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Stopped: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] overcloud-controller-1-ipmi (stonith:fence_ipmilan): Started overcloud-controller-2 overcloud-controller-0-ipmi (stonith:fence_ipmilan): Started overcloud-controller-1 overcloud-controller-2-ipmi (stonith:fence_ipmilan): Started overcloud-controller-0 nova-evacuate (ocf::openstack:NovaEvacuate): Started overcloud-controller-1 Clone Set: neutron-openvswitch-agent-compute-clone [neutron-openvswitch-agent-compute] Started: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: libvirtd-compute-clone [libvirtd-compute] Started: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: ceilometer-compute-clone [ceilometer-compute] Started: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: nova-compute-checkevacuate-clone [nova-compute-checkevacuate] Started: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: nova-compute-clone [nova-compute] Started: [ overcloud-compute-0.localdomain overcloud-compute-1.localdomain overcloud-compute-2.localdomain ] Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] ipmilan-overcloud-compute-0 (stonith:fence_ipmilan): Started overcloud-controller-2 ipmilan-overcloud-compute-1 (stonith:fence_ipmilan): Started overcloud-controller-0 ipmilan-overcloud-compute-2 (stonith:fence_ipmilan): Started overcloud-controller-1 fence-nova (stonith:fence_compute): Started overcloud-controller-2 overcloud-compute-0.localdomain (ocf::pacemaker:remote): Started overcloud-controller-0 overcloud-compute-1.localdomain (ocf::pacemaker:remote): Started overcloud-controller-1 overcloud-compute-2.localdomain (ocf::pacemaker:remote): Started overcloud-controller-2 Failed Actions: * nova-compute_monitor_0 on overcloud-compute-0.localdomain 'OCF_PENDING' (196): call=26, status=complete, exitreason='none', last-rc-change='Fri Sep 30 16:06:18 2016', queued=0ms, exec=6ms PCSD Status: overcloud-controller-0: Online overcloud-controller-1: Online overcloud-controller-2: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled Expected result: Successful minor update.
Created attachment 1206319 [details] versionlock list
I managed to update successfully. Two differences/fixes from the previous run: 1. Added "OVERCLOUD_MYSQL_CLUSTERCHECK_PASSWORD=A6uQ33sWarHDm3WuqCGrdJbVY " line to /home/stack/tripleo-overcloud-passwords file 2. Patched the /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml file the custom parts used to deploy. Not sure this is a bug. Need to see the update passing without "fix" #1 above.
(In reply to Alexander Chuzhoy from comment #3) > I managed to update successfully. > Two differences/fixes from the previous run: > 1. Added "OVERCLOUD_MYSQL_CLUSTERCHECK_PASSWORD=A6uQ33sWarHDm3WuqCGrdJbVY > " line to /home/stack/tripleo-overcloud-passwords file > 2. Patched the > /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker. > yaml file the custom parts used to deploy. > > Not sure this is a bug. > Need to see the update passing without "fix" #1 above. Hi Sasha, I knew this sounded familiar so I went digging... yes there is a bug for this upstream @ [1] and downstream @ [2] The fix at [3] is apparently in openstack-tripleo-heat-templates-0.8.14-16.el7ost.noarch [4] Looking at the version lock list you attached here sounds like the env didn't have the fixup (tht @ openstack-tripleo-heat-templates-0.8.14-11.el7ost ) thanks, [1] https://bugs.launchpad.net/tripleo/+bug/1598890 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1358779 [3] https://review.openstack.org/#/c/337304/ [4] https://bugzilla.redhat.com/show_bug.cgi?id=1358779#c3
Question: I do not see an upgrade procedure from a deployed Cloud with openstack-tripleo-heat-templates-0.8.14-11.el7ost to the openstack-tripleo-heat-templates-0.8.14-16.el7ost.noarch. As the 8.14.11 is GA are there instructions on how to update?
(In reply to Randy Perryman from comment #5) > Question: > I do not see an upgrade procedure from a deployed Cloud with > openstack-tripleo-heat-templates-0.8.14-11.el7ost to the > openstack-tripleo-heat-templates-0.8.14-16.el7ost.noarch. > > As the 8.14.11 is GA are there instructions on how to update? you would follow the minor version update procedure: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/8/html/upgrading_red_hat_openstack_platform/sect-updating_the_environment sasha, is this still an issue based on comment #4?