Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1353079

Summary: osp-director-9: Attempt to upgrade OSP 8.0-> 9.0 with SSL fails.
Product: Red Hat OpenStack Reporter: Omri Hochman <ohochman>
Component: documentationAssignee: Dan Macpherson <dmacpher>
Status: CLOSED CURRENTRELEASE QA Contact: RHOS Documentation Team <rhos-docs>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 9.0 (Mitaka)CC: akaris, dbecker, dmacpher, ebarrera, goneri, jason.dobies, jcoufal, jjoyce, josorior, lbopf, markmc, mburns, mcornea, morazi, nkinder, ohochman, panbalag, racedoro, rhel-osp-director-maint, srevivo, tvignaud
Target Milestone: gaKeywords: Documentation
Target Release: 10.0 (Newton)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-23 08:03:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
heat-engine.log from undercloud
none
Adding the /var/log/messages from the controller none

Description Omri Hochman 2016-07-06 03:16:58 UTC 
osp-director-9:  Attempt to upgrade OSP 8.0-> 9.0 with SSL fails.  

Environment:
-------------
openstack-heat-common-6.0.0-6.el7ost.noarch
openstack-heat-api-cfn-6.0.0-6.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-12.el7ost.noarch
openstack-heat-engine-6.0.0-6.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-12.el7ost.noarch
python-heatclient-1.2.0-1.el7ost.noarch
openstack-tripleo-heat-templates-kilo-2.0.0-12.el7ost.noarch
openstack-heat-api-6.0.0-6.el7ost.noarch
heat-cfntools-1.3.0-2.el7ost.noarch
openstack-heat-api-cloudwatch-6.0.0-6.el7ost.noarch
openstack-heat-templates-0-0.8.20150605git.el7ost.noarch
openstack-tripleo-puppet-elements-2.0.0-2.el7ost.noarch
puppet-3.6.2-2.el7.noarch
openstack-puppet-modules-8.1.2-1.el7ost.noarch


Steps : 
-------
(1) deploy OSP8 when undercloud/overcloud are with SSL 
(2) Attempt to upgrade from 8.0-> 9.0


Results:  Stack in UPDATE_FAILED


[root@undercloud72 ~]# heat stack-list
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:266: SubjectAltNameWarning: Certificate for 192.168.0.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+---------------+---------------------+---------------------+
| id                                   | stack_name | stack_status  | creation_time       | updated_time        |
+--------------------------------------+------------+---------------+---------------------+---------------------+
| 606f32eb-7bd4-4423-acb6-681545f3f1cd | overcloud  | UPDATE_FAILED | 2016-02-10T18:21:00 | 2016-02-10T20:52:59 |
+--------------------------------------+------------+---------------+---------------------+---------------------+
[root@undercloud72 ~]# su - stack
Last login: Wed Feb 10 12:09:23 EST 2016 on pts/0
[stack@undercloud72 ~]$ heat resource-list overcloud -n5 | grep -v COMPLETE
You must provide a username via either --os-username or env[OS_USERNAME] or a token via --os-auth-token or env[OS_AUTH_TOKEN]
[stack@undercloud72 ~]$ source stackrc 
[stack@undercloud72 ~]$ heat resource-list overcloud -n5 | grep -v COMPLETE
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:266: SubjectAltNameWarning: Certificate for 192.168.0.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+-----------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| resource_name                                | physical_resource_id                          | resource_type                                                                   | resource_status | updated_time        | stack_name                                                                                                                                      |
+----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+-----------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| ControllerNodesPostDeployment                | 3c823abe-8ee5-4ba9-9cb0-ffc218ed4a81          | OS::TripleO::ControllerPostDeployment                                           | CREATE_FAILED   | 2016-02-10T20:57:46 | overcloud                                                                                                                                       |
| ControllerLoadBalancerDeployment_Step1       | ea62255a-437f-4d3e-82d1-8240f6a0406f          | OS::Heat::StructuredDeployments                                                 | CREATE_FAILED   | 2016-02-10T20:57:52 | overcloud-ControllerNodesPostDeployment-slywoqel5o4e                                                                                            |
| 0                                            | 64a928fe-b5c0-41d8-b4a3-238011ab6c90          | OS::Heat::StructuredDeployment                                                  | CREATE_FAILED   | 2016-02-10T20:59:32 | overcloud-ControllerNodesPostDeployment-slywoqel5o4e-ControllerLoadBalancerDeployment_Step1-zs7fcze4skpx                                        |
| 1                                            | f219334a-d516-409e-ac70-7eb977a6ce34          | OS::Heat::StructuredDeployment                                                  | CREATE_FAILED   | 2016-02-10T20:59:32 | overcloud-ControllerNodesPostDeployment-slywoqel5o4e-ControllerLoadBalancerDeployment_Step1-zs7fcze4skpx                                        |
| 2                                            | a9cd6ef8-8b13-449d-b66b-7c91a7228e34          | OS::Heat::StructuredDeployment                                                  | CREATE_FAILED   | 2016-02-10T20:59:32 | overcloud-ControllerNodesPostDeployment-slywoqel5o4e-ControllerLoadBalancerDeployment_Step1-zs7fcze4skpx                                        |
+----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+-----------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
[stack@undercloud72 ~]$ heat deployment-show a9cd6ef8-8b13-449d-b66b-7c91a7228e34
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:266: SubjectAltNameWarning: Certificate for 192.168.0.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead
{
  "status": "FAILED", 
  "server_id": "a0cc0c66-f90d-4a1d-93fb-9d76280d8fbf", 
  "config_id": "93bf3b60-4836-4eda-9bc7-83159091339d", 
  "output_values": {
    "deploy_stdout": "", 
    "deploy_stderr": "Could not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass\nCould not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass\n\u001b[1;31mError: Error from DataBinding 'hiera' while looking up 'tripleo::packages::enable_install': (<unknown>): mapping values are not allowed in this context at line 318 column 42 on node overcloud-controller-2.localdomain\nWrapped exception:\n(<unknown>): mapping values are not allowed in this context at line 318 column 42\nWrapped exception:\n(<unknown>): mapping values are not allowed in this context at line 318 column 42\u001b[0m\n\u001b[1;31mError: Error from DataBinding 'hiera' while looking up 'tripleo::packages::enable_install': (<unknown>): mapping values are not allowed in this context at line 318 column 42 on node overcloud-controller-2.localdomain\u001b[0m\n", 
    "deploy_status_code": 1
  }, 
  "creation_time": "2016-02-10T20:59:39", 
  "updated_time": "2016-02-10T21:00:52", 
  "input_values": {
    "step": 1, 
    "update_identifier": {
      "deployment_identifier": 1455137567, 
      "controller_config": {
        "1": "os-apply-config deployment dc0c9682-96fe-4741-a62f-bb3bb9bfb183 completed,a60777fe1910728ddbb3feec64db38d3  /etc/pki/ca-trust/source/anchors/ca.crt.pem\n,f80c79ff4ff77b6bff81f1c3bf9b0c1d  /etc/pki/tls/private/overcloud_endpoint.pem\n,None,", 
        "0": "os-apply-config deployment 03a1fc23-3799-461f-951f-34c24be592ea completed,a60777fe1910728ddbb3feec64db38d3  /etc/pki/ca-trust/source/anchors/ca.crt.pem\n,f80c79ff4ff77b6bff81f1c3bf9b0c1d  /etc/pki/tls/private/overcloud_endpoint.pem\n,None,", 
        "2": "os-apply-config deployment 3881a93a-a72a-4ba5-b4fc-bd00e9ec288f completed,a60777fe1910728ddbb3feec64db38d3  /etc/pki/ca-trust/source/anchors/ca.crt.pem\n,f80c79ff4ff77b6bff81f1c3bf9b0c1d  /etc/pki/tls/private/overcloud_endpoint.pem\n,None,"
      }, 
      "allnodes_extra": "none"
    }
  }, 
  "action": "CREATE", 
  "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", 
  "id": "a9cd6ef8-8b13-449d-b66b-7c91a7228e34"


/var/log/messages from controller-0 (full log attach):
---------------------------------------------------------
time]/seluser: seluser changed 'unconfined_u' to 'system_u'\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Service/Service[pcsd]/ensure: ensure changed 'st
opped' to 'running'\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Exec[enable-not-start-tripleo_cluster]/returns: executed successfully\u001b[0m\
n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Exec[Set password for hacluster user on tripleo_cluster]/returns: executed successfully\u001b[0m\n\u001b[mNo
tice: /Stage[main]/Pacemaker::Corosync/Exec[auth-successful-across-all-nodes]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker:
:Corosync/Exec[Create Cluster tripleo_cluster]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Exec[Start Cluster tr
ipleo_cluster]/returns: executed succes
Jul  5 18:48:47 localhost os-collect-config: sfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/returns: executed success
fully\u001b[0m\n\u001b[mNotice: Pacemaker has reported quorum achieved\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Notify[pacemaker settled]/me
ssage: defined 'message' as 'Pacemaker has reported quorum achieved'\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Stonith/Exec[Disable STONITH]/returns: 
executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Tripleo::Loadbalancer/Haproxy::Listen[redis]/Concat::Fragment[haproxy-redis_listen_block]/File[/v
ar/lib/puppet/concat/_etc_haproxy_haproxy.cfg/fragments/20-redis-00_haproxy-redis_listen_block]/ensure: defined content as '{md5}4856695a5278618563d799c764a48
473'\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/Exec[concat_/etc/hapr
oxy/haproxy.cfg]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc
/haproxy/haproxy.cfg]/Exec[concat_/etc/haproxy/haproxy.cfg]: Triggered 'refresh' from 41 events\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy/Haproxy::Instan
ce[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]/content: content changed '{md5}1f337186b0e1ba5ee82760cb43
7fb810' to '{md5}e5096dfe95ab3474416c831a7bf5fec8'\u001b[0m\n\u001b[mNotice: /File[/etc/haproxy/haproxy.cfg]/seluser: seluser changed 'unconfined_u' to 'syste
m_u'\u001b[0m\n\u001b[mNotice: Finished catalog run in 58.62 seconds\u001b[0m\n", "deploy_stderr": "Could not retrieve fact='apache_version', resolution='<ano
nymous>': undefined method `[]' for nil:NilClass\nCould not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass\n
\u001b[1;31mWarning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.\u001b[0m\n\u001b[1;31mWarning: Scope
(Haproxy::Config[haproxy]): haproxy: Th
Jul  5 18:48:47 localhost os-collect-config: e $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n", "deploy_status_code": 0}
Jul  5 18:48:47 localhost os-collect-config: [2016-07-05 22:48:47,252] (heat-config) [DEBUG] [2016-07-05 18:47:34,135] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/dd4ea9f5-2a8e-44c0-9ed5-76b766e53820"  FACTER_fqdn="overcloud-controller-0.localdomain"  FACTER_deploy_config_name="ControllerLoadBalancerDeployment_Step1"  puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/dd4ea9f5-2a8e-44c0-9ed5-76b766e53820.pp
Comment 1 Omri Hochman 2016-07-06 03:17:28 UTC 
[root@overcloud-controller-0 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Wed Jul  6 03:15:35 2016		Last change: Wed Jul  6 00:52:01 2016 by root via cibadmin on overcloud-controller-2
Stack: corosync
Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum
3 nodes and 112 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-10.19.184.210	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0 (unmanaged)
 ip-192.168.200.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1 (unmanaged)
 Clone Set: haproxy-clone [haproxy] (unmanaged)
     haproxy	(systemd:haproxy):	Started overcloud-controller-2 (unmanaged)
     haproxy	(systemd:haproxy):	Started overcloud-controller-0 (unmanaged)
     haproxy	(systemd:haproxy):	Started overcloud-controller-1 (unmanaged)
 ip-192.168.0.6	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2 (unmanaged)
 ip-10.19.104.11	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0 (unmanaged)
 ip-10.19.105.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1 (unmanaged)
 ip-10.19.104.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2 (unmanaged)
 Master/Slave Set: redis-master [redis] (unmanaged)
     redis	(ocf::heartbeat:redis):	Master overcloud-controller-2 (unmanaged)
     redis	(ocf::heartbeat:redis):	Slave overcloud-controller-0 (unmanaged)
     redis	(ocf::heartbeat:redis):	Slave overcloud-controller-1 (unmanaged)
 Master/Slave Set: galera-master [galera] (unmanaged)
     galera	(ocf::heartbeat:galera):	Master overcloud-controller-2 (unmanaged)
     galera	(ocf::heartbeat:galera):	Master overcloud-controller-0 (unmanaged)
     galera	(ocf::heartbeat:galera):	Master overcloud-controller-1 (unmanaged)
 Clone Set: mongod-clone [mongod] (unmanaged)
     mongod	(systemd:mongod):	Started overcloud-controller-2 (unmanaged)
     mongod	(systemd:mongod):	Started overcloud-controller-0 (unmanaged)
     mongod	(systemd:mongod):	Started overcloud-controller-1 (unmanaged)
 Clone Set: rabbitmq-clone [rabbitmq] (unmanaged)
     rabbitmq	(ocf::heartbeat:rabbitmq-cluster):	Started overcloud-controller-2 (unmanaged)
     rabbitmq	(ocf::heartbeat:rabbitmq-cluster):	Started overcloud-controller-0 (unmanaged)
     rabbitmq	(ocf::heartbeat:rabbitmq-cluster):	Started overcloud-controller-1 (unmanaged)
 Clone Set: memcached-clone [memcached] (unmanaged)
     memcached	(systemd:memcached):	Started overcloud-controller-2 (unmanaged)
     memcached	(systemd:memcached):	Started overcloud-controller-0 (unmanaged)
     memcached	(systemd:memcached):	Started overcloud-controller-1 (unmanaged)
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler] (unmanaged)
     openstack-nova-scheduler	(systemd:openstack-nova-scheduler):	Started overcloud-controller-2 (unmanaged)
     openstack-nova-scheduler	(systemd:openstack-nova-scheduler):	Started overcloud-controller-0 (unmanaged)
     openstack-nova-scheduler	(systemd:openstack-nova-scheduler):	Started overcloud-controller-1 (unmanaged)
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent] (unmanaged)
     neutron-l3-agent	(systemd:neutron-l3-agent):	Started overcloud-controller-0 (unmanaged)
     neutron-l3-agent	(systemd:neutron-l3-agent):	Started overcloud-controller-1 (unmanaged)
     Stopped: [ overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-alarm-notifier-clone [openstack-ceilometer-alarm-notifier] (unmanaged)
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine] (unmanaged)
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api] (unmanaged)
     openstack-ceilometer-api	(systemd:openstack-ceilometer-api):	Started overcloud-controller-2 (unmanaged)
     openstack-ceilometer-api	(systemd:openstack-ceilometer-api):	Started overcloud-controller-0 (unmanaged)
     openstack-ceilometer-api	(systemd:openstack-ceilometer-api):	Started overcloud-controller-1 (unmanaged)
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent] (unmanaged)
     neutron-metadata-agent	(systemd:neutron-metadata-agent):	Started overcloud-controller-0 (unmanaged)
     neutron-metadata-agent	(systemd:neutron-metadata-agent):	Started overcloud-controller-1 (unmanaged)
     Stopped: [ overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup] (unmanaged)
     neutron-ovs-cleanup	(ocf::neutron:OVSCleanup):	Started overcloud-controller-2 (unmanaged)
     neutron-ovs-cleanup	(ocf::neutron:OVSCleanup):	Started overcloud-controller-0 (unmanaged)
     neutron-ovs-cleanup	(ocf::neutron:OVSCleanup):	Started overcloud-controller-1 (unmanaged)
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup] (unmanaged)
     neutron-netns-cleanup	(ocf::neutron:NetnsCleanup):	Started overcloud-controller-2 (unmanaged)
     neutron-netns-cleanup	(ocf::neutron:NetnsCleanup):	Started overcloud-controller-0 (unmanaged)
     neutron-netns-cleanup	(ocf::neutron:NetnsCleanup):	Started overcloud-controller-1 (unmanaged)
 Clone Set: openstack-heat-api-clone [openstack-heat-api] (unmanaged)
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler] (unmanaged)
     openstack-cinder-scheduler	(systemd:openstack-cinder-scheduler):	Started overcloud-controller-2 (unmanaged)
     openstack-cinder-scheduler	(systemd:openstack-cinder-scheduler):	Started overcloud-controller-0 (unmanaged)
     openstack-cinder-scheduler	(systemd:openstack-cinder-scheduler):	Started overcloud-controller-1 (unmanaged)
 Clone Set: openstack-nova-api-clone [openstack-nova-api] (unmanaged)
     openstack-nova-api	(systemd:openstack-nova-api):	Started overcloud-controller-2 (unmanaged)
     openstack-nova-api	(systemd:openstack-nova-api):	Started overcloud-controller-0 (unmanaged)
     openstack-nova-api	(systemd:openstack-nova-api):	Started overcloud-controller-1 (unmanaged)
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch] (unmanaged)
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector] (unmanaged)
     openstack-ceilometer-collector	(systemd:openstack-ceilometer-collector):	Started overcloud-controller-2 (unmanaged)
     openstack-ceilometer-collector	(systemd:openstack-ceilometer-collector):	Started overcloud-controller-0 (unmanaged)
     openstack-ceilometer-collector	(systemd:openstack-ceilometer-collector):	Started overcloud-controller-1 (unmanaged)
 Clone Set: openstack-keystone-clone [openstack-keystone] (unmanaged)
     openstack-keystone	(systemd:openstack-keystone):	Started overcloud-controller-2 (unmanaged)
     openstack-keystone	(systemd:openstack-keystone):	Started overcloud-controller-0 (unmanaged)
     openstack-keystone	(systemd:openstack-keystone):	Started overcloud-controller-1 (unmanaged)
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth] (unmanaged)
     openstack-nova-consoleauth	(systemd:openstack-nova-consoleauth):	Started overcloud-controller-2 (unmanaged)
     openstack-nova-consoleauth	(systemd:openstack-nova-consoleauth):	Started overcloud-controller-0 (unmanaged)
     openstack-nova-consoleauth	(systemd:openstack-nova-consoleauth):	Started overcloud-controller-1 (unmanaged)
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry] (unmanaged)
     openstack-glance-registry	(systemd:openstack-glance-registry):	Started overcloud-controller-2 (unmanaged)
     openstack-glance-registry	(systemd:openstack-glance-registry):	Started overcloud-controller-0 (unmanaged)
     openstack-glance-registry	(systemd:openstack-glance-registry):	Started overcloud-controller-1 (unmanaged)
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification] (unmanaged)
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api] (unmanaged)
     openstack-cinder-api	(systemd:openstack-cinder-api):	Started overcloud-controller-2 (unmanaged)
     openstack-cinder-api	(systemd:openstack-cinder-api):	Started overcloud-controller-0 (unmanaged)
     openstack-cinder-api	(systemd:openstack-cinder-api):	Started overcloud-controller-1 (unmanaged)
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent] (unmanaged)
     neutron-dhcp-agent	(systemd:neutron-dhcp-agent):	Started overcloud-controller-2 (unmanaged)
     neutron-dhcp-agent	(systemd:neutron-dhcp-agent):	Started overcloud-controller-0 (unmanaged)
     neutron-dhcp-agent	(systemd:neutron-dhcp-agent):	Started overcloud-controller-1 (unmanaged)
 Clone Set: openstack-glance-api-clone [openstack-glance-api] (unmanaged)
     openstack-glance-api	(systemd:openstack-glance-api):	Started overcloud-controller-2 (unmanaged)
     openstack-glance-api	(systemd:openstack-glance-api):	Started overcloud-controller-0 (unmanaged)
     openstack-glance-api	(systemd:openstack-glance-api):	Started overcloud-controller-1 (unmanaged)
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent] (unmanaged)
     neutron-openvswitch-agent	(systemd:neutron-openvswitch-agent):	Started overcloud-controller-2 (unmanaged)
     neutron-openvswitch-agent	(systemd:neutron-openvswitch-agent):	Started overcloud-controller-0 (unmanaged)
     neutron-openvswitch-agent	(systemd:neutron-openvswitch-agent):	Started overcloud-controller-1 (unmanaged)
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy] (unmanaged)
     openstack-nova-novncproxy	(systemd:openstack-nova-novncproxy):	Started overcloud-controller-2 (unmanaged)
     openstack-nova-novncproxy	(systemd:openstack-nova-novncproxy):	Started overcloud-controller-0 (unmanaged)
     openstack-nova-novncproxy	(systemd:openstack-nova-novncproxy):	Started overcloud-controller-1 (unmanaged)
 Clone Set: delay-clone [delay] (unmanaged)
     delay	(ocf::heartbeat:Delay):	Started overcloud-controller-2 (unmanaged)
     delay	(ocf::heartbeat:Delay):	Started overcloud-controller-0 (unmanaged)
     delay	(ocf::heartbeat:Delay):	Started overcloud-controller-1 (unmanaged)
 Clone Set: neutron-server-clone [neutron-server] (unmanaged)
     neutron-server	(systemd:neutron-server):	Started overcloud-controller-2 (unmanaged)
     neutron-server	(systemd:neutron-server):	Started overcloud-controller-0 (unmanaged)
     neutron-server	(systemd:neutron-server):	Started overcloud-controller-1 (unmanaged)
 Clone Set: httpd-clone [httpd] (unmanaged)
     httpd	(systemd:httpd):	Started overcloud-controller-2 (unmanaged)
     httpd	(systemd:httpd):	Started overcloud-controller-0 (unmanaged)
     httpd	(systemd:httpd):	Started overcloud-controller-1 (unmanaged)
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central] (unmanaged)
     openstack-ceilometer-central	(systemd:openstack-ceilometer-central):	Started overcloud-controller-2 (unmanaged)
     openstack-ceilometer-central	(systemd:openstack-ceilometer-central):	Started overcloud-controller-0 (unmanaged)
     openstack-ceilometer-central	(systemd:openstack-ceilometer-central):	Started overcloud-controller-1 (unmanaged)
 Clone Set: openstack-ceilometer-alarm-evaluator-clone [openstack-ceilometer-alarm-evaluator] (unmanaged)
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn] (unmanaged)
     Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume	(systemd:openstack-cinder-volume):	Started overcloud-controller-0 (unmanaged)
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor] (unmanaged)
     openstack-nova-conductor	(systemd:openstack-nova-conductor):	Started overcloud-controller-2 (unmanaged)
     openstack-nova-conductor	(systemd:openstack-nova-conductor):	Started overcloud-controller-0 (unmanaged)
     openstack-nova-conductor	(systemd:openstack-nova-conductor):	Started overcloud-controller-1 (unmanaged)

Failed Actions:
* openstack-ceilometer-alarm-evaluator_start_0 on overcloud-controller-2 'not installed' (5): call=617, status=Not installed, exitreason='none',
    last-rc-change='Wed Jul  6 00:20:13 2016', queued=0ms, exec=87ms
* openstack-ceilometer-alarm-evaluator_start_0 on overcloud-controller-0 'not installed' (5): call=616, status=Not installed, exitreason='none',
    last-rc-change='Wed Jul  6 00:20:13 2016', queued=0ms, exec=216ms
* openstack-ceilometer-alarm-evaluator_start_0 on overcloud-controller-1 'not installed' (5): call=639, status=Not installed, exitreason='none',
    last-rc-change='Wed Jul  6 00:20:13 2016', queued=0ms, exec=99ms


PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@overcloud-controller-0 ~]#
Comment 2 Omri Hochman 2016-07-06 03:18:41 UTC 
Created attachment 1176713 [details]
heat-engine.log from undercloud
Comment 3 Omri Hochman 2016-07-06 03:22:28 UTC 
Created attachment 1176714 [details]
Adding the /var/log/messages from the controller
Comment 4 Omri Hochman 2016-07-07 02:05:47 UTC 
reproduced again with: openstack-tripleo-heat-templates-liberty-2.0.0-13.el7ost.noarch 

happens during the AODH migration step:

openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ceph-storage-scale 2   --neutron-network-type vxlan --neutron-tunnel-types vxlan  --ntp-server 10.5.26.10 --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml -e /home/stack/ssl-heat-templates/environments/enable-tls.yaml -e /home/stack/ssl-heat-templates/environments/inject-trust-anchor.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml


results :
-----------+---------------------+
| d288684b-b4d7-4606-bac5-c63611fb262b | overcloud  | UPDATE_FAILED | 2016-02-10T18:26:12 | 2016-02-11T17:17:10 |
+--------------------------------------+------------+---------------+---------------------+---------------------+
Comment 5 Marius Cornea 2016-07-07 13:44:17 UTC 
This appears to be caused by the following hierdata in controller.yaml:

[root@overcloud-controller-0 hieradata]# sed -n 318p /etc/puppet/hieradata/controller.yaml 
ceilometer::dispatcher::gnocchi::url: ://:
Comment 6 Marius Cornea 2016-07-07 14:06:16 UTC 
Most probably this is caused by the enable-tls.yaml not containing the new services in the EndpointMap. I adjusted it like the following and the hieradata got populated with the url containing the internal vip:

[root@overcloud-controller-0 heat-admin]# sed -n 318p /etc/puppet/hieradata/controller.yaml 
ceilometer::dispatcher::gnocchi::url: http://10.0.0.10:8041

diff templates/enable-tls.yaml templates/enable-tls.yaml.pre-upgrade 
55,63d54
<     AodhAdmin: {protocol: 'http', port: '8042', host: 'IP_ADDRESS'}
<     AodhInternal: {protocol: 'http', port: '8042', host: 'IP_ADDRESS'}
<     AodhPublic: {protocol: 'https', port: '13042', host: 'IP_ADDRESS'}
<     GnocchiAdmin: {protocol: 'http', port: '8041', host: 'IP_ADDRESS'}
<     GnocchiInternal: {protocol: 'http', port: '8041', host: 'IP_ADDRESS'}
<     GnocchiPublic: {protocol: 'https', port: '13041', host: 'IP_ADDRESS'}
<     SaharaAdmin: {protocol: 'http', port: '8386', host: 'IP_ADDRESS'}
<     SaharaInternal: {protocol: 'http', port: '8386', host: 'IP_ADDRESS'}
<     SaharaPublic: {protocol: 'https', port: '13386', host: 'IP_ADDRESS'}
Comment 7 Omri Hochman 2016-07-08 14:29:54 UTC 
According to Marius the issue is that the upgrade command is changing the original enable-tls.yaml file that locates under:

/usr/share/openstack-tripleo-heat-templates/environments/enable-tls.yaml

then when we run the upgrade command (with -e) we were calling the local copy of the enable-tls.yaml file that is located under:

/home/stack/ssl-heat-templates/environments/enable-tls.yaml

the file that under the /home/stack/ - didn't change during the upgrade,. 

Workaround: 
------------
before running the upgrade command, run sasha's sed cmd 
to fix the enable-tls.yaml file that under /home/stack/ssl-heat-templates/environments/enable-tls.yaml: 

sed -i '/EndpointMap.*/a \ \ \ \ AodhAdmin: {protocol: 'http', port: '8042', host: 'IP_ADDRESS'}\n\ \ \ \ AodhInternal: {protocol: 'http', port: '8042', ho
'https', port: '13042', host: 'IP_ADDRESS'}\n\ \ \ \ GnocchiAdmin: {protocol: 'http', port: '8041', host: 'IP_ADDRESS'}\n\ \ \ \ GnocchiInternal: {protocol: 'http', port: '8041', host: 'IP_A
', port: '13041', host: 'IP_ADDRESS'\n\ \ \ \ SaharaAdmin: {protocol: 'http', port: '8386', host: 'IP_ADDRESS'}\n\ \ \ \ SaharaInternal: {protocol: 'http', port: '8386', host: 'IP_ADDRESS'\n
13386', host: 'IP_ADDRESS'}' /home/stack/ssl-heat-templates/environments/enable-tls.yaml
Comment 8 Omri Hochman 2016-07-08 21:02:08 UTC 
fixing the sed command : 
sed -i "/EndpointMap.*/a \ \ \ \ AodhAdmin: {protocol: 'http', port: '8042', host: 'IP_ADDRESS'}\n\ \ \ \ AodhInternal: {protocol: 'http', port: '8042', host: 'IP_ADDRESS'}\n\ \ \ \ AodhPublic: {protocol: 'https', port: '13042', host: 'IP_ADDRESS'}\n\ \ \ \ GnocchiAdmin: {protocol: 'http', port: '8041', host: 'IP_ADDRESS'}\n\ \ \ \ GnocchiInternal: {protocol: 'http', port: '8041', host: 'IP_ADDRESS'}\n\ \ \ \ GnocchiPublic: {protocol: 'https', port: '13041', host: 'IP_ADDRESS'}\n\ \ \ \ SaharaAdmin: {protocol: 'http', port: '8386', host: 'IP_ADDRESS'}\n\ \ \ \ SaharaInternal: {protocol: 'http', port: '8386', host: 'IP_ADDRESS'}\n\ \ \ \ SaharaPublic: {protocol: 'https', port: '13386', host: 'IP_ADDRESS'}" /home/stack/ssl-heat-templates/environments/enable-tls.yaml
Comment 9 Juan Antonio Osorio 2016-07-12 01:56:49 UTC 
My attempt as fixing these problems was to use map_merge for that https://review.openstack.org/#/c/308182/ but it wasn't well received. I can try to re-take that effort though. Else I can write a tool to update that map and keep the certs, which should be called when an update is needed.
Comment 10 Gonéri Le Bouder 2016-07-13 12:02:10 UTC 
*** Bug 1356077 has been marked as a duplicate of this bug. ***
Comment 19 Dan Macpherson 2016-11-07 12:33:40 UTC 
I think this BZ might have slipped under the radar. Grabbing this BZ and checking for the fix.
Comment 20 Andreas Karis 2017-02-02 00:07:18 UTC 
Dan Macpherson, do we have any progress with the documentation of the workaround in our upgrade doc? The workaround is known for 6 months. Given the number of customers upgrading from 8 to 9, this should be documented.

Thanks.

-ak
Comment 21 Dan Macpherson 2017-02-02 00:14:23 UTC 
Thanks for highlighting this BZ, Andreas. I had some priority work I had to focus on for OSP10, but I've got that out of the way. I'll make this BZ a priority and work on it today.
Comment 22 Andreas Karis 2017-02-02 00:32:29 UTC 
Thanks a lot :-)
Comment 24 Dan Macpherson 2017-02-03 04:55:24 UTC 
Have added the following note to the OSP9 Upgrade procedure:

"If using a custom endpoint map for enabling TLS/SSL in the overcloud, make sure to update the map with endpoints for the following new services:

    OpenStack Telemetry Metrics (gnocchi)
    OpenStack Telemetry Alarming (aodh)
    OpenStack Clustering (sahara) 

Check the latest TLS/SSL mappings from the core Heat template collection (see EndpointMap in /usr/share/openstack-tripleo-heat-templates/environments/enable-tls.yaml) and add the missing endpoints to the EndpointMap in your custom enable-tls.yaml file. For more information, see "Enabling SSL/TLS on the Overcloud" in the Red Hat OpenStack Platform Director Installation and Usage guide."

Link: https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/single/upgrading-red-hat-openstack-platform/#sect-Pre-Upgrade_Notes_for_Overcloud

Omri and Andreas, anything further required for this BZ?
Comment 25 Dan Macpherson 2017-02-23 08:03:33 UTC 
No response in over two weeks. If nothing further to add, I'll close this BZ.

If further changes are required, please feel free to reopen it.
Comment 26 Dan Macpherson 2017-03-01 12:58:11 UTC 
Hi Eduard,

Is there something in the documentation unresolved in relation to the case you posted?

Just want to make sure we have everything covered.

- Dan