osp-director-9: Attempt to upgrade OSP 8.0-> 9.0 with SSL fails. Environment: ------------- openstack-heat-common-6.0.0-6.el7ost.noarch openstack-heat-api-cfn-6.0.0-6.el7ost.noarch openstack-tripleo-heat-templates-2.0.0-12.el7ost.noarch openstack-heat-engine-6.0.0-6.el7ost.noarch openstack-tripleo-heat-templates-liberty-2.0.0-12.el7ost.noarch python-heatclient-1.2.0-1.el7ost.noarch openstack-tripleo-heat-templates-kilo-2.0.0-12.el7ost.noarch openstack-heat-api-6.0.0-6.el7ost.noarch heat-cfntools-1.3.0-2.el7ost.noarch openstack-heat-api-cloudwatch-6.0.0-6.el7ost.noarch openstack-heat-templates-0-0.8.20150605git.el7ost.noarch openstack-tripleo-puppet-elements-2.0.0-2.el7ost.noarch puppet-3.6.2-2.el7.noarch openstack-puppet-modules-8.1.2-1.el7ost.noarch Steps : ------- (1) deploy OSP8 when undercloud/overcloud are with SSL (2) Attempt to upgrade from 8.0-> 9.0 Results: Stack in UPDATE_FAILED [root@undercloud72 ~]# heat stack-list /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:266: SubjectAltNameWarning: Certificate for 192.168.0.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead +--------------------------------------+------------+---------------+---------------------+---------------------+ | id | stack_name | stack_status | creation_time | updated_time | +--------------------------------------+------------+---------------+---------------------+---------------------+ | 606f32eb-7bd4-4423-acb6-681545f3f1cd | overcloud | UPDATE_FAILED | 2016-02-10T18:21:00 | 2016-02-10T20:52:59 | +--------------------------------------+------------+---------------+---------------------+---------------------+ [root@undercloud72 ~]# su - stack Last login: Wed Feb 10 12:09:23 EST 2016 on pts/0 [stack@undercloud72 ~]$ heat resource-list overcloud -n5 | grep -v COMPLETE You must provide a username via either --os-username or env[OS_USERNAME] or a token via --os-auth-token or env[OS_AUTH_TOKEN] [stack@undercloud72 ~]$ source stackrc [stack@undercloud72 ~]$ heat resource-list overcloud -n5 | grep -v COMPLETE /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:266: SubjectAltNameWarning: Certificate for 192.168.0.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead +----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+-----------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name | +----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+-----------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+ | ControllerNodesPostDeployment | 3c823abe-8ee5-4ba9-9cb0-ffc218ed4a81 | OS::TripleO::ControllerPostDeployment | CREATE_FAILED | 2016-02-10T20:57:46 | overcloud | | ControllerLoadBalancerDeployment_Step1 | ea62255a-437f-4d3e-82d1-8240f6a0406f | OS::Heat::StructuredDeployments | CREATE_FAILED | 2016-02-10T20:57:52 | overcloud-ControllerNodesPostDeployment-slywoqel5o4e | | 0 | 64a928fe-b5c0-41d8-b4a3-238011ab6c90 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2016-02-10T20:59:32 | overcloud-ControllerNodesPostDeployment-slywoqel5o4e-ControllerLoadBalancerDeployment_Step1-zs7fcze4skpx | | 1 | f219334a-d516-409e-ac70-7eb977a6ce34 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2016-02-10T20:59:32 | overcloud-ControllerNodesPostDeployment-slywoqel5o4e-ControllerLoadBalancerDeployment_Step1-zs7fcze4skpx | | 2 | a9cd6ef8-8b13-449d-b66b-7c91a7228e34 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2016-02-10T20:59:32 | overcloud-ControllerNodesPostDeployment-slywoqel5o4e-ControllerLoadBalancerDeployment_Step1-zs7fcze4skpx | +----------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------------+-----------------+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+ [stack@undercloud72 ~]$ heat deployment-show a9cd6ef8-8b13-449d-b66b-7c91a7228e34 /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:266: SubjectAltNameWarning: Certificate for 192.168.0.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead { "status": "FAILED", "server_id": "a0cc0c66-f90d-4a1d-93fb-9d76280d8fbf", "config_id": "93bf3b60-4836-4eda-9bc7-83159091339d", "output_values": { "deploy_stdout": "", "deploy_stderr": "Could not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass\nCould not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass\n\u001b[1;31mError: Error from DataBinding 'hiera' while looking up 'tripleo::packages::enable_install': (<unknown>): mapping values are not allowed in this context at line 318 column 42 on node overcloud-controller-2.localdomain\nWrapped exception:\n(<unknown>): mapping values are not allowed in this context at line 318 column 42\nWrapped exception:\n(<unknown>): mapping values are not allowed in this context at line 318 column 42\u001b[0m\n\u001b[1;31mError: Error from DataBinding 'hiera' while looking up 'tripleo::packages::enable_install': (<unknown>): mapping values are not allowed in this context at line 318 column 42 on node overcloud-controller-2.localdomain\u001b[0m\n", "deploy_status_code": 1 }, "creation_time": "2016-02-10T20:59:39", "updated_time": "2016-02-10T21:00:52", "input_values": { "step": 1, "update_identifier": { "deployment_identifier": 1455137567, "controller_config": { "1": "os-apply-config deployment dc0c9682-96fe-4741-a62f-bb3bb9bfb183 completed,a60777fe1910728ddbb3feec64db38d3 /etc/pki/ca-trust/source/anchors/ca.crt.pem\n,f80c79ff4ff77b6bff81f1c3bf9b0c1d /etc/pki/tls/private/overcloud_endpoint.pem\n,None,", "0": "os-apply-config deployment 03a1fc23-3799-461f-951f-34c24be592ea completed,a60777fe1910728ddbb3feec64db38d3 /etc/pki/ca-trust/source/anchors/ca.crt.pem\n,f80c79ff4ff77b6bff81f1c3bf9b0c1d /etc/pki/tls/private/overcloud_endpoint.pem\n,None,", "2": "os-apply-config deployment 3881a93a-a72a-4ba5-b4fc-bd00e9ec288f completed,a60777fe1910728ddbb3feec64db38d3 /etc/pki/ca-trust/source/anchors/ca.crt.pem\n,f80c79ff4ff77b6bff81f1c3bf9b0c1d /etc/pki/tls/private/overcloud_endpoint.pem\n,None," }, "allnodes_extra": "none" } }, "action": "CREATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 1", "id": "a9cd6ef8-8b13-449d-b66b-7c91a7228e34" /var/log/messages from controller-0 (full log attach): --------------------------------------------------------- time]/seluser: seluser changed 'unconfined_u' to 'system_u'\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Service/Service[pcsd]/ensure: ensure changed 'st opped' to 'running'\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Exec[enable-not-start-tripleo_cluster]/returns: executed successfully\u001b[0m\ n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Exec[Set password for hacluster user on tripleo_cluster]/returns: executed successfully\u001b[0m\n\u001b[mNo tice: /Stage[main]/Pacemaker::Corosync/Exec[auth-successful-across-all-nodes]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker: :Corosync/Exec[Create Cluster tripleo_cluster]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Exec[Start Cluster tr ipleo_cluster]/returns: executed succes Jul 5 18:48:47 localhost os-collect-config: sfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/returns: executed success fully\u001b[0m\n\u001b[mNotice: Pacemaker has reported quorum achieved\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Corosync/Notify[pacemaker settled]/me ssage: defined 'message' as 'Pacemaker has reported quorum achieved'\u001b[0m\n\u001b[mNotice: /Stage[main]/Pacemaker::Stonith/Exec[Disable STONITH]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Tripleo::Loadbalancer/Haproxy::Listen[redis]/Concat::Fragment[haproxy-redis_listen_block]/File[/v ar/lib/puppet/concat/_etc_haproxy_haproxy.cfg/fragments/20-redis-00_haproxy-redis_listen_block]/ensure: defined content as '{md5}4856695a5278618563d799c764a48 473'\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/Exec[concat_/etc/hapr oxy/haproxy.cfg]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc /haproxy/haproxy.cfg]/Exec[concat_/etc/haproxy/haproxy.cfg]: Triggered 'refresh' from 41 events\u001b[0m\n\u001b[mNotice: /Stage[main]/Haproxy/Haproxy::Instan ce[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]/content: content changed '{md5}1f337186b0e1ba5ee82760cb43 7fb810' to '{md5}e5096dfe95ab3474416c831a7bf5fec8'\u001b[0m\n\u001b[mNotice: /File[/etc/haproxy/haproxy.cfg]/seluser: seluser changed 'unconfined_u' to 'syste m_u'\u001b[0m\n\u001b[mNotice: Finished catalog run in 58.62 seconds\u001b[0m\n", "deploy_stderr": "Could not retrieve fact='apache_version', resolution='<ano nymous>': undefined method `[]' for nil:NilClass\nCould not retrieve fact='apache_version', resolution='<anonymous>': undefined method `[]' for nil:NilClass\n \u001b[1;31mWarning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.\u001b[0m\n\u001b[1;31mWarning: Scope (Haproxy::Config[haproxy]): haproxy: Th Jul 5 18:48:47 localhost os-collect-config: e $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_database provider 'mysql': Execution of '/usr/bin/mysql -NBe show databases' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n", "deploy_status_code": 0} Jul 5 18:48:47 localhost os-collect-config: [2016-07-05 22:48:47,252] (heat-config) [DEBUG] [2016-07-05 18:47:34,135] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/dd4ea9f5-2a8e-44c0-9ed5-76b766e53820" FACTER_fqdn="overcloud-controller-0.localdomain" FACTER_deploy_config_name="ControllerLoadBalancerDeployment_Step1" puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/dd4ea9f5-2a8e-44c0-9ed5-76b766e53820.pp
[root@overcloud-controller-0 ~]# pcs status Cluster name: tripleo_cluster Last updated: Wed Jul 6 03:15:35 2016 Last change: Wed Jul 6 00:52:01 2016 by root via cibadmin on overcloud-controller-2 Stack: corosync Current DC: overcloud-controller-0 (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum 3 nodes and 112 resources configured Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Full list of resources: ip-10.19.184.210 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 (unmanaged) ip-192.168.200.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 (unmanaged) Clone Set: haproxy-clone [haproxy] (unmanaged) haproxy (systemd:haproxy): Started overcloud-controller-2 (unmanaged) haproxy (systemd:haproxy): Started overcloud-controller-0 (unmanaged) haproxy (systemd:haproxy): Started overcloud-controller-1 (unmanaged) ip-192.168.0.6 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 (unmanaged) ip-10.19.104.11 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 (unmanaged) ip-10.19.105.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-1 (unmanaged) ip-10.19.104.10 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 (unmanaged) Master/Slave Set: redis-master [redis] (unmanaged) redis (ocf::heartbeat:redis): Master overcloud-controller-2 (unmanaged) redis (ocf::heartbeat:redis): Slave overcloud-controller-0 (unmanaged) redis (ocf::heartbeat:redis): Slave overcloud-controller-1 (unmanaged) Master/Slave Set: galera-master [galera] (unmanaged) galera (ocf::heartbeat:galera): Master overcloud-controller-2 (unmanaged) galera (ocf::heartbeat:galera): Master overcloud-controller-0 (unmanaged) galera (ocf::heartbeat:galera): Master overcloud-controller-1 (unmanaged) Clone Set: mongod-clone [mongod] (unmanaged) mongod (systemd:mongod): Started overcloud-controller-2 (unmanaged) mongod (systemd:mongod): Started overcloud-controller-0 (unmanaged) mongod (systemd:mongod): Started overcloud-controller-1 (unmanaged) Clone Set: rabbitmq-clone [rabbitmq] (unmanaged) rabbitmq (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-2 (unmanaged) rabbitmq (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-0 (unmanaged) rabbitmq (ocf::heartbeat:rabbitmq-cluster): Started overcloud-controller-1 (unmanaged) Clone Set: memcached-clone [memcached] (unmanaged) memcached (systemd:memcached): Started overcloud-controller-2 (unmanaged) memcached (systemd:memcached): Started overcloud-controller-0 (unmanaged) memcached (systemd:memcached): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler] (unmanaged) openstack-nova-scheduler (systemd:openstack-nova-scheduler): Started overcloud-controller-2 (unmanaged) openstack-nova-scheduler (systemd:openstack-nova-scheduler): Started overcloud-controller-0 (unmanaged) openstack-nova-scheduler (systemd:openstack-nova-scheduler): Started overcloud-controller-1 (unmanaged) Clone Set: neutron-l3-agent-clone [neutron-l3-agent] (unmanaged) neutron-l3-agent (systemd:neutron-l3-agent): Started overcloud-controller-0 (unmanaged) neutron-l3-agent (systemd:neutron-l3-agent): Started overcloud-controller-1 (unmanaged) Stopped: [ overcloud-controller-2 ] Clone Set: openstack-ceilometer-alarm-notifier-clone [openstack-ceilometer-alarm-notifier] (unmanaged) Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-heat-engine-clone [openstack-heat-engine] (unmanaged) Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api] (unmanaged) openstack-ceilometer-api (systemd:openstack-ceilometer-api): Started overcloud-controller-2 (unmanaged) openstack-ceilometer-api (systemd:openstack-ceilometer-api): Started overcloud-controller-0 (unmanaged) openstack-ceilometer-api (systemd:openstack-ceilometer-api): Started overcloud-controller-1 (unmanaged) Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent] (unmanaged) neutron-metadata-agent (systemd:neutron-metadata-agent): Started overcloud-controller-0 (unmanaged) neutron-metadata-agent (systemd:neutron-metadata-agent): Started overcloud-controller-1 (unmanaged) Stopped: [ overcloud-controller-2 ] Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup] (unmanaged) neutron-ovs-cleanup (ocf::neutron:OVSCleanup): Started overcloud-controller-2 (unmanaged) neutron-ovs-cleanup (ocf::neutron:OVSCleanup): Started overcloud-controller-0 (unmanaged) neutron-ovs-cleanup (ocf::neutron:OVSCleanup): Started overcloud-controller-1 (unmanaged) Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup] (unmanaged) neutron-netns-cleanup (ocf::neutron:NetnsCleanup): Started overcloud-controller-2 (unmanaged) neutron-netns-cleanup (ocf::neutron:NetnsCleanup): Started overcloud-controller-0 (unmanaged) neutron-netns-cleanup (ocf::neutron:NetnsCleanup): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-heat-api-clone [openstack-heat-api] (unmanaged) Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler] (unmanaged) openstack-cinder-scheduler (systemd:openstack-cinder-scheduler): Started overcloud-controller-2 (unmanaged) openstack-cinder-scheduler (systemd:openstack-cinder-scheduler): Started overcloud-controller-0 (unmanaged) openstack-cinder-scheduler (systemd:openstack-cinder-scheduler): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-nova-api-clone [openstack-nova-api] (unmanaged) openstack-nova-api (systemd:openstack-nova-api): Started overcloud-controller-2 (unmanaged) openstack-nova-api (systemd:openstack-nova-api): Started overcloud-controller-0 (unmanaged) openstack-nova-api (systemd:openstack-nova-api): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch] (unmanaged) Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector] (unmanaged) openstack-ceilometer-collector (systemd:openstack-ceilometer-collector): Started overcloud-controller-2 (unmanaged) openstack-ceilometer-collector (systemd:openstack-ceilometer-collector): Started overcloud-controller-0 (unmanaged) openstack-ceilometer-collector (systemd:openstack-ceilometer-collector): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-keystone-clone [openstack-keystone] (unmanaged) openstack-keystone (systemd:openstack-keystone): Started overcloud-controller-2 (unmanaged) openstack-keystone (systemd:openstack-keystone): Started overcloud-controller-0 (unmanaged) openstack-keystone (systemd:openstack-keystone): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth] (unmanaged) openstack-nova-consoleauth (systemd:openstack-nova-consoleauth): Started overcloud-controller-2 (unmanaged) openstack-nova-consoleauth (systemd:openstack-nova-consoleauth): Started overcloud-controller-0 (unmanaged) openstack-nova-consoleauth (systemd:openstack-nova-consoleauth): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-glance-registry-clone [openstack-glance-registry] (unmanaged) openstack-glance-registry (systemd:openstack-glance-registry): Started overcloud-controller-2 (unmanaged) openstack-glance-registry (systemd:openstack-glance-registry): Started overcloud-controller-0 (unmanaged) openstack-glance-registry (systemd:openstack-glance-registry): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification] (unmanaged) Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-cinder-api-clone [openstack-cinder-api] (unmanaged) openstack-cinder-api (systemd:openstack-cinder-api): Started overcloud-controller-2 (unmanaged) openstack-cinder-api (systemd:openstack-cinder-api): Started overcloud-controller-0 (unmanaged) openstack-cinder-api (systemd:openstack-cinder-api): Started overcloud-controller-1 (unmanaged) Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent] (unmanaged) neutron-dhcp-agent (systemd:neutron-dhcp-agent): Started overcloud-controller-2 (unmanaged) neutron-dhcp-agent (systemd:neutron-dhcp-agent): Started overcloud-controller-0 (unmanaged) neutron-dhcp-agent (systemd:neutron-dhcp-agent): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-glance-api-clone [openstack-glance-api] (unmanaged) openstack-glance-api (systemd:openstack-glance-api): Started overcloud-controller-2 (unmanaged) openstack-glance-api (systemd:openstack-glance-api): Started overcloud-controller-0 (unmanaged) openstack-glance-api (systemd:openstack-glance-api): Started overcloud-controller-1 (unmanaged) Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent] (unmanaged) neutron-openvswitch-agent (systemd:neutron-openvswitch-agent): Started overcloud-controller-2 (unmanaged) neutron-openvswitch-agent (systemd:neutron-openvswitch-agent): Started overcloud-controller-0 (unmanaged) neutron-openvswitch-agent (systemd:neutron-openvswitch-agent): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy] (unmanaged) openstack-nova-novncproxy (systemd:openstack-nova-novncproxy): Started overcloud-controller-2 (unmanaged) openstack-nova-novncproxy (systemd:openstack-nova-novncproxy): Started overcloud-controller-0 (unmanaged) openstack-nova-novncproxy (systemd:openstack-nova-novncproxy): Started overcloud-controller-1 (unmanaged) Clone Set: delay-clone [delay] (unmanaged) delay (ocf::heartbeat:Delay): Started overcloud-controller-2 (unmanaged) delay (ocf::heartbeat:Delay): Started overcloud-controller-0 (unmanaged) delay (ocf::heartbeat:Delay): Started overcloud-controller-1 (unmanaged) Clone Set: neutron-server-clone [neutron-server] (unmanaged) neutron-server (systemd:neutron-server): Started overcloud-controller-2 (unmanaged) neutron-server (systemd:neutron-server): Started overcloud-controller-0 (unmanaged) neutron-server (systemd:neutron-server): Started overcloud-controller-1 (unmanaged) Clone Set: httpd-clone [httpd] (unmanaged) httpd (systemd:httpd): Started overcloud-controller-2 (unmanaged) httpd (systemd:httpd): Started overcloud-controller-0 (unmanaged) httpd (systemd:httpd): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central] (unmanaged) openstack-ceilometer-central (systemd:openstack-ceilometer-central): Started overcloud-controller-2 (unmanaged) openstack-ceilometer-central (systemd:openstack-ceilometer-central): Started overcloud-controller-0 (unmanaged) openstack-ceilometer-central (systemd:openstack-ceilometer-central): Started overcloud-controller-1 (unmanaged) Clone Set: openstack-ceilometer-alarm-evaluator-clone [openstack-ceilometer-alarm-evaluator] (unmanaged) Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn] (unmanaged) Stopped: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] openstack-cinder-volume (systemd:openstack-cinder-volume): Started overcloud-controller-0 (unmanaged) Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor] (unmanaged) openstack-nova-conductor (systemd:openstack-nova-conductor): Started overcloud-controller-2 (unmanaged) openstack-nova-conductor (systemd:openstack-nova-conductor): Started overcloud-controller-0 (unmanaged) openstack-nova-conductor (systemd:openstack-nova-conductor): Started overcloud-controller-1 (unmanaged) Failed Actions: * openstack-ceilometer-alarm-evaluator_start_0 on overcloud-controller-2 'not installed' (5): call=617, status=Not installed, exitreason='none', last-rc-change='Wed Jul 6 00:20:13 2016', queued=0ms, exec=87ms * openstack-ceilometer-alarm-evaluator_start_0 on overcloud-controller-0 'not installed' (5): call=616, status=Not installed, exitreason='none', last-rc-change='Wed Jul 6 00:20:13 2016', queued=0ms, exec=216ms * openstack-ceilometer-alarm-evaluator_start_0 on overcloud-controller-1 'not installed' (5): call=639, status=Not installed, exitreason='none', last-rc-change='Wed Jul 6 00:20:13 2016', queued=0ms, exec=99ms PCSD Status: overcloud-controller-0: Online overcloud-controller-1: Online overcloud-controller-2: Online Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@overcloud-controller-0 ~]#
Created attachment 1176713 [details] heat-engine.log from undercloud
Created attachment 1176714 [details] Adding the /var/log/messages from the controller
reproduced again with: openstack-tripleo-heat-templates-liberty-2.0.0-13.el7ost.noarch happens during the AODH migration step: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ceph-storage-scale 2 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server 10.5.26.10 --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml -e /home/stack/ssl-heat-templates/environments/enable-tls.yaml -e /home/stack/ssl-heat-templates/environments/inject-trust-anchor.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-aodh.yaml results : -----------+---------------------+ | d288684b-b4d7-4606-bac5-c63611fb262b | overcloud | UPDATE_FAILED | 2016-02-10T18:26:12 | 2016-02-11T17:17:10 | +--------------------------------------+------------+---------------+---------------------+---------------------+
This appears to be caused by the following hierdata in controller.yaml: [root@overcloud-controller-0 hieradata]# sed -n 318p /etc/puppet/hieradata/controller.yaml ceilometer::dispatcher::gnocchi::url: ://:
Most probably this is caused by the enable-tls.yaml not containing the new services in the EndpointMap. I adjusted it like the following and the hieradata got populated with the url containing the internal vip: [root@overcloud-controller-0 heat-admin]# sed -n 318p /etc/puppet/hieradata/controller.yaml ceilometer::dispatcher::gnocchi::url: http://10.0.0.10:8041 diff templates/enable-tls.yaml templates/enable-tls.yaml.pre-upgrade 55,63d54 < AodhAdmin: {protocol: 'http', port: '8042', host: 'IP_ADDRESS'} < AodhInternal: {protocol: 'http', port: '8042', host: 'IP_ADDRESS'} < AodhPublic: {protocol: 'https', port: '13042', host: 'IP_ADDRESS'} < GnocchiAdmin: {protocol: 'http', port: '8041', host: 'IP_ADDRESS'} < GnocchiInternal: {protocol: 'http', port: '8041', host: 'IP_ADDRESS'} < GnocchiPublic: {protocol: 'https', port: '13041', host: 'IP_ADDRESS'} < SaharaAdmin: {protocol: 'http', port: '8386', host: 'IP_ADDRESS'} < SaharaInternal: {protocol: 'http', port: '8386', host: 'IP_ADDRESS'} < SaharaPublic: {protocol: 'https', port: '13386', host: 'IP_ADDRESS'}
According to Marius the issue is that the upgrade command is changing the original enable-tls.yaml file that locates under: /usr/share/openstack-tripleo-heat-templates/environments/enable-tls.yaml then when we run the upgrade command (with -e) we were calling the local copy of the enable-tls.yaml file that is located under: /home/stack/ssl-heat-templates/environments/enable-tls.yaml the file that under the /home/stack/ - didn't change during the upgrade,. Workaround: ------------ before running the upgrade command, run sasha's sed cmd to fix the enable-tls.yaml file that under /home/stack/ssl-heat-templates/environments/enable-tls.yaml: sed -i '/EndpointMap.*/a \ \ \ \ AodhAdmin: {protocol: 'http', port: '8042', host: 'IP_ADDRESS'}\n\ \ \ \ AodhInternal: {protocol: 'http', port: '8042', ho 'https', port: '13042', host: 'IP_ADDRESS'}\n\ \ \ \ GnocchiAdmin: {protocol: 'http', port: '8041', host: 'IP_ADDRESS'}\n\ \ \ \ GnocchiInternal: {protocol: 'http', port: '8041', host: 'IP_A ', port: '13041', host: 'IP_ADDRESS'\n\ \ \ \ SaharaAdmin: {protocol: 'http', port: '8386', host: 'IP_ADDRESS'}\n\ \ \ \ SaharaInternal: {protocol: 'http', port: '8386', host: 'IP_ADDRESS'\n 13386', host: 'IP_ADDRESS'}' /home/stack/ssl-heat-templates/environments/enable-tls.yaml
fixing the sed command : sed -i "/EndpointMap.*/a \ \ \ \ AodhAdmin: {protocol: 'http', port: '8042', host: 'IP_ADDRESS'}\n\ \ \ \ AodhInternal: {protocol: 'http', port: '8042', host: 'IP_ADDRESS'}\n\ \ \ \ AodhPublic: {protocol: 'https', port: '13042', host: 'IP_ADDRESS'}\n\ \ \ \ GnocchiAdmin: {protocol: 'http', port: '8041', host: 'IP_ADDRESS'}\n\ \ \ \ GnocchiInternal: {protocol: 'http', port: '8041', host: 'IP_ADDRESS'}\n\ \ \ \ GnocchiPublic: {protocol: 'https', port: '13041', host: 'IP_ADDRESS'}\n\ \ \ \ SaharaAdmin: {protocol: 'http', port: '8386', host: 'IP_ADDRESS'}\n\ \ \ \ SaharaInternal: {protocol: 'http', port: '8386', host: 'IP_ADDRESS'}\n\ \ \ \ SaharaPublic: {protocol: 'https', port: '13386', host: 'IP_ADDRESS'}" /home/stack/ssl-heat-templates/environments/enable-tls.yaml
My attempt as fixing these problems was to use map_merge for that https://review.openstack.org/#/c/308182/ but it wasn't well received. I can try to re-take that effort though. Else I can write a tool to update that map and keep the certs, which should be called when an update is needed.
*** Bug 1356077 has been marked as a duplicate of this bug. ***
I think this BZ might have slipped under the radar. Grabbing this BZ and checking for the fix.
Dan Macpherson, do we have any progress with the documentation of the workaround in our upgrade doc? The workaround is known for 6 months. Given the number of customers upgrading from 8 to 9, this should be documented. Thanks. -ak
Thanks for highlighting this BZ, Andreas. I had some priority work I had to focus on for OSP10, but I've got that out of the way. I'll make this BZ a priority and work on it today.
Thanks a lot :-)
Have added the following note to the OSP9 Upgrade procedure: "If using a custom endpoint map for enabling TLS/SSL in the overcloud, make sure to update the map with endpoints for the following new services: OpenStack Telemetry Metrics (gnocchi) OpenStack Telemetry Alarming (aodh) OpenStack Clustering (sahara) Check the latest TLS/SSL mappings from the core Heat template collection (see EndpointMap in /usr/share/openstack-tripleo-heat-templates/environments/enable-tls.yaml) and add the missing endpoints to the EndpointMap in your custom enable-tls.yaml file. For more information, see "Enabling SSL/TLS on the Overcloud" in the Red Hat OpenStack Platform Director Installation and Usage guide." Link: https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/single/upgrading-red-hat-openstack-platform/#sect-Pre-Upgrade_Notes_for_Overcloud Omri and Andreas, anything further required for this BZ?
No response in over two weeks. If nothing further to add, I'll close this BZ. If further changes are required, please feel free to reopen it.
Hi Eduard, Is there something in the documentation unresolved in relation to the case you posted? Just want to make sure we have everything covered. - Dan