Hide Forgot
Description of problem: Deployment of a new overcloud v9, using undercloud v10 fails details: [stack@instack ~]$ heat stack-list --show-nested -f "status=FAILED" /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.0.2.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.0.2.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning +--------------------------------------+---------------------------------------------------------------------------------------------------------------+---------------+----------------------+--------------+--------------------------------------+ | id | stack_name | stack_status | creation_time | updated_time | parent | +--------------------------------------+---------------------------------------------------------------------------------------------------------------+---------------+----------------------+--------------+--------------------------------------+ | 3974b36e-9504-43c2-8426-a399634018c8 | overcloud | CREATE_FAILED | 2016-10-17T15:38:47Z | None | None | | 7b857760-4c89-49e3-a515-25358d676ba8 | overcloud-ControllerNodesPostDeployment-3is2wwbw3f6l | CREATE_FAILED | 2016-10-17T16:21:46Z | None | 3974b36e-9504-43c2-8426-a399634018c8 | | f8858df3-d233-4d52-9da2-578f981ecf2f | overcloud-ControllerNodesPostDeployment-3is2wwbw3f6l-ControllerOvercloudServicesDeployment_Step6-dhn2bppp6noc | CREATE_FAILED | 2016-10-17T16:37:13Z | None | 7b857760-4c89-49e3-a515-25358d676ba8 | +--------------------------------------+---------------------------------------------------------------------------------------------------------------+---------------+----------------------+--------------+--------------------------------------+ [stack@instack ~]$ heat resource-list --nested-depth 5 overcloud | grep FAILED /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.0.2.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.0.2.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning | ControllerNodesPostDeployment | 7b857760-4c89-49e3-a515-25358d676ba8 | OS::TripleO::ControllerPostDeployment | CREATE_FAILED | 2016-10-17T15:38:48Z | overcloud | | ControllerOvercloudServicesDeployment_Step6 | f8858df3-d233-4d52-9da2-578f981ecf2f | OS::Heat::StructuredDeployments | CREATE_FAILED | 2016-10-17T16:21:46Z | overcloud-ControllerNodesPostDeployment-3is2wwbw3f6l | | 0 | ac7ebc4f-7185-4be0-bb9d-5a233c95bc14 | OS::Heat::StructuredDeployment | CREATE_FAILED | 2016-10-17T16:37:13Z | overcloud-ControllerNodesPostDeployment-3is2wwbw3f6l-ControllerOvercloudServicesDeployment_Step6-dhn2bppp6noc | [stack@instack ~]$ heat resource-show f8858df3-d233-4d52-9da2-578f981ecf2f 0|grep resource_status_reason /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.0.2.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.0.2.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning | resource_status_reason | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6 | [stack@instack ~]$ echo -e `heat deployment-show ac7ebc4f-7185-4be0-bb9d-5a233c95bc14` /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.0.2.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning WARNING (shell) "heat deployment-show" is deprecated, please use "openstack software deployment show" instead /usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py:303: SubjectAltNameWarning: Certificate for 192.0.2.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.) SubjectAltNameWarning { "status": "FAILED", "server_id": "e9f9dc5a-a0a8-4dbc-8242-09ae08d3a928", "config_id": "b8359f87-17cc-460d-96fe-ddba66131fa7", "output_values": { "deploy_stdout": "Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 40.38 seconds Notice: /Stage[main]/Main/Exec[galera-set-root-password]/returns: executed successfully Notice: /Stage[main]/Main/File[/root/.my.cnf]/content: content changed '{md5}d41d8cd98f00b204e9800998ecf8427e' to '{md5}9d9cc9b42a0cc9cbad480734b5127f08' Notice: /Stage[main]/Main/File[/root/.my.cnf]/mode: mode changed '0644' to '0600' Notice: /File[/root/.my.cnf]/seltype: seltype changed 'admin_home_t' to 'mysqld_home_t' Notice: /Stage[main]/Main/Package_manifest[/var/lib/tripleo/installed-packages/overcloud_controller_pacemaker5]/ensure: created 8cZrKGspeB3vn3x8wqN6xgJtq password regionOne -1 True rabbit rsK9EpdXrQRcQ34QpUHCApdWF 192.168.100.22,192.168.100.19,192.168.100.15 redis://:Dcjv4yHMB67uyePCEjX89nCRW.100.10:6379/ 600 notifications 0.0.0.0 Default Default True database False http://192.168.100.18:5000/v2.0 database 4952 http://192.168.100.18:5000 http://192.0.2.8:35357 Notice: /Stage[main]/Gnocchi::Storage::Ceph/Package[python-cradox]/ensure: created /var/log/ceilometer 192.168.100.22 Notice: /Stage[main]/Aodh::Client/Package[python-aodhclient]/ensure: created service ceilometer / 60 service guest 2 rsK9EpdXrQRcQ34QpUHCApdWF ceilometer -1 mongodb://192.168.100.22:27017,192.168.100.19:27017,192.168.100.15:27017/ceilometer?replicaSet=tripleo False 8777 service http://192.168.100.18:8041 gnocchi_resources.yaml low Gc8RBtB3vrrYhy4KfTv7My4tD internalURL Notice: /Stage[main]/Main/Exec[galera-ready]/returns: executed successfully Notice: /Stage[main]/Gnocchi::Db::Sync/Exec[gnocchi-db-sync]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Gnocchi::Statsd/Service[gnocchi-statsd]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Gnocchi::Api/Service[gnocchi-api]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Gnocchi::Metricd/Service[gnocchi-metricd]: Triggered 'refresh' from 1 events Notice: /Stage[main]/Keystone::Deps/Anchor[keystone::service::end]: Triggered 'refresh' from 2 events Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: created Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/description: description changed 'Bootstrap project for initializing the cloud.' to 'admin tenant' Notice: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_user_domain_name]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin_password]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Keystone::Domain/Keystone_domain[heat_stack]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Keystone::Domain/Keystone_user[heat_stack_domain_admin::heat_stack]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Keystone::Domain/Keystone_user_role[heat_stack_domain_admin::heat_stack@::heat_stack]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_endpoint[regionOne/keystone::identity]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Deps/Anchor[heat::config::end]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Deps/Anchor[heat::db::begin]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.168.100.18]/Mysql_user[heat.100.18]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.168.100.22]/Mysql_user[heat.100.22]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.168.100.22]/Mysql_grant[heat.100.22/heat.*]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.168.100.18]/Mysql_grant[heat.100.18/heat.*]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Deps/Anchor[heat::db::end]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::begin]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::end]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::begin]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Engine/Service[heat-engine]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Api/Service[heat-api]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Dependency Keystone_user[admin] has failures: true Notice: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Dependency Keystone_user[admin] has failures: true Notice: Finished catalog run in 291.41 seconds ", "deploy_stderr": "Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided. Warning: Scope(Class[Swift]): swift_hash_suffix has been deprecated and should be replaced with swift_hash_path_suffix, this will be removed as part of the N-cycle Warning: Scope(Class[Keystone]): Execution of db_sync does not depend on $enabled anymore. Please use sync_db instead. Warning: Scope(Class[Glance::Api]): The known_stores parameter is deprecated, use stores instead Warning: Scope(Class[Glance::Api]): default_store not provided, it will be automatically set to glance.store.http.Store Warning: Scope(Class[Glance::Registry]): Execution of db_sync does not depend on $manage_service or $enabled anymore. Please use sync_db instead. Warning: Scope(Class[Nova::Api]): ec2_listen_port, ec2_workers and keystone_ec2_url are deprecated and have no effect. Deploy openstack/ec2-api instead. Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated Warning: Scope(Class[Neutron::Server]): identity_uri, auth_tenant, auth_user, auth_password, auth_region configuration options are deprecated in favor of auth_plugin and related options Warning: Scope(Class[Neutron::Agents::Dhcp]): The dhcp_delete_namespaces parameter was removed in Mitaka, it does not take any affect Warning: Scope(Class[Neutron::Agents::L3]): parameter external_network_bridge is deprecated Warning: Scope(Class[Neutron::Agents::L3]): parameter router_delete_namespaces was removed in Mitaka, it does not take any affect Warning: Scope(Class[Neutron::Agents::Metadata]): The auth_password parameter is deprecated and was removed in Mitaka release. Warning: Scope(Class[Neutron::Agents::Metadata]): The auth_tenant parameter is deprecated and was removed in Mitaka release. Warning: Scope(Class[Neutron::Agents::Metadata]): The auth_url parameter is deprecated and was removed in Mitaka release. Warning: Scope(Class[Ceilometer::Api]): The keystone_auth_uri parameter is deprecated. Please use auth_uri instead. Warning: Scope(Class[Ceilometer::Api]): The keystone_identity_uri parameter is deprecated. Please use identity_uri instead. Warning: Scope(Class[Heat]): \"admin_user\", \"admin_password\", \"admin_tenant_name\" configuration options are deprecated in favor of auth_plugin and related options Warning: You cannot collect exported resources without storeconfigs being set; the collection will be ignored on line 123 in file /etc/puppet/modules/gnocchi/manifests/api.pp Warning: Not collecting exported resources without storeconfigs Warning: Not collecting exported resources without storeconfigs Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications. Warning: Not collecting exported resources without storeconfigs Warning: Not collecting exported resources without storeconfigs Warning: Not collecting exported resources without storeconfigs Error: /Stage[main]/Neutron/Resources[neutron_config]: Failed to generate additional resources using 'generate': OpenStackConfig only support collecting instances when a file path is hard coded Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/usr/bin/openstack token issue --format value' returned 1: Unable to establish connection to http://192.168.200.189:5000/v3/auth/tokens (tried 41, for a total of 170 seconds) Warning: /Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_user_domain_name]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin_password]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Keystone::Domain/Heat_config[DEFAULT/stack_domain_admin]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Keystone::Domain/Keystone_domain[heat_stack]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Keystone::Domain/Keystone_user[heat_stack_domain_admin::heat_stack]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Keystone::Domain/Keystone_user_role[heat_stack_domain_admin::heat_stack@::heat_stack]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_service[keystone::identity]: Skipping because of failed dependencies Warning: /Stage[main]/Keystone::Endpoint/Keystone::Resource::Service_identity[keystone]/Keystone_endpoint[regionOne/keystone::identity]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Deps/Anchor[heat::config::end]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Deps/Anchor[heat::db::begin]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Mysql_database[heat]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_user[heat@%]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_%]/Mysql_grant[heat@%/heat.*]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.168.100.18]/Mysql_user[heat.100.18]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.168.100.22]/Mysql_user[heat.100.22]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.168.100.22]/Mysql_grant[heat.100.22/heat.*]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Mysql/Openstacklib::Db::Mysql[heat]/Openstacklib::Db::Mysql::Host_access[heat_192.168.100.18]/Mysql_grant[heat.100.18/heat.*]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Deps/Anchor[heat::db::end]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::begin]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Db::Sync/Exec[heat-dbsync]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Deps/Anchor[heat::dbsync::end]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Deps/Anchor[heat::service::begin]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Api_cfn/Service[heat-api-cfn]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Engine/Service[heat-engine]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Api/Service[heat-api]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Api_cloudwatch/Service[heat-api-cloudwatch]: Skipping because of failed dependencies Warning: /Stage[main]/Heat::Deps/Anchor[heat::service::end]: Skipping because of failed dependencies ", "deploy_status_code": 6 }, "creation_time": "2016-10-17T16:37:15Z", "updated_time": "2016-10-17T16:44:06Z", "input_values": { "step": 5, "update_identifier": { "deployment_identifier": "1476718715", "controller_config": { "1": "os-apply-config deployment 25bcf36c-d7fc-44b0-8a74-90303db1fd69 completed,b1bc18a1c2dc82e4beaacc4d7157dede /etc/pki/ca-trust/source/anchors/ca.crt.pem ,f6b0ec3565ed4337e0e22f636d5d6efc /etc/pki/tls/private/overcloud_endpoint.pem ,None,", "0": "os-apply-config deployment 6804ee20-bde6-4323-a057-566a18c4f4f5 completed,b1bc18a1c2dc82e4beaacc4d7157dede /etc/pki/ca-trust/source/anchors/ca.crt.pem ,f6b0ec3565ed4337e0e22f636d5d6efc /etc/pki/tls/private/overcloud_endpoint.pem ,None,", "2": "os-apply-config deployment ad014f4f-82e2-4d3c-9862-12b41bc676d2 completed,b1bc18a1c2dc82e4beaacc4d7157dede /etc/pki/ca-trust/source/anchors/ca.crt.pem ,f6b0ec3565ed4337e0e22f636d5d6efc /etc/pki/tls/private/overcloud_endpoint.pem ,None," }, "allnodes_extra": "none" } }, "action": "CREATE", "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 6", "id": "ac7ebc4f-7185-4be0-bb9d-5a233c95bc14" } Version-Release number of selected component (if applicable): openstack-selinux-0.7.11-1.el7ost.noarch openstack-mistral-common-3.0.1-0.20161006155154.6356bce.el7ost.noarch openstack-ironic-conductor-6.2.2-0.20161006174219.500a27d.el7ost.noarch openstack-glance-13.0.0-1.el7ost.noarch openstack-nova-scheduler-14.0.1-1.el7ost.noarch openstack-neutron-common-9.0.0-1.3.el7ost.noarch openstack-tripleo-heat-templates-compat-2.0.0-34.3.el7ost.noarch openstack-nova-compute-14.0.1-1.el7ost.noarch openstack-heat-api-7.0.0-2.el7ost.noarch openstack-nova-api-14.0.1-1.el7ost.noarch openstack-tripleo-puppet-elements-5.0.0-0.20161003213431.200d011.el7ost.noarch openstack-tripleo-ui-1.0.3-1.el7ost.noarch openstack-tripleo-image-elements-5.0.0-1.el7ost.noarch openstack-nova-common-14.0.1-1.el7ost.noarch openstack-neutron-ml2-9.0.0-1.3.el7ost.noarch openstack-ironic-inspector-4.2.1-0.20161005144819.9a079eb.el7ost.noarch openstack-neutron-openvswitch-9.0.0-1.3.el7ost.noarch openstack-heat-common-7.0.0-2.el7ost.noarch openstack-mistral-executor-3.0.1-0.20161006155154.6356bce.el7ost.noarch openstack-swift-container-2.10.1-0.20161003211202.3349016.el7ost.noarch openstack-nova-cert-14.0.1-1.el7ost.noarch puppet-openstack_extras-9.4.0-1.el7ost.noarch openstack-puppet-modules-9.3.0-0.20161003154825.8c758d6.el7ost.noarch python-openstackclient-3.2.0-2.el7ost.noarch openstack-tripleo-common-5.2.1-0.20161007114757.cc19d04.el7ost.noarch openstack-tripleo-0.0.8-0.2.4de13b3git.el7ost.noarch openstack-neutron-9.0.0-1.3.el7ost.noarch openstack-zaqar-3.0.0-2.el7ost.noarch openstack-nova-conductor-14.0.1-1.el7ost.noarch openstack-ironic-api-6.2.2-0.20161006174219.500a27d.el7ost.noarch openstack-heat-engine-7.0.0-2.el7ost.noarch openstack-swift-object-2.10.1-0.20161003211202.3349016.el7ost.noarch python-openstacksdk-0.9.5-1.el7ost.noarch puppet-openstacklib-9.4.0-0.20161004171440.0e58c86.el7ost.noarch openstack-tempest-12.2.1-0.20161004111913.ef2befe.1.el7ost.noarch openstack-mistral-api-3.0.1-0.20161006155154.6356bce.el7ost.noarch openstack-heat-api-cfn-7.0.0-2.el7ost.noarch openstack-swift-proxy-2.10.1-0.20161003211202.3349016.el7ost.noarch openstack-mistral-engine-3.0.1-0.20161006155154.6356bce.el7ost.noarch openstack-ironic-common-6.2.2-0.20161006174219.500a27d.el7ost.noarch python-openstack-mistral-3.0.1-0.20161006155154.6356bce.el7ost.noarch openstack-swift-account-2.10.1-0.20161003211202.3349016.el7ost.noarch openstack-tripleo-heat-templates-5.0.0-0.20161003064637.d636e3a.1.2.el7ost.noarch openstack-keystone-10.0.0-1.el7ost.noarch openstack-heat-templates-0.0.1-0.20161004223740.f123aa1.el7ost.noarch
original deployment command: openstack overcloud deploy --templates /home/stack/tht --control-scale 3 --compute-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server clock.redhat.com --timeout 90 -e /home/stack/tht/environments/puppet-pacemaker.yaml -e /home/stack/tht/environments/storage-environment.yaml -e /home/stack/tht/environments/network-isolation.yaml -e network-environment.yaml -e ~/ssl-heat-templates/environments/enable-tls.yaml -e ~/ssl-heat-templates/environments/inject-trust-anchor.yaml --ceph-storage-scale 1 /home/stack/tht holds a copy of THT from the openstack-tripleo-heat-templates-compat
marios, can someone from lifecycle take a look at this one?
OK assigned to apetrich since he's looking at the backwards compat - let's see if there was any info/triage from Sofer as per comment #5 too Adriano can you please sync with Dan and have a look at this?
So it seems that the issue is that the VIP is going to 192.168.200.188 instead of 192.168.200.180 and the cert is for 192.168.200.180 some evidence of that: Notice: /Stage[main]/Main/Pacemaker::Resource::Ip[public_vip]/Pcmk_resource[ip-192.168.200.188]/ensure: created here is the error that causes the newton error SSL exception connecting to https://192.168.200.188:13000/v3/auth/tokens: hostname '192.168.200.188' doesn't match u'192.168.200.180' Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/usr/bin/openstack token issue --format value' returned 1: Certificate did not match expected hostname: 192.168.200.188. the network-environment.yaml: ExternalAllocationPools: [{'start': '192.168.200.180', 'end': '192.168.200.200'}] and a netstat in the controller-0 [root@overcloud-controller-0 keystone]# netstat -anp | grep 188 tcp 0 0 192.168.200.188:13386 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13003 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13004 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13292 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13773 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13357 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13774 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13808 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:80 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13776 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13041 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13777 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13042 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13080 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:443 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13696 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13000 0.0.0.0:* LISTEN 18534/haproxy | tcp 0 0 192.168.200.188:13800 0.0.0.0:* LISTEN 18534/haproxy
So, I got into the node and the actual error message is: Error: /Stage[main]/Keystone::Roles::Admin/Keystone_user[admin]: Could not evaluate: Execution of '/usr/bin/openstack token issue --format value' returned 1: Certificate did not match expected hostn ame: 192.168.200.185. Certificate: {'notBefore': u'Oct 20 15:43:59 2016 GMT', 'serialNumber': u'9D54725C4D116EB7', 'notAfter': 'Oct 20 15:43:59 2017 GMT', 'version': 3L, 'subject': ((('countryName', u'US'),), (('stateOrProvinceName', u'NC'),), (('localityName', u'Raleigh'),), (('organizationName', u'Red HAt'),), (('organizationalUnitName', u'QE'),), (('commonName', u'192.168.200.180'),)), 'iss uer': ((('countryName', u'US'),), (('stateOrProvinceName', u'NC'),), (('localityName', u'Raleigh'),), (('organizationName', u'Red HAt'),), (('organizationalUnitName', u'QE'),), (('commonName', u'192 .168.200.180'),))} SSL exception connecting to https://192.168.200.185:13000/v3/auth/tokens: hostname '192.168.200.185' doesn't match u'192.168.200.180' (tried 40, for a total of 170 seconds) which indicates that the certificate has the wrong CN or SubjectAltName. Now, it was assumed that it would be 192.168.200.180, however, we need to consider that this cannot be assured unless we set the FixedIPs for the Public network (which can be done via the PublicVirtualFixedIPs). I checked the Fixed IPs and they're not set: "StorageVirtualFixedIPs": "[]", "PublicVirtualFixedIPs": "[]", "StorageMgmtVirtualFixedIPs": "[]", "ControlFixedIPs": "[]", "InternalApiVirtualFixedIPs": "[]", Setting FixedIPs to match the certificate would fix the issue.
Could you try that Dan? Also I think it only affects multiple controllers on a 1 controller 1 compute env it worked fine without the extra params
(In reply to Adriano Petrich from comment #9) > Could you try that Dan? > > Also I think it only affects multiple controllers on a 1 controller 1 > compute env it worked fine without the extra params I can try it, but this is still something new, since this sort of setup worked on previous puddles without any issues. With just one controller the VIP will not be able to change, and the cert signed for a specific IP will work, of course. I can definitely try to assign fixed IPs, but this will be a workaround, not a solution and will bring us no closer to the root cause of this issue
dan, can you confirm that this is a new deployment of an osp9 overcloud with ssl using an osp10 undercloud? I think we need to identify why the cert generation process assumed the VIP was 192.168.200.180 when it is actually 192.168.200.185. Juan, is this something you can look into? Using FixedIP's is a workaround, but it shouldn't be necessary if we are automatically generating the certs during the deployment process.
James, Yes it is a new deployment of an overcloud osp9 using an osp10 undercloud. I think it is the other way around we were expecting the VIP to be the first of the ExternalAllocationPools as it has been on the previous versions. this script is based on what was working before. It looks like the ssl is not the issue, as the certs points to the expected VIP the breakage is just showing now. if it wasn't for the ssl the endpoints are mapped to the new ips and everything in the overcloud is still working although not in the first external ip of the allocation pool. So far we are not sure on what prompted this change but I can see two possibilities: * What we assumed that was "the usual behaviour" was a glitch and we used that as the default. This is a tangible possibility since we are not defining PublicVirtualFixedIPs anything in the sense in order forcing that ip to be 192.168.200.180 * the "usual behaviour" is the correct one, and now it has changed accidentally (or not). There are going to be breakages from clients and users Outputs in the first case we might need more documentation on this. on the second we need to find where did the change happened. Anyway I don't know where to go from here besides what I'm doing right now that is try setting up those values as an workaround
Dan, adding PublicVirtualFixedIPs: [{'ip_address':'192.168.200.180'}] to the network-settings.yaml fixed the issue.
(In reply to Adriano Petrich from comment #15) > Dan, > > > adding PublicVirtualFixedIPs: [{'ip_address':'192.168.200.180'}] to the > network-settings.yaml fixed the issue. That sounds good, but we need to understand whether this workaround needs to become the default for all new deployments (or just mixed version deployments?) and then this needs to be documented, or the old behaviour is correct and we need to fix whatever broke it in the current and previous puddles. James, can your team help with that? I realize the easiest solution is to just document it, but leaving a regression alone can cause additional grief down the line, I think.
(In reply to Dan Yasny from comment #16) > (In reply to Adriano Petrich from comment #15) > > Dan, > > > > > > adding PublicVirtualFixedIPs: [{'ip_address':'192.168.200.180'}] to the > > network-settings.yaml fixed the issue. > > That sounds good, but we need to understand whether this workaround needs to > become the default for all new deployments (or just mixed version > deployments?) and then this needs to be documented, or the old behaviour is > correct and we need to fix whatever broke it in the current and previous > puddles. > > James, can your team help with that? I realize the easiest solution is to > just document it, but leaving a regression alone can cause additional grief > down the line, I think. setting PublicVirtualFixedIPs is required when deploying with ssl and using the VIP as the CN of the certificate. This is because Neutron no longer gurantees that the first IP allocated in a dhcp subnet range will be the first (lowest) IP in the range, so the VIP is not predictable. Setting the PublicVirtualFixedIPs parameter makes it predictable. This is not a regression. It's still possible to deploy with SSL and do everything that was previously possible. It is however a change in the documented instructions on how you need to deploy with SSL. This is documented in tripleo-docs: http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/ssl.html#overcloud-ssl I think the action here for this bugzilla is to make it into a docs bug to make sure that same change is reflected in the product docs.
With the suggested workaround in place, the mixed version deployment with SSL enabled works manually. I have also tested deployments of clean SSL enabled 7, 8 and 9 setups with a FixedIP parameter set, and it might be a good idea to recommend this parameter to be included in the documentation for all versions, since it causes no damage and allows for consistency between versions
I think this requirement is covered in bug 1357688. I'm closing this one as a duplicate. Please reopen if this is incorrect, or add any additional requirements in bug 1357688. *** This bug has been marked as a duplicate of bug 1357688 ***