Bug 1308542

Summary: overcloud deploy complete with Authorization Failed
Product: Red Hat OpenStack Reporter: Asaf Hirshberg <ahirshbe>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED NOTABUG QA Contact: yeylon <yeylon>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.0 (Liberty)CC: dbecker, gkadam, mburns, mcornea, morazi, rhel-osp-director-maint, srevivo
Target Milestone: ---   
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-16 13:13:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
controller-0 keystone log
none
controller-1 keystone log
none
controller-2 keystone log none

Description Asaf Hirshberg 2016-02-15 13:21:25 UTC 
Created attachment 1127273 [details]
controller-0 keystone log

Description of problem:

2016-02-15 12:44:48 [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully
Stack overcloud CREATE_COMPLETE
/home/stack/.ssh/known_hosts updated.
Original contents retained as /home/stack/.ssh/known_hosts.old
PKI initialization in init-keystone is deprecated and will be removed.
Warning: Permanently added '192.0.2.8' (ECDSA) to the list of known hosts.
 No handlers could be found for logger "oslo_config.cfg"
2016-02-15 07:45:21.257 3332 WARNING keystone.cmd.cli [-] keystone-manage pki_setup is not recommended for production use.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/ssl/certs/ca.pem already exists
/etc/keystone/ssl/private/signing_key.pem already exists
/etc/keystone/ssl/certs/signing_cert.pem already exists
Connection to 192.0.2.8 closed.
Authorization Failed: Unable to establish connection to http://10.35.180.10:5000/v2.0/tokens
[stack@puma33 ~]$ heat stack-list
+--------------------------------------+------------+-----------------+---------------------+--------------+
| id                                   | stack_name | stack_status    | creation_time       | updated_time |
+--------------------------------------+------------+-----------------+---------------------+--------------+
| 8bc81aab-75d6-474d-962d-87daa3bec1d2 | overcloud  | CREATE_COMPLETE | 2016-02-15T12:02:15 | None         |
+--------------------------------------+------------+-----------------+---------------------+--------------+

[root@overcloud-controller-0 ~]# openstack-status
== Nova services ==
openstack-nova-api:                     active    (disabled on boot)
openstack-nova-cert:                    inactive  (disabled on boot)
openstack-nova-compute:                 inactive  (disabled on boot)
openstack-nova-network:                 inactive  (disabled on boot)
openstack-nova-scheduler:               active    (disabled on boot)
openstack-nova-conductor:               active    (disabled on boot)
== Glance services ==
openstack-glance-api:                   active    (disabled on boot)
openstack-glance-registry:              active    (disabled on boot)
== Keystone service ==
openstack-keystone:                     active    (disabled on boot)
== Horizon service ==
openstack-dashboard:                    uncontactable
== neutron services ==
neutron-server:                         active    (disabled on boot)
neutron-dhcp-agent:                     active    (disabled on boot)
neutron-l3-agent:                       active    (disabled on boot)
neutron-metadata-agent:                 active    (disabled on boot)
neutron-lbaas-agent:                    inactive  (disabled on boot)
neutron-openvswitch-agent:              active    (disabled on boot)
neutron-metering-agent:                 inactive  (disabled on boot)
== Swift services ==
openstack-swift-proxy:                  active
openstack-swift-account:                active
openstack-swift-container:              active
openstack-swift-object:                 active
== Cinder services ==
openstack-cinder-api:                   active    (disabled on boot)
openstack-cinder-scheduler:             active    (disabled on boot)
openstack-cinder-volume:                active    (disabled on boot)
openstack-cinder-backup:                inactive  (disabled on boot)
== Ceilometer services ==
openstack-ceilometer-api:               active    (disabled on boot)
openstack-ceilometer-central:           active    (disabled on boot)
openstack-ceilometer-compute:           inactive  (disabled on boot)
openstack-ceilometer-collector:         active    (disabled on boot)
openstack-ceilometer-alarm-notifier:    active    (disabled on boot)
openstack-ceilometer-alarm-evaluator:   active    (disabled on boot)
openstack-ceilometer-notification:      active    (disabled on boot)
== Heat services ==
openstack-heat-api:                     active    (disabled on boot)
openstack-heat-api-cfn:                 active    (disabled on boot)
openstack-heat-api-cloudwatch:          active    (disabled on boot)
openstack-heat-engine:                  active    (disabled on boot)
== Support services ==
mysqld:                                 inactive  (disabled on boot)
libvirtd:                               active
openvswitch:                            active
dbus:                                   active
rabbitmq-server:                        inactive  (disabled on boot)
memcached:                              active    (disabled on boot)
== Keystone users ==
Warning keystonerc not sourced
[root@overcloud-controller-0 ~]# 


Version-Release number of selected component (if applicable):
director 8 puddle - 2016-02-11.1

How reproducible:
3/3

Steps to Reproduce:
setup: 3-controllers, 3-computes, external-ceph
Deploy command:
openstack overcloud deploy -e /home/stack/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e ceph-external.yaml --templates --control-scale 3 --compute-scale 3 --ntp-server clock.redhat.com --timeout 180
Comment 1 Asaf Hirshberg 2016-02-15 13:21:59 UTC 
Created attachment 1127274 [details]
controller-1 keystone log
Comment 2 Asaf Hirshberg 2016-02-15 13:22:23 UTC 
Created attachment 1127275 [details]
controller-2 keystone log
Comment 3 Marius Cornea 2016-02-15 13:35:47 UTC 
Hi Asaf, 

Are you able to ping 10.35.180.10 from the undercloud node? In case not this indicates a network connectivity issue and thus the error message:

Authorization Failed: Unable to establish connection to http://10.35.180.10:5000/v2.0/tokens
Comment 5 Asaf Hirshberg 2016-02-16 13:13:17 UTC 
Closed.

Thanks Marius, you were right, the problem have caused by asymetric routing problem; disabled by default in rhel.
configuring enp5s0f0 in the controller nic template and disabling dhcp solved the problem.