Bug 1232498 - Overcloud deploy and post deploy steps fail - some showing auth failures
Summary: Overcloud deploy and post deploy steps fail - some showing auth failures
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-rdomanager-oscplugin
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ga
: Director
Assignee: Brad P. Crochet
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-16 21:52 UTC by Ronelle Landy
Modified: 2015-08-05 13:54 UTC (History)
11 users (show)

Fixed In Version: python-rdomanager-oscplugin-0.0.8-4.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-08-05 13:54:13 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2015:1549 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform director Release 2015-08-05 17:49:10 UTC

Description Ronelle Landy 2015-06-16 21:52:41 UTC
Description of problem:

After the overcloud is deployed (possibly not completely) running any heat command returns authorization or other failures as in:

failed: [undercloud] => {"changed": true, "cmd": "source /home/stack/stackrc; heat output-show overcloud KeystoneURL | grep -E -o \"([0-9]{1,3}[\\.]){3}[0-9]{1,3}\";", "delta": "0:00:01.001769", "end": "2015-06-16 14:08:20.375668", "rc": 1, "start": "2015-06-16 14:08:19.373899", "warnings": []}
14:08:19 stderr: Unable to establish connection to http://192.0.2.6:8004/v1/%(tenant_id)s/stacks/overcloud

failed: [undercloud] => {"changed": true, "cmd": "source /home/stack/stackrc; heat output-show overcloud KeystoneURL | grep -E -o \"([0-9]{1,3}[\\.]){3}[0-9]{1,3}\";", "delta": "0:00:01.001769", "end": "2015-06-16 14:08:20.375668", "rc": 1, "start": "2015-06-16 14:08:19.373899", "warnings": []}
14:08:19 stderr: Unable to establish connection to http://192.0.2.6:8004/v1/%(tenant_id)s/stacks/overcloud

Possibly the overcloud deploy never completes correctly:

1:25:40 source /home/stack/stackrc; if [ -f "/home/stack/deploy-overcloudrc" ]; then
11:25:40  source /home/stack/deploy-overcloudrc;
11:25:40  fi; openstack overcloud deploy --plan-uuid 5b5fdea0-f001-4b8b-ae7b-d94615dc2b03;
11:25:40 
11:25:40 start:
11:25:40 2015-06-16 11:13:40.746207
11:25:40 
11:25:40 end:
11:25:40 2015-06-16 11:25:41.433885
11:25:40 
11:25:40 delta:
11:25:40 0:12:00.687678
11:25:40 
11:25:40 stdout:
11:25:40 The following cert files already exist, use --rebuild to remove the existing files before regenerating:
11:25:40 /etc/keystone/ssl/certs/ca.pem already exists
11:25:40 /etc/keystone/ssl/private/signing_key.pem already exists
11:25:40 /etc/keystone/ssl/certs/signing_cert.pem already exists
11:25:40 The following templates will be written:
11:25:40 /tmp/tmpHIH6Gw/puppet/manifests/overcloud_volume.pp
11:25:40 /tmp/tmpHIH6Gw/hieradata/object.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/hieradata/common.yaml
11:25:40 /tmp/tmpHIH6Gw/provider-Swift-Storage-1.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/controller-config-pacemaker.yaml
11:25:40 /tmp/tmpHIH6Gw/provider-Cinder-Storage-1.yaml
11:25:40 /tmp/tmpHIH6Gw/provider-Compute-1.yaml
11:25:40 /tmp/tmpHIH6Gw/network/noop.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/bootstrap-config.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/cinder-storage-post.yaml
11:25:40 /tmp/tmpHIH6Gw/provider-Ceph-Storage-1.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/controller-post-puppet.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/cinder-storage-puppet.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/manifests/overcloud_cephstorage.pp
11:25:40 /tmp/tmpHIH6Gw/puppet/hieradata/object.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/controller-puppet.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/manifests/overcloud_compute.pp
11:25:40 /tmp/tmpHIH6Gw/net-config-bridge.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/swift-storage-post.yaml
11:25:40 /tmp/tmpHIH6Gw/provider-Controller-1.yaml
11:25:40 /tmp/tmpHIH6Gw/network/networks.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/manifests/overcloud_object.pp
11:25:40 /tmp/tmpHIH6Gw/hieradata/controller.yaml
11:25:40 /tmp/tmpHIH6Gw/network/ports/ctlplane_vip.yaml
11:25:40 /tmp/tmpHIH6Gw/hieradata/volume.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/compute-post-puppet.yaml
11:25:40 /tmp/tmpHIH6Gw/extraconfig/tasks/yum_update.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/swift-storage-puppet.yaml
11:25:40 /tmp/tmpHIH6Gw/extraconfig/tasks/yum_update.sh
11:25:40 /tmp/tmpHIH6Gw/puppet/swift-devices-and-proxy-config.yaml
11:25:40 /tmp/tmpHIH6Gw/network/ports/net_ip_map.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/compute-puppet.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/hieradata/volume.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/ceph-storage-post-puppet.yaml
11:25:40 /tmp/tmpHIH6Gw/extraconfig/controller/noop.yaml
11:25:40 /tmp/tmpHIH6Gw/network/ports/noop.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/ceph-storage-puppet.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/hieradata/ceph.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/vip-config.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/hieradata/controller.yaml
11:25:40 /tmp/tmpHIH6Gw/plan.yaml
11:25:40 /tmp/tmpHIH6Gw/environment.yaml
11:25:40 /tmp/tmpHIH6Gw/network/ports/net_ip_list_map.yaml
11:25:40 /tmp/tmpHIH6Gw/hieradata/compute.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/hieradata/compute.yaml
11:25:40 /tmp/tmpHIH6Gw/hieradata/ceph.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/manifests/overcloud_controller_pacemaker.pp
11:25:40 /tmp/tmpHIH6Gw/hieradata/common.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/manifests/ringbuilder.pp
11:25:40 /tmp/tmpHIH6Gw/extraconfig/post_deploy/default.yaml
11:25:40 /tmp/tmpHIH6Gw/net-config-noop.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/ceph-cluster-config.yaml
11:25:40 /tmp/tmpHIH6Gw/firstboot/userdata_default.yaml
11:25:40 /tmp/tmpHIH6Gw/puppet/all-nodes-config.yaml
11:25:40 PKI initialization in init-keystone is deprecated and will be removed.
11:25:40 
11:25:40 stderr:
11:25:40 /usr/lib/python2.7/site-packages/novaclient/v1_1/__init__.py:30: UserWarning: Module novaclient.v1_1 is deprecated (taken as a basis for novaclient.v2). The preferable way to get client class or object you can find in novaclient.client module.
11:25:40   warnings.warn("Module novaclient.v1_1 is deprecated (taken as a basis for "
11:25:40 Warning: Permanently added '192.0.2.6' (ECDSA) to the list of known hosts.
11:25:40 Connection to 192.0.2.6 closed.
11:25:40 

(note no overcloud deployed message)


Version-Release number of selected component (if applicable):

Poodle rpms:

[stack@instack ~]$ rpm -qa | grep openstack
openstack-nova-console-2015.1.0-11.el7ost.noarch
openstack-neutron-2015.1.0-8.el7ost.noarch
openstack-ironic-conductor-2015.1.0-4.el7ost.noarch
openstack-ceilometer-alarm-2015.1.0-2.el7ost.noarch
openstack-swift-account-2.3.0-1.el7ost.noarch
python-django-openstack-auth-1.2.0-2.el7ost.noarch
openstack-tuskar-ui-0.3.0-2.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.0-3.el7ost.noarch
openstack-ceilometer-notification-2015.1.0-2.el7ost.noarch
openstack-neutron-openvswitch-2015.1.0-8.el7ost.noarch
openstack-nova-api-2015.1.0-11.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-9.el7ost.noarch
openstack-tripleo-image-elements-0.9.6-1.el7ost.noarch
python-openstackclient-1.0.3-2.el7ost.noarch
openstack-ironic-discoverd-1.1.0-3.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.1-2.el7ost.noarch
openstack-swift-object-2.3.0-1.el7ost.noarch
openstack-tripleo-0.0.6-0.1.git812abe0.el7ost.noarch
openstack-nova-common-2015.1.0-11.el7ost.noarch
openstack-heat-common-2015.1.0-3.el7ost.noarch
openstack-tuskar-0.4.18-2.el7ost.noarch
openstack-dashboard-theme-2015.1.0-10.el7ost.noarch
openstack-tuskar-ui-extras-0.0.4-1.el7ost.noarch
openstack-tempest-kilo-20150507.2.el7ost.noarch
openstack-swift-2.3.0-1.el7ost.noarch
openstack-neutron-ml2-2015.1.0-8.el7ost.noarch
openstack-nova-novncproxy-2015.1.0-11.el7ost.noarch
openstack-keystone-2015.1.0-1.el7ost.noarch
openstack-swift-plugin-swift3-1.7-3.el7ost.noarch
openstack-tripleo-common-0.0.1.dev6-0.git49b57eb.el7ost.noarch
openstack-neutron-common-2015.1.0-8.el7ost.noarch
openstack-heat-engine-2015.1.0-3.el7ost.noarch
openstack-ceilometer-common-2015.1.0-2.el7ost.noarch
openstack-heat-api-cfn-2015.1.0-3.el7ost.noarch
openstack-ceilometer-api-2015.1.0-2.el7ost.noarch
openstack-ironic-api-2015.1.0-4.el7ost.noarch
openstack-swift-proxy-2.3.0-1.el7ost.noarch
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch
openstack-ceilometer-collector-2015.1.0-2.el7ost.noarch
openstack-ironic-common-2015.1.0-4.el7ost.noarch
openstack-selinux-0.6.32-1.el7ost.noarch
openstack-nova-compute-2015.1.0-11.el7ost.noarch
openstack-nova-conductor-2015.1.0-11.el7ost.noarch
openstack-swift-container-2.3.0-1.el7ost.noarch
redhat-access-plugin-openstack-7.0.0-0.el7ost.noarch
openstack-glance-2015.1.0-6.el7ost.noarch
openstack-heat-api-2015.1.0-3.el7ost.noarch
openstack-ceilometer-central-2015.1.0-2.el7ost.noarch
openstack-puppet-modules-2015.1.6-1.el7ost.noarch
openstack-nova-scheduler-2015.1.0-11.el7ost.noarch
openstack-nova-cert-2015.1.0-11.el7ost.noarch
openstack-dashboard-2015.1.0-10.el7ost.noarch

[stack@instack ~]$ rpm -qa | grep python-rdomanager-oscplugin
python-rdomanager-oscplugin-0.0.7-2.el7ost.noarch

How reproducible:
always on poodle jobs

Steps to Reproduce:
1. Install from latest poodle
2. Deploy overcloud - see results, try heat commands afterwards
3. Compare with RDO job results

Actual results:
Failure to deploy and complete post steps

Expected results:

Deploy should pass and post deploy steps should succeed

Additional info:

Comment 3 Alexander Chuzhoy 2015-06-16 22:00:56 UTC
Reproduced.

Comment 4 chris alfonso 2015-06-19 17:08:18 UTC
Brad, will you please check to see if this is still an issue?

Comment 5 Brad P. Crochet 2015-06-22 14:48:12 UTC
python-rdomanager-oscplugin downstream was not fully up to where midstream was. After a rebase, this went away.

Comment 6 Amit Ugol 2015-07-06 10:49:50 UTC
No issues that I can see on virt or on BM

Comment 8 errata-xmlrpc 2015-08-05 13:54:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549


Note You need to log in before you can comment on or make changes to this bug.