Bug 1392054 - Overcloud deployed without Ceph nodes results in failure due to missing data item ceph::profile::params::client_keys
Summary: Overcloud deployed without Ceph nodes results in failure due to missing data ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-tripleoclient
Version: 10.0 (Newton)
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: 10.0 (Newton)
Assignee: leseb
QA Contact: Yogev Rabl
Derek
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-04 16:39 UTC by Kevin Jones
Modified: 2016-12-14 16:29 UTC (History)
13 users (show)

Fixed In Version: python-tripleoclient-5.4.0-1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-14 16:29:42 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 396524 None None None 2016-11-12 06:20:27 UTC
Launchpad 1639787 None None None 2016-11-07 17:29:54 UTC

Description Kevin Jones 2016-11-04 16:39:57 UTC
Description of problem:

Attempted RHOSP10 deployment using latest puddle where I specified 0 ceph-storage nodes and no storage-environment file in the deploy command. The deployment fails in overcloud.AllNodesDeploySteps.ComputeDeployment_Step4.0.

Willing to accept this may be a misunderstanding of what happens when no storage-environment file is included in deploy command and setting the number of ceph storage nodes to 0.

openstack overcloud deploy --templates \
        -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
        -e /home/stack/templates/network-environment.yaml \
        -e /home/stack/templates/firstboot-environment.yaml \
        --control-flavor control \
        --compute-flavor compute \
        --ceph-storage-flavor ceph-storage \
        --control-scale 1 \
        --compute-scale 1 \
        --ceph-storage-scale 0 \
        --ntp-server clock.redhat.corp \
        --neutron-tunnel-types vxlan \
        --neutron-network-type vxlan


Version-Release number of selected component (if applicable):
Loaded plugins: search-disabled-repos
Installed Packages
Name        : python-tripleoclient
Arch        : noarch
Version     : 5.3.0
Release     : 4.el7ost
Size        : 878 k
Repo        : installed
From repo   : rhelosp-10.0-puddle
Summary     : OpenstackClient plugin for tripleoclient
URL         : https://pypi.python.org/pypi/python-tripleoclient
License     : ASL 2.0
Description : python-tripleoclient is a Python plugin to OpenstackClient
            : for TripleO <https://github.com/openstack/python-tripleoclient>.

Installed Packages
Name        : openstack-heat-templates
Arch        : noarch
Version     : 0.0.1
Release     : 0.20160906185549.ac2db55.el7ost
Size        : 166 k
Repo        : installed
From repo   : local-rhel-7-server-openstack-beta-rpms
Summary     : Heat software config templates and DIB elements
URL         : https://github.com/openstack/heat-templates
License     : ASL 2.0
Description : Heat software config templates and image building elements

Name        : openstack-tripleo-heat-templates
Arch        : noarch
Version     : 5.0.0
Release     : 1.2.el7ost
Size        : 1.3 M
Repo        : installed
From repo   : rhelosp-10.0-puddle
Summary     : Heat templates for TripleO
URL         : https://wiki.openstack.org/wiki/TripleO
License     : ASL 2.0
Description : OpenStack TripleO Heat Templates is a collection of templates and tools for
            : building Heat Templates to do deployments of OpenStack.


How reproducible:
100%

Steps to Reproduce:
1. Install undercloud
2. Import nodes
3. Run introspection
4. Tag nodes and flavors
5. Issue overcloud deploy with 1 controller, 1 compute, 0 ceph and no storage-environment file

Actual results:
Overcloud deployment fails in step 4 because of the missing hieradata data item for Ceph.

Expected results:
Overcloud completes with Cinder, Glance and Nova backed by local storage or swift.

Additional info:
Failed deploy 1 controller 1 compute 0 Ceph

Failed on step 4

overcloud.AllNodesDeploySteps.ComputeDeployment_Step4.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 3eed9d1d-be53-41e0-8ab9-d9b537a1d400
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    Matching apachectl 'Server version: Apache/2.4.6 (Red Hat Enterprise Linux)
    Server built:   Aug  3 2016 08:33:27'
  deploy_stderr: |
    ...
    Warning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::cpu_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated
    Warning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::ram_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated
    Warning: Scope(Class[Nova]): Could not look up qualified variable '::nova::scheduler::filter::disk_allocation_ratio'; class ::nova::scheduler::filter has not been evaluated
    Warning: Scope(Class[Nova::Compute]): compute_manager is marked as deprecated in Nova but still needed when Ironic is used. It will be removed once Nova removes it.
    Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::host'; class ::nova::vncproxy has not been evaluated
    Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::vncproxy_protocol'; class ::nova::vncproxy has not been evaluated
    Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::port'; class ::nova::vncproxy has not been evaluated
    Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::vncproxy_path'; class ::nova::vncproxy has not been evaluated
    Error: Could not find data item ceph::profile::params::client_keys in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/nova/compute/libvirt.pp:36 on node overcloud-compute-0.localdomain
    Error: Could not find data item ceph::profile::params::client_keys in any Hiera data file and no default supplied at /etc/puppet/modules/tripleo/manifests/profile/base/nova/compute/libvirt.pp:36 on node overcloud-compute-0.localdomain

Comment 1 James Slagle 2016-11-07 14:26:25 UTC
afaict, the puppet code using ceph::profile::params::client_keys should only be triggered if NovaEnableRbdBackend or CinderEnableRbdBackend are true.

do you have those set in any of your environment files?

Comment 2 Giulio Fidente 2016-11-07 14:36:00 UTC
(In reply to James Slagle from comment #1)
> afaict, the puppet code using ceph::profile::params::client_keys should only
> be triggered if NovaEnableRbdBackend or CinderEnableRbdBackend are true.
> 
> do you have those set in any of your environment files?

also, if not, can you try to delete the pre-existing plan in swift (if any) with:

  # openstack overcloud plan delete overcloud

to make sure the plan in swift does not contain stale params from previous deployments?

Comment 3 Kevin Jones 2016-11-07 15:02:47 UTC
In the failed deployment, I did not have NovaEnabledRbdBackend or CinderEnabledRbdBackend in any of my environment files.

However, to Giulio's point, I had previously tried to deploy with those set. So it is possible it was hang over from a previously failed Ceph deployment.

It does appear that the first step in a deployment though (at least what gets output) is that the old plan files are removed and new plan files are uploaded.

Let me see if I can reproduce.

Comment 4 Giulio Fidente 2016-11-07 17:20:58 UTC
hi Kevin, indeed this seems to be an issue in the client which is not refreshing the user environment in Swift.

I've reproduced the error by deploying with storage-environment.yaml first, then deleting the stack and redeploying without. The second stack got some of the parameter from storage-environment.yaml (including NovaEnableRbdBackend and CinderEnableRbdBackend) which instead were not passed anymore.

The command in comment #2 should clean the environment. Moving to puppet-tripleoclient, adding Dougal on CC as he might help us with this.

Comment 5 Jon Schlueter 2016-11-12 06:20:28 UTC
backport proposed to stable/newton https://review.openstack.org/#/c/396524

Comment 7 Yogev Rabl 2016-11-24 14:39:24 UTC
verified on python-tripleoclient-5.4.0-1.el7ost

Comment 9 errata-xmlrpc 2016-12-14 16:29:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.