Bug 1251267

Summary: Overcloud deployment fails for unspecified reason
Product: [Community] RDO Reporter: Steve Linabery <slinaber>
Component: rdo-managerAssignee: Dougal Matthews <dmatthew>
Status: CLOSED EOL QA Contact: Shai Revivo <srevivo>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: trunkCC: apetrich, dmatthew, mburns
Target Milestone: ---   
Target Release: Kilo   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-19 16:02:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Full deployment log none

Description Steve Linabery 2015-08-06 20:56:15 UTC 
Description of problem:
The delorean rdo_management promote job is failing
https://ci.centos.org/view/rdo/job/rdo_manager-promote-build-delorean-rdo_management-kilo/209/
Unfortunately the overcloud deployment log file is not collected successfully, so there is no useful output available from the deploy command.

Version-Release number of selected component (if applicable):


How reproducible:
Failing for last several runs of the jenkins job linked above.

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Adriano Petrich 2015-08-10 06:48:15 UTC 
After getting more data the deployment is failing with

BadRequest: There are no parameters named Controller-1::ManilaPassword

with this traceback

[2015-08-10 05:03:40,337] DEBUG    requests.packages.urllib3.connectionpool "GET /v1/4cb76e548e83443ea34b9d1056627950/stacks/overcloud HTTP/1.1" 404 585
[2015-08-10 05:03:40,338] DEBUG    heatclient.common.http 
HTTP/1.1 404 Not Found
date: Mon, 10 Aug 2015 05:03:40 GMT
connection: keep-alive
content-type: application/json; charset=UTF-8
content-length: 585
x-openstack-request-id: req-21ddbb61-e1d6-4302-9a87-0e93d5480923

{"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Stack (overcloud) could not be found.", "traceback": "Traceback (most recent call last):\n\n  File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 300, in wrapped\n    return func(self, ctx, *args, **kwargs)\n\n  File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 422, in identify_stack\n    raise exception.StackNotFound(stack_name=stack_name)\n\nStackNotFound: The Stack (overcloud) could not be found.\n", "type": "StackNotFound"}, "title": "Not Found"}

[2015-08-10 05:03:40,338] INFO     rdomanager_oscplugin.v1.overcloud_deploy.DeployOvercloud No stack found, will be doing a stack create
[2015-08-10 05:03:40,339] DEBUG    openstackclient.compute.client Instantiating compute client: <class 'novaclient.v2.client.Client'>
[2015-08-10 05:03:40,339] DEBUG    rdomanager_oscplugin.v1.overcloud_deploy.DeployOvercloud Checking hypervisor stats
[2015-08-10 05:03:40,339] DEBUG    keystoneclient.session REQ: curl -g -i -X GET http://192.0.2.1:8774/v2/4cb76e548e83443ea34b9d1056627950/os-hypervisors/statistics -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}317fffe90ed2cdbbaa7aca8d65175f4e0ecd4292"
[2015-08-10 05:03:40,340] INFO     requests.packages.urllib3.connectionpool Starting new HTTP connection (1): 192.0.2.1
[2015-08-10 05:03:40,381] DEBUG    requests.packages.urllib3.connectionpool "GET /v2/4cb76e548e83443ea34b9d1056627950/os-hypervisors/statistics HTTP/1.1" 200 259
[2015-08-10 05:03:40,383] DEBUG    keystoneclient.session RESP: [200] date: Mon, 10 Aug 2015 05:03:40 GMT connection: keep-alive content-type: application/json content-length: 259 x-compute-request-id: req-be6a5f08-084b-400b-a10b-28181e5685fa 
RESP BODY: {"hypervisor_statistics": {"count": 3, "vcpus_used": 0, "local_gb_used": 0, "memory_mb": 12288, "current_workload": 0, "vcpus": 3, "running_vms": 0, "free_disk_gb": 120, "disk_available_least": 120, "local_gb": 120, "free_ram_mb": 12288, "memory_mb_used": 0}}

[2015-08-10 05:03:40,405] DEBUG    openstackclient.network.client Instantiating network client: <class 'neutronclient.v2_0.client.Client'>
[2015-08-10 05:03:40,407] DEBUG    openstackclient.network.client Instantiating network api: <class 'neutronclient.v2_0.client.Client'>
[2015-08-10 05:03:44,382] DEBUG    rdomanager_oscplugin.v1.overcloud_deploy.DeployOvercloud Generating overcloud passwords
[2015-08-10 05:03:44,386] DEBUG    rdomanager_oscplugin.v1.overcloud_deploy.DeployOvercloud Getting ctlplane from Neutron
[2015-08-10 05:03:44,386] DEBUG    keystoneclient.session REQ: curl -g -i -X GET http://192.0.2.1:9696//v2.0/networks -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: {SHA1}317fffe90ed2cdbbaa7aca8d65175f4e0ecd4292"
[2015-08-10 05:03:44,465] DEBUG    keystoneclient.session RESP: [200] date: Mon, 10 Aug 2015 05:03:44 GMT connection: keep-alive content-type: application/json; charset=UTF-8 content-length: 389 x-openstack-request-id: req-911ebeec-ac05-4b73-a957-2b7f4159cc44 
RESP BODY: {"networks": [{"status": "ACTIVE", "subnets": ["0996a30e-6219-4b41-973c-30026f25e2f8"], "name": "ctlplane", "provider:physical_network": "ctlplane", "router:external": false, "tenant_id": "4cb76e548e83443ea34b9d1056627950", "admin_state_up": true, "mtu": 0, "shared": false, "provider:network_type": "flat", "id": "921e70de-2c41-4a1d-8fff-777ff0fdc2e0", "provider:segmentation_id": null}]}

[2015-08-10 05:03:44,627] DEBUG    os_cloud_config.keystone_pki Generated CA key.
[2015-08-10 05:03:44,629] DEBUG    os_cloud_config.keystone_pki Generated CA certificate.
[2015-08-10 05:03:44,697] DEBUG    os_cloud_config.keystone_pki Generated signing key.
[2015-08-10 05:03:44,699] DEBUG    os_cloud_config.keystone_pki Generated signing certificate.
[2015-08-10 05:03:44,808] DEBUG    tuskarclient.openstack.common.apiclient.client Request returned failure status: 400
[2015-08-10 05:03:44,808] ERROR    openstack There are no parameters named Controller-1::ManilaPassword in plan 3a948883-9af6-4fcc-879f-ea431cb19f13. (HTTP 400)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 817, in take_action
    self._deploy_tuskar(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 536, in _deploy_tuskar
    for x in parameters.items()]
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans.py", line 113, in patch
    parameter_list)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 213, in _patch
    body = self.client.patch(url, json=json).json()
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 367, in patch
    return self.client_request("PATCH", url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 201, in request
    raise exceptions.from_response(resp, method, url)
BadRequest: There are no parameters named Controller-1::ManilaPassword in plan 3a948883-9af6-4fcc-879f-ea431cb19f13. (HTTP 400)
[2015-08-10 05:03:44,814] DEBUG    openstackclient.shell clean_up DeployOvercloud
[2015-08-10 05:03:44,814] DEBUG    openstackclient.shell got an error: There are no parameters named Controller-1::ManilaPassword in plan 3a948883-9af6-4fcc-879f-ea431cb19f13. (HTTP 400)
[2015-08-10 05:03:44,816] ERROR    openstackclient.shell Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 176, in run
    return super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 230, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 817, in take_action
    self._deploy_tuskar(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 536, in _deploy_tuskar
    for x in parameters.items()]
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans.py", line 113, in patch
    parameter_list)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 213, in _patch
    body = self.client.patch(url, json=json).json()
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 367, in patch
    return self.client_request("PATCH", url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 201, in request
    raise exceptions.from_response(resp, method, url)
BadRequest: There are no parameters named Controller-1::ManilaPassword in plan 3a948883-9af6-4fcc-879f-ea431cb19f13. (HTTP 400)
Comment 2 Adriano Petrich 2015-08-10 06:49:03 UTC 
Created attachment 1060955 [details]
Full deployment log
Comment 3 Adriano Petrich 2015-08-10 06:49:57 UTC 
also the full run with more debug is in https://ci.centos.org/view/rdo/job/rdo_manager-promote-build-delorean-rdo_management-kilo/225/consoleFull
Comment 4 Dougal Matthews 2015-08-12 10:07:25 UTC 
The Manilla change addition has been reverted from the midstream repo. https://review.gerrithub.io/#/c/242798/
Comment 5 Mike McCune 2016-03-28 22:30:10 UTC 
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
Comment 7 Chandan Kumar 2016-05-19 16:02:56 UTC 
This bug is against a Version which has reached End of Life.
If it's still present in supported release (http://releases.openstack.org), please update Version and reopen.