Bug 1644917

Summary: config-download takes hours to complete
Product: Red Hat OpenStack Reporter: Lars Kellogg-Stedman <lars>
Component: python-tripleoclientAssignee: James Slagle <jslagle>
Status: CLOSED DUPLICATE QA Contact: Gurenko Alex <agurenko>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: aschultz, emacchi, hbrock, jslagle, lars, mburns, sbaker
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-07 15:36:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1644920    
Attachments:
Description Flags
output of config download --debug none

Description Lars Kellogg-Stedman 2018-10-31 22:06:44 UTC 
Description of problem:

Running `openstack overcloud config download --config-dir config` takes hours
to complete.

Version-Release number of selected component (if applicable):

OSP 13

Additional info:

The download seems to proceed normally at first, but then seems to get severaly
bogged down when downloading individual resources in calls like:


GET call to orchestration for http://172.16.0.5:8004/v1/dbf7e0522d59438a936b0ccc0527b44a/stacks/overcloud-AllNodesDeploySteps-zq5zbvsplduz-ComputeDeployedServerExtraConfigPost-jue4bwaxrvay-KeystoneFederationDeployments-2zvdpmmpvioo-0-5kvttgq235s6/5984bc4d-6f7a-40f3-b05f-867eaaa51f48?resolve_outputs=False&resolve_outputs=False used request id req-cf624b16-6b47-4543-ba98-31c8df745006
REQ: curl -g -i -X GET http://172.16.0.5:8004/v1/dbf7e0522d59438a936b0ccc0527b44a/stacks/overcloud/resources?physical_resource_id=aaf39de8-d66b-4db6-bd36-6bc77bba1043&nested_depth=7 -H "User-Agent: python-heatclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}b0501da04cc9b32c884bfbc7ff17ec13c0d2e4d1"
http://172.16.0.5:8004 "GET /v1/dbf7e0522d59438a936b0ccc0527b44a/stacks/overcloud/resources?physical_resource_id=aaf39de8-d66b-4db6-bd36-6bc77bba1043&nested_depth=7 HTTP/1.1" 302 533
RESP: [302] Date: Wed, 31 Oct 2018 17:17:48 GMT Server: Apache x-openstack-request-id: req-fefb4b7a-47b2-4874-b1ad-dec5b8538fab Location: http://172.16.0.5:8004/v1/dbf7e0522d59438a936b0ccc0527b44a/stacks/overcloud/4052acc8-958d-407a-b1f2-ade1e9bd328a/resources?physical_resource_id=aaf39de8-d66b-4db6-bd36-6bc77bba1043&nested_depth=7 Content-Length: 533 Keep-Alive: timeout=15, max=62 Connection: Keep-Alive Content-Type: application/json
RESP BODY: {"message": "The resource was found at <a href=\"http://172.16.0.5:8004/v1/dbf7e0522d59438a936b0ccc0527b44a/stacks/overcloud/4052acc8-958d-407a-b1f2-ade1e9bd328a/resources?physical_resource_id=aaf39de8-d66b-4db6-bd36-6bc77bba1043&nested_depth=7\">http://172.16.0.5:8004/v1/dbf7e0522d59438a936b0ccc0527b44a/stacks/overcloud/4052acc8-958d-407a-b1f2-ade1e9bd328a/resources?physical_resource_id=aaf39de8-d66b-4db6-bd36-6bc77bba1043&nested_depth=7</a>;\nyou should be redirected automatically.\n\n", "code": "302 Found", "title": "Found"}
Comment 1 James Slagle 2018-10-31 23:54:04 UTC 
Is that captured output from when you ran the command with --debug? Either way, can you try the command with --debug, save the output and attach it for us to examine?

Even a partial log would be helpful. It shouldn't take more than a few minutes, so what you've captured after 5 or so minutes would probably be helpful for us to see what is happening.

How many nodes are in this deployment?
Comment 2 Lars Kellogg-Stedman 2018-11-01 13:04:13 UTC 
The was --debug output.  I'll re-run the download and attach the log to this bz shortly. There are 3 controllers, 2 network hosts, and 15 compute nodes in this deployment.
Comment 3 James Slagle 2018-11-01 13:31:17 UTC 
Let us know when the log is available.
Comment 4 Lars Kellogg-Stedman 2018-11-01 16:12:27 UTC 
Created attachment 1500081 [details]
output of config download --debug

This is the log from running `openstack --debug overcloud config download --config-dir config` and waiting about 10 minutes.
Comment 5 James Slagle 2018-11-01 21:26:58 UTC 
can you add a name property to each of the OS::Heat::SoftwareDeploymentGroup resources in your templates/postconfig.yaml?

I think there is a performance issue with trying to fetch the name dynamically if it's not set. The name can just be the name of the resource.

So as an example, it would look like:

  KeystoneFederationDeployments:
    type: OS::Heat::SoftwareDeploymentGroup
    depends_on: CreateCloudYamlDeployments
    properties:
      name: KeystoneFederationDeployments  # <------- new line
      servers: {get_param: servers}

That needs to be done for each OS::Heat::SoftwareDeploymentGroup in your custom template.
Comment 6 Lars Kellogg-Stedman 2018-11-06 14:36:33 UTC 
I removed most of the SoftwareDeploymentGroups from the configuration, and added a name to the remaining one [1].  This seems to have fixed things up: just now, running 'config download' completed in under two minutes.

[1]: https://github.com/CCI-MOC/rhosp-director-config/blob/master/templates/postconfig.yaml
Comment 7 Alex Schultz 2019-08-07 15:36:59 UTC 
We've improved the config download call to do something a bit more optimal if the name is not configured on a SoftwareDeploymentGroup. It's still a good idea to use a name on the SoftwareDeploymentGroup as that will use the best path in the code.

*** This bug has been marked as a duplicate of bug 1738344 ***