Description of problem: I had a failed deployment and wanted to troubleshoot it. It was impossible to get any info because 'heat resource-list -n 5 overcloud' hangs for 1 minute and then returns: ERROR: <html><body><h1>504 Gateway Time-out</h1> The server didn't respond in time. </body></html> If you don't pass '-n 5' then the operation doesn't time out. In any case, however, you can see the CPU usage of heat-engine jump to 100% when calling resource-list (also without -n 5). How reproducible: 100% Steps to Reproduce: 1. After a failed deployment, call: heat resource-list -n 5 overcloud | egrep -v COMPLETE Actual results: 100% CPU usage and the operation times out after ~1 minute. Expected results: This is not supposed to be an expensive operation and should be very fast.
Unfortunately, it is an expensive operation. What's your deployment like? How many nodes do you have in the overcloud? What's the undercloud sizing? Can you attach logs from the heat-engine? Thanks.
Yeah, this is expensive and not likely to get cheaper any time soon. It's worth looking at whether we need to adjust the timeout on the load balancer though, assuming that's what's timing out.
Created attachment 1219454 [details] heat logs I am attaching the logs. Sorry that it's all the logs and they include other tests I made also. I was trying to deploy 3 controllers and 2 computes on a bare metal setup. I tried to ssl-ize the overcloud and I think the deployment failed because I didn't set the keys and certificates in the template. Can we release with such a basic operation not working?
We can see the 2 failures in the API log when nested_depth is specified. The calls took around 64s to finish, so just after a haproxy timeout I presume. It is slow, I'm not sure we'll get around to fix that now though. The command "openstack stack failures list" is meant to do what you'd like to do I think, maybe that mitigate this issue.
A patch has been proposed upstream to bump the HAProxy timeout to 2 minutes.
So it's *really* not helping that we reduced the number of engine workers to 2. That's going to make everything painfully slow. Fix here: https://review.openstack.org/#/c/399619/
Is there any desire to backport https://review.openstack.org/399619 to stable/newton, or should we just close this bug?
*** Bug 1396391 has been marked as a duplicate of this bug. ***
Apparently we are not going to increase the number of workers back to 4 on OSP 10, so I'm going to close this bug since all of the other patches have been released.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days