Bug 1566520 - Response from stack update is "504 Gateway Timeout"
Summary: Response from stack update is "504 Gateway Timeout"
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-heat
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: ---
Assignee: Zane Bitter
QA Contact: Ronnie Rasouli
Depends On:
TreeView+ depends on / blocked
Reported: 2018-04-12 13:00 UTC by Dariusz Wojewódzki
Modified: 2021-06-10 15:47 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2018-05-29 10:02:27 UTC
Target Upstream Version:

Attachments (Terms of Use)
rhbz1566520.yaml (179 bytes, text/x-vhdl)
2018-05-18 16:06 UTC, Alex Schultz
no flags Details

Comment 2 Zane Bitter 2018-04-12 13:54:28 UTC
Yes, this sounds like the same issue as bug 1394920. 504 is not something that Heat ever returns. It's something that HAProxy returns if it doesn't get a reply back before its timeout. So the first thing to check is the HAProxy timeout - heat-api's timeout when passing requests to the engine is 600s on the undercloud, so it makes sense to set the HAProxy timeout to 600s as well.

Comment 4 Zane Bitter 2018-04-12 16:27:42 UTC
The puppet setting for this is called haproxy_default_timeout. The default value is:

  [ 'http-request 10s', 'queue 2m', 'connect 10s', 'client 2m', 'server 2m', 'check 10s' ]

I'd advise changing the 2m to 10m across the board:

  [ 'http-request 10s', 'queue 10m', 'connect 10s', 'client 10m', 'server 10m', 'check 10s' ]

IIUC you want to change this in the undercloud hieradata and then re-run puppet, otherwise it is liable to get overwritten in the future.

Comment 5 Zane Bitter 2018-04-12 16:38:31 UTC
With the default configuration on OSP10, you'll only be using 4 heat-engine worker (number of CPUs / 2). If you haven't already, you probably also want to bump this to 8 so that you're using all of your CPU capacity for cpu-bound operations. (This is the default in every release except OSP10.)

The bug that most closely resembles this is actually bug 1393802.

FWIW, the validation performance is considerably improved in OSP11 and later.

Comment 6 Thomas Hervé 2018-04-19 13:56:47 UTC
Did you push the new timeout values successfully? Thanks.

Comment 9 Alex Schultz 2018-05-18 16:06:57 UTC
Created attachment 1438735 [details]

So you don't need to update the puppet-stack-config.yaml. You can use the hieradata_override option in the undercloud.conf.  An example of this setting is can be found on the APPENDIX G. SECURITY ENHANCEMENTS of the OSP10 documentation[0].

I have attached an example of the contents of the hieradata override file that can be used.  Simply update the undercloud with "hieradata_override = rhbz1566520.yaml" and run an undercloud update. This should apply the override values.  Feel free to rename this file if this naming convention does not work for you.

[0] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html/director_installation_and_usage/appe-security_enhancements

Note You need to log in before you can comment on or make changes to this bug.