Bug 1959476
| Summary: | [OSP-13] Unable to tune cinder-api service's WSGI timeout | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alan Bishop <abishop> |
| Component: | puppet-cinder | Assignee: | Alan Bishop <abishop> |
| Status: | CLOSED ERRATA | QA Contact: | Tzach Shefi <tshefi> |
| Severity: | high | Docs Contact: | RHOS Documentation Team <rhos-docs> |
| Priority: | high | ||
| Version: | 13.0 (Queens) | CC: | astillma, gcharot, gkadam, jamsmith, jjoyce, jschluet, jvisser, pcaruana, pgrist, slinaber, spower, tshefi, tvignaud |
| Target Milestone: | async | Keywords: | Triaged, ZStream |
| Target Release: | 13.0 (Queens) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | puppet-cinder-12.4.1-8.el7ost | Doc Type: | Bug Fix |
| Doc Text: |
In previous releases, the timeout period for the Block Storage service (cinder) API requests defaulted to 60 seconds and the value could not be overridden. Cinder API requests that took longer than 60 seconds would time out, even if the request would have eventually succeeded.
+
With this release, you can increase the API timeout value to avoid premature timeout requests.
+
To deploy this fix, increase the two timeout values as shown in the following example.
+
----
1. Install the `puppet-cinder` RPM on the overcloud Controller nodes where the cinder-api service runs.
2. In an environment file, configure the remote procedure call (RPC) timeout and API wsgi timeout value to the desired settings. For example, in `myenvfile.yaml`, enter the following settings.
+
----
parameter_defaults:
ControllerExtraConfig:
cinder::rpc_response_timeout: 120
cinder::wsgi::apache::vhost_custom_fragment: 'Timeout 120'
----
+
3. Include the environment file that contains your new content in the `openstack overcloud deploy` by using the `-e` option. Ensure that you include all other environment files that are relevant to your deployment in the correct order.
+
----
$ openstack overcloud deploy \
-e my-env-file.yaml
----
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-08-02 13:36:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Alan Bishop
2021-05-11 15:36:21 UTC
Upstream backports are under way. Exception request email has been sent. Still waiting on a fresh P1 build with fix included. Latest I managed to deploy lacks the fix: 13 -p 2021-05-13.1 / puppet-cinder-12.4.1-7.el7ost.noarch On a later build of 13 -p 2021-07-09.9, still lacking fixed-in [stack@undercloud-0 ~]$ rpm -qa | grep cinder puppet-cinder-12.4.1-7.el7ost.noarch Tested on:
puppet-cinder-12.4.1-8.el7ost.noarch
Used this yaml:
(undercloud) [stack@undercloud-0 ~]$ cat virt/extra_templates.yaml
parameter_defaults:
ControllerExtraConfig:
cinder::rpc_response_timeout: 120
cinder::wsgi::apache::vhost_custom_fragment: Timeout 300
[root@controller-0 hieradata]# cat /etc/puppet/hieradata/controller_extraconfig.json
{
"cinder::rpc_response_timeout": 120,
"cinder::wsgi::apache::vhost_custom_fragment": "Timeout 300"
First line is simple we can confirm it's set on Cinder.conf
[root@controller-0 hieradata]# grep rpc_res /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf
#rpc_response_timeout = 60
rpc_response_timeout=120
Alan,
With the second parameter I'm struggling a bit,
where/how can I show it was set as expected?
From what I gather, might be wrong, it's a parameter set when we kick off a wsgi process for Cinder api.
So i figured lets restart cinder api and look at api log, but I can't say I found what I was looking for.
I tend to think we're good to verify,
just waiting on your answer so as to confirm second parameter.
Thanks
The CinderApiWsgiTimeout parameter ends up in [1] on the host and [2] inside the c-api container. [1] /var/lib/config-data/puppet-generated/cinder/etc/httpd/conf.d/10-cinder_wsgi.conf [2] /etc/httpd/conf.d/10-cinder_wsgi.conf Look for a "## Custom fragment" comment in the file, followed by the appropriate "Timeout" value. Thanks Alan,
Something isn't adding up, maybe syntax error on my yaml?
(undercloud) [stack@undercloud-0 ~]$ cat virt/extra_templates.yaml
parameter_defaults:
BarbicanSimpleCryptoGlobalDefault: true -> ignore, carried over from cloned job.
ControllerExtraConfig:
cinder::rpc_response_timeout: 120
cinder::wsgi::apache::vhost_custom_fragment: 'Timeout 300' --> noticed I was missing both apostrophes on first attempt #21.
I then added them as seen above, deleted the overcloud and deployed yet again.
Still it seams as this isn't being set, with or without apostrophes.
Maybe I have another syntax error?
No sign of "## Custom fragment" or it's timeout valve
[root@controller-2 ~]# cat /var/lib/config-data/puppet-generated/cinder/etc/httpd/conf.d/10-cinder_wsgi.conf
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************
<VirtualHost 172.17.1.79:8776>
ServerName controller-2.internalapi.redhat.local
## Vhost docroot
DocumentRoot "/var/www/cgi-bin/cinder"
## Directories, there should at least be a declaration for /var/www/cgi-bin/cinder
<Directory "/var/www/cgi-bin/cinder">
Options -Indexes +FollowSymLinks +MultiViews
AllowOverride None
Require all granted
</Directory>
## Logging
ErrorLog "/var/log/httpd/cinder_wsgi_error.log"
ServerSignature Off
CustomLog "/var/log/httpd/cinder_wsgi_access.log" combined
SetEnvIf X-Forwarded-Proto https HTTPS=1
WSGIApplicationGroup %{GLOBAL}
WSGIDaemonProcess cinder-api display-name=cinder_wsgi group=cinder processes=4 threads=1 user=cinder
WSGIProcessGroup cinder-api
WSGIScriptAlias / "/var/www/cgi-bin/cinder/cinder-api"
</VirtualHost>
Just to repeat what we discussed on IRC, the fix does appear in the undercloud but it's missing from the overcloud nodes. It seems the overcloud image does not contain the FIV. Good to verify,
After adding latest repo and running yum update,
controller nodes also updated to latest:
puppet-cinder-12.4.1-8.el7ost
Updating the overcloud resulted in the missing second parameter showing up too.
[root@controller-0 ~]# cat /var/lib/config-data/puppet-generated/cinder/etc/httpd/conf.d/10-cinder_wsgi.conf
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************
<VirtualHost 172.17.1.103:8776>
ServerName controller-0.internalapi.redhat.local
## Vhost docroot
DocumentRoot "/var/www/cgi-bin/cinder"
## Directories, there should at least be a declaration for /var/www/cgi-bin/cinder
<Directory "/var/www/cgi-bin/cinder">
Options -Indexes +FollowSymLinks +MultiViews
AllowOverride None
Require all granted
</Directory>
## Logging
ErrorLog "/var/log/httpd/cinder_wsgi_error.log"
ServerSignature Off
CustomLog "/var/log/httpd/cinder_wsgi_access.log" combined
SetEnvIf X-Forwarded-Proto https HTTPS=1
WSGIApplicationGroup %{GLOBAL}
WSGIDaemonProcess cinder-api display-name=cinder_wsgi group=cinder processes=4 threads=1 user=cinder
WSGIProcessGroup cinder-api
WSGIScriptAlias / "/var/www/cgi-bin/cinder/cinder-api"
## Custom fragment
Timeout 300
</VirtualHost>
Also set inside Cinder api
[root@controller-0 ~]# docker exec -it cinder_api grep Timeout /etc/httpd/conf.d/10-cinder_wsgi.conf
Timeout 300
Good to verify.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 13 Bug Fix and Enhancement Advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2978 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days |