Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1959476

Summary: [OSP-13] Unable to tune cinder-api service's WSGI timeout
Product: Red Hat OpenStack Reporter: Alan Bishop <abishop>
Component: puppet-cinderAssignee: Alan Bishop <abishop>
Status: CLOSED ERRATA QA Contact: Tzach Shefi <tshefi>
Severity: high Docs Contact: RHOS Documentation Team <rhos-docs>
Priority: high    
Version: 13.0 (Queens)CC: astillma, gcharot, gkadam, jamsmith, jjoyce, jschluet, jvisser, pcaruana, pgrist, slinaber, spower, tshefi, tvignaud
Target Milestone: asyncKeywords: Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: puppet-cinder-12.4.1-8.el7ost Doc Type: Bug Fix
Doc Text:
In previous releases, the timeout period for the Block Storage service (cinder) API requests defaulted to 60 seconds and the value could not be overridden. Cinder API requests that took longer than 60 seconds would time out, even if the request would have eventually succeeded. + With this release, you can increase the API timeout value to avoid premature timeout requests. + To deploy this fix, increase the two timeout values as shown in the following example. + ---- 1. Install the `puppet-cinder` RPM on the overcloud Controller nodes where the cinder-api service runs. 2. In an environment file, configure the remote procedure call (RPC) timeout and API wsgi timeout value to the desired settings. For example, in `myenvfile.yaml`, enter the following settings. + ---- parameter_defaults: ControllerExtraConfig: cinder::rpc_response_timeout: 120 cinder::wsgi::apache::vhost_custom_fragment: 'Timeout 120' ---- + 3. Include the environment file that contains your new content in the `openstack overcloud deploy` by using the `-e` option. Ensure that you include all other environment files that are relevant to your deployment in the correct order. + ---- $ openstack overcloud deploy \ -e my-env-file.yaml ----
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-02 13:36:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alan Bishop 2021-05-11 15:36:21 UTC
The OSP-13 (queens) version of puppet-cinder does not include a patch that was recently added that adds support for configuring cinder-api's WSGI timeout. That functionality is needed in OSP-13.

Background:

Bug #1947414 describes a set of problems that arise when nova invokes the cinder API and the request times out. Starting with OSP-16, it will be possible to tune cinder's RPC and WSGI timeouts with THT parameters (see bug #1930806), and the suggested workaround for OSP-13 is to directly manipulate the same puppet hiera variables. See bug #1947414 comment #9, which references bug #1930806 comment #4.

Unfortunately, the cinder::wsgi::apache::vhost_custom_fragment variable (which is used to tune the WSGI timeout) was added in a puppet-cinder patch [1] that has been backported only as far as Train

[1] https://review.opendev.org/q/If3f3f230aae7e04a774c967e1786561d70e43e50

That patch needs to be backported to queens for inclusion in OSP-13.

Comment 1 Alan Bishop 2021-05-11 15:40:42 UTC
Upstream backports are under way.

Comment 8 Alan Bishop 2021-06-03 15:34:57 UTC
Exception request email has been sent.

Comment 19 Tzach Shefi 2021-07-12 07:09:39 UTC
Still waiting on a fresh P1 build with fix included. 
Latest I managed to deploy lacks the fix:
13  -p 2021-05-13.1 / puppet-cinder-12.4.1-7.el7ost.noarch

Comment 20 Tzach Shefi 2021-07-14 11:28:09 UTC
On a later build of 13 -p 2021-07-09.9, still lacking fixed-in
[stack@undercloud-0 ~]$ rpm -qa | grep cinder
puppet-cinder-12.4.1-7.el7ost.noarch

Comment 21 Tzach Shefi 2021-07-18 11:13:47 UTC
Tested on:
puppet-cinder-12.4.1-8.el7ost.noarch

Used this yaml:
(undercloud) [stack@undercloud-0 ~]$ cat virt/extra_templates.yaml 
parameter_defaults:
    ControllerExtraConfig:
        cinder::rpc_response_timeout: 120
        cinder::wsgi::apache::vhost_custom_fragment: Timeout 300


[root@controller-0 hieradata]# cat /etc/puppet/hieradata/controller_extraconfig.json
{
    "cinder::rpc_response_timeout": 120,
    "cinder::wsgi::apache::vhost_custom_fragment": "Timeout 300"

First line is simple we can confirm it's set on Cinder.conf

[root@controller-0 hieradata]# grep rpc_res /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf 
#rpc_response_timeout = 60
rpc_response_timeout=120


Alan, 

With the second parameter I'm struggling a bit, 
where/how can I show it was set as expected?

From what I gather, might be wrong, it's a parameter set when we kick off a wsgi process for Cinder api.
So i figured lets restart cinder api and look at api log, but I can't say I found what I was looking for.

I tend to think we're good to verify,
just waiting on your answer so as to confirm second parameter.

Thanks

Comment 22 Alan Bishop 2021-07-20 15:14:29 UTC
The CinderApiWsgiTimeout parameter ends up in [1] on the host and [2] inside the c-api container.

[1] /var/lib/config-data/puppet-generated/cinder/etc/httpd/conf.d/10-cinder_wsgi.conf
[2] /etc/httpd/conf.d/10-cinder_wsgi.conf

Look for a "## Custom fragment" comment in the file, followed by the appropriate "Timeout" value.

Comment 23 Tzach Shefi 2021-07-21 10:54:15 UTC
Thanks Alan, 
Something isn't adding up, maybe syntax error on my yaml?

(undercloud) [stack@undercloud-0 ~]$ cat virt/extra_templates.yaml 
parameter_defaults:
    BarbicanSimpleCryptoGlobalDefault: true  -> ignore, carried over from cloned job. 
    ControllerExtraConfig:
        cinder::rpc_response_timeout: 120
        cinder::wsgi::apache::vhost_custom_fragment: 'Timeout 300'   --> noticed I was missing both apostrophes on first attempt #21.

I then added them as seen above, deleted the overcloud and deployed yet again.
Still it seams as this isn't being set, with or without  apostrophes. 

Maybe I have another syntax error? 
No sign of "## Custom fragment" or it's timeout valve


[root@controller-2 ~]# cat /var/lib/config-data/puppet-generated/cinder/etc/httpd/conf.d/10-cinder_wsgi.conf 
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************

<VirtualHost 172.17.1.79:8776>
  ServerName controller-2.internalapi.redhat.local

  ## Vhost docroot
  DocumentRoot "/var/www/cgi-bin/cinder"

  ## Directories, there should at least be a declaration for /var/www/cgi-bin/cinder

  <Directory "/var/www/cgi-bin/cinder">
    Options -Indexes +FollowSymLinks +MultiViews
    AllowOverride None
    Require all granted
  </Directory>

  ## Logging
  ErrorLog "/var/log/httpd/cinder_wsgi_error.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/cinder_wsgi_access.log" combined 
  SetEnvIf X-Forwarded-Proto https HTTPS=1
  WSGIApplicationGroup %{GLOBAL}
  WSGIDaemonProcess cinder-api display-name=cinder_wsgi group=cinder processes=4 threads=1 user=cinder
  WSGIProcessGroup cinder-api
  WSGIScriptAlias / "/var/www/cgi-bin/cinder/cinder-api"
</VirtualHost>

Comment 25 Alan Bishop 2021-07-21 14:52:57 UTC
Just to repeat what we discussed on IRC, the fix does appear in the undercloud but it's missing from the overcloud nodes. It seems the overcloud image does not contain the FIV.

Comment 26 Tzach Shefi 2021-07-21 19:46:57 UTC
Good to verify, 
After adding latest repo and running yum update,
controller nodes also updated to latest:
puppet-cinder-12.4.1-8.el7ost

Updating the overcloud resulted in the missing second parameter showing up too. 

[root@controller-0 ~]# cat /var/lib/config-data/puppet-generated/cinder/etc/httpd/conf.d/10-cinder_wsgi.conf 
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************

<VirtualHost 172.17.1.103:8776>
  ServerName controller-0.internalapi.redhat.local

  ## Vhost docroot
  DocumentRoot "/var/www/cgi-bin/cinder"

  ## Directories, there should at least be a declaration for /var/www/cgi-bin/cinder

  <Directory "/var/www/cgi-bin/cinder">
    Options -Indexes +FollowSymLinks +MultiViews
    AllowOverride None
    Require all granted
  </Directory>

  ## Logging
  ErrorLog "/var/log/httpd/cinder_wsgi_error.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/cinder_wsgi_access.log" combined 
  SetEnvIf X-Forwarded-Proto https HTTPS=1
  WSGIApplicationGroup %{GLOBAL}
  WSGIDaemonProcess cinder-api display-name=cinder_wsgi group=cinder processes=4 threads=1 user=cinder
  WSGIProcessGroup cinder-api
  WSGIScriptAlias / "/var/www/cgi-bin/cinder/cinder-api"

  ## Custom fragment     
  Timeout 300
</VirtualHost>

Also set inside Cinder api
[root@controller-0 ~]# docker exec -it cinder_api grep Timeout /etc/httpd/conf.d/10-cinder_wsgi.conf
  Timeout 300 

Good to verify.

Comment 28 errata-xmlrpc 2021-08-02 13:36:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 13 Bug Fix and Enhancement Advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2978

Comment 29 Red Hat Bugzilla 2023-09-15 01:34:00 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days