1394155 – RPCTimeout: Timeout while requesting capabilities from backend hostgroup.

Bug 1394155 - RPCTimeout: Timeout while requesting capabilities from backend hostgroup.

Summary: RPCTimeout: Timeout while requesting capabilities from backend hostgroup.

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-cinder
Sub Component:
Version:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Priority:	low
Severity:	low
Target Milestone:	rc
Target Release:	12.0 (Pike)
Assignee:	Eric Harney
QA Contact:	Avi Avraham
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-11-11 08:50 UTC by Attila Fazekas
Modified:	2018-02-05 19:02 UTC (History)
CC List:	7 users (show)
Fixed In Version:	openstack-cinder-11.0.0-2.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-12-13 20:49:23 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
OpenStack gerrit	460712	0	'None'	ABANDONED	Make get_capabilities look for clusters	2020-11-06 09:13:12 UTC
Red Hat Product Errata	RHEA-2017:3462	0	normal	SHIPPED_LIVE	Red Hat OpenStack Platform 12.0 Enhancement Advisory	2018-02-16 01:43:25 UTC

Description Attila Fazekas 2016-11-11 08:50:35 UTC

Description of problem:
# cinder service-list
+------------------+------------------------+------+---------+-------+----------------------------+-----------------+
| Binary           | Host                   | Zone | Status  | State | Updated_at                 | Disabled Reason |
+------------------+------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup    | hostgroup              | nova | enabled | up    | 2016-11-11T08:26:34.000000 | -               |
| cinder-scheduler | hostgroup              | nova | enabled | up    | 2016-11-11T08:26:35.000000 | -               |
| cinder-volume    | hostgroup@tripleo_ceph | nova | enabled | up    | 2016-11-11T08:26:31.000000 | -               |
+------------------+------------------------+------+---------+-------+----------------------------+-----------------+

# cinder get-capabilities hostgroup@tripleo_ceph
+---------------------+---------------------------------------------------+
| Volume stats        | Value                                             |
+---------------------+---------------------------------------------------+
| description         | None                                              |
| display_name        | None                                              |
| driver_version      | 1.2.0                                             |
| namespace           | OS::Storage::Capabilities::hostgroup@tripleo_ceph |
| pool_name           | None                                              |
| replication_targets | []                                                |
| storage_protocol    | ceph                                              |
| vendor_name         | Open Source                                       |
| visibility          | None                                              |
| volume_backend_name | tripleo_ceph                                      |
+---------------------+---------------------------------------------------+
+--------------------+---------------------------------------------------------------------------------------------------+
| Backend properties | Value                                                                                             |
+--------------------+---------------------------------------------------------------------------------------------------+
| compression        | {u'type': u'boolean', u'description': u'Enables compression.', u'title': u'Compression'}          |
| qos                | {u'type': u'boolean', u'description': u'Enables QoS.', u'title': u'QoS'}                          |
| replication        | {u'type': u'boolean', u'description': u'Enables replication.', u'title': u'Replication'}          |
| thin_provisioning  | {u'type': u'boolean', u'description': u'Sets thin provisioning.', u'title': u'Thin Provisioning'} |
+--------------------+---------------------------------------------------------------------------------------------------+
(.venv)[root@undercloud-0 tempest-dir]# cinder get-capabilities hostgroup
ERROR: Gateway Time-out (HTTP 504)

GOTO: https://bugzilla.redhat.com/show_bug.cgi?id=1391375

bypassing the proxy:
(from local)
cinder --bypass-url http://controller-0.localdomain:8776/v2/63a9a7c2f42e4d45a1c67cb3c4a8b743 get-capabilities hostgroup
ERROR: Bad gateway. (HTTP 502) (Request-ID: req-1a92f2d0-3f86-4c67-9f52-1738f561fa80)


2016-11-11 08:38:41.318 875 DEBUG cinder.api.openstack.wsgi [req-1a92f2d0-3f86-4c67-9f52-1738f561fa80 5c037e1539f0465f869ba2e7969a8f5f 63a9a7c2f42e4d45a1c67cb3c4a8b743 - default default] Calling method '<bound m
ethod CapabilitiesController.show of <cinder.api.contrib.capabilities.CapabilitiesController object at 0x692e2d0>>' _process_stack /usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py:871
2016-11-11 08:38:41.331 875 DEBUG oslo_messaging._drivers.amqpdriver [req-1a92f2d0-3f86-4c67-9f52-1738f561fa80 5c037e1539f0465f869ba2e7969a8f5f 63a9a7c2f42e4d45a1c67cb3c4a8b743 - default default] CALL msg_id: 46
f2f0d972c843aa8e31d49872439be3 exchange 'openstack' topic 'cinder-volume' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault [req-1a92f2d0-3f86-4c67-9f52-1738f561fa80 5c037e1539f0465f869ba2e7969a8f5f 63a9a7c2f42e4d45a1c67cb3c4a8b743 - default default] Caught error: <class '
cinder.exception.RPCTimeout'> Timeout while requesting capabilities from backend hostgroup.
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault Traceback (most recent call last):
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/middleware/fault.py", line 79, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     return req.get_response(self.application)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     resp = self.call_func(req, *args, **self.kwargs)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     return self.func(req, *args, **kwargs)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 108, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     return request.get_response(self.application)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     resp = self.call_func(req, *args, **self.kwargs)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     return self.func(req, *args, **kwargs)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     response = req.get_response(self._app)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     resp = self.call_func(req, *args, **self.kwargs)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     return self.func(req, *args, **kwargs)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     response = req.get_response(self._app)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     response = self.app(environ, start_response)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     resp = self.call_func(req, *args, **self.kwargs)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     return self.func(req, *args, **kwargs)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 843, in __call__
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     content_type, body, accept)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 902, in _process_stack
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     action_result = self.dispatch(meth, request, action_args)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 997, in dispatch
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     return method(req=request, **action_args)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/site-packages/cinder/api/contrib/capabilities.py", line 55, in show
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault     raise exception.RPCTimeout(service=id)
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault RPCTimeout: Timeout while requesting capabilities from backend hostgroup.
2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault 
2016-11-11 08:39:41.341 875 INFO cinder.api.middleware.fault [req-1a92f2d0-3f86-4c67-9f52-1738f561fa80 5c037e1539f0465f869ba2e7969a8f5f 63a9a7c2f42e4d45a1c67cb3c4a8b743 - default default] http://controller-0.localdomain:8776/v2/63a9a7c2f42e4d45a1c67cb3c4a8b743/capabilities/hostgroup returned with HTTP 502

Version-Release number of selected component (if applicable):
python-cinderclient-1.9.0-1.el7ost.noarch
puppet-cinder-9.4.1-2.el7ost.noarch
python-cinder-9.0.0-8.el7ost.noarch
openstack-cinder-9.0.0-8.el7ost.noarch


How reproducible:
always

Actual results:
I had to wait too much for a faulty response.

Expected results:
Not wait more than 5 sec for the response.


Additional info:
time cinder get-capabilities random_name_not_exisitng
ERROR: Can't find service: random_name_not_exisitng (HTTP 404) (Request-ID: req-4e6feb91-7d18-4b8a-bbd9-7ba2ea74c919)

real	0m1.858s
user	0m0.544s
sys	0m0.083s

Comment 1 Eric Harney 2016-12-21 14:49:20 UTC

What is the reason for using get-capabilities here?  This command has had some issues upstream and we are considering removing it altogether.

Is it something needed for a particular reason here?

Comment 2 Attila Fazekas 2017-02-01 08:20:21 UTC

Several tempest extra spec test requires to know more information about the back-ends , like available protocol, vendor , name ...

You can know these if you read the cinder config file, but it would allow us to auto-configure these options in tempest without having direct access to the controller machines.

Comment 3 Elise Gafford 2017-03-08 14:49:44 UTC

Hi Pranali,

Can you take a look at this one? Please do reach out with any questions.

Thanks,
Elise

Comment 4 Pranali Deore 2017-04-28 04:18:19 UTC

This is already fixed on master. I've backported the fix to stable/newton and submitted patch in upstream.

Comment 5 Pranali Deore 2017-04-28 04:19:26 UTC

This is already fixed on master. I've backported the fix to stable/newton and submitted patch in upstream.

Comment 8 Tzach Shefi 2017-11-21 13:52:41 UTC

Verified on:
openstack-cinder-11.0.1-1.el7ost.noarch

On a deployment with Ceph

$ time cinder get-capabilities hostgroup@tripleo_ceph
+---------------------+---------------------------------------------------+
| Volume stats        | Value                                             |
+---------------------+---------------------------------------------------+
| description         | None                                              |
| display_name        | None                                              |
| driver_version      | 1.2.0                                             |
| namespace           | OS::Storage::Capabilities::hostgroup@tripleo_ceph |
| pool_name           | None                                              |
| replication_targets | []                                                |
| storage_protocol    | ceph                                              |
| vendor_name         | Open Source                                       |
| visibility          | None                                              |
| volume_backend_name | tripleo_ceph                                      |
+---------------------+---------------------------------------------------+
+---------------------+---------------------------------------------------------------------------------------------------+
| Backend properties  | Value                                                                                             |
+---------------------+---------------------------------------------------------------------------------------------------+
| compression         | {u'type': u'boolean', u'description': u'Enables compression.', u'title': u'Compression'}          |
| qos                 | {u'type': u'boolean', u'description': u'Enables QoS.', u'title': u'QoS'}                          |
| replication_enabled | {u'type': u'boolean', u'description': u'Enables replication.', u'title': u'Replication'}          |
| thin_provisioning   | {u'type': u'boolean', u'description': u'Sets thin provisioning.', u'title': u'Thin Provisioning'} |
+---------------------+---------------------------------------------------------------------------------------------------+

real    0m2.987s
user    0m1.171s
sys     0m0.213s

$ time cinder get-capabilities hostgroup@kuku
ERROR: Can't find service: hostgroup@kuku (HTTP 404) (Request-ID: req-7c22c7fa-5892-4e02-937c-95da5f10741b)

real    0m2.867s
user    0m1.197s
sys     0m0.178s


I've tested a few time, in either case existing or none existing backend replay is returned quick well before the excepted 5 sec.

Comment 11 errata-xmlrpc 2017-12-13 20:49:23 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.