Bug 1394155
| Summary: | RPCTimeout: Timeout while requesting capabilities from backend hostgroup. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Attila Fazekas <afazekas> |
| Component: | openstack-cinder | Assignee: | Eric Harney <eharney> |
| Status: | CLOSED ERRATA | QA Contact: | Avi Avraham <aavraham> |
| Severity: | low | Docs Contact: | |
| Priority: | low | ||
| Version: | 10.0 (Newton) | CC: | afazekas, egafford, eharney, scohen, srevivo, tshefi, tvignaud |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | 12.0 (Pike) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-cinder-11.0.0-2.el7ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-13 20:49:23 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
What is the reason for using get-capabilities here? This command has had some issues upstream and we are considering removing it altogether. Is it something needed for a particular reason here? Several tempest extra spec test requires to know more information about the back-ends , like available protocol, vendor , name ... You can know these if you read the cinder config file, but it would allow us to auto-configure these options in tempest without having direct access to the controller machines. Hi Pranali, Can you take a look at this one? Please do reach out with any questions. Thanks, Elise This is already fixed on master. I've backported the fix to stable/newton and submitted patch in upstream. This is already fixed on master. I've backported the fix to stable/newton and submitted patch in upstream. Verified on:
openstack-cinder-11.0.1-1.el7ost.noarch
On a deployment with Ceph
$ time cinder get-capabilities hostgroup@tripleo_ceph
+---------------------+---------------------------------------------------+
| Volume stats | Value |
+---------------------+---------------------------------------------------+
| description | None |
| display_name | None |
| driver_version | 1.2.0 |
| namespace | OS::Storage::Capabilities::hostgroup@tripleo_ceph |
| pool_name | None |
| replication_targets | [] |
| storage_protocol | ceph |
| vendor_name | Open Source |
| visibility | None |
| volume_backend_name | tripleo_ceph |
+---------------------+---------------------------------------------------+
+---------------------+---------------------------------------------------------------------------------------------------+
| Backend properties | Value |
+---------------------+---------------------------------------------------------------------------------------------------+
| compression | {u'type': u'boolean', u'description': u'Enables compression.', u'title': u'Compression'} |
| qos | {u'type': u'boolean', u'description': u'Enables QoS.', u'title': u'QoS'} |
| replication_enabled | {u'type': u'boolean', u'description': u'Enables replication.', u'title': u'Replication'} |
| thin_provisioning | {u'type': u'boolean', u'description': u'Sets thin provisioning.', u'title': u'Thin Provisioning'} |
+---------------------+---------------------------------------------------------------------------------------------------+
real 0m2.987s
user 0m1.171s
sys 0m0.213s
$ time cinder get-capabilities hostgroup@kuku
ERROR: Can't find service: hostgroup@kuku (HTTP 404) (Request-ID: req-7c22c7fa-5892-4e02-937c-95da5f10741b)
real 0m2.867s
user 0m1.197s
sys 0m0.178s
I've tested a few time, in either case existing or none existing backend replay is returned quick well before the excepted 5 sec.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462 |
Description of problem: # cinder service-list +------------------+------------------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+------------------------+------+---------+-------+----------------------------+-----------------+ | cinder-backup | hostgroup | nova | enabled | up | 2016-11-11T08:26:34.000000 | - | | cinder-scheduler | hostgroup | nova | enabled | up | 2016-11-11T08:26:35.000000 | - | | cinder-volume | hostgroup@tripleo_ceph | nova | enabled | up | 2016-11-11T08:26:31.000000 | - | +------------------+------------------------+------+---------+-------+----------------------------+-----------------+ # cinder get-capabilities hostgroup@tripleo_ceph +---------------------+---------------------------------------------------+ | Volume stats | Value | +---------------------+---------------------------------------------------+ | description | None | | display_name | None | | driver_version | 1.2.0 | | namespace | OS::Storage::Capabilities::hostgroup@tripleo_ceph | | pool_name | None | | replication_targets | [] | | storage_protocol | ceph | | vendor_name | Open Source | | visibility | None | | volume_backend_name | tripleo_ceph | +---------------------+---------------------------------------------------+ +--------------------+---------------------------------------------------------------------------------------------------+ | Backend properties | Value | +--------------------+---------------------------------------------------------------------------------------------------+ | compression | {u'type': u'boolean', u'description': u'Enables compression.', u'title': u'Compression'} | | qos | {u'type': u'boolean', u'description': u'Enables QoS.', u'title': u'QoS'} | | replication | {u'type': u'boolean', u'description': u'Enables replication.', u'title': u'Replication'} | | thin_provisioning | {u'type': u'boolean', u'description': u'Sets thin provisioning.', u'title': u'Thin Provisioning'} | +--------------------+---------------------------------------------------------------------------------------------------+ (.venv)[root@undercloud-0 tempest-dir]# cinder get-capabilities hostgroup ERROR: Gateway Time-out (HTTP 504) GOTO: https://bugzilla.redhat.com/show_bug.cgi?id=1391375 bypassing the proxy: (from local) cinder --bypass-url http://controller-0.localdomain:8776/v2/63a9a7c2f42e4d45a1c67cb3c4a8b743 get-capabilities hostgroup ERROR: Bad gateway. (HTTP 502) (Request-ID: req-1a92f2d0-3f86-4c67-9f52-1738f561fa80) 2016-11-11 08:38:41.318 875 DEBUG cinder.api.openstack.wsgi [req-1a92f2d0-3f86-4c67-9f52-1738f561fa80 5c037e1539f0465f869ba2e7969a8f5f 63a9a7c2f42e4d45a1c67cb3c4a8b743 - default default] Calling method '<bound m ethod CapabilitiesController.show of <cinder.api.contrib.capabilities.CapabilitiesController object at 0x692e2d0>>' _process_stack /usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py:871 2016-11-11 08:38:41.331 875 DEBUG oslo_messaging._drivers.amqpdriver [req-1a92f2d0-3f86-4c67-9f52-1738f561fa80 5c037e1539f0465f869ba2e7969a8f5f 63a9a7c2f42e4d45a1c67cb3c4a8b743 - default default] CALL msg_id: 46 f2f0d972c843aa8e31d49872439be3 exchange 'openstack' topic 'cinder-volume' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault [req-1a92f2d0-3f86-4c67-9f52-1738f561fa80 5c037e1539f0465f869ba2e7969a8f5f 63a9a7c2f42e4d45a1c67cb3c4a8b743 - default default] Caught error: <class ' cinder.exception.RPCTimeout'> Timeout while requesting capabilities from backend hostgroup. 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault Traceback (most recent call last): 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/middleware/fault.py", line 79, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault return req.get_response(self.application) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault application, catch_exc_info=False) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault resp = self.call_func(req, *args, **self.kwargs) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault return self.func(req, *args, **kwargs) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 108, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault return request.get_response(self.application) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault application, catch_exc_info=False) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault resp = self.call_func(req, *args, **self.kwargs) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault return self.func(req, *args, **kwargs) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault response = req.get_response(self._app) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault application, catch_exc_info=False) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault resp = self.call_func(req, *args, **self.kwargs) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault return self.func(req, *args, **kwargs) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault response = req.get_response(self._app) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1299, in send 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault application, catch_exc_info=False) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/request.py", line 1263, in call_application 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault app_iter = application(self.environ, start_response) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault response = self.app(environ, start_response) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault return resp(environ, start_response) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault resp = self.call_func(req, *args, **self.kwargs) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault return self.func(req, *args, **kwargs) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 843, in __call__ 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault content_type, body, accept) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 902, in _process_stack 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault action_result = self.dispatch(meth, request, action_args) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/openstack/wsgi.py", line 997, in dispatch 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault return method(req=request, **action_args) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/site-packages/cinder/api/contrib/capabilities.py", line 55, in show 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault raise exception.RPCTimeout(service=id) 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault RPCTimeout: Timeout while requesting capabilities from backend hostgroup. 2016-11-11 08:39:41.334 875 ERROR cinder.api.middleware.fault 2016-11-11 08:39:41.341 875 INFO cinder.api.middleware.fault [req-1a92f2d0-3f86-4c67-9f52-1738f561fa80 5c037e1539f0465f869ba2e7969a8f5f 63a9a7c2f42e4d45a1c67cb3c4a8b743 - default default] http://controller-0.localdomain:8776/v2/63a9a7c2f42e4d45a1c67cb3c4a8b743/capabilities/hostgroup returned with HTTP 502 Version-Release number of selected component (if applicable): python-cinderclient-1.9.0-1.el7ost.noarch puppet-cinder-9.4.1-2.el7ost.noarch python-cinder-9.0.0-8.el7ost.noarch openstack-cinder-9.0.0-8.el7ost.noarch How reproducible: always Actual results: I had to wait too much for a faulty response. Expected results: Not wait more than 5 sec for the response. Additional info: time cinder get-capabilities random_name_not_exisitng ERROR: Can't find service: random_name_not_exisitng (HTTP 404) (Request-ID: req-4e6feb91-7d18-4b8a-bbd9-7ba2ea74c919) real 0m1.858s user 0m0.544s sys 0m0.083s