Bug 1343445

Summary: metricd stops renewing/refreshing token with swift backend
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: openstack-gnocchiAssignee: Pradeep Kilambi <pkilambi>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0 (Mitaka)CC: apevec, jdanjou, jschluet, lhh
Target Milestone: gaKeywords: Triaged
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-gnocchi-2.1.3-1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-11 12:24:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Yurii Prokulevych 2016-06-07 11:14:29 UTC 
Description of problem:
Gnocchi is configured to use swift backend. Also gnocchi's api is running under mod_wsgi.

After some time, gnocchi-metricd starts error'ing with next message:
...
2016-06-07 11:06:01.077 16999 DEBUG gnocchi.storage [-] Expunging deleted metrics process_background_tasks /usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py:191
2016-06-07 11:06:06.060 16999 DEBUG gnocchi.storage [-] Processing new and to delete measures process_background_tasks /usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py:183
2016-06-07 11:06:06.079 16999 INFO swiftclient [-] REQ: curl -i http://192.0.2.15:8080/v1/AUTH_a2a8b54f11bf47d7a5b8ac9be9acf5c9/measure?format=json&limit=16&delimiter=/ -X GET -H "X-Auth-Token: 4dd3cd7c8930434f..."
2016-06-07 11:06:06.079 16999 INFO swiftclient [-] RESP STATUS: 401 Unauthorized
2016-06-07 11:06:06.080 16999 INFO swiftclient [-] RESP HEADERS: {u'Date': u'Tue, 07 Jun 2016 11:06:06 GMT', u'Content-Length': u'131', u'Content-Type': u'text/html; charset=UTF-8', u'Www-Authenticate': u'Swift realm="AUTH_a2a8b54f11bf47d7a5b8ac9be9acf5c9", Keystone uri=\'http://192.0.2.15:5000/v2.0\'', u'X-Trans-Id': u'tx52d6876089464082a5389-005756aa9e'}
2016-06-07 11:06:06.080 16999 INFO swiftclient [-] RESP BODY: <html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>
2016-06-07 11:06:06.080 16999 ERROR swiftclient [-] Container GET failed: http://192.0.2.15:8080/v1/AUTH_a2a8b54f11bf47d7a5b8ac9be9acf5c9/measure?format=json&limit=16&delimiter=/ 401 Unauthorized [first 60 chars of response] <html><h1>Unauthorized</h1><p>This server could not verify t
2016-06-07 11:06:06.080 16999 ERROR swiftclient Traceback (most recent call last):
2016-06-07 11:06:06.080 16999 ERROR swiftclient File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1565, in _retry
2016-06-07 11:06:06.080 16999 ERROR swiftclient service_token=self.service_token, **kwargs)
2016-06-07 11:06:06.080 16999 ERROR swiftclient File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 884, in get_container
2016-06-07 11:06:06.080 16999 ERROR swiftclient http_response_content=body)
2016-06-07 11:06:06.080 16999 ERROR swiftclient ClientException: Container GET failed: http://192.0.2.15:8080/v1/AUTH_a2a8b54f11bf47d7a5b8ac9be9acf5c9/measure?format=json&limit=16&delimiter=/ 401 Unauthorized [first 60 chars of response] <html><h1>Unauthorized</h1><p>This server could not verify t
2016-06-07 11:06:06.080 16999 ERROR swiftclient
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage [-] Unexpected error during measures processing
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage Traceback (most recent call last):
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py", line 185, in process_background_tasks
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage self.process_measures(index, block_size, sync)
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 346, in process_measures
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage block_size, full=sync)
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py", line 138, in _list_metric_with_measures_to_process
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage limit=limit)
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1638, in get_container
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage full_listing=full_listing, headers=headers)
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1565, in _retry
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage service_token=self.service_token, **kwargs)
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 884, in get_container
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage http_response_content=body)
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage ClientException: Container GET failed: http://192.0.2.15:8080/v1/AUTH_a2a8b54f11bf47d7a5b8ac9be9acf5c9/measure?format=json&limit=16&delimiter=/ 401 Unauthorized [first 60 chars of response] <html><h1>Unauthorized</h1><p>This server could not verify t
2016-06-07 11:06:06.080 16999 ERROR gnocchi.storage
2016-06-07 11:06:06.081 16999 DEBUG gnocchi.storage [-] Expunging deleted metrics process_background_tasks /usr/lib/python2.7/site-packages/gnocchi/storage/__init__.py:191
...

And gnocchi_wsgi_error.log reports next:

[Tue Jun 07 11:06:31.662229 2016] [:error] [pid 32764] [remote 192.0.2.21:44] mod_wsgi (pid=32764): Exception occurred processing WSGI script '/var/www/cgi-bin/gnocchi/app'.
[Tue Jun 07 11:06:31.662262 2016] [:error] [pid 32764] [remote 192.0.2.21:44] Traceback (most recent call last):
[Tue Jun 07 11:06:31.662276 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__
[Tue Jun 07 11:06:31.662295 2016] [:error] [pid 32764] [remote 192.0.2.21:44] return app(environ, start_response)
[Tue Jun 07 11:06:31.662314 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
[Tue Jun 07 11:06:31.662331 2016] [:error] [pid 32764] [remote 192.0.2.21:44] resp = self.call_func(req, *args, **self.kwargs)
[Tue Jun 07 11:06:31.662340 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
[Tue Jun 07 11:06:31.662352 2016] [:error] [pid 32764] [remote 192.0.2.21:44] return self.func(req, *args, **kwargs)
[Tue Jun 07 11:06:31.662361 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__
[Tue Jun 07 11:06:31.662375 2016] [:error] [pid 32764] [remote 192.0.2.21:44] response = req.get_response(self._app)
[Tue Jun 07 11:06:31.662384 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/webob/request.py", line 1317, in send
[Tue Jun 07 11:06:31.662396 2016] [:error] [pid 32764] [remote 192.0.2.21:44] application, catch_exc_info=False)
[Tue Jun 07 11:06:31.662405 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
[Tue Jun 07 11:06:31.662417 2016] [:error] [pid 32764] [remote 192.0.2.21:44] app_iter = application(self.environ, start_response)
[Tue Jun 07 11:06:31.662425 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/webob/exc.py", line 1117, in __call__
[Tue Jun 07 11:06:31.662437 2016] [:error] [pid 32764] [remote 192.0.2.21:44] return self.application(environ, start_response)
[Tue Jun 07 11:06:31.662446 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/gnocchi/rest/app.py", line 68, in __call__
[Tue Jun 07 11:06:31.662460 2016] [:error] [pid 32764] [remote 192.0.2.21:44] return self.app(environ, start_response)
[Tue Jun 07 11:06:31.662480 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__
[Tue Jun 07 11:06:31.662525 2016] [:error] [pid 32764] [remote 192.0.2.21:44] return self.application(environ, start_response)
[Tue Jun 07 11:06:31.662538 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/pecan/core.py", line 829, in __call__
[Tue Jun 07 11:06:31.662552 2016] [:error] [pid 32764] [remote 192.0.2.21:44] return super(Pecan, self).__call__(environ, start_response)
[Tue Jun 07 11:06:31.662561 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/pecan/core.py", line 678, in __call__
[Tue Jun 07 11:06:31.662573 2016] [:error] [pid 32764] [remote 192.0.2.21:44] self.invoke_controller(controller, args, kwargs, state)
[Tue Jun 07 11:06:31.662582 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/pecan/core.py", line 572, in invoke_controller
[Tue Jun 07 11:06:31.662593 2016] [:error] [pid 32764] [remote 192.0.2.21:44] result = controller(*args, **kwargs)
[Tue Jun 07 11:06:31.662608 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/gnocchi/rest/__init__.py", line 513, in post_measures
[Tue Jun 07 11:06:31.662622 2016] [:error] [pid 32764] [remote 192.0.2.21:44] self.metric, six.moves.map(MeasureSchema, params))
[Tue Jun 07 11:06:31.662631 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py", line 248, in add_measures
[Tue Jun 07 11:06:31.662645 2016] [:error] [pid 32764] [remote 192.0.2.21:44] list(map(tuple, measures))))
[Tue Jun 07 11:06:31.662654 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/gnocchi/storage/swift.py", line 114, in _store_measures
[Tue Jun 07 11:06:31.662667 2016] [:error] [pid 32764] [remote 192.0.2.21:44] data)
[Tue Jun 07 11:06:31.662675 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1709, in put_object
[Tue Jun 07 11:06:31.662687 2016] [:error] [pid 32764] [remote 192.0.2.21:44] response_dict=response_dict)
[Tue Jun 07 11:06:31.662696 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1565, in _retry
[Tue Jun 07 11:06:31.662707 2016] [:error] [pid 32764] [remote 192.0.2.21:44] service_token=self.service_token, **kwargs)
[Tue Jun 07 11:06:31.662715 2016] [:error] [pid 32764] [remote 192.0.2.21:44] File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1281, in put_object
[Tue Jun 07 11:06:31.662726 2016] [:error] [pid 32764] [remote 192.0.2.21:44] http_response_content=body)
[Tue Jun 07 11:06:31.662755 2016] [:error] [pid 32764] [remote 192.0.2.21:44] ClientException: Object PUT failed: http://192.0.2.15:8080/v1/AUTH_a2a8b54f11bf47d7a5b8ac9be9acf5c9/measure/466e5c18-6bf3-4559-97d0-b66d30827f9e/813633d6-d30f-477e-8ffb-9d48ffad111f_20160607_11%3A06%3A31 401 Unauthorized [first 60 chars of response] <html><h1>Unauthorized</h1><p>This server could not verify t

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
python-gnocchiclient-2.2.0-1.el7ost.noarch
openstack-gnocchi-common-2.1.1-1.el7ost.noarch
openstack-gnocchi-statsd-2.1.1-1.el7ost.noarch
python-gnocchi-2.1.1-1.el7ost.noarch
openstack-gnocchi-metricd-2.1.1-1.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-2.1.1-1.el7ost.noarch
openstack-gnocchi-api-2.1.1-1.el7ost.noarch
openstack-gnocchi-carbonara-2.1.1-1.el7ost.noarch

python-swiftclient-3.0.0-1.el7ost.noarch


Additional info:
Virtual Env - 3controllers + 2computes + 2cephs
Comment 3 Yurii Prokulevych 2016-06-22 15:29:15 UTC 
Verified with packages:

openstack-gnocchi-common-2.1.3-1.el7ost.noarch
openstack-gnocchi-metricd-2.1.3-1.el7ost.noarch
openstack-gnocchi-carbonara-2.1.3-1.el7ost.noarch
openstack-gnocchi-indexer-sqlalchemy-2.1.3-1.el7ost.noarch
openstack-gnocchi-statsd-2.1.3-1.el7ost.noarch
python-gnocchiclient-2.2.0-1.el7ost.noarch
python-gnocchi-2.1.3-1.el7ost.noarch
openstack-gnocchi-api-2.1.3-1.el7ost.noarch
Comment 5 errata-xmlrpc 2016-08-11 12:24:43 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1597.html