Bug 1348201

Summary: Graceful handling of 404 error from neutron
Product: Red Hat OpenStack Reporter: Yurii Prokulevych <yprokule>
Component: openstack-ceilometerAssignee: Pradeep Kilambi <pkilambi>
Status: CLOSED ERRATA QA Contact: Yurii Prokulevych <yprokule>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0 (Mitaka)CC: jdanjou, jruzicka, mabaakou, srevivo
Target Milestone: gaKeywords: Triaged
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-ceilometer-6.1.1-1.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-11 12:26:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1353929    
Bug Blocks:    

Description Yurii Prokulevych 2016-06-20 12:31:35 UTC 
Description of problem:
-----------------------
Default pipeline.yaml contains a wild card entry - causing central agent to poll 'all' pollsters. Though if some polled object doesn't exist or is not configured it throws traceback in log:

2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager [req-1c713065-631a-4764-a2ea-a40384fe51d9 admin - - - -] Unable to discover resources: 404 Not Found

The resource could not be found.

   
Neutron server returns request_ids: ['req-d7573c0c-4939-499c-8c30-d611791e487f']
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager Traceback (most recent call last):
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/ceilometer/agent/manager.py", line 458, in discover
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager     discovered = discoverer.discover(self, param)
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/ceilometer/network/services/discovery.py", line 68, in discover
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager     loadbalancers = self.neutron_cli.list_loadbalancer()
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 401, in list_loadbalancer
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager     resp = self.client.list_loadbalancers()
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 97, in with_params
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager     ret = self.function(instance, *args, **kwargs)
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1045, in list_loadbalancers
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager     retrieve_all, **_params)
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 373, in list
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager     for r in self._pagination(collection, path, **params):
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 388, in _pagination
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager     res = self.get(path, params=params)
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 358, in get
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager     headers=headers, params=params)
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 335, in retry_request
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager     headers=headers, params=params)
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 298, in do_request
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager     self._handle_fault_response(status_code, replybody, resp)
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 273, in _handle_fault_response
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager     exception_handler_v20(status_code, error_body)
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 84, in exception_handler_v20
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager     request_ids=request_ids)
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager NotFound: 404 Not Found
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager 
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager The resource could not be found.
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager 
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager    
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager Neutron server returns request_ids: ['req-d7573c0c-4939-499c-8c30-d611791e487f']
2016-06-20 12:14:27.613 6075 ERROR ceilometer.agent.manager 
2016-06-20 12:14:27.614 6075 INFO ceilometer.agent.manager [req-1c713065-631a-4764-a2ea-a40384fe51d9 admin - - - -] Skip pollster network.services.lb.total.connections, no resources found this cycle
2016-06-20 12:14:27.950 6075 WARNING ceilometer.neutron_client [req-1c713065-631a-4764-a2ea-a40384fe51d9 admin - - - -] The resource could not be found.

This situation has to be handled gracefully.

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-ceilometer-api-6.0.0-2.el7ost.noarch
openstack-ceilometer-polling-6.0.0-2.el7ost.noarch
openstack-ceilometer-compute-6.0.0-2.el7ost.noarch
python-ceilometer-6.0.0-2.el7ost.noarch
openstack-ceilometer-common-6.0.0-2.el7ost.noarch
openstack-ceilometer-collector-6.0.0-2.el7ost.noarch
openstack-ceilometer-notification-6.0.0-2.el7ost.noarch
openstack-ceilometer-central-6.0.0-2.el7ost.noarch
python-ceilometerclient-2.3.0-1.el7ost.noarch

python-neutronclient-4.1.1-2.el7ost.noarch

Additional info:
----------------
Virtual setup: 3Controllers + 2Computes +1Ceph
Comment 1 Julien Danjou 2016-06-30 12:02:57 UTC 
Backported upstream fix to stable/mitaka at http://review.openstack.org/335973
Comment 3 Yurii Prokulevych 2016-07-20 13:48:40 UTC 
Excerpt from ceilometer/central.log:
------------------------------------
...
Skip pollster network.services.lb.total.connections, no resources found this cycle
Skip pollster network.services.lb.loadbalancer, no resources found this cycle
Skip pollster network.services.vpn.connections, no resources found this cycle
Skip pollster network.services.lb.member, no resources found this cycle

Packages:
---------
openstack-ceilometer-central-6.1.1-1.el7ost.noarch
python-ceilometerclient-2.3.0-1.el7ost.noarch
openstack-ceilometer-polling-6.1.1-1.el7ost.noarch
python-ceilometer-6.1.1-1.el7ost.noarch
openstack-ceilometer-common-6.1.1-1.el7ost.noarch
openstack-ceilometer-collector-6.1.1-1.el7ost.noarch
openstack-ceilometer-notification-6.1.1-1.el7ost.noarch
openstack-ceilometer-api-6.1.1-1.el7ost.noarch
openstack-ceilometer-compute-6.1.1-1.el7ost.noarch
Comment 5 errata-xmlrpc 2016-08-11 12:26:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1597.html