Bug 1466524

Summary: IndexError: list index out of range.
Product: Red Hat OpenStack Reporter: Siggy Sigwald <ssigwald>
Component: openstack-ceilometerAssignee: Mehdi ABAAKOUK <mabaakou>
Status: CLOSED ERRATA QA Contact: Sasha Smolyak <ssmolyak>
Severity: medium Docs Contact:
Priority: medium    
Version: 10.0 (Newton)CC: jruzicka, mabaakou, scorcora, srevivo, tvignaud
Target Milestone: rcKeywords: Triaged
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-ceilometer-8.1.0-0.20170821152204.04352f3.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-13 21:35:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Siggy Sigwald 2017-06-29 21:36:59 UTC
Description of problem:
Customer is getting this ERROR message on his logs.

2017-06-23 17:35:16.707 433170 ERROR ceilometer.neutron_client [-] list index out of range
2017-06-23 17:35:16.707 433170 ERROR ceilometer.neutron_client Traceback (most recent call last):
2017-06-23 17:35:16.707 433170 ERROR ceilometer.neutron_client   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 45, in with_logging
2017-06-23 17:35:16.707 433170 ERROR ceilometer.neutron_client     return func(*args, **kwargs)
2017-06-23 17:35:16.707 433170 ERROR ceilometer.neutron_client   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 155, in list_pools_v2
2017-06-23 17:35:16.707 433170 ERROR ceilometer.neutron_client     pool_listener_dict = self._get_pool_and_listener_ids(temp_pools)
2017-06-23 17:35:16.707 433170 ERROR ceilometer.neutron_client   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 232, in _get_pool_and_listener_ids
2017-06-23 17:35:16.707 433170 ERROR ceilometer.neutron_client     value = pool.get('listeners')[0].get('id')
2017-06-23 17:35:16.707 433170 ERROR ceilometer.neutron_client IndexError: list index out of range
2017-06-23 17:35:16.707 433170 ERROR ceilometer.neutron_client
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client [-] list index out of range
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client Traceback (most recent call last):
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 45, in with_logging
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client     return func(*args, **kwargs)
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 90, in pool_get_all
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client     resources = self.list_pools_v2()
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 45, in with_logging
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client     return func(*args, **kwargs)
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 155, in list_pools_v2
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client     pool_listener_dict = self._get_pool_and_listener_ids(temp_pools)
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 232, in _get_pool_and_listener_ids
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client     value = pool.get('listeners')[0].get('id')
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client IndexError: list index out of range
2017-06-23 17:35:16.708 433170 ERROR ceilometer.neutron_client
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager [-] Unable to discover resources: list index out of range
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager Traceback (most recent call last):
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/ceilometer/agent/manager.py", line 499, in discover
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager     discovered = discoverer.discover(self, param)
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/ceilometer/network/services/discovery.py", line 32, in discover
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager     pools = self.neutron_cli.pool_get_all()
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 45, in with_logging
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager     return func(*args, **kwargs)
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 90, in pool_get_all
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager     resources = self.list_pools_v2()
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 45, in with_logging
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager     return func(*args, **kwargs)
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 155, in list_pools_v2
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager     pool_listener_dict = self._get_pool_and_listener_ids(temp_pools)
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager   File "/usr/lib/python2.7/site-packages/ceilometer/neutron_client.py", line 232, in _get_pool_and_listener_ids
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager     value = pool.get('listeners')[0].get('id')
2017-06-23 17:35:16.709 433170 ERROR ceilometer.agent.manager IndexError: list index out of range

Version-Release number of selected component (if applicable):
openstack-ceilometer-api-7.0.1-1.el7ost.noarch
openstack-ceilometer-central-7.0.1-1.el7ost.noarch
openstack-ceilometer-collector-7.0.1-1.el7ost.noarch
openstack-ceilometer-common-7.0.1-1.el7ost.noarch
openstack-ceilometer-compute-7.0.1-1.el7ost.noarch
openstack-ceilometer-notification-7.0.1-1.el7ost.noarch
openstack-ceilometer-polling-7.0.1-1.el7ost.noarch
puppet-ceilometer-9.5.0-1.el7ost.noarch
python-ceilometer-7.0.1-1.el7ost.noarch
python-ceilometerclient-2.6.2-1.el7ost.noarch
python-ceilometermiddleware-0.5.1-1.el7ost.noarch

Comment 2 Mehdi ABAAKOUK 2017-08-01 08:32:06 UTC
Upstream patch merged, will be part of next OSP12 puddle

Comment 7 Mehdi ABAAKOUK 2017-11-27 09:27:38 UTC
On undercloud:

$ source overcloudrc
$ neutron lbaas-loadbalancer-create --name lb-BZ1466524  external_subnet
$ neutron lbaas-listener-create  --loadbalancer lb-BZ1466524 --protocol HTTP --protocol-port 80 --name listener-BZ1466524
$ neutron lbaas-pool-create --protocol HTTP --listener listener-BZ1466524 --lb-algorithm ROUND_ROBIN --name pool-BZ1466524
$ openstack server show MyFirstInstance -c addresses -f value
nova=10.0.0.213

$ neutron lbaas-member-create --subnet external_subnet --name member-BZ1466524  --address 10.0.0.213 --protocol-port 80 pool-BZ1466524
$ neutron lbaas-healthmonitor-create --delay 100 --timeout 100 --name heath-BZ1466524 --max-retries 1 --type PING --pool pool-BZ1466524
$ neutron lbaas-loadbalancer-show lb-BZ1466524 -c  vip_address -f value
10.0.0.211
$ curl 10.0.0.211
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

"No server is available" is fine since the instance doesn't have a httpd service running, but we don't care here.


On controller0:

$ docker exec -it ceilometer_agent_central vi /etc/ceilometer/polling.yaml
Add to meters:

  - network.services.lb*

$ docker exec -it -u root ceilometer_agent_central  pkill -HUP -f ceilometer-polling.*master

Wait 10 minutes for next polling cycle to run, and check network.services.lb* are polled and no more error occurs:

$ docker exec -it -u root ceilometer_agent_central  cat  /var/log/ceilometer/central.log

2017-11-27 09:15:56.981 20941 INFO ceilometer.agent.manager [-] Polling pollster network.services.lb.pool in the context of some_pollsters
2017-11-27 09:15:56.986 20941 INFO ceilometer.agent.manager [-] Polling pollster network.services.lb.outgoing.bytes in the context of some_pollsters
2017-11-27 09:15:57.727 20941 INFO ceilometer.agent.manager [-] Polling pollster network.services.lb.member in the context of some_pollsters
2017-11-27 09:15:57.731 20941 INFO ceilometer.agent.manager [-] Polling pollster network.services.lb.total.connections in the context of some_pollsters
2017-11-27 09:15:57.733 20941 INFO ceilometer.agent.manager [-] Polling pollster network.services.lb.active.connections in the context of some_pollsters
2017-11-27 09:15:58.127 20941 INFO ceilometer.agent.manager [-] Polling pollster network.services.lb.listener in the context of some_pollsters
2017-11-27 09:16:01.030 20941 INFO ceilometer.agent.manager [-] Polling pollster network.services.lb.incoming.bytes in the context of some_pollsters
2017-11-27 09:16:01.251 20941 INFO ceilometer.agent.manager [-] Polling pollster network.services.lb.loadbalancer in the context of some_pollsters

Comment 10 errata-xmlrpc 2017-12-13 21:35:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462