Bug 1512880

Summary:	openstack metric event list gives an error in undercloud
Product:	Red Hat OpenStack	Reporter:	Sasha Smolyak <ssmolyak>
Component:	instack-undercloud	Assignee:	Pradeep Kilambi <pkilambi>
Status:	CLOSED ERRATA	QA Contact:	Sasha Smolyak <ssmolyak>
Severity:	high	Docs Contact:
Priority:	medium
Version:	12.0 (Pike)	CC:	eglynn, jruzicka, jschluet, mburns, pkilambi, rhel-osp-director-maint, sclewis, srevivo, tvignaud
Target Milestone:	rc	Keywords:	Triaged
Target Release:	12.0 (Pike)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	instack-undercloud-7.4.3-3.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-12-13 22:20:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sasha Smolyak 2017-11-14 11:52:31 UTC

Description of problem:
openstack metric event list in undercloud gives an error:
Unable to establish connection to https://192.168.24.2:13779/v2/events: HTTPSConnectionPool(host='192.168.24.2', port=13779): Max retries exceeded with url: /v2/events (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x41139d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

openstack metric event type list:
Unable to establish connection to https://192.168.24.2:13779/v2/event_types/: HTTPSConnectionPool(host='192.168.24.2', port=13779): Max retries exceeded with url: /v2/event_types/ (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x37917d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

in overcloud works ok.

In The logs:
(undercloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID                                   | Name         | Status | Networks               | Image          | Flavor     |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| 3ed4fd57-abd0-406c-a7e1-2ed1c34265ca | compute-0    | ACTIVE | ctlplane=192.168.24.23 | overcloud-full | compute    |
| eb66f2ca-0a18-4fda-8a17-d64ea7f8f22c | ceph-1       | ACTIVE | ctlplane=192.168.24.16 | overcloud-full | ceph       |
| 0aebe045-4e08-432f-ae5b-14ef5ea1f629 | ceph-0       | ACTIVE | ctlplane=192.168.24.11 | overcloud-full | ceph       |
| 1d99f9c9-912e-480d-8b8e-6e6792dad00d | controller-2 | ACTIVE | ctlplane=192.168.24.18 | overcloud-full | controller |
| 02831378-0395-407d-9856-b8ac287552bd | controller-0 | ACTIVE | ctlplane=192.168.24.21 | overcloud-full | controller |
| f46ab971-2fe7-4766-aee3-cb4aca46f26a | controller-1 | ACTIVE | ctlplane=192.168.24.12 | overcloud-full | controller |
| 11569104-9053-4568-a16b-f2abd258d7a2 | ceph-2       | ACTIVE | ctlplane=192.168.24.20 | overcloud-full | ceph       |
| 369ba8fe-d019-43c6-bff1-74905f4d4d63 | compute-1    | ACTIVE | ctlplane=192.168.24.13 | overcloud-full | compute    |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
(undercloud) [stack@undercloud-0 ~]$ sudo tail -n 20 /var/log/ceilometer/agent-notification.log
2017-11-14 06:39:46.068 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 02831378-0395-407d-9856-b8ac287552bd does not exist (HTTP 404): ResourceNotFound: Resource 02831378-0395-407d-9856-b8ac287552bd does not exist (HTTP 404)
2017-11-14 06:39:46.091 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 369ba8fe-d019-43c6-bff1-74905f4d4d63 does not exist (HTTP 404): ResourceNotFound: Resource 369ba8fe-d019-43c6-bff1-74905f4d4d63 does not exist (HTTP 404)
2017-11-14 06:39:46.463 2802 DEBUG ceilometer.dispatcher.gnocchi [-] 8 measures posted against 8 metrics through 8 resources batch_measures /usr/lib/python2.7/site-packages/ceilometer/dispatcher/gnocchi.py:417
2017-11-14 06:39:46.489 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 11569104-9053-4568-a16b-f2abd258d7a2 does not exist (HTTP 404): ResourceNotFound: Resource 11569104-9053-4568-a16b-f2abd258d7a2 does not exist (HTTP 404)
2017-11-14 06:39:46.513 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 3ed4fd57-abd0-406c-a7e1-2ed1c34265ca does not exist (HTTP 404): ResourceNotFound: Resource 3ed4fd57-abd0-406c-a7e1-2ed1c34265ca does not exist (HTTP 404)
2017-11-14 06:39:46.536 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource eb66f2ca-0a18-4fda-8a17-d64ea7f8f22c does not exist (HTTP 404): ResourceNotFound: Resource eb66f2ca-0a18-4fda-8a17-d64ea7f8f22c does not exist (HTTP 404)
2017-11-14 06:39:46.561 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 1d99f9c9-912e-480d-8b8e-6e6792dad00d does not exist (HTTP 404): ResourceNotFound: Resource 1d99f9c9-912e-480d-8b8e-6e6792dad00d does not exist (HTTP 404)
2017-11-14 06:39:46.586 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource f46ab971-2fe7-4766-aee3-cb4aca46f26a does not exist (HTTP 404): ResourceNotFound: Resource f46ab971-2fe7-4766-aee3-cb4aca46f26a does not exist (HTTP 404)
2017-11-14 06:39:46.609 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 0aebe045-4e08-432f-ae5b-14ef5ea1f629 does not exist (HTTP 404): ResourceNotFound: Resource 0aebe045-4e08-432f-ae5b-14ef5ea1f629 does not exist (HTTP 404)
2017-11-14 06:39:46.635 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 02831378-0395-407d-9856-b8ac287552bd does not exist (HTTP 404): ResourceNotFound: Resource 02831378-0395-407d-9856-b8ac287552bd does not exist (HTTP 404)
2017-11-14 06:39:46.658 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 369ba8fe-d019-43c6-bff1-74905f4d4d63 does not exist (HTTP 404): ResourceNotFound: Resource 369ba8fe-d019-43c6-bff1-74905f4d4d63 does not exist (HTTP 404)
2017-11-14 06:39:47.039 2802 DEBUG ceilometer.dispatcher.gnocchi [-] 8 measures posted against 8 metrics through 8 resources batch_measures /usr/lib/python2.7/site-packages/ceilometer/dispatcher/gnocchi.py:417
2017-11-14 06:39:47.064 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 11569104-9053-4568-a16b-f2abd258d7a2 does not exist (HTTP 404): ResourceNotFound: Resource 11569104-9053-4568-a16b-f2abd258d7a2 does not exist (HTTP 404)
2017-11-14 06:39:47.087 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 3ed4fd57-abd0-406c-a7e1-2ed1c34265ca does not exist (HTTP 404): ResourceNotFound: Resource 3ed4fd57-abd0-406c-a7e1-2ed1c34265ca does not exist (HTTP 404)
2017-11-14 06:39:47.110 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource eb66f2ca-0a18-4fda-8a17-d64ea7f8f22c does not exist (HTTP 404): ResourceNotFound: Resource eb66f2ca-0a18-4fda-8a17-d64ea7f8f22c does not exist (HTTP 404)
2017-11-14 06:39:47.134 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 1d99f9c9-912e-480d-8b8e-6e6792dad00d does not exist (HTTP 404): ResourceNotFound: Resource 1d99f9c9-912e-480d-8b8e-6e6792dad00d does not exist (HTTP 404)
2017-11-14 06:39:47.157 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource f46ab971-2fe7-4766-aee3-cb4aca46f26a does not exist (HTTP 404): ResourceNotFound: Resource f46ab971-2fe7-4766-aee3-cb4aca46f26a does not exist (HTTP 404)
2017-11-14 06:39:47.182 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 0aebe045-4e08-432f-ae5b-14ef5ea1f629 does not exist (HTTP 404): ResourceNotFound: Resource 0aebe045-4e08-432f-ae5b-14ef5ea1f629 does not exist (HTTP 404)
2017-11-14 06:39:47.208 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 02831378-0395-407d-9856-b8ac287552bd does not exist (HTTP 404): ResourceNotFound: Resource 02831378-0395-407d-9856-b8ac287552bd does not exist (HTTP 404)
2017-11-14 06:39:47.232 2802 ERROR ceilometer.dispatcher.gnocchi [-] Resource 369ba8fe-d019-43c6-bff1-74905f4d4d63 does not exist (HTTP 404): ResourceNotFound: Resource 369ba8fe-d019-43c6-bff1-74905f4d4d63 does not exist (HTTP 404)

Version-Release number of selected component (if applicable):
openstack-ceilometer-notification-9.0.1-3.el7ost.noarch
openstack-ceilometer-central-9.0.1-3.el7ost.noarch
python-ceilometermiddleware-1.1.0-1.el7ost.noarch
python-ceilometerclient-2.9.0-1.el7ost.noarch
puppet-ceilometer-11.3.1-0.20170923033230.767c543.el7ost.noarch
openstack-ceilometer-common-9.0.1-3.el7ost.noarch
openstack-ceilometer-polling-9.0.1-3.el7ost.noarch
python-ceilometer-9.0.1-3.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deployed new setup. Performed manual update: 
sudo yum localinstall -y http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm

sudo yum localinstall -y http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/rhos-release/rhos-release-latest.noarch.rpm # alternative location

http://download.eng.bos.redhat.com/brewroot/packages/wget/1.14/15.el7/x86_64/wget-1.14-15.el7.x86_64.rpm

sudo rhos-release -x
sudo rhos-release -P 12
sudo yum update python-tripleoclient
su - stack openstack undercloud upgrade 2>&1 | tee undercloud_update.log


2. Tried to check events list, got the error
3.Watched the agent-notification.log, saw that the error was there from the beginning

Actual results:
error for openstack metric event type list and openstack metric event list

Expected results:
The lists

Additional info:

Comment 1 Sasha Smolyak 2017-11-15 14:00:28 UTC

Happens straight after deployment, without updates. Event list in undercloud is unreachable

Comment 2 Red Hat Bugzilla Rules Engine 2017-11-15 14:16:44 UTC

This bugzilla has been removed from the release since it has not been triaged, and needs to be reviewed for targeting another release.

Comment 6 Sasha Smolyak 2017-11-23 13:54:00 UTC

Verified, 
openstack event list works OK in undercloud

Comment 9 errata-xmlrpc 2017-12-13 22:20:31 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462