Bug 1540950

Summary:	No measures for instance in undercloud
Product:	Red Hat OpenStack	Reporter:	Sasha Smolyak <ssmolyak>
Component:	documentation	Assignee:	Stephen <stesmith>
Status:	CLOSED DUPLICATE	QA Contact:	RHOS Documentation Team <rhos-docs>
Severity:	low	Docs Contact:	Deepti Navale <dnavale>
Priority:	low
Version:	13.0 (Queens)	CC:	dcadzow, pkilambi, srevivo, ssmolyak, stesmith
Target Milestone:	z10	Keywords:	Triaged
Target Release:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-11-07 20:39:23 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sasha Smolyak 2018-02-01 11:59:23 UTC

Description of problem:
There are no cpu measures for instance in undercloud.

(undercloud) [stack@undercloud-0 gnocchi_auto]$ gnocchi resource show ef87ce85-a65a-4027-97a3-14cf1e51987e --type instance
+-----------------------+--------------------------------------------------------------------------------+
| Field                 | Value                                                                          |
+-----------------------+--------------------------------------------------------------------------------+
| created_by_project_id | 7eece16824194a24bdb8a5e50e8729c5                                               |
| created_by_user_id    | 9d15466610a74baea8a72ec8f5d80bcb                                               |
| creator               | 9d15466610a74baea8a72ec8f5d80bcb:7eece16824194a24bdb8a5e50e8729c5              |
| display_name          | controller-0                                                                   |
| ended_at              | None                                                                           |
| flavor_id             | 58034f59-65bb-442f-8a2c-92cdb176f02a                                           |
| flavor_name           | controller                                                                     |
| host                  | undercloud-0.redhat.local                                                      |
| id                    | ef87ce85-a65a-4027-97a3-14cf1e51987e                                           |
| image_ref             | ff7983c0-5af3-4374-87a3-ebfaf116b1e6                                           |
| metrics               | compute.instance.booting.time: 7ee89721-6409-447a-8488-0f92b827fd78            |
|                       | cpu.delta: da78f63a-fa1e-4344-b828-62eb5251e047                                |
|                       | cpu: 591b8d15-6cc6-4583-bd50-21d601e5bf7c                                      |
|                       | cpu_l3_cache: 5214225a-515f-48c0-8e3b-6389cfc44c45                             |
|                       | cpu_util: 20daab47-c871-44e9-a72e-605c3a9fe1a6                                 |
|                       | disk.allocation: c8b2ea75-2254-456c-840d-c002be569c06                          |
|                       | disk.capacity: d8905189-9ce5-47ac-b508-958c3371988d                            |
|                       | disk.ephemeral.size: adaec51a-f0d9-477a-8958-3fc8f2dad6e9                      |
|                       | disk.iops: dc8303f6-8e6b-4013-92aa-068a4a9191bb                                |
|                       | disk.latency: d1a4eab4-03fd-401f-b935-f32915cf4c1b                             |
|                       | disk.read.bytes.rate: deb506e7-58ff-48e4-8182-95e6dacc0527                     |
|                       | disk.read.bytes: 1d15b5a4-937b-4ff3-b24b-da6a6264d5f6                          |
|                       | disk.read.requests.rate: 0631388e-e2cc-4e80-b81e-1fc73fa74491                  |
|                       | disk.read.requests: 4ca15082-68ae-4787-b7d0-35fec1c478ff                       |
|                       | disk.root.size: dee1895c-a7fe-48b1-8152-504f6547e5a4                           |
|                       | disk.usage: 2752c397-7e88-49b9-a117-1163fe4f9136                               |
|                       | disk.write.bytes.rate: d78a2047-bdde-4d93-a7cf-f7bf67a5cf27                    |
|                       | disk.write.bytes: 524aae07-c260-4483-8dc8-4614ac825a23                         |
|                       | disk.write.requests.rate: 157eacde-af66-4d7e-9ed0-8f7a5d48f48f                 |
|                       | disk.write.requests: 2f6db038-da67-45ff-8e9a-ced68b9db28a                      |
|                       | hardware.cpu.util: d8e75015-1c3b-4b30-b33c-73f5b2d29a82                        |
|                       | hardware.memory.buffer: 8a2ce1cc-8ec9-4113-8df8-afdb6f18d914                   |
|                       | hardware.memory.cached: 793519bb-2668-4835-ab8b-d85903575ef5                   |
|                       | hardware.memory.swap.avail: c7f1d089-77a1-4d85-a7c5-7fb308b209c7               |
|                       | hardware.memory.swap.total: 05fbe99a-232c-4ccb-a615-b4b8a8b540e2               |
|                       | hardware.memory.total: 98d869f6-6199-4b9e-8294-75d988696ad3                    |
|                       | hardware.memory.used: c539d760-55c3-4619-b93b-2cf0ea7db7e9                     |
|                       | hardware.network.ip.incoming.datagrams: 2a35032c-be8e-462c-9b6b-03e3bafd7a42   |
|                       | hardware.network.ip.outgoing.datagrams: 15b69843-8da7-4a75-b05a-297fc1d03fc1   |
|                       | hardware.system_stats.io.incoming.blocks: 12aaf2bf-3d07-4ceb-b9d4-192d5c90ad0d |
|                       | hardware.system_stats.io.outgoing.blocks: 7f344df3-8551-449f-a550-16106c0ad67a |
|                       | memory.bandwidth.local: 06ef9047-eeac-4a40-87f6-49130155cebc                   |
|                       | memory.bandwidth.total: 3478e9be-6dbf-4720-8498-ca080fd15238                   |
|                       | memory.resident: f554b760-3952-4d19-b489-93de8bd63788                          |
|                       | memory.swap.in: 0038fbb3-6e43-4922-9c62-b0b6896b9c9f                           |
|                       | memory.swap.out: 8391b90f-bbe5-4b8d-99fe-cd15c833e943                          |
|                       | memory.usage: 4defb5f8-4b9c-4cc8-a9c7-1a26e616706c                             |
|                       | memory: 1393d5fe-2639-42bd-8bde-5a3af217d40c                                   |
|                       | perf.cache.misses: 446aa89f-e623-4bb6-a901-cf6a4d32bd66                        |
|                       | perf.cache.references: 8badcbca-db1e-42ef-9030-af90fc3b4512                    |
|                       | perf.cpu.cycles: d6698e5f-1a52-47c3-bdf1-994da6c581f4                          |
|                       | perf.instructions: bdabb1b2-8d9f-4710-aa43-2bcfae6fe115                        |
|                       | vcpus: e2bfe9ad-e5a9-448a-9220-47077f077d6d                                    |
| original_resource_id  | ef87ce85-a65a-4027-97a3-14cf1e51987e                                           |
| project_id            | 213d93978fd6477fba6c7b48f8cab59e                                               |
| revision_end          | None                                                                           |
| revision_start        | 2018-01-28T17:12:53.293037+00:00                                               |
| server_group          | None                                                                           |
| started_at            | 2018-01-28T17:12:53.293019+00:00                                               |
| type                  | instance                                                                       |
| user_id               | 1243ff307d924b9693534f6bd64da70a                                               |
+-----------------------+--------------------------------------------------------------------------------+
(undercloud) [stack@undercloud-0 gnocchi_auto]$ gnocchi measures show 591b8d15-6cc6-4583-bd50-21d601e5bf7c


Output is empty, the instance is working for 2 days in a row. The log shows errors for this instance:
(undercloud) [stack@undercloud-0 gnocchi_auto]$ sudo tail -n 50 /var/log/ceilometer/agent-notification.log
2018-02-01 06:47:52.680 2549 ERROR ceilometer.publisher.gnocchi [-] Resource 47350f5f-7731-4b1a-a251-61f63a75a70e does not exist (HTTP 404): ResourceNotFound: Resource 47350f5f-7731-4b1a-a251-61f63a75a70e does not exist (HTTP 404)
2018-02-01 06:47:52.685 2549 ERROR ceilometer.publisher.gnocchi [-] Resource 47350f5f-7731-4b1a-a251-61f63a75a70e does not exist (HTTP 404): ResourceNotFound: Resource 47350f5f-7731-4b1a-a251-61f63a75a70e does not exist (HTTP 404)
2018-02-01 06:47:52.703 2549 ERROR ceilometer.publisher.gnocchi [-] Resource 64298001-ee8a-439b-9080-745d77560d41 does not exist (HTTP 404): ResourceNotFound: Resource 64298001-ee8a-439b-9080-745d77560d41 does not exist (HTTP 404)
2018-02-01 06:47:52.709 2549 ERROR ceilometer.publisher.gnocchi [-] Resource 64298001-ee8a-439b-9080-745d77560d41 does not exist (HTTP 404): ResourceNotFound: Resource 64298001-ee8a-439b-9080-745d77560d41 does not exist (HTTP 404)
2018-02-01 06:47:52.728 2549 ERROR ceilometer.publisher.gnocchi [-] Resource 9d6c2cf9-0b93-4781-ba28-49fe21c2b256 does not exist (HTTP 404): ResourceNotFound: Resource 9d6c2cf9-0b93-4781-ba28-49fe21c2b256 does not exist (HTTP 404)
2018-02-01 06:47:52.734 2549 ERROR ceilometer.publisher.gnocchi [-] Resource 9d6c2cf9-0b93-4781-ba28-49fe21c2b256 does not exist (HTTP 404): ResourceNotFound: Resource 9d6c2cf9-0b93-4781-ba28-49fe21c2b256 does not exist (HTTP 404)
2018-02-01 06:47:52.755 2549 ERROR ceilometer.publisher.gnocchi [-] Resource 29cd6a5d-0e60-4529-b773-131b52ffd34b does not exist (HTTP 404): ResourceNotFound: Resource 29cd6a5d-0e60-4529-b773-131b52ffd34b does not exist (HTTP 404)
2018-02-01 06:47:52.758 2549 ERROR ceilometer.publisher.gnocchi [-] Resource 29cd6a5d-0e60-4529-b773-131b52ffd34b does not exist (HTTP 404): ResourceNotFound: Resource 29cd6a5d-0e60-4529-b773-131b52ffd34b does not exist (HTTP 404)
2018-02-01 06:47:52.786 2549 ERROR ceilometer.publisher.gnocchi [-] Resource 67cccc43-08c3-4131-a6dd-28f1db920006 does not exist (HTTP 404): ResourceNotFound: Resource 67cccc43-08c3-4131-a6dd-28f1db920006 does not exist (HTTP 404)
2018-02-01 06:47:52.789 2549 ERROR ceilometer.publisher.gnocchi [-] Resource 67cccc43-08c3-4131-a6dd-28f1db920006 does not exist (HTTP 404): ResourceNotFound: Resource 67cccc43-08c3-4131-a6dd-28f1db920006 does not exist (HTTP 404)
2018-02-01 06:47:52.820 2549 ERROR ceilometer.publisher.gnocchi [-] Resource cecf52e9-2cdd-4a44-a2e5-541f404ec352 does not exist (HTTP 404): ResourceNotFound: Resource cecf52e9-2cdd-4a44-a2e5-541f404ec352 does not exist (HTTP 404)
2018-02-01 06:47:52.823 2549 ERROR ceilometer.publisher.gnocchi [-] Resource cecf52e9-2cdd-4a44-a2e5-541f404ec352 does not exist (HTTP 404): ResourceNotFound: Resource cecf52e9-2cdd-4a44-a2e5-541f404ec352 does not exist (HTTP 404)
2018-02-01 06:47:52.851 2549 ERROR ceilometer.publisher.gnocchi [-] Resource ef87ce85-a65a-4027-97a3-14cf1e51987e does not exist (HTTP 404): ResourceNotFound: Resource ef87ce85-a65a-4027-97a3-14cf1e51987e does not exist (HTTP 404)
2018-02-01 06:47:52.858 2549 ERROR ceilometer.publisher.gnocchi [-] Resource ef87ce85-a65a-4027-97a3-14cf1e51987e does not exist (HTTP 404): ResourceNotFound: Resource ef87ce85-a65a-4027-97a3-14cf1e51987e does not exist (HTTP 404)


Version-Release number of selected component (if applicable):
openstack-ceilometer-common-9.0.1-0.20180103125229.342feaa.el7ost.noarch
openstack-ceilometer-polling-9.0.1-0.20180103125229.342feaa.el7ost.noarch
python-ceilometer-9.0.1-0.20180103125229.342feaa.el7ost.noarch
python-ceilometermiddleware-1.1.0-0.20170811152243.0cea730.el7ost.noarch
python-ceilometerclient-2.9.0-0.20170811140520.3b4e35a.el7ost.noarch
openstack-ceilometer-notification-9.0.1-0.20180103125229.342feaa.el7ost.noarch
openstack-ceilometer-central-9.0.1-0.20180103125229.342feaa.el7ost.noarch
puppet-ceilometer-12.2.1-0.20171223022736.f562bff.el7ost.noarch


How reproducible:
100%

Steps to Reproduce:
1. Deploy openstack with enable_telemetry=true
2. Observe undercloud and watch cpu measures for controller-0
3.

Actual results:
No measures

Expected results:
There are supposed to be measures of cpu

Additional info:

Comment 2 Julien Danjou 2018-02-14 17:07:53 UTC

If controller-0 is a physical host (Ironic I guess) and not a physical host, the cpu metric (reported by libvirt) will never have any data since it's not a VM. It might be created as an instance because it's reported by Nova, but it's not AFAIU.

This has never worked and will never work.

Comment 6 Sasha Smolyak 2018-02-28 10:52:37 UTC

Need to add to the undercloud docs explanation about measures:

Why don’t I see cpu.*, memory.*, etc measures for undercloud nodes, although they are present in the list of resource metrics?

Undercloud nodes might be listed as instances in Gnocchi because they are managed by Nova, making Ceilometer create those resources as instances, with all the associated metrics. However, as they are not managed by Nova/libvirt, those resources cannot be polled for CPU or memory usage via libvirt as done for the overcloud instances.
Therefore they are polled via SNMP: you can check hardware.cpu.util and others hardware.* metrics for undercloud nodes, that will give you the desired measures.

Comment 14 Stephen 2019-11-07 20:39:23 UTC

Duplicate of BZ 1659453. Closing this BZ.

*** This bug has been marked as a duplicate of bug 1659453 ***