Description of problem: gnocchi not able to show the instance measures hence aodh alarm is also showing "insufficient data " Version-Release number of selected component (if applicable): RHEL OSP 10 ]# awk '/gnocchi/ {print $1}' installed-rpms openstack-gnocchi-api-3.0.2-1.el7ost.noarch openstack-gnocchi-carbonara-3.0.2-1.el7ost.noarch openstack-gnocchi-common-3.0.2-1.el7ost.noarch openstack-gnocchi-indexer-sqlalchemy-3.0.2-1.el7ost.noarch openstack-gnocchi-metricd-3.0.2-1.el7ost.noarch openstack-gnocchi-statsd-3.0.2-1.el7ost.noarch puppet-gnocchi-9.4.1-1.el7ost.noarch python-gnocchi-3.0.2-1.el7ost.noarch python-gnocchiclient-2.6.0-1.el7ost.noarch How reproducible: Everytime for customer. Steps to Reproduce: 1. Spawn an instance. 2. Check gnocchi measures but nothing is shown in them. 3. Created aodh alarm using same granularity as of metric but alarm also remains in insufficent data state. Actual results: gnocchi not showing anything in measures for instance. Expected results: gnocchi should showing something in measures for instance. Additional info: [stack@openstack-director templates]$ nova list +--------------------------------------+---------------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+---------------+--------+------------+-------------+---------------------+ | 8ba46760-42b8-4513-bf79-c571ef17125d | testinstance1 | ACTIVE | - | Running | swadmin=10.10.10.11 | +--------------------------------------+---------------+--------+------------+-------------+---------------------+ [stack@openstack-director templates]$ gnocchi resource list --type instance| grep '8ba46760-42b8-4513-bf79-c571ef17125d' | 8ba46760-42b8-4513-bf79-c571ef17125d | instance | 007c4a4c5c2848b897c3b8b52b317242 | cce4d171efc340cf89847aa0cea615ef | 8ba46760-42b8-4513-bf79-c571ef17125d | 2017-03-31T10:01:57.547510+00:00 | None | 2017-03-31T10:02:03.863194+00:00 | None | [stack@openstack-director templates]$ gnocchi resource show --type instance 8ba46760-42b8-4513-bf79-c571ef17125d +-----------------------+----------------------------------------------------------------+ | Field | Value | +-----------------------+----------------------------------------------------------------+ | created_by_project_id | d131ae1ac428476dbd2648257f0c9a8b | | created_by_user_id | adddeb0570144747bd6e4e34b40f324a | | display_name | testinstance1 | | ended_at | None | | flavor_id | bb26b8e9-a3d3-4c40-a897-273b7934306b | | host | compute.overcloud-compute-0.localdomain | | id | 8ba46760-42b8-4513-bf79-c571ef17125d | | image_ref | None | | metrics | cpu.delta: 5ab559da-95c3-4409-8764-bcdf1bb5a1ef | | | cpu: 61dcbbf7-4268-4aff-88b5-e24186f20de8 | | | cpu_util: c084385e-408a-4eeb-8783-a032129e7702 | | | disk.allocation: a61310c8-3b27-4976-bced-42214412d5f7 | | | disk.capacity: 3c91d2ba-1054-4f30-8534-5581237ac770 | | | disk.ephemeral.size: cc00c585-ab2f-4651-88f9-090f7a6e804b | | | disk.iops: f13fedf0-6e3c-4e7a-ac2a-6a9aa07c08fa | | | disk.latency: baa9a8a1-9a5a-4d4e-82ce-24eee92d31eb | | | disk.read.bytes.rate: ea4b775a-f84c-4fa0-a13c-049345771761 | | | disk.read.bytes: b83a216c-9c55-4c34-a9f8-cd148c6e0dfa | | | disk.read.requests.rate: 575ab558-6eb8-450b-a613-a9ae701c0d03 | | | disk.read.requests: bb773bd0-336e-4674-a424-7300295c4f93 | | | disk.root.size: 44a101a0-eca7-439f-a576-03cc99de5b5f | | | disk.usage: f264f3b8-08a1-4b5c-96b5-29a8ca13e8e4 | | | disk.write.bytes.rate: fa7f5d3c-96d7-42ca-b8e3-255ddfe813cb | | | disk.write.bytes: 4d67d247-6237-42af-ac1a-8431eb47317c | | | disk.write.requests.rate: 06fcf56c-9da1-4a02-874a-a7471121070b | | | disk.write.requests: fb6bc7a8-ff24-4109-b1e2-338131b82beb | | | instance: 5f0bba57-0912-407f-87f4-5d3cfd58ff6e | | | memory.resident: 219f564b-20f5-41ac-a5f1-3f81775305d9 | | | memory.usage: 84b02c20-4f3d-4faa-bf4c-ccfd08b65d82 | | | memory: 840a6dce-9fc2-4889-97c7-2ffc900815e8 | | | vcpus: 0f542352-aa77-4158-a0cd-09562e7fcafd | | original_resource_id | 8ba46760-42b8-4513-bf79-c571ef17125d | | project_id | 007c4a4c5c2848b897c3b8b52b317242 | | revision_end | None | | revision_start | 2017-03-31T10:02:03.863194+00:00 | | server_group | None | | started_at | 2017-03-31T10:01:57.547510+00:00 | | type | instance | | user_id | cce4d171efc340cf89847aa0cea615ef | +-----------------------+----------------------------------------------------------------+ stack@openstack-director templates]$ gnocchi metric show c084385e-408a-4eeb-8783-a032129e7702 +------------------------------------+-----------------------------------------------------------------------+ | Field | Value | +------------------------------------+-----------------------------------------------------------------------+ | archive_policy/aggregation_methods | std, count, 95pct, min, max, sum, median, mean | | archive_policy/back_window | 0 | | archive_policy/definition | - points: 12, granularity: 0:05:00, timespan: 1:00:00 | | | - points: 24, granularity: 1:00:00, timespan: 1 day, 0:00:00 | | | - points: 30, granularity: 1 day, 0:00:00, timespan: 30 days, 0:00:00 | | archive_policy/name | low | | created_by_project_id | d131ae1ac428476dbd2648257f0c9a8b | | created_by_user_id | adddeb0570144747bd6e4e34b40f324a | | id | c084385e-408a-4eeb-8783-a032129e7702 | | name | cpu_util | | resource/created_by_project_id | d131ae1ac428476dbd2648257f0c9a8b | | resource/created_by_user_id | adddeb0570144747bd6e4e34b40f324a | | resource/ended_at | None | | resource/id | 8ba46760-42b8-4513-bf79-c571ef17125d | | resource/original_resource_id | 8ba46760-42b8-4513-bf79-c571ef17125d | | resource/project_id | 007c4a4c5c2848b897c3b8b52b317242 | | resource/revision_end | None | | resource/revision_start | 2017-03-31T10:02:03.863194+00:00 | | resource/started_at | 2017-03-31T10:01:57.547510+00:00 | | resource/type | instance | | resource/user_id | cce4d171efc340cf89847aa0cea615ef | | unit | % | +------------------------------------+-----------------------------------------------------------------------+ aodh alarm remains in "insufficient data" state. $ aodh --debug alarm create --type gnocchi_aggregation_by_resources_threshold --name cputhreshold-2 --description 'GnocchiAggregationByResourceThreshold' --enabled True --alarm-action 'log://' --ok-action 'log://' --insufficient-data-action 'log://' --comparison-operator 'ge' --evaluation-periods 3 --threshold 50.0 --granularity 300 --aggregation-method mean --metric cpu_util --query '{"=": {"id": "8ba46760-42b8-4513-bf79-c571ef17125d"}}' --resource-type instance +---------------------------+-------------------------------------------------------+ | Field | Value | +---------------------------+-------------------------------------------------------+ | aggregation_method | mean | | alarm_actions | [u'log://'] | | alarm_id | 0cd3aa28-304f-4e60-b274-c6939bdfa227 | | comparison_operator | ge | | description | GnocchiAggregationByResourceThreshold | | enabled | True | | evaluation_periods | 3 | | granularity | 300 | | insufficient_data_actions | [u'log://'] | | metric | cpu_util | | name | cputhreshold-2 | | ok_actions | [u'log://'] | | project_id | 007c4a4c5c2848b897c3b8b52b317242 | | query | {"=": {"id": "8ba46760-42b8-4513-bf79-c571ef17125d"}} | | repeat_actions | False | | resource_type | instance | | severity | low | | state | insufficient data | | state_timestamp | 2017-03-31T10:14:18.834284 | | threshold | 50.0 | | time_constraints | [] | | timestamp | 2017-03-31T10:14:18.834284 | | type | gnocchi_aggregation_by_resources_threshold | | user_id | cce4d171efc340cf89847aa0cea615ef | +---------------------------+-------------------------------------------------------+ nothing displayed in gnocchi measures output. [stack@openstack-director templates]$ gnocchi measures show c084385e-408a-4eeb-8783-a032129e7702 [stack@openstack-director templates]$ [stack@openstack-director templates]$ gnocchi status +-----------------------------------------------------+-------+ | Field | Value | +-----------------------------------------------------+-------+ | storage/number of metric having measures to process | 0 | | storage/total number of measures to process | 0 | +-----------------------------------------------------+-------+
Request some help here to identify why gnocchi is not able to collect the measures.
I don't find the the logs on collab shell. Can I also have the logs of ceilometer-collector and ceilometer-central-agent controller nodes and ceilometer-compute-agent on controller nodes ? Or even better the sosreports of the controllers and the compute where the instance run.
I got the files now, but I also need the content of /var/log/ceilometer on the controller and the compute ?
Sorry, I just have found them.
I have found the following backtrace in central.log on compute node, this is a known bug, I mark this one as duplicate. 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager [-] Unable to discover resources: 'ascii' codec can't encode character u'\xe9' in position 37: ordinal not in range(128) 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager Traceback (most recent call last): 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/site-packages/ceilometer/agent/manager.py", line 502, in discover 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager discovered) 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 223, in call 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager return attempt.get(self._wrap_exception) 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/site-packages/ceilometer/coordination.py", line 227, in extract_my_subset 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager if hr.get_node(str(v)) == self._my_id] 2017-03-30 14:20:14.475 748734 ERROR ceilometer.agent.manager UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 37: ordinal not in range(128) *** This bug has been marked as a duplicate of bug 1434413 ***