Bug 1358005
| Summary: | Invalid message signature warnings in collector log | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | nalmond |
| Component: | openstack-ceilometer | Assignee: | Mehdi ABAAKOUK <mabaakou> |
| Status: | CLOSED NOTABUG | QA Contact: | Yurii Prokulevych <yprokule> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.0 (Kilo) | CC: | fwissing, jruzicka, mabaakou, nalmond, pkilambi, srevivo |
| Target Milestone: | --- | Keywords: | Triaged, ZStream |
| Target Release: | 8.0 (Liberty) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-11-28 11:58:42 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
These messages continue to appear even after setting the telemetry secret to '' on each node: [publisher] # # Options defined in ceilometer.publisher.utils # # Secret value for signing metering messages. (string value) # Deprecated group/name - [DEFAULT]/metering_secret # Deprecated group/name - [publisher_rpc]/metering_secret telemetry_secret= I can't access to the case on the customer portail and on collab-shell. When the secret is empty the whole signature checking and generation are skipped. So the 'message_signature' should be empty too, and that doesn't seem your case as the example show. Also telemetry_secret have to be in sync everywhere and all ceilometer-XXX restart after the change. Does the invalid signature occur on all samples, or only on a particular subset ? On both nodes the ceilometer.conf looks incorrect and unloadable by ceilometer
sosreport have hidden a string just after telemetry_secret that is not prefixed by an option name, this is dubious:
#metering_secret=change this or be hacked
telemetry_secret=
*********
#
# Options defined in ceilometer.publisher.messaging
#
If this ***** is really garbage, so I don't think that ceilometer have not been restart after the telemetry_secret have been emptied, a service can't start with this kind of misconfiguration.
Also all samples in logs have a signature, so telemetry_secret (or one of its deprecation version) was not empty when their was generated.
From agent-notification logs, the last restart was at 2016-07-15 12:18 and a deprecation for metering_secret in publisher section is mentioned, while the configuration doesn't have it.
This make me think that services were not restarted after the configuration change.
agent-notification.log-20160717.gz:2016-07-15 12:18:58.488 12095 WARNING oslo_config.cfg [-] Option "metering_secret" from group "publisher" is deprecated. Use option "telemetry_secret" from group "publisher".
agent-notification.log-20160717.gz:2016-07-15 12:18:58.488 12095 DEBUG oslo_service.service [-] publisher.telemetry_secret = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
I see the same thing on collector.log files, the [publisher]/metering_secret was set too.
collector.log-20160718.gz:2016-07-15 12:18:44.884 11607 WARNING oslo_config.cfg [-] Option "metering_secret" from group "publisher" is deprecated. Use option "telemetry_secret" from group "publisher".
collector.log-20160718.gz:2016-07-15 12:18:44.884 11607 DEBUG oslo_service.service [-] publisher.telemetry_secret = **** log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2237
On compute node I don't even see any restart in log files, the older line is 2016-06-19, so I don't think serives have been restart here too.
For me the next step is to fix the wrong ceilometer.conf (or understand what is this the garbage line) and restart ceilometer services after this change and see if that still occurs.
If that still doesn't work, a new sosreport with everything properly configured and restarted will be helpful.
I close this, this is not a bug but a configuration issue and customer case is closed. |
Description of problem: Observe frequent 'WARNING ceilometer.dispatcher.database [-] message signature invalid, discarding message:' messages in /var/log/ceilometer/collector.log on the controller. Version-Release number of selected component (if applicable): openstack-ceilometer-collector-5.0.2-2.el7ost openstack-ceilometer-compute-5.0.2-2.el7ost How reproducible: Always Steps to Reproduce: 1. Set up mongo on a separate node 2. Configure ceilometer publisher on controller and compute nodes to use a telemetry secret 3. Observe that sample are being collected Actual results: Warnings appear in collector log Expected results: Warnings do not appear Additional info: The system time on each node is synchronized. The secret hash on each node is identical. example full message with uuids redacted: 2016-07-18 15:48:03.433 21494 WARNING ceilometer.dispatcher.database [-] message signature invalid, discarding message: {u'counter_name': u'image', u'user_id': None, u'resource_id': u'<id>', u'timestamp': u'2016-07-18T19:48:03.169978', u'message_signature': u'<uuid>', u'resource_metadata': {u'status': u'active', u'name': u'image-<id>', u'deleted': False, u'container_format': u'bare', u'created_at': u'2016-03-02T19:00:24.000000', u'disk_format': u'raw', u'updated_at': u'2016-03-02T19:00:48.000000', u'properties': {u'os_version': u'alt1', u'architecture': u'value3', u'os_distro': u'value2'}, u'protected': False, u'checksum': u'<checksum>', u'min_disk': 0, u'is_public': False, u'deleted_at': None, u'min_ram': 0, u'size': 1024}, u'source': u'openstack', u'counter_unit': u'image', u'counter_volume': 1, u'project_id': u'<id>', u'message_id': u'<id>', u'counter_type': u'gauge'}