Description of problem: Undercloud instances don't have hardware.* metrics enabled in polling.yaml Version-Release number of selected component (if applicable): Performed update from puddle of 2018-05-15.2 to puppet-gnocchi-12.4.0-0.20180329032858.5dfa350.el7ost.noarch gnocchi-metricd-4.2.3-2.el7ost.noarch python-gnocchi-4.2.3-2.el7ost.noarch gnocchi-api-4.2.3-2.el7ost.noarch python2-gnocchiclient-7.0.1-1.el7ost.noarch gnocchi-statsd-4.2.3-2.el7ost.noarch gnocchi-common-4.2.3-2.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Install 2018-05-15.2 2. Update to 2018-05-29.2 3. Check /etc/ceilometer/polling.yaml in undercloud machine Actual results: No hardware.* metrics hence no cpu or memory measures of undercloud machine collected. Expected results: hardware* metrics present in the file. Additional info: First seen it after update, but I think it's worth checking whether the metrics disappeared from the polling.yaml in the build at some point. I do know that after upgrade from 12 they were there.
My current investigation leads to a snmpd issue or configuration: * /etc/ceilometer/polling.yaml have hardware metrics * We see 3 instances instead of 8 in /var/log/ceilometer/agent-notification.log * The 3 ceph nodes have the hardware.* metrics, but not the compute and controller nodes * /var/log/ceilometer/central.log is full of: 2018-06-04 08:48:23.838 2952 WARNING ceilometer.hardware.pollsters.generic [-] inspector call failed for hardware.cpu.util host 192.168.24.11: An error occurred, oids ['1.3.6.1.4.1.2021.11.9.0'], host 192.168.24.11, No SNMP response received before timeout: SNMPException: An error occurred, oids ['1.3.6.1.4.1.2021.11.9.0'], host 192.168.24.11, No SNMP response received before timeout 2018-06-04 08:48:30.033 2952 WARNING ceilometer.hardware.pollsters.generic [-] inspector call failed for hardware.cpu.util host 192.168.24.19: An error occurred, oids ['1.3.6.1.4.1.2021.11.9.0'], host 192.168.24.19, No SNMP response received before timeout: SNMPException: An error occurred, oids ['1.3.6.1.4.1.2021.11.9.0'], host 192.168.24.19, No SNMP response received before timeout This suggest that snmpd on the controller and compute nodes is not answering to ceilometer-agent-central.
This is a firewall issue. The firewall rules created on controller/compute nodes doesn't use the correct subnet (it's 172.17.1.0/24 instead of 92.168.24.0/24 : * Broken node: [root@compute-1 ~]# iptables -nL |grep 161 ACCEPT udp -- 172.17.1.0/24 0.0.0.0/0 multiport dports 161 state NEW /* 124 snmp ipv4 */ * Working node: [root@ceph-0 snmp]# iptables -nL |grep 161 ACCEPT udp -- 192.168.24.0/24 0.0.0.0/0 multiport dports 161 state NEW /* 124 snmp ipv4 */
This have been broken by: https://github.com/openstack/tripleo-heat-templates/commit/43155ed1462a8e27c9efdbb345bfc5832c50bd2f
The workaround works OK, now it's important to add this to documentation for installation/update/upgrade
I confirm the regression have been backported to OSP10, so I have cloned it for OSP10 #1622839
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2574