Bug 1884478

Summary:

Collectd process plugin settings not configured when using THT

Product:

Red Hat OpenStack

Reporter:

NaveenRaj Navaratna Raj <nnavarat>

Component:

puppet-collectd

Assignee:

Martin Magr <mmagr>

Status:

CLOSED ERRATA

QA Contact:

Leonid Natapov <lnatapov>

Severity:

low

Docs Contact:

Priority:

low

Version:

13.0 (Queens)

CC:

fdaencar, jbadiapa, jjoyce, jschluet, lmadsen, manrodri, mburns, mmagr, mrunge, rmccabe, slinaber, tvignaud

Target Milestone:

z14

Keywords:

Triaged, ZStream

Target Release:

13.0 (Queens)

Hardware:

Unspecified

OS:

Linux

Whiteboard:

Fixed In Version:

puppet-collectd-10.1.0-6.el7ost

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2021-03-18 13:08:47 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
sosreport from a node where the changes are not applied	none

Comment 1 NaveenRaj Navaratna Raj 2020-10-02 07:19:56 UTC

Created attachment 1718382 [details]
sosreport from a node where the changes are not applied

Comment 2 NaveenRaj Navaratna Raj 2020-10-02 07:21:44 UTC

Created attachment 1718383 [details]
templates

Comment 4 Manuel Rodriguez 2020-10-12 16:34:31 UTC

Just to add some context about this problem, I'm setting up collectd templates in my OSP13 deployment to gather metrics from certain plugins and send this data to kafka brokers. 

The problem I'm facing is that collectd processes plugin is not configured when passing parameters via THT, I'm passing these parameters[1] and when I update the stack, I do not see any errors, stack is updated, but the resulting file /var/lib/config-data/puppet-generated/collectd/etc/collectd.d/10-processes.conf does not contain any content of the processes I specified. 

On the other hand I see the extraconfig.json gets populated with hieradata[2]

I managed to have a working configuration, with the THT parameters I passed, but I had to apply this upstream patch[3] from 6 months ago that is not present in collectd puppet modules in my servers.

I did a manual change in the puppet modules to tests in one server, then I update the stack and this time I got a collectd config file with the information I provided. The resulting configuration is not in the default file /var/lib/config-data/puppet-generated/collectd/etc/collectd.d/10-processes.conf, but a new file gets created in /var/lib/config-data/puppet-generated/collectd/etc/collectd.d/processes_config.conf[4]


Thanks,

(undercloud) [stack@slq-nfdv1-001 qa-pod1-osp13]$ rpm -qa | grep tripleo
openstack-tripleo-validations-8.5.0-4.el7ost.noarch
openstack-tripleo-common-8.7.1-20.el7ost.noarch
openstack-tripleo-heat-templates-8.4.1-58.1.el7ost.noarch
puppet-tripleo-8.5.1-14.el7ost.noarch
openstack-tripleo-image-elements-8.0.3-1.el7ost.noarch
ansible-tripleo-ipsec-8.1.1-0.20190513184007.7eb892c.el7ost.noarch
openstack-tripleo-ui-8.3.2-3.el7ost.noarch
python-tripleoclient-9.3.1-7.el7ost.noarch
openstack-tripleo-common-containers-8.7.1-20.el7ost.noarch
openstack-tripleo-puppet-elements-8.1.1-2.el7ost.noarch
(undercloud) [stack@slq-nfdv1-001 qa-pod1-osp13]$ rpm -qa | grep collectd
puppet-collectd-10.1.0-4.el7ost.noarch



[1]
  ExtraConfig:
    collectd::plugin::processes::processes:
      - {"name": "dockerd-current"}
    collectd::plugin::processes::process_matches:
      - {"name": "collectd", "regex": "/usr/sbin/collectd"}


[2]
    "collectd::plugin::processes::process_matches": [
        {
            "name": "collectd",
            "regex": "/usr/sbin/collectd"
        }
    ],
    "collectd::plugin::processes::processes": [
        {
            "name": "dockerd-current"
        }
    ],


[3]
https://github.com/voxpupuli/puppet-collectd/commit/f1d7ee5c122d7b6168250639cdcd2459494c2b51#diff-cd99d28d9e42cf7e1b6f2b7c35a533b5


[4]
[root@slq-nfcp1-003 tripleo-config]# cat /var/lib/config-data/puppet-generated/collectd/etc/collectd.d/processes_config.conf
<Plugin processes>
  <Process "dockerd-current">
  </Process>
  <ProcessMatch "collectd" "/usr/sbin/collectd">
  </ProcessMatch>
</Plugin>

Comment 5 Matthias Runge 2020-10-14 15:24:42 UTC

Can you please output ps -aux from inside the collectd container?

Comment 7 Manuel Rodriguez 2020-10-14 16:34:35 UTC

I ran the following in a compute node with the collectd puppet patch inside the container:

[root@slq-nfcp1-003 ~]# docker exec -ti collectd bash
()[root@slq-nfcp1-003 /]# ps aux > /tmp/ps_aux_from_container.txt 

See full output in the following link:

http://paste.openstack.org/show/fzZMvgfYY7qJJzVLnTkM/

Thanks,

Comment 8 Matthias Runge 2020-10-14 19:27:30 UTC

Thank you.

Comment 11 Martin Magr 2020-12-08 13:38:33 UTC

Reproduced. I'm not sure if the package provided config files removal process is the cause here, since I don't see it in a catalog application in my case:

Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.conf]/content) content changed '{md5}2d446135c74bf830a470c6c82e2b018c' to '{md5}3e549afd22d9ee0888b8eed8c1951e43'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.conf]/owner) owner changed 'collectd' to 'root'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.conf]/mode) mode changed '0644' to '0640'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.d]/mode) mode changed '0755' to '0750'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-cpu.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-interface.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-load.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-memory.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-syslog.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/apache.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/hugepages.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ipmi.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/libvirt.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/mysql.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ovs-events.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ovs-stats.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ping.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/rdt.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/snmp.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Logfile/Collectd::Plugin[logfile]/File[logfile.load]/ensure) defined content as '{md5}fe28d10cb452a9016530531b00a35d48'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Amqp1/Collectd::Plugin[amqp1]/File[amqp1.load]/ensure) defined content as '{md5}b58345bbf0d08eecbe75627c48406915'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Disk/Collectd::Plugin[disk]/File[disk.load]/ensure) defined content as '{md5}bcd29dd5e41c48e589831845da1eb648'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Interface/Collectd::Plugin[interface]/File[interface.load]/ensure) defined content as '{md5}ae0f91f8ea544aac9d0389a093765c0d'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Load/Collectd::Plugin[load]/File[load.load]/ensure) defined content as '{md5}98cd4523a355f4ea6a49c381408380e3'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Memory/Collectd::Plugin[memory]/File[memory.load]/ensure) defined content as '{md5}e303246961970ca7aaad379146fa0a4f'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Processes/Collectd::Plugin[processes]/File[processes.load]/ensure) defined content as '{md5}f56739142fda8aad962dd71f844936ae'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Tcpconns/Collectd::Plugin[tcpconns]/File[tcpconns.load]/ensure) defined content as '{md5}a9068ca311647205817c0ad318dd3677'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Unixsock/Collectd::Plugin[unixsock]/File[unixsock.load]/ensure) defined content as '{md5}757b85d78e9a1796468c0ed2c094d761'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Memcached/Collectd::Plugin[memcached]/File[memcached.load]/ensure) defined content as '{md5}7b0342d8999f6b22ebd297bb266a6ec3'
Dec  8 10:43:52 controller-0 puppet-user[14]: Applied catalog in 0.58 seconds

To me it seems that for some reason the appropriate concat resource is not being applied, but will of course try the patch from comment #4. cOntinuing with investigation.

Comment 12 Martin Magr 2020-12-08 14:42:10 UTC

So when patch is applied puppet starts applying the concat resource:

Dec  8 14:03:29 controller-0 journal: Notice: Compiled catalog for controller-0.redhat.local in environment production in 3.91 seconds
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.conf]/content) content changed '{md5}2d446135c74bf830a470c6c82e2b018c' to '{md5}3e549afd22d9ee0888b8eed8c1951e43'
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.conf]/owner) owner changed 'collectd' to 'root'
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.conf]/mode) mode changed '0644' to '0640'
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.d]/owner) owner changed 'collectd' to 'root'
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.d]/mode) mode changed '0755' to '0750'
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-cpu.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-interface.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-load.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-memory.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-syslog.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/apache.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/hugepages.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ipmi.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/libvirt.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/mysql.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ovs-events.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ovs-stats.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ping.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/rdt.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/snmp.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Logfile/Collectd::Plugin[logfile]/File[logfile.load]/ensure) defined content as '{md5}fe28d10cb452a9016530531b00a35d48'
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Amqp1/Collectd::Plugin[amqp1]/File[amqp1.load]/ensure) defined content as '{md5}b58345bbf0d08eecbe75627c48406915'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Disk/Collectd::Plugin[disk]/File[disk.load]/ensure) defined content as '{md5}bcd29dd5e41c48e589831845da1eb648'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Interface/Collectd::Plugin[interface]/File[interface.load]/ensure) defined content as '{md5}ae0f91f8ea544aac9d0389a093765c0d'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Load/Collectd::Plugin[load]/File[load.load]/ensure) defined content as '{md5}98cd4523a355f4ea6a49c381408380e3'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Memory/Collectd::Plugin[memory]/File[memory.load]/ensure) defined content as '{md5}e303246961970ca7aaad379146fa0a4f'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Processes/Collectd::Plugin[processes]/File[processes.load]/ensure) defined content as '{md5}f56739142fda8aad962dd71f844936ae'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Processes/Concat[/etc/collectd.d/processes_config.conf]/File[/etc/collectd.d/processes_config.conf]/ensure) defined content as '{md5}f6110e5ce088a1fc39c40239c20275f0'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Tcpconns/Collectd::Plugin[tcpconns]/File[tcpconns.load]/ensure) defined content as '{md5}a9068ca311647205817c0ad318dd3677'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Unixsock/Collectd::Plugin[unixsock]/File[unixsock.load]/ensure) defined content as '{md5}757b85d78e9a1796468c0ed2c094d761'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Memcached/Collectd::Plugin[memcached]/File[memcached.load]/ensure) defined content as '{md5}7b0342d8999f6b22ebd297bb266a6ec3'
Dec  8 14:03:31 controller-0 puppet-user[14]: Applied catalog in 1.18 seconds

That is interesting behaviour, since there is still missing the packaging config removal in the catalog application (something like "(/Stage[main]/Collectd::Config/File[/etc/collectd.d/processes-config.conf]/ensure) removed"). Nevertheless I'm going to backport the patch to have this fix sooner rather than later.

Comment 22 errata-xmlrpc 2021-03-18 13:08:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 13.0 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0932