Bug 1884478 - Collectd process plugin settings not configured when using THT
Summary: Collectd process plugin settings not configured when using THT
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-collectd
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Linux
low
low
Target Milestone: z14
: 13.0 (Queens)
Assignee: Martin Magr
QA Contact: Leonid Natapov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-02 06:56 UTC by NaveenRaj Navaratna Raj
Modified: 2021-03-18 20:39 UTC (History)
12 users (show)

Fixed In Version: puppet-collectd-10.1.0-6.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-18 13:08:47 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
sosreport from a node where the changes are not applied (14.34 MB, application/x-xz)
2020-10-02 07:19 UTC, NaveenRaj Navaratna Raj 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2021:0932 0 None None None 2021-03-18 13:10:06 UTC

Comment 1 NaveenRaj Navaratna Raj 2020-10-02 07:19:56 UTC 
Created attachment 1718382 [details]
sosreport from a node where the changes are not applied
Comment 2 NaveenRaj Navaratna Raj 2020-10-02 07:21:44 UTC 
Created attachment 1718383 [details]
templates
Comment 4 Manuel Rodriguez 2020-10-12 16:34:31 UTC 
Just to add some context about this problem, I'm setting up collectd templates in my OSP13 deployment to gather metrics from certain plugins and send this data to kafka brokers. 

The problem I'm facing is that collectd processes plugin is not configured when passing parameters via THT, I'm passing these parameters[1] and when I update the stack, I do not see any errors, stack is updated, but the resulting file /var/lib/config-data/puppet-generated/collectd/etc/collectd.d/10-processes.conf does not contain any content of the processes I specified. 

On the other hand I see the extraconfig.json gets populated with hieradata[2]

I managed to have a working configuration, with the THT parameters I passed, but I had to apply this upstream patch[3] from 6 months ago that is not present in collectd puppet modules in my servers.

I did a manual change in the puppet modules to tests in one server, then I update the stack and this time I got a collectd config file with the information I provided. The resulting configuration is not in the default file /var/lib/config-data/puppet-generated/collectd/etc/collectd.d/10-processes.conf, but a new file gets created in /var/lib/config-data/puppet-generated/collectd/etc/collectd.d/processes_config.conf[4]


Thanks,

(undercloud) [stack@slq-nfdv1-001 qa-pod1-osp13]$ rpm -qa | grep tripleo
openstack-tripleo-validations-8.5.0-4.el7ost.noarch
openstack-tripleo-common-8.7.1-20.el7ost.noarch
openstack-tripleo-heat-templates-8.4.1-58.1.el7ost.noarch
puppet-tripleo-8.5.1-14.el7ost.noarch
openstack-tripleo-image-elements-8.0.3-1.el7ost.noarch
ansible-tripleo-ipsec-8.1.1-0.20190513184007.7eb892c.el7ost.noarch
openstack-tripleo-ui-8.3.2-3.el7ost.noarch
python-tripleoclient-9.3.1-7.el7ost.noarch
openstack-tripleo-common-containers-8.7.1-20.el7ost.noarch
openstack-tripleo-puppet-elements-8.1.1-2.el7ost.noarch
(undercloud) [stack@slq-nfdv1-001 qa-pod1-osp13]$ rpm -qa | grep collectd
puppet-collectd-10.1.0-4.el7ost.noarch



[1]
  ExtraConfig:
    collectd::plugin::processes::processes:
      - {"name": "dockerd-current"}
    collectd::plugin::processes::process_matches:
      - {"name": "collectd", "regex": "/usr/sbin/collectd"}


[2]
    "collectd::plugin::processes::process_matches": [
        {
            "name": "collectd",
            "regex": "/usr/sbin/collectd"
        }
    ],
    "collectd::plugin::processes::processes": [
        {
            "name": "dockerd-current"
        }
    ],


[3]
https://github.com/voxpupuli/puppet-collectd/commit/f1d7ee5c122d7b6168250639cdcd2459494c2b51#diff-cd99d28d9e42cf7e1b6f2b7c35a533b5


[4]
[root@slq-nfcp1-003 tripleo-config]# cat /var/lib/config-data/puppet-generated/collectd/etc/collectd.d/processes_config.conf
<Plugin processes>
  <Process "dockerd-current">
  </Process>
  <ProcessMatch "collectd" "/usr/sbin/collectd">
  </ProcessMatch>
</Plugin>
Comment 5 Matthias Runge 2020-10-14 15:24:42 UTC 
Can you please output ps -aux from inside the collectd container?
Comment 7 Manuel Rodriguez 2020-10-14 16:34:35 UTC 
I ran the following in a compute node with the collectd puppet patch inside the container:

[root@slq-nfcp1-003 ~]# docker exec -ti collectd bash
()[root@slq-nfcp1-003 /]# ps aux > /tmp/ps_aux_from_container.txt 

See full output in the following link:

http://paste.openstack.org/show/fzZMvgfYY7qJJzVLnTkM/

Thanks,
Comment 8 Matthias Runge 2020-10-14 19:27:30 UTC 
Thank you.
Comment 11 Martin Magr 2020-12-08 13:38:33 UTC 
Reproduced. I'm not sure if the package provided config files removal process is the cause here, since I don't see it in a catalog application in my case:

Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.conf]/content) content changed '{md5}2d446135c74bf830a470c6c82e2b018c' to '{md5}3e549afd22d9ee0888b8eed8c1951e43'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.conf]/owner) owner changed 'collectd' to 'root'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.conf]/mode) mode changed '0644' to '0640'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.d]/mode) mode changed '0755' to '0750'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-cpu.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-interface.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-load.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-memory.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-syslog.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/apache.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/hugepages.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ipmi.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/libvirt.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/mysql.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ovs-events.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ovs-stats.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ping.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/rdt.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/snmp.conf]/ensure) removed
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Logfile/Collectd::Plugin[logfile]/File[logfile.load]/ensure) defined content as '{md5}fe28d10cb452a9016530531b00a35d48'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Amqp1/Collectd::Plugin[amqp1]/File[amqp1.load]/ensure) defined content as '{md5}b58345bbf0d08eecbe75627c48406915'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Disk/Collectd::Plugin[disk]/File[disk.load]/ensure) defined content as '{md5}bcd29dd5e41c48e589831845da1eb648'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Interface/Collectd::Plugin[interface]/File[interface.load]/ensure) defined content as '{md5}ae0f91f8ea544aac9d0389a093765c0d'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Load/Collectd::Plugin[load]/File[load.load]/ensure) defined content as '{md5}98cd4523a355f4ea6a49c381408380e3'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Memory/Collectd::Plugin[memory]/File[memory.load]/ensure) defined content as '{md5}e303246961970ca7aaad379146fa0a4f'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Processes/Collectd::Plugin[processes]/File[processes.load]/ensure) defined content as '{md5}f56739142fda8aad962dd71f844936ae'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Tcpconns/Collectd::Plugin[tcpconns]/File[tcpconns.load]/ensure) defined content as '{md5}a9068ca311647205817c0ad318dd3677'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Unixsock/Collectd::Plugin[unixsock]/File[unixsock.load]/ensure) defined content as '{md5}757b85d78e9a1796468c0ed2c094d761'
Dec  8 10:43:52 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Memcached/Collectd::Plugin[memcached]/File[memcached.load]/ensure) defined content as '{md5}7b0342d8999f6b22ebd297bb266a6ec3'
Dec  8 10:43:52 controller-0 puppet-user[14]: Applied catalog in 0.58 seconds

To me it seems that for some reason the appropriate concat resource is not being applied, but will of course try the patch from comment #4. cOntinuing with investigation.
Comment 12 Martin Magr 2020-12-08 14:42:10 UTC 
So when patch is applied puppet starts applying the concat resource:

Dec  8 14:03:29 controller-0 journal: Notice: Compiled catalog for controller-0.redhat.local in environment production in 3.91 seconds
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.conf]/content) content changed '{md5}2d446135c74bf830a470c6c82e2b018c' to '{md5}3e549afd22d9ee0888b8eed8c1951e43'
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.conf]/owner) owner changed 'collectd' to 'root'
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.conf]/mode) mode changed '0644' to '0640'
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.d]/owner) owner changed 'collectd' to 'root'
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[collectd.d]/mode) mode changed '0755' to '0750'
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-cpu.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-interface.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-load.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-memory.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/90-default-plugins-syslog.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/apache.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/hugepages.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ipmi.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/libvirt.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/mysql.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ovs-events.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ovs-stats.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/ping.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/rdt.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Config/File[/etc/collectd.d/snmp.conf]/ensure) removed
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Logfile/Collectd::Plugin[logfile]/File[logfile.load]/ensure) defined content as '{md5}fe28d10cb452a9016530531b00a35d48'
Dec  8 14:03:30 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Amqp1/Collectd::Plugin[amqp1]/File[amqp1.load]/ensure) defined content as '{md5}b58345bbf0d08eecbe75627c48406915'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Disk/Collectd::Plugin[disk]/File[disk.load]/ensure) defined content as '{md5}bcd29dd5e41c48e589831845da1eb648'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Interface/Collectd::Plugin[interface]/File[interface.load]/ensure) defined content as '{md5}ae0f91f8ea544aac9d0389a093765c0d'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Load/Collectd::Plugin[load]/File[load.load]/ensure) defined content as '{md5}98cd4523a355f4ea6a49c381408380e3'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Memory/Collectd::Plugin[memory]/File[memory.load]/ensure) defined content as '{md5}e303246961970ca7aaad379146fa0a4f'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Processes/Collectd::Plugin[processes]/File[processes.load]/ensure) defined content as '{md5}f56739142fda8aad962dd71f844936ae'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Processes/Concat[/etc/collectd.d/processes_config.conf]/File[/etc/collectd.d/processes_config.conf]/ensure) defined content as '{md5}f6110e5ce088a1fc39c40239c20275f0'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Tcpconns/Collectd::Plugin[tcpconns]/File[tcpconns.load]/ensure) defined content as '{md5}a9068ca311647205817c0ad318dd3677'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Unixsock/Collectd::Plugin[unixsock]/File[unixsock.load]/ensure) defined content as '{md5}757b85d78e9a1796468c0ed2c094d761'
Dec  8 14:03:31 controller-0 puppet-user[14]: (/Stage[main]/Collectd::Plugin::Memcached/Collectd::Plugin[memcached]/File[memcached.load]/ensure) defined content as '{md5}7b0342d8999f6b22ebd297bb266a6ec3'
Dec  8 14:03:31 controller-0 puppet-user[14]: Applied catalog in 1.18 seconds

That is interesting behaviour, since there is still missing the packaging config removal in the catalog application (something like "(/Stage[main]/Collectd::Config/File[/etc/collectd.d/processes-config.conf]/ensure) removed"). Nevertheless I'm going to backport the patch to have this fix sooner rather than later.
Comment 22 errata-xmlrpc 2021-03-18 13:08:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 13.0 bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0932
Note You need to log in before you can comment on or make changes to this bug.