Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1991896

Summary: Pmlogger processes run in different cgroups
Product: Red Hat Enterprise Linux 8 Reporter: Flos Qi Guo <qguo>
Component: pcpAssignee: pcp-maint <pcp-maint>
Status: CLOSED ERRATA QA Contact: Jan Kurik <jkurik>
Severity: medium Docs Contact: Apurva Bhide <abhide>
Priority: unspecified    
Version: 8.4CC: agerstmayr, jkurik, nathans, peter.vreman
Target Milestone: betaKeywords: FutureFeature, Triaged
Target Release: 8.6Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pcp-5.3.5-1.el8 Doc Type: Release Note
Doc Text:
The PCP pmlogger systemd service now only manages the primary pmlogger instance (localhost) - other pmlogger instances for logging remote hosts are in the pmlogger_farm service. Individual pmlogger instances in a pmlogger farm are best managed with the pmlogctl(1) command. The same change has been made for the pmie service by splitting off the pmie_farm service.
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-10 13:31:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Flos Qi Guo 2021-08-10 11:06:33 UTC
> Description of problem:
The pmlogger instances run in different cgroups. These instances seem running similar options.

> Version-Release number of selected component (if applicable):
pcp-5.2.5-4.el8.x86_64

> How reproducible:
Always when running with large clients.

> Steps to Reproduce:
1. Install pcp-* packages
2. Configure large number of remote loggers
3. Set pmlogger units as follows:
----------8< ----------8< ----------8< ----------
pmlogger.service                           enabled  
pmlogger_check.service                     disabled 
pmlogger_daily-poll.service                static   
pmlogger_daily.service                     static   
pmlogger_daily_report-poll.service         disabled 
pmlogger_daily_report.service              disabled 
pmlogger_check.timer                       enabled  
pmlogger_daily-poll.timer                  disabled 
pmlogger_daily.timer                       enabled  
pmlogger_daily_report-poll.timer           disabled 
pmlogger_daily_report.timer                disabled 
----------8< ----------8< ----------8< ----------

> Actual results:
pmlogger instances with nearly identical parameters are started by two services and run in different cgroups:
----------------8< ----------------8< ----------------8< ----------------8< ----------------
root@test:~# systemctl status pmlogger pmlogger_check pmlogger_check.timer
● pmlogger.service - Performance Metrics Archive Logger
   Loaded: loaded (/etc/systemd/system/pmlogger.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-07-21 08:40:32 UTC; 1h 50min ago
     Docs: man:pmlogger(1)
 Main PID: 2315 (pmlogger)
    Tasks: 18 (limit: 203885)
   Memory: 130.6M
   CGroup: /system.slice/pmlogger.service
           ├─  1732 /bin/sh /usr/libexec/pcp/lib/pmlogger start-systemd
           ├─  1970 /bin/sh /usr/libexec/pcp/bin/pmlogger_check --quick -V
           ├─  2315 /usr/libexec/pcp/bin/pmlogger -N -P -r -T24h10m -c config.default -v 100mb -m pmlogger_check %Y%m%d.%H.%M
           ├─  2749 /bin/sh /usr/libexec/pcp/bin/pmlogger_check --quick -V
           ├─  2792 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1010.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1010.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─116394 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1013.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1013.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─162865 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1437.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1437.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─240718 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1637.ii.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1637.ii.bb.com -v 100mb -m pmlogger_chec>
           ├─340811 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1680.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1680.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─412732 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1726.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1726.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─485412 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1803.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1803.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─537125 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1823.ii.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1823.ii.bb.com -v 100mb -m pmlogger_chec>
           ├─611397 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1875.ii.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1875.ii.bb.com -v 100mb -m pmlogger_chec>
           ├─710811 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1913.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1913.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─751007 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1921.ii.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1921.ii.bb.com -v 100mb -m pmlogger_chec>
           ├─837207 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1923.ii.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1923.ii.bb.com -v 100mb -m pmlogger_chec>
           ├─874040 pmlc
           └─874041 grep Unable to connect

Jul 21 08:40:29 xxx-yyy-2635 systemd[1]: Starting Performance Metrics Archive Logger...
Jul 21 08:40:32 xxx-yyy-2635 systemd[1]: Started Performance Metrics Archive Logger.

● pmlogger_check.service - Check pmlogger instances are running
   Loaded: loaded (/etc/systemd/system/pmlogger_check.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-07-21 08:41:23 UTC; 1h 50min ago
     Docs: man:pmlogger_check(1)
 Main PID: 12124 (pmlogger_check)
    Tasks: 16 (limit: 203885)
   Memory: 126.2M
   CGroup: /system.slice/pmlogger_check.service
           ├─ 12124 /bin/sh /usr/libexec/pcp/bin/pmlogger_check -VV -C
           ├─ 12511 /bin/sh /usr/libexec/pcp/bin/pmlogger_check -VV -C
           ├─ 17390 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1011.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1011.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─111565 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1012.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1012.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─167846 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1636.ii.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1636.ii.bb.com -v 100mb -m pmlogger_chec>
           ├─269462 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1667.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1667.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─306743 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1679.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1679.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─383600 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1681.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1681.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─449261 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1727.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1727.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─512264 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1804.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1804.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─575094 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1824.ii.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1824.ii.bb.com -v 100mb -m pmlogger_chec>
           ├─658871 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1876.ii.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1876.ii.bb.com -v 100mb -m pmlogger_chec>
           ├─739180 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1914.aa.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1914.aa.bb.com -v 100mb -m pmlogger_ch>
           ├─800864 /usr/libexec/pcp/bin/pmlogger -h xxx-yyy-1922.ii.bb.com -r -T24h10m -c /var/lib/pcp/config/pmlogger/remote/config.xxx-yyy-1922.ii.bb.com -v 100mb -m pmlogger_chec>
           ├─836338 pmlc
           └─836339 grep Unable to connect

Jul 21 08:41:23 xxx-yyy-2635 systemd[1]: Started Check pmlogger instances are running.

● pmlogger_check.timer - Half-hourly check of pmlogger instances
   Loaded: loaded (/usr/lib/systemd/system/pmlogger_check.timer; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2021-07-21 08:40:32 UTC; 1h 50min ago
  Trigger: n/a

Jul 21 08:40:32 xxx-yyy-2635 systemd[1]: Started Half-hourly check of pmlogger instances.
lines 1-63/63 (END)
----------------8< ----------------8< ----------------8< ----------------8< ----------------

> Expected results:
pmlogger instances should run in the same cgroup. And these processes should be controlled under the same service.

> Additional info:
N/A.

Comment 1 Mark Goodwin 2021-10-08 02:01:55 UTC
mostly resolved with upstream commit 215bdb10f5831, though a bit more to come yet as mentioned in the commit message:

commit 215bdb10f5831f1164438580811c733ac86177a8
Author: Mark Goodwin <mgoodwin>
Date:   Tue Oct 5 11:41:14 2021 +1100

    logutil: cleanly separate primary and farm logger services
    
    Cleanly separate primary pmlogger (using pmlogger.service) from
    farm pmloggers (using pmlogger_check.service). Further changes
    are coming for the pmlogger-farm.service which will manage only
    farm loggers (if any). Both the pmlogger and pmlogger-farm
    services will need to carefully inherit the same prior service
    state in all upgrade and virgin install scenarios.
    
    Also tweak timeouts in pmlogger.service for the local primary
    and in pmlogger_check.service for farm loggers.
    
    The KillMode=process setting in pmlogger_check.service is temporary
    (will not be needed by pmlogger-farm.service).
    
    Resolves: https://github.com/performancecopilot/pcp/issues/1435
    Related BZ#1973833

Comment 7 errata-xmlrpc 2022-05-10 13:31:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pcp bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1765