RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1775373 - [RFE] pcp-zeroconf should gather per-thread info by default
Summary: [RFE] pcp-zeroconf should gather per-thread info by default
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: pcp
Version: 7.8
Hardware: All
OS: Linux
unspecified
medium
Target Milestone: rc
: 7.9
Assignee: Nathan Scott
QA Contact: Jan Kurik
Apurva Bhide
URL:
Whiteboard:
Depends On:
Blocks: 1782202
TreeView+ depends on / blocked
 
Reported: 2019-11-21 20:01 UTC by Charles Haithcock
Modified: 2023-03-24 16:08 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
.The `per-thread` metrics is now available for historical analysis Optionally, enable logging of the `per-thread` and `per-process` performance metric values in the Performance Co-Pilot (PCP) using the `pcp-zeroconf` package and `pmieconf` utility. Previously, only the `per-process` metric values were logged by `pmlogger` through the `pcp-zeroconf` package, but some analysis situation also requires `per-thread` values. As a result, the `per-thread` metrics are now available for historical analysis, after executing the following command: ---- # pmieconf -c enable zeroconf.all_threads ----
Clone Of:
Environment:
Last Closed: 2020-09-29 19:24:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2020:3869 0 None None None 2020-09-29 19:25:25 UTC

Description Charles Haithcock 2019-11-21 20:01:06 UTC
1. Proposed title of this feature request

pcp-zeroconf gathers per-thread info by default 


2. Who is the customer behind the request?

Red Hat CEE


Account: name and acct N/A
TAM customer: N/A
CSM customer: N/A
Strategic: N/A


3. What is the nature and description of the request?

To automatically enable per-thread resource utilisation metric recording when installing pcp-zeroconf for performance troubleshooting. 


4. Why does the customer need this? (List the business requirements here)

When enacting performance troubleshooting analysis, sbr-kernel often needs a per-thread breakdown of resource consumption due to many threading implementations creating a LWP. Such functionality is implemented with proc.control.{all,perclient}.threads but is disabled by default. For typical installation of PCP this should remain disabled by default but enabling it for pcp-zeroconf will help improve performance troubleshooting support delivery engagements. 


5. How would the customer like to achieve this? (List the functional requirements here)

Maybe add into the post-install scripts for pcp-zeroconf execution of 'pmstore proc.control.all.threads 1'


6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.

Simply spin up a fresh vm/container with RHEL 7/8, install pcp-zeroconf, and check if threads are being tracked. A modified pmrep config precedes this comment that recreates pidstat -t and will easily demonstrate if threads are being reported or not. 


7. Is there already an existing RFE upstream or in Red Hat Bugzilla?

No


8. Does the customer have any specific timeline dependencies and which release would they like to target (i.e. RHEL5, RHEL6)?

Soon within RHEL 7 and 8 at least 


9. Is the sales team involved in this request and do they have any additional input?

No


10. List any affected packages or components.

pcp-zeroconf 


11. Would the customer be able to assist in testing this functionality if implemented?

I can absolutely assist. 






 $ cat /etc/pcp/pmrep/pmrep.conf
- - - - - [SNIP] - - - - -
[pidstat-t]
header = yes
unitinfo = no
globals = no
timestamp = yes
precision = 2
delimiter = " "
colxrow = "   PID"
sort_metric           = -proc.psinfo.tgid
proc.id.uid           = UID,,,,11
proc.psinfo.tgid      = TGID,,,,11
proc.psinfo.pid       = PID,,,,11
upct                  = proc.psinfo.cpu.util.user
upct.formula          = 100 * rate(proc.psinfo.utime)
upct.label            = %%usr
upct.width            = 7
spct                  = proc.psinfo.cpu.util.sys
spct.formula          = 100 * rate(proc.psinfo.stime)
spct.label            = %%system
spct.width            = 7
gpct                  = proc.psinfo.cpu.util.guest_time
gpct.formula          = 100 * rate(proc.psinfo.guest_time)
gpct.label            = %%guest
gpct.width            = 7
allcpu                = proc.psinfo.alltime
allcpu.label          = %%CPU
allcpu.formula        = 100 * (proc.psinfo.utime + proc.psinfo.stime + proc.psinfo.guest_time)
allcpu.unit           = s
allcpu.width          = 7
proc.psinfo.processor = CPU,,,,5
proc.psinfo.cmd       = Command,,,,16

Comment 3 Mark Goodwin 2019-11-21 21:02:45 UTC
Hi Charles, thanks for the RFE. On systems where per-thread monitoring is needed, would it be sufficient to just edit /etc/pcp/pmcd/rc.local and under the line "Add startup actions here" near line 57 add:

pmstore proc.control.all.threads 1

i.e.

 54 case $1 in
 55 
 56         'start')
 57             # Add startup actions here
                pmstore proc.control.all.threads 1
 58             ;;
 59 
 60         'stop')
 61             # Add shutdown actions here
 62             ;;
 63 
 64         *)
 65             echo "Usage: $0 {start|stop}"
 66             ;;
 67 esac

This is a fairly simple admin task and should be retained across PCP upgrades because the rc.local file is marked as a config file in the RPM spec.

Comment 4 Nathan Scott 2019-11-25 00:16:42 UTC
Capturing some IRC discussion - there were concerns raised about the potential for large increases in logged data volumes, though it was pointed out the same potential issue already exists with large numbers of logged processes.

An alternative, opt-in approach was suggested using pmieconf to add a pmie rule to enable thread metrics.  It would be activated like:

# pmieconf -c enable all_threads

enabling a pmie rule something like 'proc.control.all.threads != 0 -> shell "pmstore proc.control.all.threads 1";'
[ src/pmieconf/rules/primary/pmda_status could be used as a reference example if someone wants to experiment with this]

cheers.

Comment 5 Nathan Scott 2019-11-26 01:27:51 UTC
I wrote a little patch as its relatively simple to add - I'll push this into pcp-5.0.2.

commit 7e33c95326aed25c83ae59105d67bc2e7dbd57af
Author: Nathan Scott <nathans>
Date:   Tue Nov 26 10:36:19 2019 +1100

    pmieconf: add a pcp-zeroconf option for enabling all_threads
    
    For customer support scenarios, we now provide an option to
    enable proc.* metrics in the per-thread fashion.  Not doing
    this by default because of concerns about logging overhead.
    
    Enabled using: # pmieconf -c enable zeroconf.all_threads
    
    Noticed an incorrect path to the default system pmie config
    in the pmieconf(1) man page - also fixed here.
    
    Resolves Red Hat BZ #1775373.

Comment 14 errata-xmlrpc 2020-09-29 19:24:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: pcp security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:3869


Note You need to log in before you can comment on or make changes to this bug.