Bug 1775373
| Summary: | [RFE] pcp-zeroconf should gather per-thread info by default | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Charles Haithcock <chaithco> |
| Component: | pcp | Assignee: | Nathan Scott <nathans> |
| Status: | CLOSED ERRATA | QA Contact: | Jan Kurik <jkurik> |
| Severity: | medium | Docs Contact: | Apurva Bhide <abhide> |
| Priority: | unspecified | ||
| Version: | 7.8 | CC: | abhide, agerstmayr, aklimov, jkurik, mgoodwin, nathans, patrickm, yuokada |
| Target Milestone: | rc | Keywords: | FutureFeature, Triaged |
| Target Release: | 7.9 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Enhancement | |
| Doc Text: |
.The `per-thread` metrics is now available for historical analysis
Optionally, enable logging of the `per-thread` and `per-process` performance metric values in the Performance Co-Pilot (PCP) using the `pcp-zeroconf` package and `pmieconf` utility. Previously, only the `per-process` metric values were logged by `pmlogger` through the `pcp-zeroconf` package, but some analysis situation also requires `per-thread` values. As a result, the `per-thread` metrics are now available for historical analysis, after executing the following command:
----
# pmieconf -c enable zeroconf.all_threads
----
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-09-29 19:24:55 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1782202 | ||
Hi Charles, thanks for the RFE. On systems where per-thread monitoring is needed, would it be sufficient to just edit /etc/pcp/pmcd/rc.local and under the line "Add startup actions here" near line 57 add:
pmstore proc.control.all.threads 1
i.e.
54 case $1 in
55
56 'start')
57 # Add startup actions here
pmstore proc.control.all.threads 1
58 ;;
59
60 'stop')
61 # Add shutdown actions here
62 ;;
63
64 *)
65 echo "Usage: $0 {start|stop}"
66 ;;
67 esac
This is a fairly simple admin task and should be retained across PCP upgrades because the rc.local file is marked as a config file in the RPM spec.
Capturing some IRC discussion - there were concerns raised about the potential for large increases in logged data volumes, though it was pointed out the same potential issue already exists with large numbers of logged processes. An alternative, opt-in approach was suggested using pmieconf to add a pmie rule to enable thread metrics. It would be activated like: # pmieconf -c enable all_threads enabling a pmie rule something like 'proc.control.all.threads != 0 -> shell "pmstore proc.control.all.threads 1";' [ src/pmieconf/rules/primary/pmda_status could be used as a reference example if someone wants to experiment with this] cheers. I wrote a little patch as its relatively simple to add - I'll push this into pcp-5.0.2.
commit 7e33c95326aed25c83ae59105d67bc2e7dbd57af
Author: Nathan Scott <nathans>
Date: Tue Nov 26 10:36:19 2019 +1100
pmieconf: add a pcp-zeroconf option for enabling all_threads
For customer support scenarios, we now provide an option to
enable proc.* metrics in the per-thread fashion. Not doing
this by default because of concerns about logging overhead.
Enabled using: # pmieconf -c enable zeroconf.all_threads
Noticed an incorrect path to the default system pmie config
in the pmieconf(1) man page - also fixed here.
Resolves Red Hat BZ #1775373.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Low: pcp security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:3869 |
1. Proposed title of this feature request pcp-zeroconf gathers per-thread info by default 2. Who is the customer behind the request? Red Hat CEE Account: name and acct N/A TAM customer: N/A CSM customer: N/A Strategic: N/A 3. What is the nature and description of the request? To automatically enable per-thread resource utilisation metric recording when installing pcp-zeroconf for performance troubleshooting. 4. Why does the customer need this? (List the business requirements here) When enacting performance troubleshooting analysis, sbr-kernel often needs a per-thread breakdown of resource consumption due to many threading implementations creating a LWP. Such functionality is implemented with proc.control.{all,perclient}.threads but is disabled by default. For typical installation of PCP this should remain disabled by default but enabling it for pcp-zeroconf will help improve performance troubleshooting support delivery engagements. 5. How would the customer like to achieve this? (List the functional requirements here) Maybe add into the post-install scripts for pcp-zeroconf execution of 'pmstore proc.control.all.threads 1' 6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented. Simply spin up a fresh vm/container with RHEL 7/8, install pcp-zeroconf, and check if threads are being tracked. A modified pmrep config precedes this comment that recreates pidstat -t and will easily demonstrate if threads are being reported or not. 7. Is there already an existing RFE upstream or in Red Hat Bugzilla? No 8. Does the customer have any specific timeline dependencies and which release would they like to target (i.e. RHEL5, RHEL6)? Soon within RHEL 7 and 8 at least 9. Is the sales team involved in this request and do they have any additional input? No 10. List any affected packages or components. pcp-zeroconf 11. Would the customer be able to assist in testing this functionality if implemented? I can absolutely assist. $ cat /etc/pcp/pmrep/pmrep.conf - - - - - [SNIP] - - - - - [pidstat-t] header = yes unitinfo = no globals = no timestamp = yes precision = 2 delimiter = " " colxrow = " PID" sort_metric = -proc.psinfo.tgid proc.id.uid = UID,,,,11 proc.psinfo.tgid = TGID,,,,11 proc.psinfo.pid = PID,,,,11 upct = proc.psinfo.cpu.util.user upct.formula = 100 * rate(proc.psinfo.utime) upct.label = %%usr upct.width = 7 spct = proc.psinfo.cpu.util.sys spct.formula = 100 * rate(proc.psinfo.stime) spct.label = %%system spct.width = 7 gpct = proc.psinfo.cpu.util.guest_time gpct.formula = 100 * rate(proc.psinfo.guest_time) gpct.label = %%guest gpct.width = 7 allcpu = proc.psinfo.alltime allcpu.label = %%CPU allcpu.formula = 100 * (proc.psinfo.utime + proc.psinfo.stime + proc.psinfo.guest_time) allcpu.unit = s allcpu.width = 7 proc.psinfo.processor = CPU,,,,5 proc.psinfo.cmd = Command,,,,16