Bug 1673250

Summary: pminfo -f proc.psinfo.* kernel processes missing for root users
Product: Red Hat Enterprise Linux 7 Reporter: nikhil kshirsagar <nkshirsa>
Component: pcpAssignee: Nathan Scott <nathans>
Status: CLOSED DUPLICATE QA Contact: qe-baseos-tools-bugs
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.6CC: fche, lberk, mgoodwin, nathans, patrickm, rupatel
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-17 00:41:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1594286    

Description nikhil kshirsagar 2019-02-07 06:32:27 UTC
Description of problem:
Fresh install of pcp on 7.6

root user doesnt get a lot of processes show up when we run,

[root@rhel7u6-3 ~]# pminfo -f proc.psinfo.cutime

proc.psinfo.cutime
    inst [1495 or "001495 /usr/lib/systemd/systemd-journald"] value 0
    inst [1513 or "001513 /usr/sbin/lvmetad"] value 0
    inst [1523 or "001523 /usr/lib/systemd/systemd-udevd"] value 550
    inst [2718 or "002718 /sbin/auditd"] value 0
...
...

as a normal user,

[root@rhel7u6-3 ~]# adduser nkshirsa_test
[root@rhel7u6-3 ~]# su - nkshirsa_test
[nkshirsa_test@rhel7u6-3 ~]$ pminfo -f proc.psinfo.cutime

proc.psinfo.cutime
    inst [1 or "000001 /usr/lib/systemd/systemd"] value 6880
    inst [2 or "000002 (kthreadd)"] value 0
    inst [3 or "000003 (ksoftirqd/0)"] value 0
    inst [5 or "000005 (kworker/0:0H)"] value 0
...
...



[root@rhel7u6-3 ~]# tail -f /var/log/pcp/pmcd/proc.log
[Thu Feb  7 16:26:41] pmdaproc(8627) Error: pmdaFetch: Fetch callback error from metric PMID 3.8.45[3]: No data available
[Thu Feb  7 16:26:41] pmdaproc(8627) Error: pmdaFetch: Fetch callback error from metric PMID 3.8.45[9]: No data available
[Thu Feb  7 16:26:51] pmdaproc(8627) Error: pmdaFetch: Fetch callback error from metric PMID 3.8.45[3]: No data available
[Thu Feb  7 16:26:51] pmdaproc(8627) Error: pmdaFetch: Fetch callback error from metric PMID 3.8.45[9]: No data available
[Thu Feb  7 16:27:01] pmdaproc(8627) Error: pmdaFetch: Fetch callback error from metric PMID 3.8.45[3]: No data available
[Thu Feb  7 16:27:01] pmdaproc(8627) Error: pmdaFetch: Fetch callback error from metric PMID 3.8.45[9]: No data available
[Thu Feb  7 16:27:11] pmdaproc(8627) Error: pmdaFetch: Fetch callback error from metric PMID 3.8.45[3]: No data available
[Thu Feb  7 16:27:11] pmdaproc(8627) Error: pmdaFetch: Fetch callback error from metric PMID 3.8.45[9]: No data available
[Thu Feb  7 16:27:21] pmdaproc(8627) Error: pmdaFetch: Fetch callback error from metric PMID 3.8.45[9]: No data available
[Thu Feb  7 16:27:21] pmdaproc(8627) Error: pmdaFetch: Fetch callback error from metric PMID 3.8.45[88]: No data available


Version-Release number of selected component (if applicable):
pcp-4.1.0-5.el7_6.x86_64

How reproducible:
Install pcp with yum install pcp*
compare output of pminfo -f proc.psinfo.cutime as root and non root user.

Additional info:
Works fine on latest upstream 

reproducers available at 

10.10.96.126 root/RedHat1!

Comment 2 nikhil kshirsagar 2019-02-07 06:51:51 UTC
It's always the first one in a list that seems to fail as root.

[root@rhel7u6-3 ~]# pminfo -f proc.psinfo.cutime proc.psinfo.stime proc.psinfo.utime

proc.psinfo.cutime
    inst [1495 or "001495 /usr/lib/systemd/systemd-journald"] value 0
    inst [1513 or "001513 /usr/sbin/lvmetad"] value 0
    inst [1523 or "001523 /usr/lib/systemd/systemd-udevd"] value 550
   
  

proc.psinfo.stime
    inst [1 or "000001 /usr/lib/systemd/systemd"] value 610
    inst [2 or "000002 (kthreadd)"] value 0
    inst [3 or "000003 (ksoftirqd/0)"] value 50
    inst [5 or "000005 (kworker/0:0H)"] value 0
    inst [7 or "000007 (migration/0)"] value 0


proc.psinfo.utime
    inst [1 or "000001 /usr/lib/systemd/systemd"] value 370
    inst [2 or "000002 (kthreadd)"] value 0
    inst [3 or "000003 (ksoftirqd/0)"] value 0
    inst [5 or "000005 (kworker/0:0H)"] value 0
    inst [7 or "000007 (migration/0)"] value 0

[root@rhel7u6-3 ~]# pminfo -f proc.psinfo.stime proc.psinfo.utime

proc.psinfo.stime
    inst [1495 or "001495 /usr/lib/systemd/systemd-journald"] value 100
    inst [1513 or "001513 /usr/sbin/lvmetad"] value 0
    inst [1523 or "001523 /usr/lib/systemd/systemd-udevd"] value 20
    inst [2718 or "002718 /sbin/auditd"] value 0
    inst [2741 or "002741 /usr/sbin/smartd"] value 0
   

proc.psinfo.utime
    inst [1 or "000001 /usr/lib/systemd/systemd"] value 370
    inst [2 or "000002 (kthreadd)"] value 0
    inst [3 or "000003 (ksoftirqd/0)"] value 0
    inst [5 or "000005 (kworker/0:0H)"] value 0
    inst [7 or "000007 (migration/0)"] value 0
    inst [8 or "000008 (rcu_bh)"] value 0
 
[root@rhel7u6-3 ~]# pminfo -f proc.psinfo.utime proc.psinfo.stime

proc.psinfo.utime
    inst [1495 or "001495 /usr/lib/systemd/systemd-journald"] value 50
    inst [1513 or "001513 /usr/sbin/lvmetad"] value 0
    inst [1523 or "001523 /usr/lib/systemd/systemd-udevd"] value 120
    inst [2718 or "002718 /sbin/auditd"] value 0

proc.psinfo.stime
    inst [1 or "000001 /usr/lib/systemd/systemd"] value 610
    inst [2 or "000002 (kthreadd)"] value 0
    inst [3 or "000003 (ksoftirqd/0)"] value 50
    inst [5 or "000005 (kworker/0:0H)"] value 0
    inst [7 or "000007 (migration/0)"] value 0

Comment 3 Nathan Scott 2019-02-07 06:54:30 UTC
Thanks Nikhil - as discussed on IRC, hopefully we can use that as a temporary workaround for the customer.

We've also confirmed the current RHEL 7.7 build of pmdaproc(1) does not exhibit this problem.

cheers.

Comment 5 Nathan Scott 2019-05-17 00:39:11 UTC
*** Bug 1603143 has been marked as a duplicate of this bug. ***

Comment 6 Nathan Scott 2019-05-17 00:41:40 UTC
Resolved by updating PCP through the 7.7 rebase.

*** This bug has been marked as a duplicate of bug 1647308 ***