Bug 1977740

I could also reproduce with RHEL 8.4 and RHEL 9 Beta.

I can reproduce this as well. However, I am not sure if this is a bug.

There is a kvm pmda installed as one of the defaults pmdas (part of pcp rpm). The kvm pmda needs the kvm kernel module to be able to provide kvm metrics.

If this blocks your kvm update, then you do not need to uninstall pcp. Stopping pmcd is sufficient.

I.e. this works for me on my test bed: 

# systemctl stop pmcd
# modprobe -r kvm
# systemct start pmcd


If you do not need the kvm metrics, there is also a possibility to uninstall the kvm pmda:

# systemctl start pmcd ### Make sure PMCD is running
# cd /var/lib/pcp/pmdas/kvm
# ./Remove

This will uninstall the kvm pmda and even with running pcp (pmcd) the kvm kernel module can be removed.

(In reply to smitterl from comment #0)
> Expected results:
> The user can identify which process uses KVM resources and take action to
> unload the kvm module successfully.

I'm not sure what we should do about that. We don't have control over the 'lsof' tool.
Afaics the kvm PMDA works by using the perf_event subsystem, which explains why it's not using /dev/kvm.

As Jan explained, the kvm PMDA is part of the default PMDAs installed by PCP (the 'pcp' rpm package). If this PMDA is not wanted on the system, it can be uninstalled.

(In reply to Andreas Gerstmayr from comment #3)
> (In reply to smitterl from comment #0)
> > Expected results:
> > The user can identify which process uses KVM resources and take action to
> > unload the kvm module successfully.
> 
> I'm not sure what we should do about that. We don't have control over the
> 'lsof' tool.
> Afaics the kvm PMDA works by using the perf_event subsystem, which explains
> why it's not using /dev/kvm.
> 
> As Jan explained, the kvm PMDA is part of the default PMDAs installed by PCP
> (the 'pcp' rpm package). If this PMDA is not wanted on the system, it can be
> uninstalled.

I understand. I'm wondering how a customer could find out they can stop the pmcd temporarily to unblock resources. I would never have occurred to me but then I didn't install pcp on purpose either.

I think it's not unlikely the customer would go through the following steps:

1. discovers can't unload kvm
2. runs 'lsof|grep kvm'
3. sees pmda and pmdakvm in the output
4. runs 'man pmda'
==> No manual entry for pmda

The pmda manpage is in the pcp-doc package which is not pulled in when installing pcp.

# rpm -q --whatrequires pcp-doc
no package requires pcp-doc

I'm wondering if making sure pcp-doc is pulled in when installing pcp could help the customer. At least they would be able to read in the pmda manpage:

"       A  PMDA  is responsible for a set of performance metrics, in the sense that it must respond to requests from pmcd(1) for informa‐
       tion about performance metrics, instance domains, and instantiated values.
"

So, there's some reference to pmcd.


Finally, if additionally either pmda or pmcd manpage could mention that a PMDA might block resources and that they can be released temporarily by stopping the daemon, that could be helpful - but it could also be too much. I don't know. Maybe getting a result for pmda per default could be enough.

(Setting lower severity because the workaround is documented on this BZ. Please shout if you disagree. Thanks.)

The only thing we can do here is add more information to the pmdakvm(1) man page about this situation - as Andreas mentioned, the PMDA has file descriptors registered with perf_event that increase the device driver reference count.

There is a slightly better way to tackle the problem than the earlier-mentioned pmcd restart.  Its best to use the ./Remove and ./Install scripts for this task as that is less disruptive to all PCP client tools connected to pmcd - when we use those scripts, pmcd is not restarted it is sent SIGHUP to remove/install the agent.  This way only clients actively using the kvm metrics are affected, and even those see localized errors relating to kvm, not complete loss of pmcd metrics service for all metrics.

I'll improve the docs as a low priority task (most likely for 8.6 since its getting late in the 8.5 release cycle for us now).

Oh, regarding:

| I think it's not unlikely the customer would go through the following steps:
|
| 1. discovers can't unload kvm
| 2. runs 'lsof|grep kvm'
| 3. sees pmda and pmdakvm in the output
| 4. runs 'man pmda'
| ==> No manual entry for pmda


I think we went sideways a bit from point 3 - looking at the lsof output, its all references to 'pmdakvm' - I'd expect (hope!) customers / support would look specifically for that man page - pmdakvm(1).  This already has a section describing the Install and Remove process I mentioned - but it would help if this specifically mention this is a good approach to allow kernel module reloading.  This is what I will add more details about.

Fixed upstream via ...

commit f97db1304440eb7d5b580f355808698fa716bcc0
Author: Nathan Scott <nathans>
Date:   Tue Jul 6 13:24:05 2021 +1000

    docs: add a CAVEATS section to pmdakvm(1) about module reloading
    
    Resolves Red Hat BZ #1977740

The CAVEATS section is present in the man page of the pmdakvm and describes the situation as well as a solution.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pcp bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1765

Thank you!