Description of problem:

PCP collector host the pmproxy process stopped working.
Looking in the log file showed 'too many open files'

Version-Release number of selected component (if applicable):

$ cat installed-rpms | grep pcp
grafana-pcp-2.0.2-3.el8.noarch                              Thu Dec 10 10:11:43 2020
pcp-5.1.1-4.el8_3.x86_64                                    Mon Feb 22 17:03:28 2021

How reproducible:


Steps to Reproduce:


- setup rhel8.4 KVM guest 

  # dnf -y install pcp-zeroconf
  # vi /etc/sysconfig/pmcd # PMCD_LOCAL=0
  # systemctl restart pmcd
  # for i in {100..200}; do ip addr add 192.168.4.$i dev enp1s0 ; done
- setup rhel8.3 KVM guest
  # yum -y install pcp-zeroconf
  # CLIENT=rhel8u4a
  # echo "$CLIENT.local n n PCP_LOG_DIR/pmlogger/$CLIENT.local" " -r -T30d -c config.$CLIENT" >/etc/pcp/pmlogger/control.d/$CLIENT
  # echo '192.168.4.13 rhel8u4a.local rhel8u4a' >>/etc/hosts

  # for i in {100..200}; do 
      echo "192.168.4.$i  ce$i.local ce$i" >>/etc/hosts; 
      /usr/libexec/pcp/bin/pmlogconf /var/lib/pcp/config/pmlogger/config.ce${i};
      CLIENT="ce$i"; 
      echo "$CLIENT.local n n PCP_LOG_DIR/pmlogger/$CLIENT.local" " -r -T30d -c config.$CLIENT" >/etc/pcp/pmlogger/control.d/$CLIENT ;
    done
  # /usr/libexec/pcp/bin/pmlogger_check
  # yum -y install redis grafana grafana-pcp
  # systemctl restart redis pmproxy grafana-server
  # systemctl enable redis pmproxy grafana-server
Then login at <ip>:3000, setup a redis/host dashboard.
Displaying rhel8u4a works fine, ce200.local not.

PCP is not distinguishing the single pmcd as 200 separate
pmcd instances, that leads to the error here.
We also see that here:
[root@rhel8u3a pmproxy]# pcp
 [..]
 pmlogger: rhel8u4a: /var/log/pcp/pmlogger/ce100.local/20210414.16.50
           rhel8u4a: /var/log/pcp/pmlogger/ce101.local/20210414.16.50
           rhel8u4a: /var/log/pcp/pmlogger/ce102.local/20210414.16.50
 [..]

But it's enough to see the pmproxy errors:

[root@rhel8u3a pmproxy]# grep open /var/log/pcp/pmproxy/pmproxy.log |tail -3
[Wed Apr 14 16:43:17] pmproxy(125947) Error: pmNewContext failed for /var/log/pcp/pmlogger/ce126.local/20210414.16.35: Too many open files
[Wed Apr 14 16:43:17] pmproxy(125947) Error: pmNewContext failed for /var/log/pcp/pmlogger/ce126.local/20210414.16.41: Too many open files
[Wed Apr 14 16:43:17] pmproxy(125947) Error: pmNewContext failed for /var/log/pcp/pmlogger/ce126.local/20210414.16.37: Too many open files
[root@rhel8u3a pmproxy]#

Actual results:

Too many open files

Expected results:

It should work smoothly.

Additional info: