Bug 1954711
| Summary: | pmproxy nofile limit too low | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Rajesh Dulhani <rdulhani> |
| Component: | pcp | Assignee: | Mark Goodwin <mgoodwin> |
| Status: | CLOSED ERRATA | QA Contact: | Jan Kurik <jkurik> |
| Severity: | medium | Docs Contact: | Apurva Bhide <abhide> |
| Priority: | unspecified | ||
| Version: | 8.3 | CC: | agerstmayr, jkurik, mgoodwin, nathans, patrickm, peter.vreman |
| Target Milestone: | beta | Keywords: | Bugfix, Triaged |
| Target Release: | 8.5 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | pcp-5.3.0-1.el8 | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-11-09 17:50:37 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1922040, 1957575 | ||
| Bug Blocks: | |||
This has been tackled upstream via this commit:
commit 5ae002bae77b3a93304299a49d1c300f8028b9a3
Author: Mark Goodwin <mgoodwin>
Date: Wed Mar 10 20:03:41 2021 +1100
pmproxy: increase open files soft limit to maxfiles
The pmproxy discovery module needs approx 30 + 4 times
the number of pmloggers configured file descriptors,
plus spares for RESTAPI client sockets and briefly more
during daily log rolling. This can exceed the default
soft limit on some platforms at ~ 200 pmloggers.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (pcp bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4171 |
Description of problem: PCP collector host the pmproxy process stopped working. Looking in the log file showed 'too many open files' Version-Release number of selected component (if applicable): $ cat installed-rpms | grep pcp grafana-pcp-2.0.2-3.el8.noarch Thu Dec 10 10:11:43 2020 pcp-5.1.1-4.el8_3.x86_64 Mon Feb 22 17:03:28 2021 How reproducible: Steps to Reproduce: - setup rhel8.4 KVM guest # dnf -y install pcp-zeroconf # vi /etc/sysconfig/pmcd # PMCD_LOCAL=0 # systemctl restart pmcd # for i in {100..200}; do ip addr add 192.168.4.$i dev enp1s0 ; done - setup rhel8.3 KVM guest # yum -y install pcp-zeroconf # CLIENT=rhel8u4a # echo "$CLIENT.local n n PCP_LOG_DIR/pmlogger/$CLIENT.local" " -r -T30d -c config.$CLIENT" >/etc/pcp/pmlogger/control.d/$CLIENT # echo '192.168.4.13 rhel8u4a.local rhel8u4a' >>/etc/hosts # for i in {100..200}; do echo "192.168.4.$i ce$i.local ce$i" >>/etc/hosts; /usr/libexec/pcp/bin/pmlogconf /var/lib/pcp/config/pmlogger/config.ce${i}; CLIENT="ce$i"; echo "$CLIENT.local n n PCP_LOG_DIR/pmlogger/$CLIENT.local" " -r -T30d -c config.$CLIENT" >/etc/pcp/pmlogger/control.d/$CLIENT ; done # /usr/libexec/pcp/bin/pmlogger_check # yum -y install redis grafana grafana-pcp # systemctl restart redis pmproxy grafana-server # systemctl enable redis pmproxy grafana-server Then login at <ip>:3000, setup a redis/host dashboard. Displaying rhel8u4a works fine, ce200.local not. PCP is not distinguishing the single pmcd as 200 separate pmcd instances, that leads to the error here. We also see that here: [root@rhel8u3a pmproxy]# pcp [..] pmlogger: rhel8u4a: /var/log/pcp/pmlogger/ce100.local/20210414.16.50 rhel8u4a: /var/log/pcp/pmlogger/ce101.local/20210414.16.50 rhel8u4a: /var/log/pcp/pmlogger/ce102.local/20210414.16.50 [..] But it's enough to see the pmproxy errors: [root@rhel8u3a pmproxy]# grep open /var/log/pcp/pmproxy/pmproxy.log |tail -3 [Wed Apr 14 16:43:17] pmproxy(125947) Error: pmNewContext failed for /var/log/pcp/pmlogger/ce126.local/20210414.16.35: Too many open files [Wed Apr 14 16:43:17] pmproxy(125947) Error: pmNewContext failed for /var/log/pcp/pmlogger/ce126.local/20210414.16.41: Too many open files [Wed Apr 14 16:43:17] pmproxy(125947) Error: pmNewContext failed for /var/log/pcp/pmlogger/ce126.local/20210414.16.37: Too many open files [root@rhel8u3a pmproxy]# Actual results: Too many open files Expected results: It should work smoothly. Additional info: