Bug 2210491
| Summary: | dtablesize being set to soft maxfiledescriptor limit causing massive slowdown in large enviroments [8.9.0] | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Jerone Young <jyoung> | |
| Component: | 389-ds-base | Assignee: | Jamie Chapman <jachapma> | |
| Status: | CLOSED ERRATA | QA Contact: | LDAP QA Team <idm-ds-qe-bugs> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 8.8 | CC: | afarley, apeddire, emartyny, gkimetto, idm-ds-dev-bugs, jachapma, jonmoore, jwooten, mreynolds, mrhodes, msauton, tbordaz, tmihinto, vashirov | |
| Target Milestone: | rc | Keywords: | Triaged, ZStream | |
| Target Release: | 8.9 | Flags: | pm-rhel:
mirror+
|
|
| Hardware: | All | |||
| OS: | Linux | |||
| Whiteboard: | sync-to-jira | |||
| Fixed In Version: | 389-ds-1.4-820230816162424-17499975 | Doc Type: | Bug Fix | |
| Doc Text: |
.Directory Server now calculates the `dtablesize` based on the maximum number of opened descriptors
Previously, an administrator could set the connection table size manually by using the `nsslapd-conntablesize` configuration parameter. Consequently, when the connection table size was set too low, it affected the number of connections the server was able to support. With this update, Directory Server now calculates the size of the connection table dynamically effectively resolving the issue with too small connection table size. In addition, you no longer need to manually change the connection table size.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 2235330 2247075 2251375 2265538 (view as bug list) | Environment: | ||
| Last Closed: | 2023-11-14 15:32:25 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 2235330, 2247075, 2251375, 2265538 | |||
the dtablesize has been documented in the RHDS perf guide until RHDS-11 ( no longer in RHDS-12 ) https://access.redhat.com/documentation/en-us/red_hat_directory_server/11/pdf/performance_tuning_guide/red_hat_directory_server-11-performance_tuning_guide-en-us.pdf 2.1.1. Monitoring the Directory Server Using the Command Line a note could be added in https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/pdf/tuning_performance_in_identity_management/red_hat_enterprise_linux-9-tuning_performance_in_identity_management-en-us.pdf for a system general configuration with sysctl for somaxconn dtablesize and/or Chapter 6. Adjusting IdM Directory Server performance https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/tuning_performance_in_identity_management/adjusting-idm-directory-server-performance_tuning-performance-in-idm and/or Chapter 7. Adjusting the performance of the KDC https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/tuning_performance_in_identity_management/assembly_adjusting-the-performance-of-the-kdc_tuning-performance-in-idm but note such tuning may displace a bottleneck where a prior connection throttling will now hammer the LDAP worker threads and cause a lot more contention with plug-ins. @Marc
Upstream has already confirmed this as a bug and are working on it now.
dtablesize is in RHDS-12 I show it in the example.
The workaround provided is being used and has been proven by customers.
@Marc
What I get for multi-tasking. Yes you are correct that notes should be added to documentation.
Patch was merged upstream, moving to POST. Automated test passed: =============================================================================================== test session starts ================================================================================================ platform linux -- Python 3.6.8, pytest-3.4.2, py-1.5.3, pluggy-0.6.0 -- /usr/libexec/platform-python cachedir: dirsrvtests/.pytest_cache 389-ds-base: 1.4.3.37-1.module+el8.9.0+19689+7d653af8 nss: 3.90.0-3.el8_8 nspr: 4.35.0-1.el8_8 openldap: 2.4.46-18.el8 cyrus-sasl: not installed FIPS: disabled rootdir: /root/ds/dirsrvtests, inifile: pytest.ini collected 1 item dirsrvtests/tests/suites/resource_limits/fdlimits_test.py::test_reserve_descriptor_validation PASSED [100%] ============================================================================================ 1 passed in 39.59 seconds ============================================================================================= Marking as VERIFIED:Tested. Hello, For the customer tam would like to know what this test verified? " I saw that the latest private message Bug 2210491 can you help understand what "automated test passed" means? Thanks Hi, it verified that the next 389-ds-base build for RHEL 8.9 contains the fix for this bz. HTH Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (389-ds:1.4 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:6965 This issue is now resolved in RHEL 8. But RHEL 9 still does not have this issue resolved. Issue is resolved in RHEL 8.8 & 9.3 for IDM installs. The issue is still there for RHDS installs as it looks like it installs an older version of 389-ds-base packages that do not contain the fix. |
Description of problem: It looks like dtablesize is being set to the maxfiledescriptors soft limit. This problem was found while migrating from RHEL 7 IDM to RHEL 8 IDM at large client site. I came up at first with systemd override for maxfiledescriptors with value 65535 that resolved issues at client site. Though after further investigation was able to see the issue was not the value of the maxfiledescriptors. But that dtablesize was being set to the soft limit. Version-Release number of selected component (if applicable): Affects both RHEL 8 & RHEL 9. How reproducible: It looks like dtablesize is being set to the maxfiledescriptors soft limit. Problem: ======= 389ds is not setting dtablesize properly based when systemd is setting the maxfiledescriptors with it's default settings. dtablesize stays 1024 which causes massive slowdown once you hit around 950 connection. Basically queries are very slow. Also can't go above around ~970 concurrent connections. It looks like dtablesize is being set to the soft limit when it needs to be set to the actual limit. So with no changes settings you see running commands: systemctl show dirsrv@<INSTANCE> |grep -i Limitnofile ----------------------------------------------------------------------------- - RHEL 8 (Directory Server) LimitNOFILE=262144 LimitNOFILESoft=1024 < ---- Notice - RHEL 9 (IDM system) LimitNOFILE=524288 LimitNOFILESoft=1024 <--- Notice dsconf <instance> config get nsslapd-maxdescriptors -------------------------------------------------------------------------- - RHEL 8 nsslapd-maxdescriptors: 262144 - RHEL 9 nsslapd-maxdescriptors: 524288 - (THE PROBLEM) dsconf <instance> monitor server |grep dtablesize -------------------------------------------------------------------- - RHEL 8 dtablesize: 1024 - RHEL 9 dtablesize: 1024 WorkAround: ========== Once in place connections can go very very high. Testing showed 7,000+ at client site. Also queries very fast. Looks like this works because the override sets both hard and soft limit. So dtablesize gets a properly. Workaround ----------------- - RHEL 8 mkdir -p /etc/systemd/system/dirsrv@.service.d/ cat > /etc/systemd/system/dirsrv@.service.d/filelimts.conf << EOF [Service] LimitNOFILE=262144 EOF systemctl daemon-reload systemctl restart dirsrv@<instance> - RHEL 9 mkdir -p /etc/systemd/system/dirsrv@.service.d/ cat > /etc/systemd/system/dirsrv@.service.d/filelimts.conf << EOF [Service] LimitNOFILE=524288 EOF systemctl daemon-reload systemctl restart dirsrv@<instance> systemctl show dirsrv@<INSTANCE> |grep -i Limitnofile ----------------------------------------------------------------------------- - RHEL 8 (Directory Server) LimitNOFILE=262144 LimitNOFILESoft=262144 <---- THIS !! - RHEL 9 (IDM system) LimitNOFILE=524288 LimitNOFILESoft=524288 <-- THIS !! dsconf <instance> config get nsslapd-maxdescriptors -------------------------------------------------------------------------- - RHEL 8 nsslapd-maxdescriptors: 262144 - RHEL 9 nsslapd-maxdescriptors: 524288 dsconf <instance> monitor server |grep dtablesize -------------------------------------------------------------------- - RHEL 8 dtablesize: 262144 - RHEL 9 dtablesize: 524288 Possible FIX ========== Looks like the fix would be to have dtablesize set to maxfiledescriptors or the hard maxfiledescriptor limit.