RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2210491 - dtablesize being set to soft maxfiledescriptor limit causing massive slowdown in large enviroments [8.9.0]
Summary: dtablesize being set to soft maxfiledescriptor limit causing massive slowdown...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: 389-ds-base
Version: 8.8
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: 8.9
Assignee: Jamie Chapman
QA Contact: LDAP QA Team
URL:
Whiteboard: sync-to-jira
Depends On:
Blocks: 2235330 2247075 2251375 2265538
TreeView+ depends on / blocked
 
Reported: 2023-05-27 22:02 UTC by Jerone Young
Modified: 2024-02-22 15:48 UTC (History)
14 users (show)

Fixed In Version: 389-ds-1.4-820230816162424-17499975
Doc Type: Bug Fix
Doc Text:
.Directory Server now calculates the `dtablesize` based on the maximum number of opened descriptors Previously, an administrator could set the connection table size manually by using the `nsslapd-conntablesize` configuration parameter. Consequently, when the connection table size was set too low, it affected the number of connections the server was able to support. With this update, Directory Server now calculates the size of the connection table dynamically effectively resolving the issue with too small connection table size. In addition, you no longer need to manually change the connection table size.
Clone Of:
: 2235330 2247075 2251375 2265538 (view as bug list)
Environment:
Last Closed: 2023-11-14 15:32:25 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github 389ds 389-ds-base issues 5804 0 None open dtablesize being set to soft maxfiledescriptor limit 2023-07-19 13:04:50 UTC
Github 389ds 389-ds-base pull 5806 0 None open Issue 5804 - dtablesize being set to soft maxfiledescriptor limit 2023-07-05 23:17:56 UTC
Red Hat Issue Tracker IDMDS-3414 0 None None None 2023-07-19 13:06:36 UTC
Red Hat Issue Tracker IDMDS-3518 0 None None None 2023-08-07 20:49:10 UTC
Red Hat Issue Tracker IDMDS-3662 0 None None None 2023-09-20 12:45:33 UTC
Red Hat Issue Tracker IDMDS-3663 0 None None None 2023-09-20 12:45:40 UTC
Red Hat Issue Tracker RHELPLAN-158330 0 None None None 2023-05-27 22:03:45 UTC
Red Hat Issue Tracker RHELPLAN-160135 0 None None None 2023-06-20 19:01:03 UTC
Red Hat Product Errata RHBA-2023:6965 0 None None None 2023-11-14 15:33:08 UTC

Description Jerone Young 2023-05-27 22:02:36 UTC
Description of problem:
 It looks like dtablesize is being set to the maxfiledescriptors soft limit. 

 This problem was found while migrating from RHEL 7 IDM to RHEL 8 IDM at large client site. I came up at first with systemd override for maxfiledescriptors with value 65535 that resolved issues at client site. Though after further investigation was able to see the issue was not the value of the maxfiledescriptors. But that dtablesize was being set to the soft limit. 


Version-Release number of selected component (if applicable):
     Affects both RHEL 8 & RHEL 9.


How reproducible:

It looks like dtablesize is being set to the maxfiledescriptors soft limit. 

Problem:
=======
     389ds is not setting dtablesize properly based when systemd is setting the maxfiledescriptors with it's default settings.
     dtablesize stays 1024 which causes massive slowdown once you hit around 950 connection. Basically queries are very slow. Also can't
     go above around ~970 concurrent connections.

      It looks like dtablesize is being set to the soft limit when it needs to be set to the actual limit. 

      So with no changes settings you see running commands:
                systemctl show dirsrv@<INSTANCE> |grep -i Limitnofile
                -----------------------------------------------------------------------------
                - RHEL 8 (Directory Server)
                      LimitNOFILE=262144
                      LimitNOFILESoft=1024      < ---- Notice

                - RHEL 9 (IDM system)
                     LimitNOFILE=524288
                     LimitNOFILESoft=1024        <--- Notice

               dsconf  <instance> config get nsslapd-maxdescriptors
               --------------------------------------------------------------------------
                - RHEL 8 
                      nsslapd-maxdescriptors: 262144

                - RHEL 9
                       nsslapd-maxdescriptors: 524288


                - 
 (THE PROBLEM)     dsconf <instance> monitor server |grep dtablesize 
                                   --------------------------------------------------------------------
                                     - RHEL 8
                                            dtablesize: 1024

                                     - RHEL 9
                                            dtablesize: 1024


WorkAround:
==========
        Once in place connections can go very very high. Testing showed 7,000+ at client site. Also queries very fast.

        Looks like this works because the override sets both hard and soft limit. So dtablesize gets a properly.
         
         Workaround
         -----------------
         - RHEL 8
                    mkdir -p /etc/systemd/system/dirsrv@.service.d/
                    cat > /etc/systemd/system/dirsrv@.service.d/filelimts.conf << EOF
                    [Service]
                    LimitNOFILE=262144
                    EOF
                  
                    systemctl daemon-reload
                    systemctl restart dirsrv@<instance>
                      
          - RHEL 9
                    mkdir -p /etc/systemd/system/dirsrv@.service.d/
                    cat > /etc/systemd/system/dirsrv@.service.d/filelimts.conf << EOF
                    [Service]
                    LimitNOFILE=524288
                    EOF
                  
                    systemctl daemon-reload
                    systemctl restart dirsrv@<instance>       
          
               systemctl show dirsrv@<INSTANCE> |grep -i Limitnofile
                -----------------------------------------------------------------------------
                - RHEL 8 (Directory Server)
                          LimitNOFILE=262144
                          LimitNOFILESoft=262144    <---- THIS !!
 
                - RHEL 9 (IDM system)
                           LimitNOFILE=524288             
                           LimitNOFILESoft=524288   <-- THIS !!
                           
                    
               dsconf  <instance> config get nsslapd-maxdescriptors
               --------------------------------------------------------------------------
                - RHEL 8 
                      nsslapd-maxdescriptors: 262144

                - RHEL 9
                       nsslapd-maxdescriptors: 524288

              
                dsconf <instance> monitor server |grep dtablesize 
                --------------------------------------------------------------------
                - RHEL 8
                         dtablesize:  262144

                 - RHEL 9
                         dtablesize: 524288



Possible FIX
==========
     Looks like the fix would be to have dtablesize set to maxfiledescriptors or the hard maxfiledescriptor limit.

Comment 6 Marc Sauton 2023-07-20 17:29:47 UTC
the dtablesize has been documented in the RHDS perf guide until RHDS-11 ( no longer in RHDS-12 )
https://access.redhat.com/documentation/en-us/red_hat_directory_server/11/pdf/performance_tuning_guide/red_hat_directory_server-11-performance_tuning_guide-en-us.pdf
2.1.1. Monitoring the Directory Server Using the Command Line

a note could be added in

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/pdf/tuning_performance_in_identity_management/red_hat_enterprise_linux-9-tuning_performance_in_identity_management-en-us.pdf
for
a system general configuration with sysctl for
somaxconn
dtablesize
and/or
Chapter 6. Adjusting IdM Directory Server performance
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/tuning_performance_in_identity_management/adjusting-idm-directory-server-performance_tuning-performance-in-idm
and/or
Chapter 7. Adjusting the performance of the KDC
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/tuning_performance_in_identity_management/assembly_adjusting-the-performance-of-the-kdc_tuning-performance-in-idm

but note such tuning may displace a bottleneck where a prior connection throttling will now hammer the LDAP worker threads and cause a lot more contention with plug-ins.

Comment 7 Jerone Young 2023-07-20 17:35:34 UTC
@Marc
     Upstream has already confirmed this as a bug and are working on it now.
 
     dtablesize is in RHDS-12 I show it in the example.

     The workaround provided is being used and has been proven by customers.

Comment 8 Jerone Young 2023-07-20 17:41:59 UTC
@Marc
     What I get for multi-tasking. Yes you are correct that notes should be added to documentation.

Comment 10 Viktor Ashirov 2023-07-24 14:57:46 UTC
Patch was merged upstream, moving to POST.

Comment 19 Viktor Ashirov 2023-08-21 12:48:02 UTC
Automated test passed:
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.6.8, pytest-3.4.2, py-1.5.3, pluggy-0.6.0 -- /usr/libexec/platform-python
cachedir: dirsrvtests/.pytest_cache
389-ds-base: 1.4.3.37-1.module+el8.9.0+19689+7d653af8
nss: 3.90.0-3.el8_8
nspr: 4.35.0-1.el8_8
openldap: 2.4.46-18.el8
cyrus-sasl: not installed
FIPS: disabled
rootdir: /root/ds/dirsrvtests, inifile: pytest.ini
collected 1 item

dirsrvtests/tests/suites/resource_limits/fdlimits_test.py::test_reserve_descriptor_validation PASSED                                                                                                         [100%]

============================================================================================ 1 passed in 39.59 seconds =============================================================================================

Marking as VERIFIED:Tested.

Comment 20 joel 2023-08-21 17:49:21 UTC
Hello,

For the customer tam would like to know what this test verified?

 "  I saw that the latest private message Bug 2210491 can you help understand what "automated test passed" means?

Thanks

Comment 21 Viktor Ashirov 2023-08-21 18:20:34 UTC
Hi,

it verified that the next 389-ds-base build for RHEL 8.9 contains the fix for this bz.

HTH

Comment 34 errata-xmlrpc 2023-11-14 15:32:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (389-ds:1.4 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6965

Comment 35 Jerone Young 2023-11-14 18:31:12 UTC
This issue is now resolved in RHEL 8. But RHEL 9 still does not have this issue resolved.

Comment 36 Jerone Young 2023-11-16 14:09:52 UTC
Issue is resolved in RHEL 8.8 & 9.3 for IDM installs.

The issue is still there for RHDS installs as it looks like it installs an older version of 389-ds-base packages that do not contain the fix.


Note You need to log in before you can comment on or make changes to this bug.