Bug 2165605

Summary: RHDS 12 Performance notably worse than RHDS 10
Product: Red Hat Directory Server Reporter: George Srutkowski <gsrutkow>
Component: 389-ds-baseAssignee: LDAP Maintainers <idm-ds-dev-bugs>
Status: CLOSED INSUFFICIENT_DATA QA Contact: LDAP QA Team <idm-ds-qe-bugs>
Severity: medium Docs Contact: Evgenia Martynyuk <emartyny>
Priority: unspecified    
Version: 12.0CC: czinda, dchen, idm-ds-dev-bugs, jyoung, mharmsen, mreynolds, msauton, musoni, tbordaz, tmihinto, vashirov, vvanhaft
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-08-16 16:12:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
SLAMD testing configuration
none
Internal Red Hat Testing Results none

Description George Srutkowski 2023-01-30 14:26:26 UTC
Created attachment 1941095 [details]
SLAMD testing configuration

Description of problem:
Customer was expecting improvement to performance RHDS 12, and has instead found performance to be anywhere from negligable to worse than RHDS 10 via SLAMD testing results.

Version-Release number of selected component (if applicable):
RHDS 12 -- 
389-ds-base-2.1.5-4.module+el9dsrv+16995+8a75ed16.x86_64    Thu Jan 12 09:25:27 2023
389-ds-base-libs-2.1.5-4.module+el9dsrv+16995+8a75ed16.x86_64 Thu Jan 12 09:25:26 2023

RHDS 10 --
389-adminutil-1.1.23-2.el7dsrv.x86_64                       Tue Mar  5 11:43:45 2019
389-admin-1.1.46-4.el7.x86_64                               Tue Mar 29 19:13:38 2022
389-admin-console-1.1.12-2.el7dsrv.noarch                   Tue Mar  5 11:43:59 2019
389-admin-console-doc-1.1.12-2.el7dsrv.noarch               Tue Mar  5 11:43:59 2019
389-console-1.1.19-6.el7.noarch                             Thu Oct 24 19:29:11 2019
389-ds-base-1.3.10.2-17.el7_9.x86_64                        Tue Nov 22 19:11:32 2022
389-ds-base-libs-1.3.10.2-17.el7_9.x86_64                   Tue Nov 22 19:11:24 2022
389-ds-console-1.2.16-1.el7dsrv.noarch                      Tue Mar  5 11:43:59 2019
389-ds-console-doc-1.2.16-1.el7dsrv.noarch                  Tue Mar  5 11:43:59 2019

How reproducible:
SLAMD is a performance testing platform for a variety of applications, including LDAP/Kerberos.

Steps to Reproduce:
1. Install SLAMD latest version from GitHub
2. Follow instructions for testing.
3. See attachments for configuration of SLAMD testing

Actual results:
See attachments for customer SLAMD testing results; "a  25% decrease in performance over RHDS 10"

Expected results:
Notable performance increase over RHDS 10

Additional info:
customer is NYS ITS
GSS Support case: https://gss--c.vf.force.com/apex/Case_View?id=5006R00001
Customer Portal: https://access.redhat.com/support/cases/#/case/03420041

Comment 10 George Srutkowski 2023-02-06 16:33:43 UTC
*** Copied from Support Ticket 03420041 ***

(replying to Srutkowski, George) 
> I apologize for not providing an update sooner; I've been communicating with the engineering team to come up with a better solution than what they've provided;  currently, the only solution they've been able to provide is to switch the password hash for all users to the less secure solution. We're continuing to examine the data you've provided while methodically testing and recording results.  I understand that the safety and security of the users in your database are of the utmost importance; Will this be a feasible solution for you?
George, 
Thank you for the update. Switching password storage scheme to SSHA512 is not an option we will be considering. We will wait for other options from engineering.

Comment 11 mreynolds 2023-02-06 18:14:01 UTC
(In reply to George Srutkowski from comment #10)
> *** Copied from Support Ticket 03420041 ***
> 
> (replying to Srutkowski, George) 
> > I apologize for not providing an update sooner; I've been communicating with the engineering team to come up with a better solution than what they've provided;  currently, the only solution they've been able to provide is to switch the password hash for all users to the less secure solution. We're continuing to examine the data you've provided while methodically testing and recording results.  I understand that the safety and security of the users in your database are of the utmost importance; Will this be a feasible solution for you?
> George, 
> Thank you for the update. Switching password storage scheme to SSHA512 is
> not an option we will be considering. We will wait for other options from
> engineering.

There are no other options if they want to compare RHDS 10 to RHDS 12.  The config and password storage scheme must be the same between the versions of RHDS as the password storage scheme PBKDF2-SHA512 is designed to be slower (but more secure) than SSHA512.  No idea why they are refusing to make this simple change, because from the engineering standpoint what they are seeing (this perf drop on binds) is expected with the default password scheme used in RHDS 12.  If they want the binds to be more performant then they need to change the password storage scheme to what is used in RHDS 10 "before" loading the entries into RHDS 12.  Or, don't change the storage scheme and remove the "binds" from the slamd configuration and rerun the tests on both RHDS 10 and 12.

Comment 13 George Srutkowski 2023-02-22 15:01:22 UTC
Customer has followed steps and provided SLAMD testing results.  Latest round (Round 2) has been uploaded to bugzilla.  

Comment from customer:
Steps followed: 

1. Verified that the password storage scheme is PBKDF2-SHA512 and this is reflected as such on the slamd accounts, those paasswords show up as PBKDF2-SHA512.
2. All RHDS 12 servers have the password storage scheme changed to SSHA512. Restarted.
3. Verified that slamd accounts still have the password storage scheme as PBKDF2-SHA512. 
4.  First round of slamd test performed. This is the first bind for slamd acccounts since the change to the password storage scheme. Verified that they now        reflect the change to SSHA512.
5. Another round of slamd test performed. This has slightly better results, presumably because the passwords have already been changed to SSHA512. But still very much lower     than RHDS 10.

Comment 15 Ding-Yi Chen 2023-03-02 04:34:14 UTC
> 1. Verified that the password storage scheme is PBKDF2-SHA512 and this is reflected as such on the slamd accounts, those paasswords show up as PBKDF2-SHA512.
> 2. All RHDS 12 servers have the password storage scheme changed to SSHA512. Restarted.
> 3. Verified that slamd accounts still have the password storage scheme as PBKDF2-SHA512. 

Changing the default password storage scheme won't change the password store scheme of existing password.

You will need to change passwords to use the new default password storage scheme.

Comment 19 mreynolds 2023-04-02 22:00:44 UTC
Created attachment 1955349 [details]
Internal Red Hat Testing Results

Red Hat Internal Testing Update

Using the same slamd job I ran tests against RHDS 10, 12.1, and 12.3.  I made sure all the user entry passwords were hashed using SSHA512.  

RHDS 12.3 is 32%-37% faster than RHDS 10.  All RHDS 12 versions were faster than RHDS 10.  I suspect in the customer tests they still did not have passwords hashed using the same algorithm.  The attached document has all the testing steps, data, and results.