Bug 2165605 - RHDS 12 Performance notably worse than RHDS 10
Summary: RHDS 12 Performance notably worse than RHDS 10
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Directory Server
Classification: Red Hat
Component: 389-ds-base
Version: 12.0
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: ---
Assignee: LDAP Maintainers
QA Contact: LDAP QA Team
Evgenia Martynyuk
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-01-30 14:26 UTC by George Srutkowski
Modified: 2023-08-16 16:12 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-08-16 16:12:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
SLAMD testing configuration (182.98 KB, application/pdf)
2023-01-30 14:26 UTC, George Srutkowski
no flags Details
Internal Red Hat Testing Results (54.92 KB, application/vnd.oasis.opendocument.text)
2023-04-02 22:00 UTC, mreynolds
no flags Details

Description George Srutkowski 2023-01-30 14:26:26 UTC
Created attachment 1941095 [details]
SLAMD testing configuration

Description of problem:
Customer was expecting improvement to performance RHDS 12, and has instead found performance to be anywhere from negligable to worse than RHDS 10 via SLAMD testing results.

Version-Release number of selected component (if applicable):
RHDS 12 -- 
389-ds-base-2.1.5-4.module+el9dsrv+16995+8a75ed16.x86_64    Thu Jan 12 09:25:27 2023
389-ds-base-libs-2.1.5-4.module+el9dsrv+16995+8a75ed16.x86_64 Thu Jan 12 09:25:26 2023

RHDS 10 --
389-adminutil-1.1.23-2.el7dsrv.x86_64                       Tue Mar  5 11:43:45 2019
389-admin-1.1.46-4.el7.x86_64                               Tue Mar 29 19:13:38 2022
389-admin-console-1.1.12-2.el7dsrv.noarch                   Tue Mar  5 11:43:59 2019
389-admin-console-doc-1.1.12-2.el7dsrv.noarch               Tue Mar  5 11:43:59 2019
389-console-1.1.19-6.el7.noarch                             Thu Oct 24 19:29:11 2019
389-ds-base-1.3.10.2-17.el7_9.x86_64                        Tue Nov 22 19:11:32 2022
389-ds-base-libs-1.3.10.2-17.el7_9.x86_64                   Tue Nov 22 19:11:24 2022
389-ds-console-1.2.16-1.el7dsrv.noarch                      Tue Mar  5 11:43:59 2019
389-ds-console-doc-1.2.16-1.el7dsrv.noarch                  Tue Mar  5 11:43:59 2019

How reproducible:
SLAMD is a performance testing platform for a variety of applications, including LDAP/Kerberos.

Steps to Reproduce:
1. Install SLAMD latest version from GitHub
2. Follow instructions for testing.
3. See attachments for configuration of SLAMD testing

Actual results:
See attachments for customer SLAMD testing results; "a  25% decrease in performance over RHDS 10"

Expected results:
Notable performance increase over RHDS 10

Additional info:
customer is NYS ITS
GSS Support case: https://gss--c.vf.force.com/apex/Case_View?id=5006R00001
Customer Portal: https://access.redhat.com/support/cases/#/case/03420041

Comment 10 George Srutkowski 2023-02-06 16:33:43 UTC
*** Copied from Support Ticket 03420041 ***

(replying to Srutkowski, George) 
> I apologize for not providing an update sooner; I've been communicating with the engineering team to come up with a better solution than what they've provided;  currently, the only solution they've been able to provide is to switch the password hash for all users to the less secure solution. We're continuing to examine the data you've provided while methodically testing and recording results.  I understand that the safety and security of the users in your database are of the utmost importance; Will this be a feasible solution for you?
George, 
Thank you for the update. Switching password storage scheme to SSHA512 is not an option we will be considering. We will wait for other options from engineering.

Comment 11 mreynolds 2023-02-06 18:14:01 UTC
(In reply to George Srutkowski from comment #10)
> *** Copied from Support Ticket 03420041 ***
> 
> (replying to Srutkowski, George) 
> > I apologize for not providing an update sooner; I've been communicating with the engineering team to come up with a better solution than what they've provided;  currently, the only solution they've been able to provide is to switch the password hash for all users to the less secure solution. We're continuing to examine the data you've provided while methodically testing and recording results.  I understand that the safety and security of the users in your database are of the utmost importance; Will this be a feasible solution for you?
> George, 
> Thank you for the update. Switching password storage scheme to SSHA512 is
> not an option we will be considering. We will wait for other options from
> engineering.

There are no other options if they want to compare RHDS 10 to RHDS 12.  The config and password storage scheme must be the same between the versions of RHDS as the password storage scheme PBKDF2-SHA512 is designed to be slower (but more secure) than SSHA512.  No idea why they are refusing to make this simple change, because from the engineering standpoint what they are seeing (this perf drop on binds) is expected with the default password scheme used in RHDS 12.  If they want the binds to be more performant then they need to change the password storage scheme to what is used in RHDS 10 "before" loading the entries into RHDS 12.  Or, don't change the storage scheme and remove the "binds" from the slamd configuration and rerun the tests on both RHDS 10 and 12.

Comment 13 George Srutkowski 2023-02-22 15:01:22 UTC
Customer has followed steps and provided SLAMD testing results.  Latest round (Round 2) has been uploaded to bugzilla.  

Comment from customer:
Steps followed: 

1. Verified that the password storage scheme is PBKDF2-SHA512 and this is reflected as such on the slamd accounts, those paasswords show up as PBKDF2-SHA512.
2. All RHDS 12 servers have the password storage scheme changed to SSHA512. Restarted.
3. Verified that slamd accounts still have the password storage scheme as PBKDF2-SHA512. 
4.  First round of slamd test performed. This is the first bind for slamd acccounts since the change to the password storage scheme. Verified that they now        reflect the change to SSHA512.
5. Another round of slamd test performed. This has slightly better results, presumably because the passwords have already been changed to SSHA512. But still very much lower     than RHDS 10.

Comment 15 Ding-Yi Chen 2023-03-02 04:34:14 UTC
> 1. Verified that the password storage scheme is PBKDF2-SHA512 and this is reflected as such on the slamd accounts, those paasswords show up as PBKDF2-SHA512.
> 2. All RHDS 12 servers have the password storage scheme changed to SSHA512. Restarted.
> 3. Verified that slamd accounts still have the password storage scheme as PBKDF2-SHA512. 

Changing the default password storage scheme won't change the password store scheme of existing password.

You will need to change passwords to use the new default password storage scheme.

Comment 19 mreynolds 2023-04-02 22:00:44 UTC
Created attachment 1955349 [details]
Internal Red Hat Testing Results

Red Hat Internal Testing Update

Using the same slamd job I ran tests against RHDS 10, 12.1, and 12.3.  I made sure all the user entry passwords were hashed using SSHA512.  

RHDS 12.3 is 32%-37% faster than RHDS 10.  All RHDS 12 versions were faster than RHDS 10.  I suspect in the customer tests they still did not have passwords hashed using the same algorithm.  The attached document has all the testing steps, data, and results.


Note You need to log in before you can comment on or make changes to this bug.