Bug 1472255

Summary:	Improve SSSD performance in the 7.5 release
Product:	Red Hat Enterprise Linux 7	Reporter:	Jakub Hrozek <jhrozek>
Component:	sssd	Assignee:	SSSD Maintainers <sssd-maint>
Status:	CLOSED ERRATA	QA Contact:	Amith <apeetham>
Severity:	unspecified	Docs Contact:	Aneta Šteflová Petrová <apetrova>
Priority:	unspecified
Version:	7.4	CC:	enewland, grajaiya, gscott, jhrozek, lslebodn, mkosek, mzidek, pasik, pbrezina, rlsmith, rmanes, sgoveas, sssd-maint, tscherf
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	sssd-1.16.0-7.el7	Doc Type:	Enhancement
Doc Text:	SSSD performance has improved This update provides several performance-related enhancements for the System Security Services Daemon (SSSD). Most notably: * Several missing indexes have been added in the SSSD cache, which makes lookups of cached objects faster. * Changes to how users and groups are saved prevent the SSSD cache performance degradation that occurred after the cache was populated with a large number of cached objects. As a result, SSSD reads user and group objects, especially large groups, faster. Also, the SSSD cache performance can now remain stable even when the cache size and the number of cache objects increase.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-04-10 17:13:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1482555
Bug Blocks:	1472344, 1477926, 1490412

Description Jakub Hrozek 2017-07-18 10:13:11 UTC

Description of problem:
This is sort of a general catch-all bugzilla to help during 7.5 planning. While we might not know yet what exact performance enhancements we might do, we do know that we need to work on performance.

Candidates for the 7.5 work include the indexing and objectclass/objectcategory fixes Sumit has been working on lately, improving the LDAP parsing performance (https://bugzilla.redhat.com/show_bug.cgi?id=1399135) or improvements around the logic in LDAP nested group code (https://pagure.io/SSSD/sssd/issue/3211) or using fewer allocations (https://pagure.io/SSSD/sssd/issue/2882)

The bugzilla will be linked with upstream tickets later.

The extent might be decided as late as 7.5 development phase, because we might find out that some enhancements require larger refactoring than possible in 7.x

Version-Release number of selected component (if applicable):
this will be implemented in 1.16.x

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Jakub Hrozek 2017-08-14 13:43:51 UTC

Upstream ticket:
https://pagure.io/SSSD/sssd/issue/3472

Comment 4 Martin Kosek 2017-08-17 15:13:49 UTC

There is now special Bugzilla for https://pagure.io/SSSD/sssd/issue/3472 --> Bug 1482555.

Comment 6 Joe Vlcek 2017-10-09 14:14:11 UTC

*** Bug 1486943 has been marked as a duplicate of this bug. ***

Comment 10 Greg Scott 2017-10-24 11:57:09 UTC

Oh - yes - I also need to ask about the possibility of back-porting any patches from this BZ into RHEL 7.4.z.

thanks

- Greg

Comment 11 Jakub Hrozek 2017-10-24 12:28:50 UTC

(In reply to Greg Scott from comment #10)
> Oh - yes - I also need to ask about the possibility of back-porting any
> patches from this BZ into RHEL 7.4.z.
> 
> thanks
> 
> - Greg

No, sorry, only 7.5

Comment 18 Amith 2018-01-23 19:29:02 UTC

Verified this bug on SSSD Version : sssd-1.16.0-14.el7.x86_64

Automated tests were executed on RHEL-7.2, RHEL-7.4 and RHEL-7.5 to generate the below performance report. The table below shows the best results collected from multiple rounds of test.

GLOSSARY OF VARIABLES USED:
----------------------------------------------------------------------------------
User login time statistics for member of 200 groups is   : TEST_200_LOGIN_STAT
User login time statistics for member of 500 groups is   : TEST_500_LOGIN_STAT
User login time statistics for member of 1000 groups is  : TEST_1000_LOGIN_STAT
User login time statistics for member of 1500 groups is  : TEST_1500_LOGIN_STAT
User login time statistics for member of 2000 groups is  : TEST_2000_LOGIN_STAT
User login time statistics for member of 3000 groups is  : TEST_3000_LOGIN_STAT
ID cmd time statistics for member of 200 groups is       : TEST_200_ID_STAT
ID cmd time statistics for member of 500 groups is       : TEST_500_ID_STAT
ID cmd time statistics for member of 1000 groups is      : TEST_1000_ID_STAT
ID cmd time statistics for member of 1500 groups is      : TEST_1500_ID_STAT
ID cmd time statistics for member of 2000 groups is      : TEST_2000_ID_STAT
ID cmd time statistics for member of 3000 groups is      : TEST_3000_ID_STAT
Group lookup stats with 200 users in a group is          : TEST_200_GRP_LKP
Group lookup stats with 500 users in a group is          : TEST_500_GRP_LKP
Group lookup stats with 1000 users in a group is         : TEST_1000_GRP_LKP
Group lookup stats with 1500 users in a group is         : TEST_1500_GRP_LKP
Group lookup stats with 2000 users in a group is         : TEST_2000_GRP_LKP
Group lookup stats with 3000 users in a group is         : TEST_3000_GRP_LKP
Time statistics for listing files owned by 200 users is  : TEST_200_LIST_STAT
Time statistics for listing files owned by 500 users is  : TEST_500_LIST_STAT
Time statistics for listing files owned by 1000 users is : TEST_1000_LIST_STAT
Time statistics for listing files owned by 1500 users is : TEST_1500_LIST_STAT
Time statistics for listing files owned by 2000 users is : TEST_2000_LIST_STAT
Time statistics for listing files owned by 3000 users is : TEST_3000_LIST_STAT
----------------------------------------------------------------------------------

PERFORMANCE REPORT IN COMPARISON WITH RHEL-7.2 & RHEL-7.4 VERSION:
---------------------------------------------------------------------------------
VARIABLES            |    RHEL-7.2     |     RHEL-7.4    | RHEL-7.5 Beta (Maipo)
---------------------------------------------------------------------------------
TEST_200_LOGIN_STAT  |  00m : 0.8046s  |  00m : 0.9540s  |   00m : 0.7743s    
TEST_500_LOGIN_STAT  |  00m : 2.1793s  |  00m : 1.6884s  |   00m : 1.3115s 
TEST_1000_LOGIN_STAT |  00m : 5.0977s  |  00m : 4.4408s  |   00m : 2.9948s 
TEST_1500_LOGIN_STAT |  00m : 7.3003s  |  00m : 6.5553s  |   00m : 4.3138s 
TEST_2000_LOGIN_STAT |  00m : 11.7889s |  00m : 8.7154s  |   00m : 5.1645s 
TEST_3000_LOGIN_STAT |  00m : 49.2608s |  00m : 10.3661s |   00m : 7.9282s 
                     |                 |                 |
TEST_200_ID_STAT     |  01m : 57.9297s |  00m : 8.7441s  |   00m : 6.3670s 
TEST_500_ID_STAT     |  04m : 54.1192s |  00m : 19.9281s |   00m : 18.5315s
TEST_1000_ID_STAT    |  09m : 47.6430s |  00m : 42.6871s |   00m : 34.1009s
TEST_1500_ID_STAT    |  14m : 46.5822s |  01m : 5.9490s  |   00m : 55.4492s
TEST_2000_ID_STAT    |  19m : 54.5008s |  01m : 41.0708s |   01m : 18.3255s
TEST_3000_ID_STAT    |  30m : 3.6612s  |  02m : 39.2325s |   02m : 5.7168s 
                     |                 |                 |
TEST_200_GRP_LKP     |  00m : 0.3210s  |  00m : 0.2295s  |   00m : 0.1384s 
TEST_500_GRP_LKP     |  00m : 0.5551s  |  00m : 0.3717s  |   00m : 0.2113s 
TEST_1000_GRP_LKP    |  00m : 1.0775s  |  00m : 0.5761s  |   00m : 0.3944s 
TEST_1500_GRP_LKP    |  00m : 1.6610s  |  00m : 0.8278s  |   00m : 0.4977s 
TEST_2000_GRP_LKP    |  00m : 2.3798s  |  00m : 1.1068s  |   00m : 0.6232s 
TEST_3000_GRP_LKP    |  00m : 4.4606s  |  00m : 1.5501s  |   00m : 0.9165s 
                     |                 |                 |
TEST_200_LIST_STAT   |  02m : 26.4913s |  00m : 21.1743s |   00m : 17.8590s
TEST_500_LIST_STAT   |  05m : 51.9399s |  00m : 42.2419s |   00m : 36.6246s
TEST_1000_LIST_STAT  |  11m : 44.4135s |  01m : 30.8535s |   01m : 18.1785s
TEST_1500_LIST_STAT  |  17m : 36.4835s |  02m : 22.5192s |   02m : 1.2889s 
TEST_2000_LIST_STAT  |  21m : 2.8683s  |  03m : 13.3497s |   02m : 43.8526s
TEST_3000_LIST_STAT  |  30m : 1.1197s  |  04m : 38.3401s |   04m : 50.5478s
--------------------------------------------------------------------------------- 

Test results from latest RHEL-7.5 release show slightly improved performance than RHEL-7.4. 
User lookup tests show improvements at a range of 8 - 30 seconds compared to RHEL-7.4, wherein lookup was conducted for member of 1000 - 3000 groups.
Group lookup shows improved performance at a range of 0.2 - 0.6 seconds compared to RHEL-7.4.

Comment 19 Lukas Slebodnik 2018-01-25 12:51:18 UTC

(In reply to Amith from comment #18)
> Verified this bug on SSSD Version : sssd-1.16.0-14.el7.x86_64
> 
> Automated tests were executed on RHEL-7.2, RHEL-7.4 and RHEL-7.5 to generate
> the below performance report. The table below shows the best results
> collected from multiple rounds of test.
> 
> GLOSSARY OF VARIABLES USED:
> -----------------------------------------------------------------------------
> -----
> User login time statistics for member of 200 groups is   :
> TEST_200_LOGIN_STAT
> User login time statistics for member of 500 groups is   :
> TEST_500_LOGIN_STAT
> User login time statistics for member of 1000 groups is  :
> TEST_1000_LOGIN_STAT
> User login time statistics for member of 1500 groups is  :
> TEST_1500_LOGIN_STAT
> User login time statistics for member of 2000 groups is  :
> TEST_2000_LOGIN_STAT
> User login time statistics for member of 3000 groups is  :
> TEST_3000_LOGIN_STAT

> ID cmd time statistics for member of 200 groups is       : TEST_200_ID_STAT
> ID cmd time statistics for member of 500 groups is       : TEST_500_ID_STAT
> ID cmd time statistics for member of 1000 groups is      : TEST_1000_ID_STAT
> ID cmd time statistics for member of 1500 groups is      : TEST_1500_ID_STAT
> ID cmd time statistics for member of 2000 groups is      : TEST_2000_ID_STAT
> ID cmd time statistics for member of 3000 groups is      : TEST_3000_ID_STAT
> Group lookup stats with 200 users in a group is          : TEST_200_GRP_LKP
> Group lookup stats with 500 users in a group is          : TEST_500_GRP_LKP
> Group lookup stats with 1000 users in a group is         : TEST_1000_GRP_LKP
> Group lookup stats with 1500 users in a group is         : TEST_1500_GRP_LKP
> Group lookup stats with 2000 users in a group is         : TEST_2000_GRP_LKP
> Group lookup stats with 3000 users in a group is         : TEST_3000_GRP_LKP
> Time statistics for listing files owned by 200 users is  : TEST_200_LIST_STAT
> Time statistics for listing files owned by 500 users is  : TEST_500_LIST_STAT
> Time statistics for listing files owned by 1000 users is :
> TEST_1000_LIST_STAT
> Time statistics for listing files owned by 1500 users is :
> TEST_1500_LIST_STAT
> Time statistics for listing files owned by 2000 users is :
> TEST_2000_LIST_STAT
> Time statistics for listing files owned by 3000 users is :
> TEST_3000_LIST_STAT
> -----------------------------------------------------------------------------
> -----
> 
> PERFORMANCE REPORT IN COMPARISON WITH RHEL-7.2 & RHEL-7.4 VERSION:
> -----------------------------------------------------------------------------
> ----
> VARIABLES            |    RHEL-7.2     |     RHEL-7.4    | RHEL-7.5 Beta
> (Maipo)
> -----------------------------------------------------------------------------
> ----
> TEST_200_LOGIN_STAT  |  00m : 0.8046s  |  00m : 0.9540s  |   00m : 0.7743s  
> 
If I understand correctly it is a time of login of single user which is a member  in 200 groups. But it is a little bit artificial test case. The real production usage is a little bit complicated. There is not just single user but hundreads of users. And there were complains that customer need to remove cache to get faster response. And We should really tried to avoid removing sssd cache in performance tests. (I know we need to do that sometimes to measure the same state.

It would be good to see also numbers for sequential authentication of 1000 users
which are member of 200 .. 500 groups and ideally have different group membership.

I also have a few questions. How many times did you run such tests?
Do you have some statistics? (min, max, average ...)
Did you test on real hardware for each distribution? (otherwise tests could be affected by virtualisation)
Were different distro tested in the same conditions? The same hardware (or at least the same resources provided by VM)

Comment 20 Amith 2018-01-26 15:01:32 UTC

(In reply to Lukas Slebodnik from comment #19)
> (In reply to Amith from comment #18)
> > Verified this bug on SSSD Version : sssd-1.16.0-14.el7.x86_64
> > 
> > Automated tests were executed on RHEL-7.2, RHEL-7.4 and RHEL-7.5 to generate
> > the below performance report. The table below shows the best results
> > collected from multiple rounds of test.
> > 
> > GLOSSARY OF VARIABLES USED:
> > -----------------------------------------------------------------------------
> > -----
> > User login time statistics for member of 200 groups is   :
> > TEST_200_LOGIN_STAT
> > User login time statistics for member of 500 groups is   :
> > TEST_500_LOGIN_STAT
> > User login time statistics for member of 1000 groups is  :
> > TEST_1000_LOGIN_STAT
> > User login time statistics for member of 1500 groups is  :
> > TEST_1500_LOGIN_STAT
> > User login time statistics for member of 2000 groups is  :
> > TEST_2000_LOGIN_STAT
> > User login time statistics for member of 3000 groups is  :
> > TEST_3000_LOGIN_STAT
> 
> > ID cmd time statistics for member of 200 groups is       : TEST_200_ID_STAT
> > ID cmd time statistics for member of 500 groups is       : TEST_500_ID_STAT
> > ID cmd time statistics for member of 1000 groups is      : TEST_1000_ID_STAT
> > ID cmd time statistics for member of 1500 groups is      : TEST_1500_ID_STAT
> > ID cmd time statistics for member of 2000 groups is      : TEST_2000_ID_STAT
> > ID cmd time statistics for member of 3000 groups is      : TEST_3000_ID_STAT
> > Group lookup stats with 200 users in a group is          : TEST_200_GRP_LKP
> > Group lookup stats with 500 users in a group is          : TEST_500_GRP_LKP
> > Group lookup stats with 1000 users in a group is         : TEST_1000_GRP_LKP
> > Group lookup stats with 1500 users in a group is         : TEST_1500_GRP_LKP
> > Group lookup stats with 2000 users in a group is         : TEST_2000_GRP_LKP
> > Group lookup stats with 3000 users in a group is         : TEST_3000_GRP_LKP
> > Time statistics for listing files owned by 200 users is  : TEST_200_LIST_STAT
> > Time statistics for listing files owned by 500 users is  : TEST_500_LIST_STAT
> > Time statistics for listing files owned by 1000 users is :
> > TEST_1000_LIST_STAT
> > Time statistics for listing files owned by 1500 users is :
> > TEST_1500_LIST_STAT
> > Time statistics for listing files owned by 2000 users is :
> > TEST_2000_LIST_STAT
> > Time statistics for listing files owned by 3000 users is :
> > TEST_3000_LIST_STAT
> > -----------------------------------------------------------------------------
> > -----
> > 
> > PERFORMANCE REPORT IN COMPARISON WITH RHEL-7.2 & RHEL-7.4 VERSION:
> > -----------------------------------------------------------------------------
> > ----
> > VARIABLES            |    RHEL-7.2     |     RHEL-7.4    | RHEL-7.5 Beta
> > (Maipo)
> > -----------------------------------------------------------------------------
> > ----
> > TEST_200_LOGIN_STAT  |  00m : 0.8046s  |  00m : 0.9540s  |   00m : 0.7743s  
> > 
> If I understand correctly it is a time of login of single user which is a
> member  in 200 groups. But it is a little bit artificial test case. The real
> production usage is a little bit complicated. There is not just single user
> but hundreads of users. And there were complains that customer need to
> remove cache to get faster response. And We should really tried to avoid
> removing sssd cache in performance tests. (I know we need to do that
> sometimes to measure the same state.
> 

Yes, user login tests were conducted for single user (member of 200 groups). Steps are as folows:
1. Execute User login to cache credential.
2. Expire cache by running, "sss_cache - E"
3. Run user login again, and calculate the login time.

> It would be good to see also numbers for sequential authentication of 1000
> users
> which are member of 200 .. 500 groups and ideally have different group
> membership.
> 

Sure. I have designed a test where 99 users simultaneously login into SSSD System. Then, the login time for the 100th user will be calculated.
But when this test is conducted with more than 200 test user, everything boils down to System hardware configurations. All our tests are executed on beaker VMs.

> I also have a few questions. How many times did you run such tests?

On RHEL-72, tests were executed many times, i lost the count. Everytime we get different results, so i reported only the best times. You are right, we should classify them into min, max and avg. On RHEL-74 and RHEL-75, test were executed atleast 3 times.

> Do you have some statistics? (min, max, average ...)

I will include them in the report.

> Did you test on real hardware for each distribution? (otherwise tests could
> be affected by virtualisation)

Tests were conducted on Beaker VMs. We don't have separate Server / Hardware configuration for this tests. The entire test suite takes 4 - 5 hours to finish execution.

> Were different distro tested in the same conditions? The same hardware (or
> at least the same resources provided by VM)

I ensured that the Beaker VMs, both server and client exist within the same lab. Hardware configurations were consistent. Overall my impression is that we have much better performance in RHEL-7.5, when compared with RHEL-7.2 and RHEL-7.4.

Comment 21 Amith 2018-02-06 16:39:23 UTC

Verified the bug on SSSD Version - sssd-1.16.0-15.el7.x86_64

All the tests were re-executed on latest SSSD build. Following scenarios were included:
1. Verify login time for a user who is member of large number of groups.
2. Verify id command execution time for a member of large number of groups.
3. Verify group lookup execution time for a group with large number of users.
4. Verify time taken by a user to list files owned by large number of users from different groups.
5. Verify login time for a user when large number of users attempt simultaneous login.

The systems used for testing SSSD Client had Multi-processors, RAM > 8000 MB and were non-virtualized.
Following table lists the glossary of VARIABLES which equates to time stats in min:sec, calculated from each test scenario.

GLOSSARY OF VARIABLES:
--------------------------------------------------------------------------
TEST_200_LOGIN_STAT : Login time for user who is member of 200 groups
TEST_500_LOGIN_STAT : Login time for user who is member of 500 groups
TEST_1000_LOGIN_STAT: Login time for user who is member of 1000 groups
TEST_1500_LOGIN_STAT: Login time for user who is member of 1500 groups
TEST_2000_LOGIN_STAT: Login time for user who is member of 2000 groups
TEST_3000_LOGIN_STAT: Login time for user who is member of 3000 groups

TEST_200_ID_STAT  : ID cmd execution time for member of 200 groups
TEST_500_ID_STAT  : ID cmd execution time for member of 500 groups
TEST_1000_ID_STAT : ID cmd execution time for member of 1000 groups
TEST_1500_ID_STAT : ID cmd execution time for member of 1500 groups
TEST_2000_ID_STAT : ID cmd execution time for member of 2000 groups
TEST_3000_ID_STAT : ID cmd execution time for member of 3000 groups

TEST_200_GRP_LKP  : Lookup time for fetching a group with 200 users
TEST_500_GRP_LKP  : Lookup time for fetching a group with 500 users
TEST_1000_GRP_LKP : Lookup time for fetching a group with 1000 users
TEST_1500_GRP_LKP : Lookup time for fetching a group with 1500 users
TEST_2000_GRP_LKP : Lookup time for fetching a group with 2000 users
TEST_3000_GRP_LKP : Lookup time for fetching a group with 3000 users

TEST_200_LIST_STAT : Time statistics for listing files owned by 200 users
TEST_500_LIST_STAT : Time statistics for listing files owned by 500 users
TEST_1000_LIST_STAT: Time statistics for listing files owned by 1000 users
TEST_1500_LIST_STAT: Time statistics for listing files owned by 1500 users
TEST_2000_LIST_STAT: Time statistics for listing files owned by 2000 users
TEST_3000_LIST_STAT: Time statistics for listing files owned by 3000 users

USER_LOGIN_100 : User login time when 100 users attempt simultaneous login 
USER_LOGIN_200 : User login time when 200 users attempt simultaneous login
USER_LOGIN_500 : User login time when 500 users attempt simultaneous login
USER_LOGIN_1000: User login time when 1000 users attempt simultaneous login
----------------------------------------------------------------------------

PERFORMANCE REPORT FROM RHEL SERVER -7.2 (Maipo)
-------------------------------------------------------------------------
VARIABLES            |    MINIMUM     |   MAXIMUM      |   AVERAGE      |
-------------------------------------------------------------------------
TEST_200_LOGIN_STAT  | 00m : 0.5492s  | 00m : 1.0234s  | 00m : 0.7284   |
TEST_500_LOGIN_STAT  | 00m : 1.0203s  | 00m : 2.1862s  | 00m : 1.4035s  |
TEST_1000_LOGIN_STAT | 00m : 4.4319s  | 00m : 5.5375s  | 00m : 4.8374s  |
TEST_1500_LOGIN_STAT | 00m : 7.8172s  | 00m : 9.8444s  | 00m : 8.5945s  |
TEST_2000_LOGIN_STAT | 00m : 11.1738s | 00m : 14.5479s | 00m : 12.5165s |
TEST_3000_LOGIN_STAT | 00m : 20.6529s | 00m : 27.2313s | 00m : 23.4657s |
                     |                |                |                |
TEST_200_ID_STAT     | 00m : 56.1155s | 01m : 47.6253s | 01m : 16.9608s |
TEST_500_ID_STAT     | 02m : 22.6959s | 04m : 40.6726s | 03m : 14.2868s |
TEST_1000_ID_STAT    | 05m : 29.8220s | 09m : 19.2609s | 06m : 26.5660s |
TEST_1500_ID_STAT    | 08m : 11.5678s | 14m : 38.4713s | 09m : 50.9305s |
TEST_2000_ID_STAT    | 11m : 4.3834s  | 19m : 43.1408s | 12m : 24.1885s |
TEST_3000_ID_STAT    | 14m : 8.0808s  | 26m : 19.5486s | 18m : 24.6781s |
                     |                |                |                |
TEST_200_GRP_LKP     | 00m : 0.0963s  | 00m : 0.1866s  | 00m : 0.1197s  |
TEST_500_GRP_LKP     | 00m : 0.2039s  | 00m : 0.3535s  | 00m : 0.2537s  |
TEST_1000_GRP_LKP    | 00m : 0.4426s  | 00m : 0.6722s  | 00m : 0.5509s  |
TEST_1500_GRP_LKP    | 00m : 0.7601s  | 00m : 1.0404s  | 00m : 0.9454s  |
TEST_2000_GRP_LKP    | 00m : 1.1345s  | 00m : 1.6496s  | 00m : 1.4276s  |
TEST_3000_GRP_LKP    | 00m : 2.2784s  | 00m : 3.2537s  | 00m : 2.8282s  |
                     |                |                |                |
TEST_200_LIST_STAT   | 02m : 19.7681s | 04m : 5.1930s  | 02m : 37.0602s |
TEST_500_LIST_STAT   | 05m : 46.0752s | 09m : 16.6836s | 06m : 14.7438s |
TEST_1000_LIST_STAT  | 10m : 19.9916s | 16m : 54.2219s | 12m : 26.0275s |
TEST_1500_LIST_STAT  | 15m : 35.2081s | 25m : 50.4367s | 18m : 28.2850s |
TEST_2000_LIST_STAT  | 19m : 9.9535s  | 35m : 20.5150s | 25m : 59.2109s |
TEST_3000_LIST_STAT  | 28m : 36.3182s | 43m : 43.6117s | 38m : 5.8602s  |
                     |                |                |                |
USER_LOGIN_100       | 00m : 2.0364s  | 00m : 6.8351s  | 00m : 2.5318s  |
USER_LOGIN_200       | 00m : 16.8303s | 00m : 28.9885s | 00m : 21.9612s |
USER_LOGIN_500       | 00m : 45.5015s | 01m : 37.3261s | 01m : 15.5131s |
USER_LOGIN_1000      | 00m : 50.9712s | 01m : 52.2236s | 01m : 31.8406s |
-------------------------------------------------------------------------

PERFORMANCE REPORT FROM RHEL SERVER RELEASE-7.4 (Maipo)
-------------------------------------------------------------------------
VARIABLES            |    MINIMUM     |   MAXIMUM      |   AVERAGE      |
-------------------------------------------------------------------------
TEST_200_LOGIN_STAT  | 00m : 0.7717s  | 00m : 0.7882s  | 00m : 0.7868s  |
TEST_500_LOGIN_STAT  | 00m : 1.1607s  | 00m : 1.2199s  | 00m : 1.2123s  |
TEST_1000_LOGIN_STAT | 00m : 2.6292s  | 00m : 2.9523s  | 00m : 2.6686s  |
TEST_1500_LOGIN_STAT | 00m : 3.8105s  | 00m : 4.1058s  | 00m : 3.8420s  |
TEST_2000_LOGIN_STAT | 00m : 5.0525s  | 00m : 5.6384s  | 00m : 5.1001s  |
TEST_3000_LOGIN_STAT | 00m : 7.9140s  | 00m : 8.5346s  | 00m : 8.1569s  |
                     |                |                |                |
TEST_200_ID_STAT     | 00m : 7.5649s  | 00m : 7.7717s  | 00m : 7.6051s  |
TEST_500_ID_STAT     | 00m : 19.7120s | 00m : 20.1928s | 00m : 20.0008s |
TEST_1000_ID_STAT    | 00m : 41.2511s | 00m : 43.5740s | 00m : 42.3056s |
TEST_1500_ID_STAT    | 01m : 6.0672s  | 01m : 9.0710s  | 01m : 8.0002s  |
TEST_2000_ID_STAT    | 01m : 31.5167s | 01m : 36.3051s | 01m : 35.2627s |
TEST_3000_ID_STAT    | 02m : 28.9140s | 02m : 39.5463s | 02m : 37.8036s |
                     |                |                |                |
TEST_200_GRP_LKP     | 00m : 0.0742s  | 00m : 0.0819s  | 00m : 0.0777s  |
TEST_500_GRP_LKP     | 00m : 0.1571s  | 00m : 0.1719s  | 00m : 0.1635s  |
TEST_1000_GRP_LKP    | 00m : 0.2957s  | 00m : 0.3677s  | 00m : 0.3137s  |
TEST_1500_GRP_LKP    | 00m : 0.4309s  | 00m : 0.4413s  | 00m : 0.4367s  |
TEST_2000_GRP_LKP    | 00m : 0.5742s  | 00m : 0.6063s  | 00m : 0.5872s  |
TEST_3000_GRP_LKP    | 00m : 0.8659s  | 00m : 0.9156s  | 00m : 0.8824s  |
                     |                |                |                |
TEST_200_LIST_STAT   | 00m : 17.5295s | 00m : 18.5104s | 00m : 18.0789s |
TEST_500_LIST_STAT   | 00m : 38.0053s | 00m : 39.2215s | 00m : 38.6181s |
TEST_1000_LIST_STAT  | 01m : 17.6583s | 01m : 23.8506s | 01m : 22.9394s |
TEST_1500_LIST_STAT  | 02m : 7.4562s  | 02m : 11.3178s | 02m : 11.1098s |
TEST_2000_LIST_STAT  | 03m : 3.8232s  | 03m : 8.9974s  | 03m : 6.0832s  |
TEST_3000_LIST_STAT  | 05m : 7.8501s  | 05m : 9.2810s  | 05m : 8.0724s  |
                     |                |                |                |
USER_LOGIN_100       | 00m : 1.0360s  | 00m : 1.1261s  | 00m : 1.0800s  |
USER_LOGIN_200       | 00m : 1.0176s  | 00m : 1.1098s  | 00m : 1.0729s  |
USER_LOGIN_500       | 00m : 3.1912s  | 00m : 6.6778s  | 00m : 4.3300s  |
USER_LOGIN_1000      | 00m : 12.4543s | 01m : 2.7378s  | 00m : 18.3621s |
-------------------------------------------------------------------------

PERFORMANCE REPORT FROM RHEL SERVER RELEASE-7.5 Beta (Maipo)
-------------------------------------------------------------------------
VARIABLES            |    MINIMUM     |   MAXIMUM      |   AVERAGE      |
-------------------------------------------------------------------------
TEST_200_LOGIN_STAT  | 00m : 0.7901s  | 00m : 0.7990s  | 00m : 0.7969s  |
TEST_500_LOGIN_STAT  | 00m : 1.2261s  | 00m : 1.3786s  | 00m : 1.2606s  |
TEST_1000_LOGIN_STAT | 00m : 2.6492s  | 00m : 3.0958s  | 00m : 2.7656s  |
TEST_1500_LOGIN_STAT | 00m : 3.8413s  | 00m : 4.3778s  | 00m : 3.9588s  |
TEST_2000_LOGIN_STAT | 00m : 4.7917s  | 00m : 6.0984s  | 00m : 4.9422s  |
TEST_3000_LOGIN_STAT | 00m : 7.3604s  | 00m : 8.7218s  | 00m : 7.4922s	|
                     |                |                |                |
TEST_200_ID_STAT     | 00m : 7.8253s  | 00m : 8.1509s  | 00m : 8.1130s  |
TEST_500_ID_STAT     | 00m : 19.9911s | 00m : 21.1010s | 00m : 20.8610s |
TEST_1000_ID_STAT    | 00m : 44.0296s | 00m : 45.5463s | 00m : 44.6411s |
TEST_1500_ID_STAT    | 01m : 8.7837s  | 01m : 11.3954s | 01m : 10.4171s |
TEST_2000_ID_STAT    | 01m : 35.0533s | 01m : 39.7182s | 01m : 38.4446s |
TEST_3000_ID_STAT    | 02m : 38.0361s | 02m : 41.7147s | 02m : 41.0588s |
                     |                |                |                |
TEST_200_GRP_LKP     | 00m : 0.0854s  | 00m : 0.0932s  | 00m : 0.0912s  |
TEST_500_GRP_LKP     | 00m : 0.1659s  | 00m : 0.2104s  | 00m : 0.1784s  |
TEST_1000_GRP_LKP    | 00m : 0.3298s  | 00m : 0.3482s  | 00m : 0.3331s  |
TEST_1500_GRP_LKP    | 00m : 0.4666s  | 00m : 0.5030s  | 00m : 0.4876s  |
TEST_2000_GRP_LKP    | 00m : 0.6103s  | 00m : 0.8174s  | 00m : 0.6484s  |
TEST_3000_GRP_LKP    | 00m : 0.9493s  | 00m : 0.9814s  | 00m : 0.9571s  |
                     |                |                |                |
TEST_200_LIST_STAT   | 00m : 18.7861s | 00m : 19.1635s | 00m : 18.9727s |
TEST_500_LIST_STAT   | 00m : 40.2218s | 00m : 41.0201s | 00m : 40.4128s |
TEST_1000_LIST_STAT  | 01m : 24.2030s | 01m : 26.8291s | 01m : 26.3331s |
TEST_1500_LIST_STAT  | 02m : 12.1911s | 02m : 19.3097s | 02m : 18.7828s |
TEST_2000_LIST_STAT  | 03m : 6.5289s  | 03m : 13.0436s | 03m : 11.4994s |
TEST_3000_LIST_STAT  | 05m : 8.1912s  | 05m : 19.3881s | 05m : 17.3072s |
                     |                |                |                |
USER_LOGIN_100       | 00m : 1.0337s  | 00m : 1.1438s  | 00m : 1.0585s  |
USER_LOGIN_200       | 00m : 1.0340s  | 00m : 1.3176s  | 00m : 1.0434s  |
USER_LOGIN_500       | 00m : 4.2392s  | 00m : 8.6088s  | 00m : 6.6729s  |
USER_LOGIN_1000      | 00m : 15.7396s | 01m : 7.7146s  | 00m : 17.9213s |
-------------------------------------------------------------------------

OBSERVATION:
1) In RHEL-7.5, average login time for a user who is member of 3000 groups is 7.4922 seconds compared to 8.1569 seconds in 7.4, which shows a marginal improvement in terms of microseconds.

2) In RHEL-7.5, avg. execution time of id command to fetch a member of 3000 groups is 2m : 41.0588s which is higher, when compared to 2m : 37.8036s in RHEL-7.4. This difference is close to 3 seconds which is negligible.

3) In RHEL-7.5, avg. time to list files owned by 3000 users from different groups is 5m : 17.3072s which is almost 8 seconds higher than the avg. time in RHEL-7.4.

4) In RHEL-7.5 shows marginally better performance in the case of avg. login time for a user when 1000 other users attempt simultaneous login. Value is 17.9213 secs which is slightly lower than RHEL-7.4 (18.3621s).

5) Overall RHEL-7.4 performs slightly better OR almost equivalent to 7.5 build. Based on the above mentioned cases, the margin of difference is in seconds which can be neglected. However, when the results are compared to RHEL-7.2 statistics then we see that the latest SSSD build is miles ahead and much more stable during load testing.

Comment 24 errata-xmlrpc 2018-04-10 17:13:24 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0929