Bug 904119

Summary: Significant decrease in migration performance
Product: Red Hat Enterprise Linux 6 Reporter: Jenny Severance <jgalipea>
Component: ipaAssignee: Martin Kosek <mkosek>
Status: CLOSED ERRATA QA Contact: Namita Soman <nsoman>
Severity: high Docs Contact:
Priority: urgent    
Version: 6.4CC: dpal, jwest, mkosek, nhosoi, tlavigne
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ipa-3.0.0-26.el6 Doc Type: Bug Fix
Doc Text:
During migration, users were added to the default users group for each user added. Adding users to a group which had thousands consumed time and caused a significant performance slowdown. With this patch, users are now added in batches of 100, which provides a considerable performance boost over the previous method.
Story Points: ---
Clone Of:
: 910554 (view as bug list) Environment:
Last Closed: 2013-11-21 20:48:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 908308, 910554    

Description Jenny Severance 2013-01-25 14:00:12 UTC
Description of problem:
Migration scenario is 10,000 users and 12 groups

RHEL 6.4

with compat plugin enabled

10 hours 14 minutes

:: [   LOG    ] :: slapd pid : 19259
:: [   LOG    ] :: Before migration free memory : 14951256064
:: [   LOG    ] :: Before migration slapd VmRSS : 28620kB
:: [   LOG    ] :: Before migration slapd VmHWM : 29096kB
:: [   LOG    ] :: ======================= Migration started: Thu Jan 17 05:13:32 EST 2013 ========================
:: [   LOG    ] :: EXECUTING: time -p echo mysecret | ipa migrate-ds --with-compat ldap://directoryserver.testrelm.com:389
:: [   PASS   ] :: Migration completed successfully.
:: [   LOG    ] :: ======================= Migration finished: Thu Jan 17 15:28:02 EST 2013 ========================
:: [   LOG    ] :: slapd pid : 19259
:: [   LOG    ] :: After migration free memory : 12555595776
:: [   LOG    ] :: After migration slapd VmRSS : 1183860kB
:: [   LOG    ] :: After migration slapd VmHWM : 1188992kB

with compat plugin disabled

18 hours 39 minutes

:: [   LOG    ] :: slapd pid : 19443
:: [   LOG    ] :: Before migration free memory : 15077851136
:: [   LOG    ] :: Before migration slapd VmRSS : 24744kB
:: [   LOG    ] :: Before migration slapd VmHWM : 24744kB
:: [   LOG    ] :: ======================= Migration started: Sat Jan 19 01:43:02 EST 2013 ========================
:: [   LOG    ] :: EXECUTING: echo mysecret | ipa migrate-ds ldap://directoryserver.testrelm.com:389
:: [   PASS   ] :: Migration completed successfully.
:: [   LOG    ] :: ======================= Migration finished: Sat Jan 19 20:22:05 EST 2013 ========================
:: [   LOG    ] :: slapd pid : 19443
:: [   LOG    ] :: After migration free memory : 12738027520
:: [   LOG    ] :: After migration slapd VmRSS : 1212460kB
:: [   LOG    ] :: After migration slapd VmHWM : 1217564kB



RHEL 6.3

with compat plugin enabled

7 hours 17 minutes

:: [   LOG    ] :: slapd pid : 21414
:: [   LOG    ] :: Before migration free memory : 14818889728
:: [   LOG    ] :: Before migration slapd VmRSS : 38200kB
:: [   LOG    ] :: Before migration slapd VmHWM : 41652kB
:: [   LOG    ] :: ======================= Migration started: Mon Apr  2 11:12:05 EDT 2012 ========================
:: [   LOG    ] :: EXECUTING: time -p echo mysecret | ipa migrate-ds --with-compat ldap://directoryserver.testrelm.com:389
:: [   PASS   ] :: Migration completed successfully.
:: [   LOG    ] :: ======================= Migration finished: Mon Apr  2 18:29:56 EDT 2012 ========================
:: [   LOG    ] :: slapd pid : 21414
:: [   LOG    ] :: After migration free memory : 13182754816
:: [   LOG    ] :: After migration slapd VmRSS : 772656kB
:: [   LOG    ] :: After migration slapd VmHWM : 772656kB


with compat plugin disabled

4 hours 36 minutes

:: [   LOG    ] :: slapd pid : 21409
:: [   LOG    ] :: Before migration free memory : 132102623232
:: [   LOG    ] :: Before migration slapd VmRSS : 25168kB
:: [   LOG    ] :: Before migration slapd VmHWM : 25168kB
:: [   LOG    ] :: ======================= Migration started: Mon Apr  9 13:26:12 EDT 2012 ========================
:: [   LOG    ] :: EXECUTING: echo mysecret | ipa migrate-ds ldap://directoryserver.testrelm.com:389
:: [   PASS   ] :: Migration completed successfully.
:: [   LOG    ] :: ======================= Migration finished: Mon Apr  9 18:02:18 EDT 2012 ========================
:: [   LOG    ] :: slapd pid : 21409
:: [   LOG    ] :: After migration free memory : 130853560320
:: [   LOG    ] :: After migration slapd VmRSS : 331900kB
:: [   LOG    ] :: After migration slapd VmHWM : 332432kB 

Version-Release number of selected component (if applicable):
ipa-server-3.0.0-22.el6.x86_64
389-ds-base-1.2.11.15-10.el6.x86_64

How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 2 RHEL Program Management 2013-01-29 06:49:24 UTC
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 3 Dmitri Pal 2013-01-31 22:10:22 UTC
Upstream ticket:
https://fedorahosted.org/freeipa/ticket/3386

Comment 8 Martin Kosek 2013-02-08 15:28:13 UTC
Regression fixed. migrate-ds command could crash with Internal Error if there was no group to migrate and --continue option was used.

master: https://fedorahosted.org/freeipa/changeset/08b84befbc40faca21ffe44dcc69aaf498a2f9f2
ipa-3-1: https://fedorahosted.org/freeipa/changeset/aa798f3ca0a08e457e7316b7017db5647d1df550
ipa-3-0: https://fedorahosted.org/freeipa/changeset/f0566b3856320334119968b81b6ea31a25bf8ea6

Comment 9 Namita Soman 2013-02-12 19:00:11 UTC
This bug also needs to be cloned for inclusion in RHEL 7.0

Comment 11 Namita Soman 2013-09-12 02:35:12 UTC
Verified using ipa-server-3.0.0-35

with compat plugin enabled

2 hours 44 minutes

:: [ 13:25:05 ] ::  slapd pid : 20288
:: [ 13:25:05 ] ::  Before migration free memory : 3498647552
:: [ 13:25:05 ] ::  Before migration slapd VmRSS : 30772kB
:: [ 13:25:05 ] ::  Before migration slapd VmHWM : 30872kB
:: [ 13:25:05 ] ::  ======================= Migration started: Mon Sep  9 13:25:05 EDT 2013 ========================
:: [ 13:25:05 ] ::  EXECUTING: time -p echo Secret123 | ipa migrate-ds --with-compat ldap://ipaqa64vmh.testrelm.com:389
:: [   PASS   ] :: Migration completed successfully. 
:: [ 16:09:32 ] ::  ======================= Migration finished: Mon Sep  9 16:09:32 EDT 2013 ========================
:: [ 16:09:32 ] ::  slapd pid : 20288
:: [ 16:09:32 ] ::  After migration free memory : 1738301440
:: [ 16:09:32 ] ::  After migration slapd VmRSS : 1184088kB
:: [ 16:09:32 ] ::  After migration slapd VmHWM : 1188280kB

with compat plugin disabled

2 hours 36 minutes

:: [ 10:53:24 ] ::  slapd pid : 18548
:: [ 10:53:24 ] ::  Before migration free memory : 15083929600
:: [ 10:53:24 ] ::  Before migration slapd VmRSS : 28908kB
:: [ 10:53:24 ] ::  Before migration slapd VmHWM : 28908kB
:: [ 10:53:24 ] ::  ======================= Migration started: Wed Sep 11 10:53:24 EDT 2013 ========================
:: [ 10:53:24 ] ::  EXECUTING: echo Secret123 | ipa migrate-ds ldap://cloud-qe-1-vm-1.testrelm.com:389
:: [   PASS   ] :: Migration completed successfully. 
:: [ 12:29:08 ] ::  ======================= Migration finished: Wed Sep 11 12:29:08 EDT 2013 ========================
:: [ 12:29:08 ] ::  slapd pid : 18548
:: [ 12:29:08 ] ::  After migration free memory : 13061668864
:: [ 12:29:08 ] ::  After migration slapd VmRSS : 1055668kB
:: [ 12:29:08 ] ::  After migration slapd VmHWM : 1061276kB

Comment 13 errata-xmlrpc 2013-11-21 20:48:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1651.html