698421 – IPA Replica Installing failing on during replication update

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 698421 - IPA Replica Installing failing on during replication update

Summary: IPA Replica Installing failing on during replication update

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	ipa
Sub Component:
Version:	6.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	rc
Target Release:	---
Assignee:	Rob Crittenden
QA Contact:	Chandrasekar Kannan
Docs Contact:
URL:
Whiteboard:
Duplicates (2):	694540 701763 (view as bug list)
Depends On:
Blocks:	709329
TreeView+	depends on / blocked

Reported:	2011-04-20 20:44 UTC by Jenny Severance
Modified:	2015-01-04 23:48 UTC (History)
CC List:	7 users (show)
Fixed In Version:	ipa-2.1.0-1.el6
Doc Type:	Bug Fix
Doc Text:	Cause: An IPA replica would sometimes fail to install while trying to initialize replication with the remote IPA server. Consequence: The IPA replica installation would be unsuccessful. Fix: The memberOf attribute is rebuilt during installation. 389-ds may crash if it is restarted while this task is running. Wait for this task to complete before requesting a restart. Result: The IPA replica installation will be successful.
Clone Of:
Environment:
Last Closed:	2011-12-06 18:21:50 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
replica install log (373.58 KB, application/octet-stream) 2011-04-20 20:44 UTC, Jenny Severance	no flags	Details
master ds error log (9.37 KB, text/plain) 2011-04-20 20:45 UTC, Jenny Severance	no flags	Details
replica ds error log (8.12 KB, text/plain) 2011-04-20 20:45 UTC, Jenny Severance	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2011:1533	0	normal	SHIPPED_LIVE	Moderate: ipa security and bug fix update	2011-12-06 01:23:31 UTC

Description Jenny Severance 2011-04-20 20:44:48 UTC

Created attachment 493626 [details]
replica install log

Description of problem:
Replica is failing to install.

ERROR from install log:

2011-04-20 16:30:45,179 DEBUG list index out of range
  File "/usr/sbin/ipa-replica-install", line 540, in <module>
    main()

  File "/usr/sbin/ipa-replica-install", line 501, in main
    install_krb(config, setup_pkinit=options.setup_pkinit)

  File "/usr/sbin/ipa-replica-install", line 242, in install_krb
    setup_pkinit, pkcs12_info)

  File "/usr/lib/python2.6/site-packages/ipaserver/install/krbinstance.py", line 217, in create_replica
    self.start_creation("Configuring Kerberos KDC", 30)

  File "/usr/lib/python2.6/site-packages/ipaserver/install/service.py", line 301, in start_creation
    method()

  File "/usr/lib/python2.6/site-packages/ipaserver/install/krbinstance.py", line 562, in __convert_to_gssapi_replication
    r_bindpw=self.dm_password)

  File "/usr/lib/python2.6/site-packages/ipaserver/install/replication.py", line 714, in convert_to_gssapi_replication
    self.gssapi_update_agreements(self.conn, r_conn)

  File "/usr/lib/python2.6/site-packages/ipaserver/install/replication.py", line 456, in gssapi_update_agreements
    self.setup_krb_princs_as_replica_binddns(a, b)

  File "/usr/lib/python2.6/site-packages/ipaserver/install/replication.py", line 449, in setup_krb_princs_as_replica_binddns
    mod = [(ldap.MOD_ADD, "nsds5replicabinddn", a_pn[0].dn)]



Version-Release number of selected component (if applicable):
ipa-server-2.0.0-23.el6.x86_64
ds-replication-1.2.8.0-1.el6.x86_64
389-ds-base-1.2.8.2-1.el6.x86_64


How reproducible:
with new -23 packages - on 1st replica install attempt everytime

Steps to Reproduce:
1. install IPA server and generate replica package for replica server
2. install replica server
3.
  
Actual results:
2011-04-20 16:30:45,179 DEBUG list index out of range

Expected results:
successful replica installation

Additional info:

Comment 1 Jenny Severance 2011-04-20 20:45:09 UTC

Created attachment 493627 [details]
master ds error log

Comment 2 Jenny Severance 2011-04-20 20:45:31 UTC

Created attachment 493628 [details]
replica ds error log

Comment 3 Rob Crittenden 2011-04-20 20:53:53 UTC

This is the same problem as reported in 694283. The fact that it happens with -23 often is coincidence, it adds literally 1 line of code unrelated to replication and installation in every way.

Comment 5 Rich Megginson 2011-04-20 21:23:23 UTC

can you attach the access logs from the master and replica?

Comment 6 Rich Megginson 2011-04-20 22:03:21 UTC

Looks like the replica is crashing?  Do you have any core files?  Do you have any segfault messages in /var/log/messages?  If not, can you enable core files on the systems?

Comment 7 RHEL Program Management 2011-04-21 06:00:22 UTC

Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Comment 8 Rich Megginson 2011-04-21 17:49:24 UTC

I think this is related and possibly a dup of https://bugzilla.redhat.com/show_bug.cgi?id=694283

The problem is that the replica directory server is crashing during shutdown.  For a normal restart, you should see something like this:

[20/Apr/2011:16:30:11 -0400] - slapd stopped.
[20/Apr/2011:16:30:13 -0400] - 389-Directory/1.2.8.2 B2011.104.2252 starting up

but the replica shows this:

[20/Apr/2011:16:30:28 -0400] - slapd shutting down - closing down internal subsystems and plugins
[20/Apr/2011:16:30:30 -0400] - Waiting for 4 database threads to stop
[20/Apr/2011:16:30:31 -0400] - 389-Directory/1.2.8.2 B2011.104.2252 starting up
[20/Apr/2011:16:30:31 -0400] - Detected Disorderly Shutdown last time Directory Server was running, recovering database.

Note that the error log does not have "slapd stopped." but has "Detected Disorderly Shutdown".  This means either a crash or a kill -9.

The problem is that the ipa installer starts a memberof fixup task on the replica but does not wait for it to finish.  Then it restarts the server while the task is running.  https://fedorahosted.org/freeipa/ticket/1188  I believe if ipa waits for this and other tasks to complete, this problem will be resolved.

Comment 9 Jenny Severance 2011-05-11 17:50:02 UTC

*** Bug 701763 has been marked as a duplicate of this bug. ***

Comment 10 Dmitri Pal 2011-05-11 22:32:59 UTC

*** Bug 694540 has been marked as a duplicate of this bug. ***

Comment 11 Dmitri Pal 2011-05-13 21:57:07 UTC

master: 46a341142079d1722647d24d06155346fc1c8442

ipa-2-0: 8c8934f45b6a24684fc9f7060f161875aa9bd700

Comment 16 Rob Crittenden 2011-05-27 16:19:08 UTC

master: 46a341142079d1722647d24d06155346fc1c8442

ipa-2-0: 8c8934f45b6a24684fc9f7060f161875aa9bd700

Comment 19 Jenny Severance 2011-09-21 16:54:11 UTC

verified :

# ipa-replica-install -U --setup-dns --forwarder=10.14.63.12 -p mysecret /dev/shm/replica-info-ipaqavmh.testrelm.gpg
Run connection check to master
Check connection from replica to remote master 'ipaqavme.testrelm':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos KDC: UDP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   Kerberos Kpasswd: UDP (464): OK
   HTTP Server: port 80 (80): OK
   HTTP Server: port 443(https) (443): OK

Connection from replica to master is OK.
Start listening on required ports for remote master check
Get credentials to log in to remote master
Password for admin@TESTRELM: 
Execute check on remote master
Check connection from master to remote replica 'ipaqavmh.testrelm':
   Directory Service: Unsecure port (389): OK
   Directory Service: Secure port (636): OK
   Kerberos KDC: TCP (88): OK
   Kerberos KDC: UDP (88): OK
   Kerberos Kpasswd: TCP (464): OK
   Kerberos Kpasswd: UDP (464): OK
   HTTP Server: port 80 (80): OK
   HTTP Server: port 443(https) (443): OK

Connection from master to replica is OK.

Connection check OK
Configuring ntpd
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
done configuring ntpd.
Configuring directory server: Estimated time 1 minute
  [1/28]: creating directory server user
  [2/28]: creating directory server instance
  [3/28]: adding default schema
  [4/28]: enabling memberof plugin
  [5/28]: enabling referential integrity plugin
  [6/28]: enabling winsync plugin
  [7/28]: configuring replication version plugin
  [8/28]: enabling IPA enrollment plugin
  [9/28]: enabling ldapi
  [10/28]: configuring uniqueness plugin
  [11/28]: configuring uuid plugin
  [12/28]: configuring modrdn plugin
  [13/28]: enabling entryUSN plugin
  [14/28]: configuring lockout plugin
  [15/28]: creating indices
  [16/28]: configuring ssl for ds instance
  [17/28]: configuring certmap.conf
  [18/28]: configure autobind for root
  [19/28]: restarting directory server
  [20/28]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress
Update in progress
Update in progress
Update succeeded
  [21/28]: adding replication acis
  [22/28]: setting Auto Member configuration
  [23/28]: initializing group membership
  [24/28]: adding master entry
  [25/28]: configuring Posix uid/gid generation
  [26/28]: enabling compatibility plugin
  [27/28]: tuning directory server
  [28/28]: configuring directory to start on boot
done configuring dirsrv.
Configuring Kerberos KDC: Estimated time 30 seconds
  [1/9]: adding sasl mappings to the directory
  [2/9]: writing stash file from DS
  [3/9]: configuring KDC
  [4/9]: creating a keytab for the directory
  [5/9]: creating a keytab for the machine
  [6/9]: adding the password extension to the directory
  [7/9]: enable GSSAPI for replication
  [8/9]: starting the KDC
  [9/9]: configuring KDC to start on boot
done configuring krb5kdc.
Configuring ipa_kpasswd
  [1/2]: starting ipa_kpasswd 
  [2/2]: configuring ipa_kpasswd to start on boot
done configuring ipa_kpasswd.
Configuring the web interface: Estimated time 1 minute
  [1/12]: disabling mod_ssl in httpd
  [2/12]: setting mod_nss port to 443
  [3/12]: setting mod_nss password file
  [4/12]: enabling mod_nss renegotiate
  [5/12]: adding URL rewriting rules
  [6/12]: configuring httpd
  [7/12]: setting up ssl
  [8/12]: publish CA cert
  [9/12]: creating a keytab for httpd
  [10/12]: configuring SELinux for httpd
  [11/12]: restarting httpd
  [12/12]: configuring httpd to start on boot
done configuring httpd.
Applying LDAP updates
Using reverse zone 98.16.10.in-addr.arpa.
Configuring named:
  [1/8]: adding NS record to the zone
  [2/8]: setting up reverse zone
  [3/8]: setting up our own record
  [4/8]: setting up kerberos principal
  [5/8]: setting up named.conf
  [6/8]: restarting named
  [7/8]: configuring named to start on boot
  [8/8]: changing resolv.conf to point to ourselves
done configuring named.


version:

389-ds-base-1.2.9.11-1.el6.x86_64
ipa-server-2.1.1-3.el6.x86_64

Comment 20 Rob Crittenden 2011-10-31 18:19:56 UTC

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause: An IPA replica would sometimes fail to install while trying to initialize replication with the remote IPA server.
Consequence: The IPA replica installation would be unsuccessful.
Fix: The memberOf attribute is rebuilt during installation. 389-ds may crash if it is restarted while this task is running. Wait for this task to complete before requesting a restart.
Result: The IPA replica installation will be successful.

Comment 21 errata-xmlrpc 2011-12-06 18:21:50 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2011-1533.html

Note You need to log in before you can comment on or make changes to this bug.