Bug 1714076

Summary: Issue with adding multiple RHEL 7 IPA replica to RHEL 6 IPA master
Product: Red Hat Enterprise Linux 7 Reporter: Glen Babiano <gbabiano>
Component: ipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED ERRATA QA Contact: ipa-qe <ipa-qe>
Severity: unspecified Docs Contact:
Priority: high    
Version: 7.6CC: fcami, frenaud, msauton, myusuf, ndehadra, pasik, pvoborni, rcritten, tmihinto, tscherf
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ipa-4.6.6-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1770728 (view as bug list) Environment:
Last Closed: 2020-03-31 19:55:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1770728    

Description Glen Babiano 2019-05-27 04:19:41 UTC
Description of problem:
When migrating IPA from RHEL 6 to RHEL 7, it is only working on the first RHEL 7 IPA server replica install, the succeeding RHEL 7 replica install fails consistently

We are following the documentation link below:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/migrate-6-to-7

Version-Release number of selected component (if applicable):

ipaserver6.example.local
RHEL 6.10
ipa-server-3.0.0-51
IP = 10.10.92.254

ipaserver7.example.local 
RHEL 7.6 
ipa-server-4.6.4-10
IP = 10.74.177.255

ipaserver8.example.local 
RHEL 7.6 
ipa-server-4.6.4-10
IP = 10.74.176.168


How reproducible:
Reproducible everytime


Steps to Reproduce:
-----
ipaserver6.example.local
-----
### Update the system to the latest version and install IPA packages
# hostnamectl set-hostname ipaserver6.example.local
# subscription-manager register --auto-attach --force
# yum update -y
# yum install "*ipa-server" "*ipa-server-trust-ad" bind bind-dyndb-ldap ipa-server-dns -y
### Add the host entry of the server's ip address
# cat /etc/hosts
  10.10.92.254 ipaserver6.example.local ipaserver6
# ipa-server-install
# ipa-dns-install

-----
ipaserver7.example.local
-----
### Update the system to the latest version and install IPA packages
# hostnamectl set-hostname ipaserver7.example.local
# subscription-manager register --auto-attach --force
# yum update -y

# yum install ipa-server ipa-server-dns -y

### Note: ensure that the file below only contains the following lines
# cat /etc/resolv.conf 
   search example.local
   nameserver 10.10.92.254

### Note: add the following line under NSSCipherSuite
# cat /etc/httpd/conf.d/nss.conf 
  +ecdhe_rsa_aes_128_sha,+ecdhe_rsa_aes_256_sha
# scp /usr/share/ipa/copy-schema-to-ca.py root.local:/root

-----
ipaserver6.example.local
-----
# cd /root
# python copy-schema-to-ca.py
# ipa-replica-prepare ipaserver7.example.local --ip-address 10.74.177.255
# scp /var/lib/ipa/replica-info-ipaserver7.example.local.gpg root.local:/var/lib/ipa/

-----
ipaserver7.example.local
-----
# ipa-replica-install /var/lib/ipa/replica-info-ipaserver7.example.local.gpg --setup-ca --setup-dns --no-forwarders --ip-address 10.74.177.255


WE WILL ADD A NEW IPA RHEL 7 REPLICA, THIS IS WHERE THE ISSUE APPEARS:
-----
ipaserver8.example.local
-----
### Update the system to the latest version and install IPA packages
# hostnamectl set-hostname ipaserver8.example.local
# subscription-manager register --auto-attach --force
# yum update -y
# yum install ipa-server ipa-server-dns -y

### Note: ensure that the file below only contains the following lines
# cat /etc/resolv.conf 
   search example.local
   nameserver 10.10.92.254

### Note: add the following line under NSSCipherSuite
# cat /etc/httpd/conf.d/nss.conf 
  +ecdhe_rsa_aes_128_sha,+ecdhe_rsa_aes_256_sha

-----
ipaserver6.example.local
-----
# ipa-replica-prepare ipaserver8.example.local --ip-address 10.74.176.168
# scp /var/lib/ipa/replica-info-ipaserver8.example.local.gpg root.local:/var/lib/ipa/


-----
ipaserver8.example.local
-----
# ipa-replica-install /var/lib/ipa/replica-info-ipaserver8.example.local.gpg --setup-ca --setup-dns --no-forwarders --ip-address 10.74.176.168
[...]
Configuring the web interface (httpd)
  [1/22]: stopping httpd
  [2/22]: setting mod_nss port to 443
  [3/22]: setting mod_nss cipher suite
  [4/22]: setting mod_nss protocol list to TLSv1.0 - TLSv1.2
  [5/22]: setting mod_nss password file
  [6/22]: enabling mod_nss renegotiate
  [7/22]: disabling mod_nss OCSP
  [8/22]: adding URL rewriting rules
  [9/22]: configuring httpd
  [10/22]: setting up httpd keytab
  [error] NotFound: wait_for_entry timeout on ldap://ipaserver6.example.local:389 for krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipapython.admintool: ERROR    wait_for_entry timeout on ldap://ipaserver6.example.local:389 for krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
ipapython.admintool: ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

Actual results:
Error appears as above

Expected results:
Expecting to be able to add IPA RHEL 7 replica multiple times on a RHEL 6 IPA Master 

Additional info:

Comment 2 Glen Babiano 2019-05-27 04:31:53 UTC
ipareplica-install.log shows:


2019-05-27T03:01:53Z DEBUG args=/usr/sbin/ipa-getkeytab -k /var/lib/ipa/gssproxy/http.keytab -p HTTP/ipaserver8.example.local -H ldapi://%2fvar%2frun%2fslapd-EXAMPLE-LOCAL.sock
et -Y EXTERNAL
2019-05-27T03:01:54Z DEBUG Process finished, return code=0
2019-05-27T03:01:54Z DEBUG stdout=
2019-05-27T03:01:54Z DEBUG stderr=Keytab successfully retrieved and stored in: /var/lib/ipa/gssproxy/http.keytab

2019-05-27T03:01:54Z DEBUG Waiting for replication (ldap://ipaserver6.example.local:389) krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=lo
cal (objectclass=*)
2019-05-27T03:02:03Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:02:13Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:02:23Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:02:33Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:02:43Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:02:53Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:03:03Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:03:13Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:03:23Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:03:33Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:03:43Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:03:53Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:04:03Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:04:13Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:04:23Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:04:33Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:04:43Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:04:53Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:05:03Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:05:13Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:05:23Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:05:33Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:05:43Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:05:53Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:06:03Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:06:13Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:06:24Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:06:34Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:06:44Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:06:54Z DEBUG Still waiting for replication of krbprincipalname=HTTP/ipaserver8.example.local,cn=services,cn=accounts,dc=example,dc=local
2019-05-27T03:06:55Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 570, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 560, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/httpinstance.py", line 656, in request_service_keytab
    timeout=api.env.replication_wait_timeout

Comment 3 Florence Blanc-Renaud 2019-05-27 07:53:08 UTC
Issue is reproducible with RHEL 7.6 z-stream (4.6.4-10.el7_6.3).

On the 6.x master, the entry cn=replica,cn=dc\3Ddomain\2Cdc\3Dcom,cn=mapping tree,cn=config does not contain nsDS5ReplicaBindDN: krbprincipalname=ldap/replica2.domain.com,cn=services,cn=accounts,dc=domain,dc=com as I would expect.

The code ipaserver/install/replication.py::ensure_replication_managers should add this attribute but doesn't. The function is checking first if the entry cn=replication managers,cn=sysaccounts,cn=etc,$BASEDN exists.
* If the entry is found, the function adds krbprincipalname=ldap/replica2.domain.com,cn=services,cn=accounts,dc=domain,dc=com as a member of the cn=replication managers entry, considering that the replication supports replicabinddngroup
* If the entry is not found, the function add krbprincipalname=ldap/replica2.domain.com,cn=services,cn=accounts,dc=domain,dc=com directly to the attribute nsDS5ReplicaBindDN of cn=replica entry, considering that the replication does not suport replicabinddngroup, but also creates the entry cn=replication managers. Because of this second step, the install of replica2 will later find cn=replication managers and will not add the nsDS5ReplicaBindDN to cn=replica.

Comment 7 Florence Blanc-Renaud 2019-06-13 20:05:24 UTC
Upstream ticket:
https://pagure.io/freeipa/issue/7976

Comment 19 Mohammad Rizwan 2019-12-16 10:28:57 UTC
version:
master :   ipa-server-3.0.0-51.el6.x86_64
replicas : ipa-server-4.6.6-11.el7.x86_64

Steps:
- install a RHEL 6 master
- install a RHEL7 replica with ipa-replica-prepare (on rhel6)/ipa-replica-install (on rhel7) (do not forget to copy and run the script copy-schema-to-ca.py as described in https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/migrate-6-to-7#migrate-6-7-schema-update-script)
- check that replication is working by creating a user on RHEL6 master, then ipa user-show on RHEL7 and vice versa
- check that the entry cn=replica,cn=...,cn=mapping tree,cn=config on the rhel6 master contains the attribute nsDS5ReplicaBindDN: krbprincipalname=ldap/replica@DOMAIN
- install another RHEL7 replica from the RHEL6 master with ipa-replica-prepare (on rhel6)/ipa-replica-install (on rhel7)
- check that replication is working by creating a user on RHEL6 master, then ipa user-show on the second RHEL7 replica, and vice-versa
- check that the entry cn=replica,cn=...,cn=mapping tree,cn=config on the rhel6 master contains the attribute nsDS5ReplicaBindDN: krbprincipalname=ldap/replica2@DOMAIN

Actual result:

User replicated from and to the server. Based on the observation, marking the bug as verified.

Comment 22 errata-xmlrpc 2020-03-31 19:55:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1083