Bug 1404338

Summary: Check IdM Topology for broken record caused by replication conflict before upgrading it
Product: Red Hat Enterprise Linux 7 Reporter: Marcel Kolaja <mkolaja>
Component: ipaAssignee: IPA Maintainers <ipa-maint>
Status: CLOSED ERRATA QA Contact: Kaleem <ksiddiqu>
Severity: urgent Docs Contact: Marc Muehlfeld <mmuehlfe>
Priority: urgent    
Version: 7.3CC: ekeck, gparente, ipa-maint, jcholast, lkrispen, mbabinsk, mkosek, ndehadra, pvoborni, rcritten, tbordaz
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: ipa-4.4.0-14.el7_3.3 Doc Type: Bug Fix
Doc Text:
Previously, if an Identity Management (IdM) upgrade ran simultaneously on multiple servers, replication conflict entries were sometimes generated in the "cn=topology" subtree. If the domain level was raised while the conflict entries existed, the generated topology segment was sometimes distributed between correct and conflict entries. Also, one-directional segments fail to receive the data. As a consequence, IdM clients and commands fail. A patch has been applied to reject raising the domain level if replication conflicts exists. As a result, topology segments are created now only in a database without conflict entries.
Story Points: ---
Clone Of: 1398670 Environment:
Last Closed: 2017-01-17 18:23:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1398670    
Bug Blocks:    

Description Marcel Kolaja 2016-12-13 15:56:16 UTC
This bug has been copied from bug #1398670 and has been proposed
to be backported to 7.3 z-stream (EUS).

Comment 7 Nikhil Dehadrai 2017-01-05 16:06:53 UTC
IPA server version: ipa-server-4.4.0-14.el7_3.4.x86_64

Tested the bug on the basis of following points:

Steps: (Upgrade from 7.2.z > 7.3.2)
====================================
1) Install master on RHEL 7.2.z. (In my case ipa-server.x86_64 0:4.2.0-15.el7_2.19).
2) Install replica on RHEL 7.2.z against master in step1, with ipa-replica-prepare command.
3) Stop replica server using "ipactl stop".
4) Configure repos for RHEL 7.3.2 on Master and Replica.
5) Upgrade master to RHEL 7.3.2 and stop master using command "ipactl stop".
6) Start replica using command "ipactl start" and Upgrade replica to Rhel 7.3.2 using command "yum -y update 'ipa*' sssd".
7) Start master
 server using command "ipactl start"
8) Run "kinit admin" both on master and replica.
9) Run "ipa domainlevel-set 1" both on Master and Replica.

Observations:
==============
1) Both Master and Replica are upgraded successfully after step5 and step6.
2) After step9, following error message is received both on Master:
#ipa domainlevel-set 1
ipa: ERROR: Domain Level cannot be raised to 1, server <replica.testrelm.test> does not support it.

3) After step9, following error message is received both on REPLICA:
ipa domainlevel-set 1
ipa: ERROR: Major (851968): Unspecified GSS failure.  Minor code may provide more information, Minor (2529639068): Cannot contact any KDC for realm 'TESTRELM.TEST'

Thus on the basis of above observations, marking the status of bug to "ASSIGNED"

Comment 8 thierry bordaz 2017-01-06 10:52:31 UTC
Verification of 1404338 depends on 1410514.

I can imagine a very poor workaround to verify 1404338, but I am not sure it is acceptable and it will work.

1) Install master on RHEL 7.2.z. (In my case ipa-server.x86_64 0:4.2.0-15.el7_2.19).
2) Install replica on RHEL 7.2.z against master in step1, with ipa-replica-prepare command.
   2-1) Configure repos for RHEL 7.3.2 on Master and Replica.

3) stop master using command "ipactl stop"

4) Upgrade replica to Rhel 7.3.2 using command "yum -y update 'ipa*' sssd".

5) Stop replica server using "ipactl stop".

  5-1) edit dse.ldif to disable cos plugin on replica
dn: cn=Class of Service,cn=plugins,cn=config
nsslapd-pluginEnabled: off

dn: cn=Legacy Replication Plugin,cn=plugins,cn=config
nsslapd-plugin-depends-on-named: Class of Service  <-- remove

dn: cn=Multimaster Replication Plugin,cn=plugins,cn=config
nsslapd-plugin-depends-on-named: Class of Service  <-- remove

dn: cn=Retro Changelog Plugin,cn=plugins,cn=config
nsslapd-plugin-depends-on-named: Class of Service  <-- remove


6) start master using command "ipactl start
7) Upgrade master to RHEL 7.3.2 and stop master using command "ipactl stop".

  7-1) edit dse.ldif to disable cos plugin on master
dn: cn=Class of Service,cn=plugins,cn=config
nsslapd-pluginEnabled: off

dn: cn=Legacy Replication Plugin,cn=plugins,cn=config
nsslapd-plugin-depends-on-named: Class of Service  <-- remove

dn: cn=Multimaster Replication Plugin,cn=plugins,cn=config
nsslapd-plugin-depends-on-named: Class of Service  <-- remove

dn: cn=Retro Changelog Plugin,cn=plugins,cn=config
nsslapd-plugin-depends-on-named: Class of Service  <-- remove


8) Start replica using command "ipactl start" 
9) Start master server using command "ipactl start"
10) wait few minutes for replication to occur

11) Stop replica server using "ipactl stop".

  11-1) edit dse.ldif to enable cos plugin on replica
dn: cn=Class of Service,cn=plugins,cn=config
nsslapd-pluginEnabled: on

dn: cn=Legacy Replication Plugin,cn=plugins,cn=config
nsslapd-plugin-depends-on-named: Class of Service  <-- add

dn: cn=Multimaster Replication Plugin,cn=plugins,cn=config
nsslapd-plugin-depends-on-named: Class of Service  <-- add

dn: cn=Retro Changelog Plugin,cn=plugins,cn=config
nsslapd-plugin-depends-on-named: Class of Service  <-- add

12) Stop master server using "ipactl stop".

  12-1) edit dse.ldif to enable cos plugin on replica
dn: cn=Class of Service,cn=plugins,cn=config
nsslapd-pluginEnabled: on

dn: cn=Legacy Replication Plugin,cn=plugins,cn=config
nsslapd-plugin-depends-on-named: Class of Service  <-- add

dn: cn=Multimaster Replication Plugin,cn=plugins,cn=config
nsslapd-plugin-depends-on-named: Class of Service  <-- add

dn: cn=Retro Changelog Plugin,cn=plugins,cn=config
nsslapd-plugin-depends-on-named: Class of Service  <-- add



13) Start replica using command "ipactl start" 
14) Start master server using command "ipactl start"
15) Run "kinit admin" both on master and replica.
16) Run "ipa domainlevel-set 1" both on Master and Replica.

Comment 9 Nikhil Dehadrai 2017-01-06 14:33:53 UTC
Hi Thierry,

As per the steps/workaround mentioned in Comment#8, I was able to verify the bug:

ON MASTER (after upgrade to 7.3.2):
=====================================
[root@vm-idm-030 slapd-TESTRELM-TEST]# ipa domainlevel-set 1
ipa: ERROR: Domain Level cannot be raised to 1, existing replication conflicts have to be resolved.
[root@vm-idm-030 slapd-TESTRELM-TEST]# ipa-replica-manage list
vm-idm-030.testrelm.test: master
auto-hv-01-guest01.testrelm.test: master
[root@vm-idm-030 slapd-TESTRELM-TEST]# ipa domainlevel-get
-----------------------
Current domain level: 0
-----------------------

ON REPLICA (after upgrade to 7.3.2):
=====================================
[root@auto-hv-01-guest01 slapd-TESTRELM-TEST]# ipa domainlevel-set 1
ipa: ERROR: Domain Level cannot be raised to 1, existing replication conflicts have to be resolved.
[root@auto-hv-01-guest01 slapd-TESTRELM-TEST]# ipa-replica-manage list
vm-idm-030.testrelm.test: master
auto-hv-01-guest01.testrelm.test: master
[root@auto-hv-01-guest01 slapd-TESTRELM-TEST]# ipa domainlevel-get
-----------------------
Current domain level: 0
-----------------------

Comment 10 Nikhil Dehadrai 2017-01-09 06:17:10 UTC
Thus on basis of steps provided in Comment#8 and respective observations in Comment#9, marking the status of bug to "VERIFIED".

Comment 12 errata-xmlrpc 2017-01-17 18:23:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0089.html