RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1542960 - restored data from backup on master vanished when replica is re-initialized
Summary: restored data from backup on master vanished when replica is re-initialized
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: ipa
Version: 7.5
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: IPA Maintainers
QA Contact: ipa-qe
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-02-07 13:07 UTC by Mohammad Rizwan
Modified: 2018-11-21 12:34 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-05-02 07:42:10 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Mohammad Rizwan 2018-02-07 13:07:00 UTC 
Description of problem:
restored data from backup on master vanished when replica is re-initialized. See the steps below

Version-Release number of selected component (if applicable):
ipa-server-4.5.4-9.el7.x86_64
389-ds-base-1.3.7.5-14.el7.x86_64

How reproducible:
always

Steps to Reproduce:
1. Install master and replica
2. add user test1Master on master and test1Replica on replica
3. ipa-backup on master
4. delete user test1Master on master and add test2Master
5. delete user test1Replica on replica and add test2Replica
6. disable replication agreement on master

[root@master ~]# ldapmodify -h localhost -D "cn=Directory Manager" -p 389 -w Secret123 -f a.ldif 
modifying entry "cn=meToreplica1.testrelm.test,cn=replica,cn=dc\3Dtestrelm\2Cdc\3Dtest,cn=mapping tree,cn=config"

modifying entry "cn=caToreplica1.testrelm.test,cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config"

[root@master ~]# cat a.ldif 
dn: cn=meToreplica1.testrelm.test,cn=replica,cn=dc\3Dtestrelm\2Cdc\3Dtest,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaEnabled
nsds5ReplicaEnabled: off

dn: cn=caToreplica1.testrelm.test,cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaEnabled
nsds5ReplicaEnabled: off

7. uninstall master
8. restore master $ ipa-restore path
9. check user test1Master on master, it will be listed there
10. re-initialize replica

[root@replica1 ~]# ipa topologysegment-reinitialize domain master.testrelm.test-to-replica1.testrelm.test --left
--------------------------------------------------------------------------------------------
Replication refresh for segment: "master.testrelm.test-to-replica1.testrelm.test" requested.
--------------------------------------------------------------------------------------------
[root@replica1 ~]# ipa topologysegment-reinitialize ca master.testrelm.test-to-replica1.testrelm.test --left
--------------------------------------------------------------------------------------------
Replication refresh for segment: "master.testrelm.test-to-replica1.testrelm.test" requested.
--------------------------------------------------------------------------------------------

11. check user test1Master and test1Replica, It is not listed on replica
12. again check user test1Master and test1Replica on master (step9), Now it wont be there
13. add user on master and check on replica
14. add user on replica and check on master


Actual results:
step 11 and step 12

Expected results:
user added before backup should be shown after restore and re-initialization

Additional info:
as per step 13 and 14 replication is working fine.
Comment 2 Mohammad Rizwan 2018-02-08 13:02:30 UTC 
version:
ipa-server-4.5.0-20.el7.x86_64
389-ds-base-1.3.6.1-16.el7.x86_64

same behaviour observed on rhel7.4 too.
Comment 3 thierry bordaz 2018-02-12 12:37:02 UTC 
I think the testcase could hit a timing issue.

I suspect that after step 8, the replica can connect to restored master and send DEL test1Master (from step 4) and DEL test1Replica (from step 5). So depending how fast is replication Replica->Master. The test users may appear or not. 

I suggest that you disable RA Replica->Master in step 6 (rather than the opposite). And reenable it between steps 12 and 13.
Comment 4 Mohammad Rizwan 2018-02-12 12:54:16 UTC 

(In reply to thierry bordaz from comment #3)
> I think the testcase could hit a timing issue.
> 
> I suspect that after step 8, the replica can connect to restored master and
> send DEL test1Master (from step 4) and DEL test1Replica (from step 5). So
> depending how fast is replication Replica->Master. The test users may appear
> or not. 
> 
> I suggest that you disable RA Replica->Master in step 6 (rather than the
> opposite). And reenable it between steps 12 and 13.

Okay, I can try disabling the RA from replica itself (and not from master).

As per my understanding, RA becomes active (enable) when topologysegment-reinitialize executed (as per my observations in https://bugzilla.redhat.com/show_bug.cgi?id=1498523#c36)

How exactly you are suggesting to re-enable it between step 12 and 13?
Comment 5 thierry bordaz 2018-02-12 12:58:58 UTC 
Hi Mohammad,

You are likely right, topology plugin should handle semi establish RAs and reenable RAs on both direction.

I think you may check the RA replica->master status and reenable it if it is not enabled.
Comment 6 Mohammad Rizwan 2018-02-13 12:19:23 UTC 
I did followed the same steps from the description except step6. I tried disabling the RA from replica as following:

[root@replica ~]# cat a.ldif 
dn: cn=meTomaster.testrelm.test,cn=replica,cn=dc\3Dtestrelm\2Cdc\3Dtest,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaEnabled
nsds5ReplicaEnabled: off

dn: cn=caTomaster.testrelm.test,cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
changetype: modify
replace: nsds5ReplicaEnabled
nsds5ReplicaEnabled: off

[root@replica ~]# ldapmodify -h localhost -D "cn=Directory Manager" -p 389 -w Secret123 -f a.ldif


After restore I can see following entries on master:
~~~~~~~~~~~~~~~~~~~
Entries on master after restore
~~~~~~~~~~~~~~~~~~~
[root@master ~]# ipa user-find
---------------
3 users matched
---------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 1137200000
  GID: 1137200000
  Account disabled: False

  User login: test1master
  First name: test1
  Last name: before_backup
  Home directory: /home/test1master
  Login shell: /bin/sh
  Principal name: test1master
  Principal alias: test1master
  Email address: test1master
  UID: 1137200001
  GID: 1137200001
  Account disabled: False

  User login: test1replica
  First name: test1
  Last name: before_backup
  Home directory: /home/test1replica
  Login shell: /bin/sh
  Principal name: test1replica
  Principal alias: test1replica
  Email address: test1replica
  UID: 1137300500
  GID: 1137300500
  Account disabled: False
----------------------------
Number of entries returned 3
----------------------------

~~~~~~~~~~~~~~~~~~~~~~~~~
Entries on replica after restore
~~~~~~~~~~~~~~~~~~~~~~~~~

[root@replica ~]# ipa user-find
---------------
3 users matched
---------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 1137200000
  GID: 1137200000
  Account disabled: False

  User login: test2master
  First name: test2
  Last name: after_backup
  Home directory: /home/test2master
  Login shell: /bin/sh
  Principal name: test2master
  Principal alias: test2master
  Email address: test2master
  UID: 1137200003
  GID: 1137200003
  Account disabled: False

  User login: test2replica
  First name: test2
  Last name: after_backup
  Home directory: /home/test2replica
  Login shell: /bin/sh
  Principal name: test2replica
  Principal alias: test2replica
  Email address: test2replica
  UID: 1137300501
  GID: 1137300501
  Account disabled: False
----------------------------
Number of entries returned 3
----------------------------


Then I added test3master on master to check if replication is working


[root@master ~]# ipa user-find
---------------
4 users matched
---------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 1137200000
  GID: 1137200000
  Account disabled: False

  User login: test1master
  First name: test1
  Last name: before_backup
  Home directory: /home/test1master
  Login shell: /bin/sh
  Principal name: test1master
  Principal alias: test1master
  Email address: test1master
  UID: 1137200001
  GID: 1137200001
  Account disabled: False

  User login: test1replica
  First name: test1
  Last name: before_backup
  Home directory: /home/test1replica
  Login shell: /bin/sh
  Principal name: test1replica
  Principal alias: test1replica
  Email address: test1replica
  UID: 1137300500
  GID: 1137300500
  Account disabled: False

  User login: test3master
  First name: test3
  Last name: after_restore
  Home directory: /home/test3master
  Login shell: /bin/sh
  Principal name: test3master
  Principal alias: test3master
  Email address: test3master
  UID: 1137200004
  GID: 1137200004
  Account disabled: False
----------------------------
Number of entries returned 4
----------------------------


But the user test3master  not listed on replica. I enabled the replication by setting "nsds5ReplicaEnabled: on" in ldif on replica, still replication is not working.

(I have ran the topologysegment-reinitialize on both the server before enabling by ldif.)

So, if replication is disabled from replica then after re-init, replication is not working.
Comment 7 thierry bordaz 2018-02-13 14:29:50 UTC 
At step 8, Master is restored from a backup. So it is late compare to Replica.
This is why we disabled (step 6) RA Replica->Master to keep Master as it was at backup time.

But Master being "in the past", is not able to update Replica that is more in advance. If you want Master to replicate to Replica, you need to continue the procedure, step 9 and 10. At step 10, Master will reinit Replica. So later, update on Master (test3master) will be correctly replciated
Comment 8 Mohammad Rizwan 2018-02-19 08:57:34 UTC 
when master reinit replica at step 10, no replication is happening between master and replica i.e user test3master didn't listed on replica.

Also, the user entries on master and replica also differs:

~~~~~~~~~~
On master:
~~~~~~~~~~
[root@master ~]# ipa user-find
---------------
4 users matched
---------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 1274000000
  GID: 1274000000
  Account disabled: False

  User login: test1master
  First name: test1
  Last name: before_backup
  Home directory: /home/test1master
  Login shell: /bin/sh
  Principal name: test1master
  Principal alias: test1master
  Email address: test1master
  UID: 1274000001
  GID: 1274000001
  Account disabled: False

  User login: test1replica
  First name: test1
  Last name: before_backup
  Home directory: /home/test1replica
  Login shell: /bin/sh
  Principal name: test1replica
  Principal alias: test1replica
  Email address: test1replica
  UID: 1274100500
  GID: 1274100500
  Account disabled: False

  User login: test3master
  First name: test3
  Last name: after_restore
  Home directory: /home/test3master
  Login shell: /bin/sh
  Principal name: test3master
  Principal alias: test3master
  Email address: test3master
  UID: 1274000003
  GID: 1274000003
  Account disabled: False
----------------------------
Number of entries returned 4
----------------------------


~~~~~~~~~~
On replica:
~~~~~~~~~~
[root@replica ~]# ipa user-find
---------------
3 users matched
---------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 1274000000
  GID: 1274000000
  Account disabled: False

  User login: test2master
  First name: test2
  Last name: after_backup
  Home directory: /home/test2master
  Login shell: /bin/sh
  Principal name: test2master
  Principal alias: test2master
  Email address: test2master
  UID: 1274000003
  GID: 1274000003
  Account disabled: False

  User login: test2replica
  First name: test2
  Last name: after_backup
  Home directory: /home/test2replica
  Login shell: /bin/sh
  Principal name: test2replica
  Principal alias: test2replica
  Email address: test2replica
  UID: 1274100501
  GID: 1274100501
  Account disabled: False
----------------------------
Number of entries returned 3
----------------------------
Comment 9 Florence Blanc-Renaud 2018-03-01 20:12:43 UTC 
Hi,

in your last tests, did you uninstall the master with ipa-server-install --uninstall? I am trying to follow the same steps, but if I uninstall the master, the replica removes the topologysegment and the server from its configuration and is not able to re-enable the replication after the master is restored.

On the contrary, if I simply call restore (without uninstall), then perform on the replica:
ipa-replica-manage re-initialize --from=restored_master_FQDN

the replica gets re-initialized and contains the entries from the backup. Same thing for the master, it contains the entries from the backup.
Comment 10 Mohammad Rizwan 2018-03-02 06:30:24 UTC 
(In reply to Florence Blanc-Renaud from comment #9)
> Hi,
> 
> in your last tests, did you uninstall the master with ipa-server-install
> --uninstall? I am trying to follow the same steps, but if I uninstall the
> master, the replica removes the topologysegment and the server from its
> configuration and is not able to re-enable the replication after the master
> is restored.

yes, I uninstalled master.

> On the contrary, if I simply call restore (without uninstall), then perform
> on the replica:
> ipa-replica-manage re-initialize --from=restored_master_FQDN
> 
> the replica gets re-initialized and contains the entries from the backup.
> Same thing for the master, it contains the entries from the backup.

Recommended process is to uninstall the master before restore[1], so we need to uninstall the server before restore.

I used topologysegment command because of domain-level 1, so in this particular case (re-init), do we have to use ipa-replica-manage command?


[1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/linux_domain_identity_authentication_and_policy_guide/#restoring-full-server
Comment 11 Mohammad Rizwan 2018-03-02 10:13:30 UTC 
Hi

I tried the scenario mentioned in description, but this time I used ipa-replica-manage instead of topologysegment. 

Now I can see the user (i.e created before backup) after restore.

~~~~~~~~~~~~~~
before restore
~~~~~~~~~~~~~~

[root@replica1 ~]# ipa user-find
---------------
3 users matched
---------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 1364600000
  GID: 1364600000
  Account disabled: False

  User login: test2master
  First name: test
  Last name: after_backup
  Home directory: /home/test2master
  Login shell: /bin/sh
  Principal name: test2master
  Principal alias: test2master
  Email address: test2master
  UID: 1364600003
  GID: 1364600003
  Account disabled: False

  User login: test2replica
  First name: test
  Last name: after_backup
  Home directory: /home/test2replica
  Login shell: /bin/sh
  Principal name: test2replica
  Principal alias: test2replica
  Email address: test2replica
  UID: 1364700501
  GID: 1364700501
  Account disabled: False
----------------------------
Number of entries returned 3
----------------------------

~~~~~~~~~~~~~
after restore
~~~~~~~~~~~~~

[root@replica1 ~]# ipa user-find
---------------
3 users matched
---------------
  User login: admin
  Last name: Administrator
  Home directory: /home/admin
  Login shell: /bin/bash
  Principal alias: admin
  UID: 1364600000
  GID: 1364600000
  Account disabled: False

  User login: test1master
  First name: test
  Last name: before_backup
  Home directory: /home/test1master
  Login shell: /bin/sh
  Principal name: test1master
  Principal alias: test1master
  Email address: test1master
  UID: 1364600001
  GID: 1364600001
  Account disabled: False

  User login: test1replica
  First name: test
  Last name: before_backup
  Home directory: /home/test1replica
  Login shell: /bin/sh
  Principal name: test1replica
  Principal alias: test1replica
  Email address: test1replica
  UID: 1364700500
  GID: 1364700500
  Account disabled: False
----------------------------
Number of entries returned 3
----------------------------

Replication working fine after restore (I checked by adding user on replica1 and  listing it on master)
Comment 12 Florence Blanc-Renaud 2018-03-02 16:56:28 UTC 
A few observations:
1- if the master is uninstalled with ipa-server-install --uninstall -U, and the replication is still enabled from master to replica, then it will delete the master entry from the replica and the topologysegment, making it more difficult to re-establish the replication.

For this test scenario, we should either:
- disable replication both ways, then uninstall the master
or
- do not call uninstall on the master

2- when the master is restored, there are 2 possible paths to re-establish the replication:
- with ipa-replica-manage re-initialize --from=restored_master_FQDN called on the replica
or
- with ipa topologysegment-reinitialize domain <segment name> --left (assuming that left is the replica)

It looks like performing topologysegment-reinitialize does not reset nsds5ReplicaEnabled: on. Ludwig, could you have a look and tell us if we should manually modify this attribute?
Comment 13 Ludwig 2018-03-05 10:09:29 UTC 
Not sure I fully understand the scenario, but if you have agreements 

A --> B and
B --> A

If A-->B is disabled, reinit A-->B will be rejected with unwilling to perform

If you do reinit B-->A thenit will work, but the agreement A-->B will not be modified
Comment 14 Petr Vobornik 2018-03-20 16:52:31 UTC 
IIUC this bugs wait on resolution of the team list discussion about restore behavior. Or other things. Setting needs info so that we won't forget.
Comment 16 Florence Blanc-Renaud 2018-03-26 13:30:16 UTC 
Upstream ticket:
https://pagure.io/freeipa/issue/7455
Comment 17 Mohammad Rizwan 2018-04-02 08:35:19 UTC 
Steps in https://pagure.io/freeipa/issue/7455 works for me.

Need to add the mention scenario from above ticket in downstream test suite use of 'ipactl stop' on replica before master uninstall.
Comment 19 Standa Laznicka 2018-04-18 12:37:20 UTC 
Seems like the issue was resolved and appropriate doc bug filed. Rizwan, is there anything else to do here?

P.S.: I removed the "Private" flag from all the comments that did not contain any sensitive information and could be helpful to anyone looking for information on this issue.
Comment 20 Mohammad Rizwan 2018-04-30 13:11:04 UTC 
As I have mentioned earlier, steps provided works for me.
This can be closed.
Comment 21 Sergey Orlov 2018-11-21 12:34:00 UTC 
Fixed upstream
master:
https://pagure.io/freeipa/c/8182ebc6c3ca636276fc277186cfbff4ea9cf5c6
Note You need to log in before you can comment on or make changes to this bug.