Bug 1542960
Summary: | restored data from backup on master vanished when replica is re-initialized | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Mohammad Rizwan <myusuf> |
Component: | ipa | Assignee: | IPA Maintainers <ipa-maint> |
Status: | CLOSED NOTABUG | QA Contact: | ipa-qe <ipa-qe> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 7.5 | CC: | frenaud, lkrispen, myusuf, pvoborni, rcritten, slaznick, sorlov, tbordaz, tscherf |
Target Milestone: | rc | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-05-02 07:42:10 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Mohammad Rizwan
2018-02-07 13:07:00 UTC
version: ipa-server-4.5.0-20.el7.x86_64 389-ds-base-1.3.6.1-16.el7.x86_64 same behaviour observed on rhel7.4 too. I think the testcase could hit a timing issue. I suspect that after step 8, the replica can connect to restored master and send DEL test1Master (from step 4) and DEL test1Replica (from step 5). So depending how fast is replication Replica->Master. The test users may appear or not. I suggest that you disable RA Replica->Master in step 6 (rather than the opposite). And reenable it between steps 12 and 13. (In reply to thierry bordaz from comment #3) > I think the testcase could hit a timing issue. > > I suspect that after step 8, the replica can connect to restored master and > send DEL test1Master (from step 4) and DEL test1Replica (from step 5). So > depending how fast is replication Replica->Master. The test users may appear > or not. > > I suggest that you disable RA Replica->Master in step 6 (rather than the > opposite). And reenable it between steps 12 and 13. Okay, I can try disabling the RA from replica itself (and not from master). As per my understanding, RA becomes active (enable) when topologysegment-reinitialize executed (as per my observations in https://bugzilla.redhat.com/show_bug.cgi?id=1498523#c36) How exactly you are suggesting to re-enable it between step 12 and 13? Hi Mohammad, You are likely right, topology plugin should handle semi establish RAs and reenable RAs on both direction. I think you may check the RA replica->master status and reenable it if it is not enabled. I did followed the same steps from the description except step6. I tried disabling the RA from replica as following: [root@replica ~]# cat a.ldif dn: cn=meTomaster.testrelm.test,cn=replica,cn=dc\3Dtestrelm\2Cdc\3Dtest,cn=mapping tree,cn=config changetype: modify replace: nsds5ReplicaEnabled nsds5ReplicaEnabled: off dn: cn=caTomaster.testrelm.test,cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config changetype: modify replace: nsds5ReplicaEnabled nsds5ReplicaEnabled: off [root@replica ~]# ldapmodify -h localhost -D "cn=Directory Manager" -p 389 -w Secret123 -f a.ldif After restore I can see following entries on master: ~~~~~~~~~~~~~~~~~~~ Entries on master after restore ~~~~~~~~~~~~~~~~~~~ [root@master ~]# ipa user-find --------------- 3 users matched --------------- User login: admin Last name: Administrator Home directory: /home/admin Login shell: /bin/bash Principal alias: admin UID: 1137200000 GID: 1137200000 Account disabled: False User login: test1master First name: test1 Last name: before_backup Home directory: /home/test1master Login shell: /bin/sh Principal name: test1master Principal alias: test1master Email address: test1master UID: 1137200001 GID: 1137200001 Account disabled: False User login: test1replica First name: test1 Last name: before_backup Home directory: /home/test1replica Login shell: /bin/sh Principal name: test1replica Principal alias: test1replica Email address: test1replica UID: 1137300500 GID: 1137300500 Account disabled: False ---------------------------- Number of entries returned 3 ---------------------------- ~~~~~~~~~~~~~~~~~~~~~~~~~ Entries on replica after restore ~~~~~~~~~~~~~~~~~~~~~~~~~ [root@replica ~]# ipa user-find --------------- 3 users matched --------------- User login: admin Last name: Administrator Home directory: /home/admin Login shell: /bin/bash Principal alias: admin UID: 1137200000 GID: 1137200000 Account disabled: False User login: test2master First name: test2 Last name: after_backup Home directory: /home/test2master Login shell: /bin/sh Principal name: test2master Principal alias: test2master Email address: test2master UID: 1137200003 GID: 1137200003 Account disabled: False User login: test2replica First name: test2 Last name: after_backup Home directory: /home/test2replica Login shell: /bin/sh Principal name: test2replica Principal alias: test2replica Email address: test2replica UID: 1137300501 GID: 1137300501 Account disabled: False ---------------------------- Number of entries returned 3 ---------------------------- Then I added test3master on master to check if replication is working [root@master ~]# ipa user-find --------------- 4 users matched --------------- User login: admin Last name: Administrator Home directory: /home/admin Login shell: /bin/bash Principal alias: admin UID: 1137200000 GID: 1137200000 Account disabled: False User login: test1master First name: test1 Last name: before_backup Home directory: /home/test1master Login shell: /bin/sh Principal name: test1master Principal alias: test1master Email address: test1master UID: 1137200001 GID: 1137200001 Account disabled: False User login: test1replica First name: test1 Last name: before_backup Home directory: /home/test1replica Login shell: /bin/sh Principal name: test1replica Principal alias: test1replica Email address: test1replica UID: 1137300500 GID: 1137300500 Account disabled: False User login: test3master First name: test3 Last name: after_restore Home directory: /home/test3master Login shell: /bin/sh Principal name: test3master Principal alias: test3master Email address: test3master UID: 1137200004 GID: 1137200004 Account disabled: False ---------------------------- Number of entries returned 4 ---------------------------- But the user test3master not listed on replica. I enabled the replication by setting "nsds5ReplicaEnabled: on" in ldif on replica, still replication is not working. (I have ran the topologysegment-reinitialize on both the server before enabling by ldif.) So, if replication is disabled from replica then after re-init, replication is not working. At step 8, Master is restored from a backup. So it is late compare to Replica. This is why we disabled (step 6) RA Replica->Master to keep Master as it was at backup time. But Master being "in the past", is not able to update Replica that is more in advance. If you want Master to replicate to Replica, you need to continue the procedure, step 9 and 10. At step 10, Master will reinit Replica. So later, update on Master (test3master) will be correctly replciated when master reinit replica at step 10, no replication is happening between master and replica i.e user test3master didn't listed on replica. Also, the user entries on master and replica also differs: ~~~~~~~~~~ On master: ~~~~~~~~~~ [root@master ~]# ipa user-find --------------- 4 users matched --------------- User login: admin Last name: Administrator Home directory: /home/admin Login shell: /bin/bash Principal alias: admin UID: 1274000000 GID: 1274000000 Account disabled: False User login: test1master First name: test1 Last name: before_backup Home directory: /home/test1master Login shell: /bin/sh Principal name: test1master Principal alias: test1master Email address: test1master UID: 1274000001 GID: 1274000001 Account disabled: False User login: test1replica First name: test1 Last name: before_backup Home directory: /home/test1replica Login shell: /bin/sh Principal name: test1replica Principal alias: test1replica Email address: test1replica UID: 1274100500 GID: 1274100500 Account disabled: False User login: test3master First name: test3 Last name: after_restore Home directory: /home/test3master Login shell: /bin/sh Principal name: test3master Principal alias: test3master Email address: test3master UID: 1274000003 GID: 1274000003 Account disabled: False ---------------------------- Number of entries returned 4 ---------------------------- ~~~~~~~~~~ On replica: ~~~~~~~~~~ [root@replica ~]# ipa user-find --------------- 3 users matched --------------- User login: admin Last name: Administrator Home directory: /home/admin Login shell: /bin/bash Principal alias: admin UID: 1274000000 GID: 1274000000 Account disabled: False User login: test2master First name: test2 Last name: after_backup Home directory: /home/test2master Login shell: /bin/sh Principal name: test2master Principal alias: test2master Email address: test2master UID: 1274000003 GID: 1274000003 Account disabled: False User login: test2replica First name: test2 Last name: after_backup Home directory: /home/test2replica Login shell: /bin/sh Principal name: test2replica Principal alias: test2replica Email address: test2replica UID: 1274100501 GID: 1274100501 Account disabled: False ---------------------------- Number of entries returned 3 ---------------------------- Hi, in your last tests, did you uninstall the master with ipa-server-install --uninstall? I am trying to follow the same steps, but if I uninstall the master, the replica removes the topologysegment and the server from its configuration and is not able to re-enable the replication after the master is restored. On the contrary, if I simply call restore (without uninstall), then perform on the replica: ipa-replica-manage re-initialize --from=restored_master_FQDN the replica gets re-initialized and contains the entries from the backup. Same thing for the master, it contains the entries from the backup. (In reply to Florence Blanc-Renaud from comment #9) > Hi, > > in your last tests, did you uninstall the master with ipa-server-install > --uninstall? I am trying to follow the same steps, but if I uninstall the > master, the replica removes the topologysegment and the server from its > configuration and is not able to re-enable the replication after the master > is restored. yes, I uninstalled master. > On the contrary, if I simply call restore (without uninstall), then perform > on the replica: > ipa-replica-manage re-initialize --from=restored_master_FQDN > > the replica gets re-initialized and contains the entries from the backup. > Same thing for the master, it contains the entries from the backup. Recommended process is to uninstall the master before restore[1], so we need to uninstall the server before restore. I used topologysegment command because of domain-level 1, so in this particular case (re-init), do we have to use ipa-replica-manage command? [1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/linux_domain_identity_authentication_and_policy_guide/#restoring-full-server Hi I tried the scenario mentioned in description, but this time I used ipa-replica-manage instead of topologysegment. Now I can see the user (i.e created before backup) after restore. ~~~~~~~~~~~~~~ before restore ~~~~~~~~~~~~~~ [root@replica1 ~]# ipa user-find --------------- 3 users matched --------------- User login: admin Last name: Administrator Home directory: /home/admin Login shell: /bin/bash Principal alias: admin UID: 1364600000 GID: 1364600000 Account disabled: False User login: test2master First name: test Last name: after_backup Home directory: /home/test2master Login shell: /bin/sh Principal name: test2master Principal alias: test2master Email address: test2master UID: 1364600003 GID: 1364600003 Account disabled: False User login: test2replica First name: test Last name: after_backup Home directory: /home/test2replica Login shell: /bin/sh Principal name: test2replica Principal alias: test2replica Email address: test2replica UID: 1364700501 GID: 1364700501 Account disabled: False ---------------------------- Number of entries returned 3 ---------------------------- ~~~~~~~~~~~~~ after restore ~~~~~~~~~~~~~ [root@replica1 ~]# ipa user-find --------------- 3 users matched --------------- User login: admin Last name: Administrator Home directory: /home/admin Login shell: /bin/bash Principal alias: admin UID: 1364600000 GID: 1364600000 Account disabled: False User login: test1master First name: test Last name: before_backup Home directory: /home/test1master Login shell: /bin/sh Principal name: test1master Principal alias: test1master Email address: test1master UID: 1364600001 GID: 1364600001 Account disabled: False User login: test1replica First name: test Last name: before_backup Home directory: /home/test1replica Login shell: /bin/sh Principal name: test1replica Principal alias: test1replica Email address: test1replica UID: 1364700500 GID: 1364700500 Account disabled: False ---------------------------- Number of entries returned 3 ---------------------------- Replication working fine after restore (I checked by adding user on replica1 and listing it on master) A few observations: 1- if the master is uninstalled with ipa-server-install --uninstall -U, and the replication is still enabled from master to replica, then it will delete the master entry from the replica and the topologysegment, making it more difficult to re-establish the replication. For this test scenario, we should either: - disable replication both ways, then uninstall the master or - do not call uninstall on the master 2- when the master is restored, there are 2 possible paths to re-establish the replication: - with ipa-replica-manage re-initialize --from=restored_master_FQDN called on the replica or - with ipa topologysegment-reinitialize domain <segment name> --left (assuming that left is the replica) It looks like performing topologysegment-reinitialize does not reset nsds5ReplicaEnabled: on. Ludwig, could you have a look and tell us if we should manually modify this attribute? Not sure I fully understand the scenario, but if you have agreements A --> B and B --> A If A-->B is disabled, reinit A-->B will be rejected with unwilling to perform If you do reinit B-->A thenit will work, but the agreement A-->B will not be modified IIUC this bugs wait on resolution of the team list discussion about restore behavior. Or other things. Setting needs info so that we won't forget. Upstream ticket: https://pagure.io/freeipa/issue/7455 Steps in https://pagure.io/freeipa/issue/7455 works for me. Need to add the mention scenario from above ticket in downstream test suite use of 'ipactl stop' on replica before master uninstall. Seems like the issue was resolved and appropriate doc bug filed. Rizwan, is there anything else to do here? P.S.: I removed the "Private" flag from all the comments that did not contain any sensitive information and could be helpful to anyone looking for information on this issue. As I have mentioned earlier, steps provided works for me. This can be closed. Fixed upstream master: https://pagure.io/freeipa/c/8182ebc6c3ca636276fc277186cfbff4ea9cf5c6 |