Bug 1511607

Upstream ticket:
https://pagure.io/freeipa/issue/7247

Possible workaround (not confirmed) is hinted in bug 1498523:

"""
Backup the tob e edited config files). Edit sysrestore state (in /var/lib/ipa/sysrestore dir) and delete "custodia" entry. Try (on the master) to delete /etc/ipa/custodia/server.keys then run ipa-
server-update (this should re-generate the server.keys file), then retry replica2 install.
"""

Fixed upstream
master:
https://pagure.io/freeipa/c/8bbeedc93fd442cbbb9bb70e5f446011e95211db

ipa-4-6:
    a926a00 Backup ipa-custodia conf and keys


ipa-4-5:
    07c0825 Backup ipa-custodia conf and keys

IPA-server-version: ipa-server-4.5.4-7.el7.x86_64

Tested the bug with following steps:
1. Setup ipa master with latest version.
2. Run the following backup command:
#ipa-backup -v --logs --log-file=ipabackup_test.log

Tested the bug for following scenarios:

Scenario-1: (Check backup log)
-----------------------------------
3. [root@ndipa ~]# cat ipabackup_test.log | grep custodia
Stopping ipa-custodia Service
2017-12-18T10:47:58Z DEBUG args=tar --exclude=/var/lib/ipa/backup --xattrs --selinux -cf /tmp/tmpJh0vt9ipa/ipa/files.tar /usr/share/ipa/html /etc/pki/pki-tomcat /etc/sysconfig/pki /etc/httpd/alias /var/lib/pki /var/lib/ipa/sysrestore /var/lib/ipa-client/sysrestore /var/lib/ipa/dnssec /var/lib/sss/pubconf/krb5.include.d/ /var/lib/authconfig/last /var/lib/certmonger /var/lib/ipa /var/run/dirsrv /var/lock/dirsrv /etc/dirsrv/slapd-TESTRELM-TEST /var/lib/dirsrv/scripts-TESTRELM-TEST /var/lib/dirsrv/slapd-TESTRELM-TEST /etc/named.conf /etc/named.keytab /etc/resolv.conf /etc/sysconfig/pki-tomcat /etc/sysconfig/dirsrv /etc/sysconfig/ntpd /etc/sysconfig/krb5kdc /etc/sysconfig/ipa-dnskeysyncd /etc/sysconfig/ipa-ods-exporter /etc/sysconfig/named /etc/sysconfig/ods /etc/sysconfig/authconfig /etc/ipa/nssdb/pwdfile.txt /etc/pki/ca-trust/source/ipa.p11-kit /etc/nsswitch.conf /etc/krb5.keytab /etc/sssd/sssd.conf /etc/openldap/ldap.conf /etc/security/limits.conf /etc/httpd/conf/password.conf /var/lib/ipa/gssproxy/http.keytab /etc/ipa/kdcproxy/ipa-kdc-proxy.conf /etc/httpd/conf.d/ipa-pki-proxy.conf /etc/httpd/conf.d/ipa-rewrite.conf /etc/httpd/conf.d/nss.conf /etc/httpd/conf.d/ipa.conf /etc/ssh/sshd_config /etc/ssh/ssh_config /etc/krb5.conf /var/lib/ipa-client/pki/kdc-ca-bundle.pem /var/lib/ipa-client/pki/ca-bundle.pem /etc/ipa/ca.crt /etc/ipa/default.conf /etc/dirsrv/ds.keytab /etc/ntp.conf /etc/samba/smb.conf /root/ca-agent.p12 /var/lib/ipa/ra-agent.pem /var/lib/ipa/ra-agent.key /root/cacert.p12 /var/kerberos/krb5kdc/kdc.conf /var/kerberos/krb5kdc/kdc.crt /var/kerberos/krb5kdc/kdc.key /var/kerberos/krb5kdc/cacert.pem /etc/systemd/system/multi-user.target.wants/ipa.service /etc/systemd/system/httpd.service.d/ipa.conf /etc/systemd/system/multi-user.target.wants/sssd.service /etc/systemd/system/multi-user.target.wants/certmonger.service /etc/systemd/system/pki-tomcatd.target.wants/pki-tomcatd /etc/opendnssec/conf.xml /etc/opendnssec/kasp.xml /etc/opendnssec/zonelist.xml /var/opendnssec/kasp.db /etc/ipa/dnssec/softhsm2.conf /etc/ipa/dnssec/softhsm_pin_so /etc/ipa/dnssec/ipa-dnskeysyncd.keytab /etc/ipa/custodia/server.keys /etc/ipa/custodia/custodia.conf /etc/hosts /etc/ipa/nssdb/cert8.db /etc/ipa/nssdb/key3.db /etc/ipa/nssdb/secmod.db /etc/sysconfig/dirsrv-TESTRELM-TEST /etc/tmpfiles.d/dirsrv-TESTRELM-TEST.conf /var/log/pki/ /var/log/httpd /var/log/ipaserver-install.log /var/log/kadmind.log /var/log/messages /var/log/ipaclient-install.log /var/log/secure /var/named/data/named.run /var/log/dirsrv/slapd-TESTRELM-TEST
Starting ipa-custodia Service

[root@ndipa ~]# echo $?
0

I am able to successfully grep custodia files:
/etc/ipa/custodia/server.keys 
/etc/ipa/custodia/custodia.conf


Scenario-2: (Check backup tar files) 
---------------------------------------
[root@ndipa ipa-full-2017-12-18-16-18-00]# ls -l
total 11268
drwxr-xr-x. 15 root   root      4096 Dec 18 17:35 etc
-rw-r--r--.  1 root   root   4232529 Dec 18 16:17 files.tar
-rw-r--r--.  1 root   root       165 Dec 18 16:17 header
-rw-r--r--.  1 root   root   6050789 Dec 18 16:18 ipa-full.tar
drwxr-xr-x.  2 root   root        44 Dec 18 17:35 root
drwx------.  5 dirsrv dirsrv     138 Dec 18 16:17 TESTRELM-TEST
-rw-------.  1 dirsrv dirsrv  770273 Dec 18 16:17 TESTRELM-TEST-ipaca.ldif
-rw-------.  1 dirsrv dirsrv  464358 Dec 18 16:17 TESTRELM-TEST-userRoot.ldif
drwxr-xr-x.  3 root   root        19 Dec 18 17:35 usr
drwxr-xr-x.  9 root   root        98 Dec 18 17:35 var

[root@ndipa custodia]# pwd
/var/lib/ipa/backup/ipa-full-2017-12-18-16-18-00/etc/ipa/custodia
[root@ndipa custodia]# ls -l
total 8
-rw-r--r--. 1 root root  638 Dec 18 16:03 custodia.conf
-rw-------. 1 root root 3351 Dec 12 15:39 server.keys


Upon extracting the tar file, we could see the custodia files as well.


Scenario-3: (Restore backup)
--------------------------------
[root@ndipa backup]# ls -l /etc/ipa/custodia/
total 8
-rw-r--r--. 1 root root  638 Dec 18 16:03 custodia.conf
-rw-------. 1 root root 3351 Dec 12 15:39 server.keys
[root@ndipa backup]# rm -rf /etc/ipa/custodia/custodia.conf /etc/ipa/custodia/server.keys 
[root@ndipa backup]# ls -l /etc/ipa/custodia/
total 0
[root@ndipa backup]# ls -l
total 0
drwxr-x---. 7 dirsrv dirsrv 190 Dec 18 17:35 ipa-full-2017-12-18-16-18-00
ipa.ipaserver.install.ipa_restore.Restore: INFO: The ipa-restore command was successful
[root@ndipa ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
[root@ndipa ~]# kinit admin
Password for admin: 

[root@ndipa ~]# ls -l /etc/ipa/custodia/
total 8
-rw-r--r--. 1 root root  638 Dec 18 16:03 custodia.conf
-rw-------. 1 root root 3351 Dec 12 15:39 server.keys
[root@ndipa ~]# 


Scenario-4: (Install Replica against restored IPA-Master)
----------------------------------------------------------

On Replica system:
-------------------
[root@ndclient ~]# tail -1 /var/log/ipareplica-install.log 
2017-12-18T14:36:58Z INFO The ipa-replica-install command was successful

[root@ndclient ~]# kinit admin
Password for admin: 

[root@ndclient ~]# ipa host-find
---------------
2 hosts matched
---------------
  Host name: ndclient.testrelm.test
  Principal name: host/ndclient.testrelm.test
  Principal alias: host/ndclient.testrelm.test
  SSH public key fingerprint: SHA256:1EnGEUdQ/gv6LzXvPbc8XxLAjRKRtAhe7up5KV54//Y (ssh-rsa),
                              SHA256:+96k5fM+g3sOyaoO5r9SNTzJIkrL7j7V+VR8mt7hprY (ecdsa-sha2-nistp256),
                              SHA256:Wy5x6kY/Zfk2gnQfW2hvs/Tio8IYe8qwhpF4ge/TxKQ (ssh-ed25519)

  Host name: ndipa.testrelm.test
  Principal name: host/ndipa.testrelm.test
  Principal alias: host/ndipa.testrelm.test
  SSH public key fingerprint: SHA256:j+1dwHR7vTsQcI1sJNjgOh5pvw/NHTHxbAq8q9jOytc (ssh-rsa),
                              SHA256:9vd5BcSfN7ss09EcxAWxVIsyddT/xK/2ZIxXLiCwBy0 (ecdsa-sha2-nistp256),
                              SHA256:aW1IpxYl/WB9hMnYV2mE3dtvCflu8qICkCrHXK/Erwk (ssh-ed25519)
----------------------------
Number of entries returned 2
----------------------------
[root@ndclient ~]# ipa-replica-manage list
ndipa.testrelm.test: master
ndclient.testrelm.test: master
[root@ndclient ~]# 
[root@ndclient ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
pki-tomcatd Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
[root@ndclient ~]# ipactl restart
Stopping pki-tomcatd Service
Restarting Directory Service
Restarting krb5kdc Service
Restarting kadmin Service
Restarting named Service
Restarting httpd Service
Restarting ipa-custodia Service
Restarting ntpd Service
Restarting pki-tomcatd Service
Restarting ipa-otpd Service
Restarting ipa-dnskeysyncd Service
ipa: INFO: The ipactl command was successful
[root@ndclient ~]# 

Thus on the basis of above observations, marking the status of bug to "VERIFIED"

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0918