Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: When using ipa-replica-install to install a replication server, it fails at a particularly vulnerable moment and leaves the entire ipa instance broken. Version-Release number of selected component (if applicable): 4.7.2-8.fc30 How reproducible: 100% (in my two attempts) Steps to Reproduce: 1. Have an existing freeipa environment with f29 based servers 2. Run ipa-replica-install on a clean f30 server beta install 3. Wait until pki-tomcatd is being configured and it will fail Actual results: Installs aborts, and does not unwind cleanly Expected results: Replication completes Additional info: See attached log files
Install script contents: [root@nas Install-Common]# more setup-ipa.sh #!/bin/bash if [ -f /etc/ipa/ca.crt ]; then echo "It appears that the ipa server is already installed. If this is not" echo "the case, then remove the file /etc/ipa/ca.crt and re-run this script" exit 0 fi firewall-cmd --zone=FedoraServer --add-service=dns --add-service=freeipa-ldap --add-service=freeipa-ldaps --add-service=freeipa-trust --add-service=freeipa-replication --add-service=kadmin --add-service=klogin --add-service=kpasswd --add-service=ntp [ $? -ne 0 ] && exit 1 firewall-cmd --runtime-to-permanent [ $? -ne 0 ] && exit 1 ipa-replica-install --ip-address=aa.bb.cc.dd -n xxx.com --server=ipa-master.xxx.com -r XXX.COM --hostname=nas.xxx.com -P admin -p 'XXXXXXXXXXXXXXX' --setup-adtrust --setup-ca --setup-kra --setup-dns --mkhomedir --ntp-server=ntp.xxx.com --ntp-pool=0.fedora.pool.ntp.org --ssh-trust-dns --netbios-name=XXX --add-sids --no-forwarders --skip-conncheck
Doug, are you planning to attach logs? I'm not seeing any.
Note that it purposely does not uninstall the remains to make troubleshooting easier so this part is working as expected unless you're saying that running ipa-server-install --uninstall does not clean things up.
Are there any AVCs? What is the mode and ownership of /tmp?
(In reply to Alexander Bokovoy from comment #2) > Doug, > > are you planning to attach logs? I'm not seeing any. I needed to copy them from the server to my workstation and sanitize them some. Done. When this failure happens, the entire ipa cluster is left mildly broken as the replicant entry is partially setup and attempts to unwind it manually using the web ui pop up errors about the machine being in an invalid state (sorry, don't want to repeat that issue, learned my lesson on that yesterday, it took several hours for me to reconstruct my previous IPA state once I restored from my last good backup...at least today when I gathered these logs I had an up2date backup).
(In reply to Rob Crittenden from comment #6) > Are there any AVCs? What is the mode and ownership of /tmp? [root@nas Install-Common]# ls -l /tmp/ca.p12 -rw-r--r--. 1 pkiuser pkiuser 10318 Apr 8 13:28 /tmp/ca.p12 I don't see any AVCs
(In reply to Rob Crittenden from comment #5) > Note that it purposely does not uninstall the remains to make > troubleshooting easier so this part is working as expected unless you're > saying that running ipa-server-install --uninstall does not clean things up. Unless I mis-remember yesterday (which is possible, it turned out to be a long day), yes, ipa-server-uninstall cleaned up the client, but it was not able to clean up the rest of the IPA cluster properly, so I still had to restore from backup to clean out items. At one point, the IPA server web ui was reduced to nothing but a runtime error failure message and nothing else.
Doug, Rob, I'm building a reproducer and will let you know.
The trace looks similar to https://bugzilla.redhat.com/show_bug.cgi?id=1677027 where ipa-server-install had to be modified to chown the configuration file only after writing it. File "/usr/lib/python3.7/site-packages/ipaserver/install/cainstance.py", line 600, in __spawn_instance shutil.copy(cafile, paths.TMP_CA_P12) File "/usr/lib64/python3.7/shutil.py", line 245, in copy copyfile(src, dst, follow_symlinks=follow_symlinks) File "/usr/lib64/python3.7/shutil.py", line 121, in copyfile with open(dst, 'wb') as fdst: PermissionError: [Errno 13] Permission denied: '/tmp/ca.p12' and /tmp/ca.p12 is owned by pkiuser:pkiuser while ipa-replica-install runs as root. However: * that file is chowned after shutil.copy in cainstance.py: os.chown(paths.TMP_CA_P12, pent.pw_uid, pent.pw_gid) * so so far I haven't been able to reproduce the issue. Doug, are you 100% sure that replica OS was clean, with no chance of weird permissions on /tmp, full file systems, and no leftover file in /tmp?
I'm tempted to chown that file to root:root if it exists before invoking shutil.copy.
(In reply to François Cami from comment #11) > > Doug, are you 100% sure that replica OS was clean, with no chance of weird > permissions on /tmp, full file systems, and no leftover file in /tmp? Yes. System was a fresh install. However, I'm now dealing with the fact that what I did to recover from the second install didn't work. What I did was this: ipa-server-1: ipa-backup ipa-server-2: ipa-backup test-machine: ipa-replica-install <wait for failure> ipa-server-1: ipa-restore <full-backup> ipa-server-2: ipa-restore <full-backup> and that left my system broken. Now, all of my user ids that used to appear as just <userid> are showing up as <userid>@<domain>, which has caused my mail server to stop delivering mail as the formerly existing Maildir directories are no longer named the same as what the system expects. Any idea on how I should resolve this? Was a backup/test/restore cycle not acceptable use of the ipa server? It appears the above process also caused all of the ssh keys on the servers to get reset.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/restore Restoring a cluster from a backup is typically done on a single master, then other replica are reinitialized from that master. However your current issue is not related to this bug.
(In reply to François Cami from comment #12) > I'm tempted to chown that file to root:root if it exists before invoking > shutil.copy. I don't believe that will resolve the issue. I say this because I did the install once, it failed, I took half a day recovering my then broken IPA database, then I backed everything up, then did the install again in order to make this bug. The first time, it failed on a tmp file created by a unique tmpfile operation (like /tmp/tmp8k763cpk) and the second time it failed on /tmp/ca.p12. So the specific file is not the issue. There is something about a file owned by pkiuser that is failing. I'll do a fresh install and try a third time if you like.
> The first time, it failed on a tmp file created by a unique tmpfile operation (like /tmp/tmp8k763cpk) Ok. Then everything makes sense. --setup-kra was broken on f30. I discovered this yesterday: https://bugzilla.redhat.com/show_bug.cgi?id=1698384 https://pagure.io/freeipa/issue/7906 This will be fixed by https://github.com/freeipa/freeipa/pull/3022 and is the original root cause. It left /tmp/ca.p12 owned by pkiuser:pkiuser. This made that shutil.copy call in cainstance.py choke the 2nd time. So the solution is indeed to protect that call to shutil.copy when the file is already present, either by chowning it to root or removing it altogether, or choosing a proper temp file.
Upstream ticket: https://pagure.io/freeipa/issue/7907
Reproducer: # touch /tmp/ca.p12 # chown pkiuser:pkiuser /tmp/ca.p12 # ipa-replica-install ... Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes [1/28]: creating certificate server db [2/28]: setting up initial replication Starting replication, please wait until this has completed. Update in progress, 4 seconds elapsed Update succeeded [3/28]: creating ACIs for admin [4/28]: creating installation admin user [5/28]: configuring certificate server instance [error] PermissionError: [Errno 13] Permission denied: '/tmp/ca.p12' Your system may be partly configured.
Forgot to post full trace. This is from my reproducer above: 2019-04-10T15:12:05Z DEBUG Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/ipaserver/install/service.py", line 605, in start_creation run_step(full_msg, method) File "/usr/lib/python3.7/site-packages/ipaserver/install/service.py", line 591, in run_step method() File "/usr/lib/python3.7/site-packages/ipaserver/install/cainstance.py", line 600, in __spawn_instance shutil.copy(cafile, paths.TMP_CA_P12) File "/usr/lib64/python3.7/shutil.py", line 245, in copy copyfile(src, dst, follow_symlinks=follow_symlinks) File "/usr/lib64/python3.7/shutil.py", line 121, in copyfile with open(dst, 'wb') as fdst: PermissionError: [Errno 13] Permission denied: '/tmp/ca.p12' 2019-04-10T15:12:05Z DEBUG [error] PermissionError: [Errno 13] Permission denied: '/tmp/ca.p12' 2019-04-10T15:12:05Z DEBUG Removing /root/.dogtag/pki-tomcat/ca 2019-04-10T15:12:05Z DEBUG File "/usr/lib/python3.7/site-packages/ipapython/admintool.py", line 179, in execute return_value = self.run() File "/usr/lib/python3.7/site-packages/ipapython/install/cli.py", line 347, in run return cfgr.run() File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 360, in run return self.execute() File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 386, in execute for rval in self._executor(): File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.7/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python3.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.7/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 655, in _configure next(executor) File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception self._handle_exception(exc_info) File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.7/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 515, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python3.7/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python3.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python3.7/site-packages/six.py", line 693, in reraise raise value File "/usr/lib/python3.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python3.7/site-packages/ipapython/install/common.py", line 65, in _install for unused in self._installer(self.parent): File "/usr/lib/python3.7/site-packages/ipaserver/install/server/__init__.py", line 583, in main replica_install(self) File "/usr/lib/python3.7/site-packages/ipaserver/install/server/replicainstall.py", line 400, in decorated func(installer) File "/usr/lib/python3.7/site-packages/ipaserver/install/server/replicainstall.py", line 1262, in install ca.install(False, config, options, custodia=custodia) File "/usr/lib/python3.7/site-packages/ipaserver/install/ca.py", line 239, in install install_step_0(standalone, replica_config, options, custodia=custodia) File "/usr/lib/python3.7/site-packages/ipaserver/install/ca.py", line 318, in install_step_0 use_ldaps=standalone) File "/usr/lib/python3.7/site-packages/ipaserver/install/cainstance.py", line 484, in configure_instance self.start_creation(runtime=runtime) File "/usr/lib/python3.7/site-packages/ipaserver/install/service.py", line 605, in start_creation run_step(full_msg, method) File "/usr/lib/python3.7/site-packages/ipaserver/install/service.py", line 591, in run_step method() File "/usr/lib/python3.7/site-packages/ipaserver/install/cainstance.py", line 600, in __spawn_instance shutil.copy(cafile, paths.TMP_CA_P12) File "/usr/lib64/python3.7/shutil.py", line 245, in copy copyfile(src, dst, follow_symlinks=follow_symlinks) File "/usr/lib64/python3.7/shutil.py", line 121, in copyfile with open(dst, 'wb') as fdst:
Fixed upstream master: https://pagure.io/freeipa/c/d68fe6b981ba1698aee35518b844eb5584a0f302
Fixed upstream ipa-4-7: https://pagure.io/freeipa/c/eeeed847b7bbd28b5883b6a3f9cd6827bdc242a4
Fixed in 4.8.0-1