Bug 1698188

Install script contents:

[root@nas Install-Common]# more setup-ipa.sh 
#!/bin/bash

if [ -f /etc/ipa/ca.crt ]; then
  echo "It appears that the ipa server is already installed.  If this is not"
  echo "the case, then remove the file /etc/ipa/ca.crt and re-run this script"
  exit 0
fi

firewall-cmd --zone=FedoraServer --add-service=dns --add-service=freeipa-ldap --add-service=freeipa-ldaps --add-service=freeipa-trust --add-service=freeipa-replication --add-service=kadmin --add-service=klogin --add-service=kpasswd --add-service=ntp
[ $? -ne 0 ] && exit 1
firewall-cmd --runtime-to-permanent
[ $? -ne 0 ] && exit 1

ipa-replica-install --ip-address=aa.bb.cc.dd -n xxx.com --server=ipa-master.xxx.com -r XXX.COM --hostname=nas.xxx.com -P admin -p 'XXXXXXXXXXXXXXX' --setup-adtrust --setup-ca --setup-kra --setup-dns --mkhomedir --ntp-server=ntp.xxx.com --ntp-pool=0.fedora.pool.ntp.org --ssh-trust-dns --netbios-name=XXX --add-sids --no-forwarders --skip-conncheck

Doug,

are you planning to attach logs? I'm not seeing any.

Note that it purposely does not uninstall the remains to make troubleshooting easier so this part is working as expected unless you're saying that running ipa-server-install --uninstall does not clean things up.

Are there any AVCs? What is the mode and ownership of /tmp?

(In reply to Alexander Bokovoy from comment #2)
> Doug,
> 
> are you planning to attach logs? I'm not seeing any.

I needed to copy them from the server to my workstation and sanitize them some.  Done.  When this failure happens, the entire ipa cluster is left mildly broken as the replicant entry is partially setup and attempts to unwind it manually using the web ui pop up errors about the machine being in an invalid state (sorry, don't want to repeat that issue, learned my lesson on that yesterday, it took several hours for me to reconstruct my previous IPA state once I restored from my last good backup...at least today when I gathered these logs I had an up2date backup).

(In reply to Rob Crittenden from comment #6)
> Are there any AVCs? What is the mode and ownership of /tmp?

[root@nas Install-Common]# ls -l /tmp/ca.p12 
-rw-r--r--. 1 pkiuser pkiuser 10318 Apr  8 13:28 /tmp/ca.p12

I don't see any AVCs

(In reply to Rob Crittenden from comment #5)
> Note that it purposely does not uninstall the remains to make
> troubleshooting easier so this part is working as expected unless you're
> saying that running ipa-server-install --uninstall does not clean things up.

Unless I mis-remember yesterday (which is possible, it turned out to be a long day), yes, ipa-server-uninstall cleaned up the client, but it was not able to clean up the rest of the IPA cluster properly, so I still had to restore from backup to clean out items.  At one point, the IPA server web ui was reduced to nothing but a runtime error failure message and nothing else.

Doug, Rob, I'm building a reproducer and will let you know.

The trace looks similar to https://bugzilla.redhat.com/show_bug.cgi?id=1677027 where ipa-server-install had to be modified to chown the configuration file only after writing it.

  File "/usr/lib/python3.7/site-packages/ipaserver/install/cainstance.py", line 600, in __spawn_instance
    shutil.copy(cafile, paths.TMP_CA_P12)
  File "/usr/lib64/python3.7/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.7/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: '/tmp/ca.p12'

and /tmp/ca.p12 is owned by pkiuser:pkiuser while ipa-replica-install runs as root. 
However:
* that file is chowned after shutil.copy in cainstance.py: os.chown(paths.TMP_CA_P12, pent.pw_uid, pent.pw_gid)
* so so far I haven't been able to reproduce the issue.

Doug, are you 100% sure that replica OS was clean, with no chance of weird permissions on /tmp, full file systems, and no leftover file in /tmp?

I'm tempted to chown that file to root:root if it exists before invoking shutil.copy.

(In reply to François Cami from comment #11)
> 
> Doug, are you 100% sure that replica OS was clean, with no chance of weird
> permissions on /tmp, full file systems, and no leftover file in /tmp?

Yes.  System was a fresh install.  However, I'm now dealing with the fact that what I did to recover from the second install didn't work.  What I did was this:

ipa-server-1:
ipa-backup

ipa-server-2:
ipa-backup

test-machine:
ipa-replica-install
<wait for failure>

ipa-server-1:
ipa-restore <full-backup>

ipa-server-2:
ipa-restore <full-backup>

and that left my system broken.  Now, all of my user ids that used to appear as just <userid> are showing up as <userid>@<domain>, which has caused my mail server to stop delivering mail as the formerly existing Maildir directories are no longer named the same as what the system expects.  Any idea on how I should resolve this?  Was a backup/test/restore cycle not acceptable use of the ipa server?  It appears the above process also caused all of the ssh keys on the servers to get reset.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/restore

Restoring a cluster from a backup is typically done on a single master, then other replica are reinitialized from that master.
However your current issue is not related to this bug.

(In reply to François Cami from comment #12)
> I'm tempted to chown that file to root:root if it exists before invoking
> shutil.copy.

I don't believe that will resolve the issue.  I say this because I did the install once, it failed, I took half a day recovering my then broken IPA database, then I backed everything up, then did the install again in order to make this bug.  The first time, it failed on a tmp file created by a unique tmpfile operation (like /tmp/tmp8k763cpk) and the second time it failed on /tmp/ca.p12.  So the specific file is not the issue.  There is something about a file owned by pkiuser that is failing.  I'll do a fresh install and try a third time if you like.

> The first time, it failed on a tmp file created by a unique tmpfile operation (like /tmp/tmp8k763cpk)

Ok. Then everything makes sense.

--setup-kra was broken on f30. I discovered this yesterday:
https://bugzilla.redhat.com/show_bug.cgi?id=1698384
https://pagure.io/freeipa/issue/7906

This will be fixed by https://github.com/freeipa/freeipa/pull/3022 and is the original root cause.
It left /tmp/ca.p12 owned by pkiuser:pkiuser.
This made that shutil.copy call in cainstance.py choke the 2nd time.

So the solution is indeed to protect that call to shutil.copy when the file is already present, either by chowning it to root or removing it altogether, or choosing a proper temp file.

Upstream ticket:
https://pagure.io/freeipa/issue/7907

Reproducer:

# touch /tmp/ca.p12
# chown pkiuser:pkiuser /tmp/ca.p12
# ipa-replica-install ...

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/28]: creating certificate server db
  [2/28]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 4 seconds elapsed
Update succeeded

  [3/28]: creating ACIs for admin
  [4/28]: creating installation admin user
  [5/28]: configuring certificate server instance
  [error] PermissionError: [Errno 13] Permission denied: '/tmp/ca.p12'
Your system may be partly configured.

Forgot to post full trace. This is from my reproducer above:

2019-04-10T15:12:05Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/ipaserver/install/service.py", line 605, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python3.7/site-packages/ipaserver/install/service.py", line 591, in run_step
    method()
  File "/usr/lib/python3.7/site-packages/ipaserver/install/cainstance.py", line 600, in __spawn_instance
    shutil.copy(cafile, paths.TMP_CA_P12)
  File "/usr/lib64/python3.7/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.7/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: '/tmp/ca.p12'

2019-04-10T15:12:05Z DEBUG   [error] PermissionError: [Errno 13] Permission denied: '/tmp/ca.p12'
2019-04-10T15:12:05Z DEBUG Removing /root/.dogtag/pki-tomcat/ca
2019-04-10T15:12:05Z DEBUG   File "/usr/lib/python3.7/site-packages/ipapython/admintool.py", line 179, in execute
    return_value = self.run()
  File "/usr/lib/python3.7/site-packages/ipapython/install/cli.py", line 347, in run
    return cfgr.run()
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 360, in run
    return self.execute()
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 386, in execute
    for rval in self._executor():
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 655, in _configure
    next(executor)
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 431, in __runner
    exc_handler(exc_info)
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 460, in _handle_execute_exception
    self._handle_exception(exc_info)
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 515, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 421, in __runner
    step()
  File "/usr/lib/python3.7/site-packages/ipapython/install/core.py", line 418, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python3.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python3.7/site-packages/ipapython/install/common.py", line 65, in _install
    for unused in self._installer(self.parent):
  File "/usr/lib/python3.7/site-packages/ipaserver/install/server/__init__.py", line 583, in main
    replica_install(self)
  File "/usr/lib/python3.7/site-packages/ipaserver/install/server/replicainstall.py", line 400, in decorated
    func(installer)
  File "/usr/lib/python3.7/site-packages/ipaserver/install/server/replicainstall.py", line 1262, in install
    ca.install(False, config, options, custodia=custodia)
  File "/usr/lib/python3.7/site-packages/ipaserver/install/ca.py", line 239, in install
    install_step_0(standalone, replica_config, options, custodia=custodia)
  File "/usr/lib/python3.7/site-packages/ipaserver/install/ca.py", line 318, in install_step_0
    use_ldaps=standalone)
  File "/usr/lib/python3.7/site-packages/ipaserver/install/cainstance.py", line 484, in configure_instance
    self.start_creation(runtime=runtime)
  File "/usr/lib/python3.7/site-packages/ipaserver/install/service.py", line 605, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python3.7/site-packages/ipaserver/install/service.py", line 591, in run_step
    method()
  File "/usr/lib/python3.7/site-packages/ipaserver/install/cainstance.py", line 600, in __spawn_instance
    shutil.copy(cafile, paths.TMP_CA_P12)
  File "/usr/lib64/python3.7/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib64/python3.7/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:

Fixed upstream
master:
https://pagure.io/freeipa/c/d68fe6b981ba1698aee35518b844eb5584a0f302

Fixed upstream
ipa-4-7:
https://pagure.io/freeipa/c/eeeed847b7bbd28b5883b6a3f9cd6827bdc242a4

Fixed in 4.8.0-1