Bug 1409806

Summary: [RFE] Adding --force option for pkispawn
Product: Red Hat Enterprise Linux 8 Reporter: Jan Pazdziora <jpazdziora>
Component: pki-coreAssignee: Endi Sukma Dewata <edewata>
Status: CLOSED UPSTREAM QA Contact: Asha Akkiangady <aakkiang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.3CC: abokovoy, ascheel, cheimes, edewata, frenaud, jpazdziora, mharmsen, pvoborni, rcritten, tdudlak
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-10 20:45:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1405325    

Description Jan Pazdziora 2017-01-03 13:30:08 UTC
Description of problem:

When /etc/pki/pki-tomcat already exists, running ipa-server-install fails.

Version-Release number of selected component (if applicable):

ipa-server-4.4.0-14.el7_3.x86_64
pki-server-10.3.3-14.el7_3.noarch

How reproducible:

Deterministic.

Steps to Reproduce:
1. yum install -y ipa-server
2. mkdir /etc/pki/pki-tomcat
3. ipa-server-install -r EXAMPLE.TEST -n example.test -p Secret123 -a Secret123 -U

Actual results:

  [37/47]: adding entries for topology management
  [38/47]: initializing group membership
  [39/47]: adding master entry
  [40/47]: initializing domain level
  [41/47]: configuring Posix uid/gid generation
  [42/47]: adding replication acis
  [43/47]: enabling compatibility plugin
  [44/47]: activating sidgen plugin
  [45/47]: activating extdom plugin
  [46/47]: tuning directory server
  [47/47]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes 30 seconds
  [1/31]: creating certificate server user
  [2/31]: configuring certificate server instance
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpdoJ4Dm' returned non-zero exit status 1
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL See the installation logs and the following files/directories for more information:
ipa.ipaserver.install.cainstance.CAInstance: CRITICAL   /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.
ipa.ipapython.install.cli.install_tool(Server): ERROR    CA configuration failed.
ipa.ipapython.install.cli.install_tool(Server): ERROR    The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

The /var/log/ipaserver-install.log ends with

2017-01-03T13:25:15Z DEBUG Starting external process
2017-01-03T13:25:15Z DEBUG args=/usr/sbin/pkispawn -s CA -f /tmp/tmpdoJ4Dm
2017-01-03T13:25:16Z DEBUG Process finished, return code=1
2017-01-03T13:25:16Z DEBUG stdout=Log file: /var/log/pki/pki-ca-spawn.20170103082515.log
Loading deployment configuration from /tmp/tmpdoJ4Dm.
Installing CA into /var/lib/pki/pki-tomcat.
Storing deployment configuration into /etc/sysconfig/pki/tomcat/pki-tomcat/ca/deployment.cfg.

Installation failed: Directory '/etc/pki/pki-tomcat' already exists!


2017-01-03T13:25:16Z DEBUG stderr=pkispawn    : ERROR    ....... Directory '/etc/pki/pki-tomcat' already exists!

2017-01-03T13:25:16Z CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmpdoJ4Dm' returned non-zero exit status 1
2017-01-03T13:25:16Z CRITICAL See the installation logs and the following files/directories for more information:
2017-01-03T13:25:16Z CRITICAL   /var/log/pki/pki-tomcat
2017-01-03T13:25:16Z DEBUG Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 448, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 438, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 590, in __spawn_instance
    DogtagInstance.spawn_instance(self, cfg_file)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 181, in spawn_instance
    self.handle_setup_error(e)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 420, in handle_setup_error
    raise RuntimeError("%s configuration failed." % self.subsystem)
RuntimeError: CA configuration failed.

2017-01-03T13:25:16Z DEBUG   [error] RuntimeError: CA configuration failed.
2017-01-03T13:25:16Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 171, in execute
    return_value = self.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 318, in run
    cfgr.run()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 310, in run
    self.execute()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 332, in execute
    for nothing in self._executor():
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 372, in __runner
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 394, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 362, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 359, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 586, in _configure
    next(executor)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 372, in __runner
    self._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 449, in _handle_exception
    self.__parent._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 394, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 446, in _handle_exception
    super(ComponentBase, self)._handle_exception(exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 394, in _handle_exception
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 362, in __runner
    step()
  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 359, in <lambda>
    step = lambda: next(self.__gen)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from
    six.reraise(*exc_info)
  File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from
    value = gen.send(prev_value)
  File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 63, in _install
    for nothing in self._installer(self.parent):
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 1357, in main
    install(self)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 267, in decorated
    func(installer)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 773, in install
    ca.install_step_0(False, None, options)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 161, in install_step_0
    ca_signing_algorithm=options.ca_signing_algorithm)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 437, in configure_instance
    self.start_creation(runtime=210)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 448, in start_creation
    run_step(full_msg, method)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 438, in run_step
    method()
  File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", line 590, in __spawn_instance
    DogtagInstance.spawn_instance(self, cfg_file)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 181, in spawn_instance
    self.handle_setup_error(e)
  File "/usr/lib/python2.7/site-packages/ipaserver/install/dogtaginstance.py", line 420, in handle_setup_error
    raise RuntimeError("%s configuration failed." % self.subsystem)

2017-01-03T13:25:16Z DEBUG The ipa-server-install command failed, exception: RuntimeError: CA configuration failed.
2017-01-03T13:25:16Z ERROR CA configuration failed.
2017-01-03T13:25:16Z ERROR The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information

Expected results:

No error.

Additional info:

This issue causes problem especially in containerized environments when we might want that directory bind-mounted or symlinked to some volume.

Filing against ipa even if pki* might be the ultimate component, since we likely want to keep this as dependency for future ipa-server-docker work.

Comment 2 Petr Vobornik 2017-01-13 17:34:19 UTC
Upstream tracking ticket created: https://fedorahosted.org/freeipa/ticket/6612

Moving to pki-core.

Comment 3 Matthew Harmsen 2017-02-09 02:43:56 UTC
Upstream ticket:
https://fedorahosted.org/pki/ticket/2589

Comment 5 Matthew Harmsen 2017-10-25 21:11:48 UTC
[20171025] - RHEL 7.5 pre-Alpha Offline Triage ==> 7.6

Comment 6 Matthew Harmsen 2018-04-27 01:22:23 UTC
alee: needed for containerization of IPA.  We need to check with Jan Pazdziora if this is still needed. 

setting NEEDINFO jpazdziora

Comment 7 Jan Pazdziora 2018-04-27 06:55:35 UTC
I'm no longer owning the containerization effort, moving needinfo to Petr.

Comment 11 Endi Sukma Dewata 2018-09-27 18:24:55 UTC
Petr,

This is currently the correct behavior of PKI. During installation and runtime PKI will need to be able to write into the filesystem including /etc/pki/pki-tomcat, so if that folder already exists the installation will fail to prevent it from overwriting existing instance.

Containerization is not currently supported by PKI, and installation/running with read-only filesystem (regardless of containerization) is also not supported either, so this would be an RFE. Could someone provide a design document explaining which files/folders are expected to be present already, and how PKI should use them?

Thanks.

Comment 14 Endi Sukma Dewata 2018-10-18 17:31:41 UTC
Alexander, would you be able to provide the requirements or the design document for the new expected behavior?

Without the new design document, we would have to close this bug as NOTABUG since it works as designed.

Comment 16 Jan Pazdziora 2018-10-19 07:01:13 UTC
The software shouldn't actively prevent situations when the user/admin needs to have the configuration and data on different volume, and symlinks to the final location. We have such a situation in containers because we need to separate the software (image) from the data.

This is maintainability issue. If the FreeIPA team is asked to move forward with the containerization effort for IdM to get it from its current tech-preview limbo, without the fix the workaround will need to be carried forward indefinitely.

Please note the I'm not the right person to push for this fix -- this should really be coming from the FreeIPA/IdM team.

Comment 17 Rob Crittenden 2018-10-19 12:10:49 UTC
It seems like you have a test to see if the instance is already configured. I think the ask is to change what you are looking for from that directory to perhaps a file, say CS.cfg.

Comment 18 Endi Sukma Dewata 2018-10-19 13:42:56 UTC
I don't think that will solve the problem since the installation will still fail. Based on Jan's description, the idea is to mount the /etc/pki/pki-tomcat from another volume, so it will already contain files from an existing PKI instance including the CS.cfg.

Comment 19 Rob Crittenden 2018-10-19 13:47:06 UTC
I don't believe that is the problem he is describing. The way I read it was that if the installer sees the instance directory as existing it blows up. He is trying to stand up a new IPA master in a container but parts of it (the data) are bind mounted into the container so some directories will pre-exist.

Comment 20 Jan Pazdziora 2018-10-19 13:51:10 UTC
Right. In the containerized setup, /etc/pki/pki-tomcat is a symlink to /data/etc/pki/pki-tomcat (where /data is a separate volume), and that directory /data/etc/pki/pki-tomcat is initially empty and later populated by ipa-server-install process.

Comment 21 Endi Sukma Dewata 2018-10-19 14:17:54 UTC
This is the level of details that I was looking for. If someone could put this into a requirement/design document that will be great (i.e. what folders will be mounted, what folders will be read-only, etc.). That way we can evaluate what changes we will need to make as a whole to make sure the whole design is valid instead of just fixing one failure at a time as we find it.

I'm not sure if it's possible to change the bug reporter to someone else, so maybe we should close this ticket and have someone in IPA team to create a new ticket with a link to the design document.

Thanks!

Comment 22 Rob Crittenden 2018-10-19 14:37:38 UTC
By design do you mean the requirements we have? (I use "we" here loosely because I don't know the container implementation well).

I think it boils down to loosening the restriction on pre-existing directories. Given it is a symlink I think once CS starts writing files, etc then it won't distinguish things. I suppose there could be SELinux issues but we won't know until it gets that far I think.

As I read this the design/RFE is: don't fail in the installer if /etc/pki/pki-tomcat exists.

If there is some straghtforward way to disable/modify this check which would let the installation continue so we could see if there are additional problems I'm sure we'd be happy to give it a try.

Comment 23 Jan Pazdziora 2018-10-19 14:54:59 UTC
(In reply to Endi Sukma Dewata from comment #21)
> (i.e. what
> folders will be mounted, what folders will be read-only, etc.).

Actually, it might be best to turn this around. Assuming the admin (or person containerizing the setup) would like to have dogtag running on system which would be as read-only as possible -- what directories and/or files would dogtag absolutely need to have write access for to keep working, and what locations can safely be expected read-only? Is there a difference between the initial "setup" stage, normal runtime stage, and upgrade stage?

Note that I talk about dogtag here, not FreeIPA/IdM. When initially developing the freeipa-container solution, we relied on docker diff to "discover" files and directories that are written to during various stages listed above. It might be useful from long term point of view to have the expectations of the FreeIPA components like dogtag spelled out.

Comment 24 Endi Sukma Dewata 2018-10-19 15:53:23 UTC
Per current design, pkispawn will create a new /etc/pki/<instance> folder for each new instance. If the folder already exists, it's assumed that the instance already exists, so pkispawn should fail. We may have other tools that rely on this assumption too.

The exception was triggered by this copy() method:
https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/deployment/pkihelper.py#L1453

I'm guessing that it actually happened here:
https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/deployment/scriptlets/instance_layout.py#L54

If you run pkispawn in verbose mode you should get a stack trace that will show the exact location so you can try bypassing the code.

pkispawn only creates the folders/files that it needs to install and run PKI server subsequently. That's basically "as read-only as possible" from PKI's perspective, and docker diff probably can provide the most accurate information. During runtime PKI server can only access the files/folders that were already set up during installation based on the permissions. During upgrade things might change, but there's no way to tell what will change in the future.

From containerization's perspective, "as read-only as possible" might have a different meaning. If we only use standard mounting configuration, pkispawn would work just fine (we are in fact running PKI CI tests in docker containers). If there's a non-standard mounting configuration, we need to know what they are so we can evaluate whether it's possible to accommodate that.

Comment 25 Jan Pazdziora 2018-10-22 06:56:51 UTC
(In reply to Endi Sukma Dewata from comment #24)
> Per current design, pkispawn will create a new /etc/pki/<instance> folder
> for each new instance. If the folder already exists, it's assumed that the

[...]

> pkispawn only creates the folders/files that it needs to install and run PKI
> server subsequently. That's basically "as read-only as possible" from PKI's
> perspective,

It is not because it forces you to have writable /etc/pki, instead of making it possible to have /etc/pki read-only and only (pre)create the /etc/pki/<instance> writable for the instance you are setting up.

Comment 26 Endi Sukma Dewata 2018-10-22 14:04:17 UTC
PKI was not designed for a read-only /etc/pki folder. There are some other files that will be created in that folder too (e.g. pki.conf) during package install and upgrade. There are other folders as well (e.g. /etc/systemd/system) where pkispawn will create new files/folders in them.

Are you proposing to move all files/folders created by pkispawn into PKI package-owned subfolders (e.g. /etc/pki/pki-server/<instance>)? That will be a major redesign since many code/docs/tests will have to be updated.

Comment 27 Rob Crittenden 2018-10-22 15:26:15 UTC
Not exactly scientific but in my /etc/pki directories (across a few releases) nothing in the /etc/pki directory is written to beyond the instance dir. A typical IdM installation doesn't need to tweak the CS configuration via pki.conf.

I think the ask is just: relax or change the current existence detection so the instance-directory without contents can pre-exist.

Comment 28 Endi Sukma Dewata 2018-10-22 15:35:00 UTC
It's still a change of behavior. Normally during uninstall pkidestroy would remove /etc/pki/<instance>. How should it work now?

Also, how about other folders such as /etc/systemd/system, would that be read-only as well?

Comment 29 Jan Pazdziora 2018-10-29 10:17:52 UTC
(In reply to Endi Sukma Dewata from comment #28)
> It's still a change of behavior. Normally during uninstall pkidestroy would
> remove /etc/pki/<instance>. How should it work now?

From containerization point of view, I have no opinion one way or another because we don't expect uninstall to be ever done in containers -- people just drop the container image and data without running any extra process.

> Also, how about other folders such as /etc/systemd/system, would that be
> read-only as well?

The ipa-server-install runs a series of "systemctl enable" operations that end up  modifying /etc/systemd/system content so in the containerized setup, that directory is symlink to writable directory. Does dogtag setup ever write to that directory directly (and assume it's not a symlink, for example) or does it always invoke systemctl to do so?

Comment 30 Endi Sukma Dewata 2018-10-29 18:35:54 UTC
I don't know the specifics without deeper investigation, but if /etc/systemd/system can be linked to a writable directory, why not do the same with /etc/pki? Why not do that to all folders used by PKI?

It looks like there is already some kind of "design/requirements" which include the following:
* no uninstall to be done in containers
* /etc/systemd/system to be linked to a writable directory

Is this documented somewhere?

What other folders are planned to be converted into links like above?

Why is it important to make most directories read-only? Is it to minimize the storage requirement? Or is it to speed up package installation? Or is it to support instance recovery?

Will pki package-owned files be part of the read-only or writable folders? How will upgrades be executed? Will it run with the writable folders mounted?

There are so many questions. We need to review the overall requirements first, then design the overall solution instead of making piecemeal changes.

Comment 31 Rob Crittenden 2018-10-29 19:08:07 UTC
The requirement is that CS be containerized.

What you are asking for are all natural outputs of that process.

Jan has managed to shoehorn CS into the IPA container through sheer willpower and reverse engineering. It would be far more straightforward if CS natively supported containers otherwise it is just peeking through holes in the black box.

Comment 32 Endi Sukma Dewata 2018-10-29 20:45:22 UTC
The "containerization" requirement is too generic that it does not translate into technical design. Does it simply mean running in a container?

As mentioned earlier, PKI can already run in a container, but that is apparently not enough. There are additional requirements such as some folders must be read-only.

If this is a "true" containerization, PKI would be running on a separate container from IPA. Is that what we want? If not, what kind of "containerization" do we want?

Let me simplify my questions:

1. Could you define containerization?
2. Could you describe the use cases?
3. Could you provide the test scenarios?

Comment 33 Endi Sukma Dewata 2018-11-01 16:41:10 UTC
Another idea, we may be able to support this using pkispawn --force option. We don't have a --force option yet, but once it's implemented, if the --force option is specified it will overwrite the existing installation. So if a folder already exists, it will reuse the folder. Is this an acceptable solution? This has nothing to do with containerization. It's simply an option to overwrite existing installation.

Comment 34 Jan Pazdziora 2018-11-02 08:04:22 UTC
(In reply to Endi Sukma Dewata from comment #32)
> The "containerization" requirement is too generic that it does not translate
> into technical design. Does it simply mean running in a container?
> 
> As mentioned earlier, PKI can already run in a container, but that is
> apparently not enough. There are additional requirements such as some
> folders must be read-only.

No. The whole container must be read-only, except for minimal set of directories for config and data that would be either mounted into the container, or would be symlink to some mounted volume.

IOW, the requirement is to be able to run PKI / FreeIPA using docker run --read-only.

> If this is a "true" containerization, PKI would be running on a separate
> container from IPA. Is that what we want? If not, what kind of
> "containerization" do we want?

I don't care. I don't think FreeIPA supports *setting up* and running PKI on different machine (which is basically a prerequisite to support PKI in separate container) though.

> 1. Could you define containerization?

Being able to run the docker run commands from

   https://github.com/freeipa/freeipa-container

but with --read-only added.

> 2. Could you describe the use cases?

Have the FreeIPA setup (meaning its config and data) including CA stored in single directory which can be easily "passed" to the container when running it, for example via docker run's -v option.

> 3. Could you provide the test scenarios?

Travis configuration in the https://github.com/freeipa/freeipa-container repository.

Comment 36 Jan Pazdziora 2018-11-02 08:06:48 UTC
(In reply to Endi Sukma Dewata from comment #33)
> Another idea, we may be able to support this using pkispawn --force option.
> We don't have a --force option yet, but once it's implemented, if the
> --force option is specified it will overwrite the existing installation. So
> if a folder already exists, it will reuse the folder. Is this an acceptable
> solution? This has nothing to do with containerization. It's simply an
> option to overwrite existing installation.

If the net effect will be that the installer will not complain that the directory already exists, that should work.

Comment 37 Endi Sukma Dewata 2018-11-05 21:42:40 UTC
Thanks for the info. Unfortunately, the containerization proposal above is very IPA-specific and still does not provide enough details about what's involved without having to dig deep into IPA code. Also, the functionality can only be tested in IPA environment, which makes it difficult for PKI development/testing.

So I think we'll go ahead with the --force option for pkispawn which should work in any environment. Basically it will overwrite existing installation (and reuse existing folders if possible). It can be tested as follows:
1. Install CA
2. Modify various things in CA
3. Install CA with --force option

Expected result: Step 3 should complete successfully producing a new CA and the changes in step 2 should be overwritten.

Comment 38 Jan Pazdziora 2018-11-06 07:16:18 UTC
Well, specifically, what is needed is behaviour in steps:

1) mkdir -p /etc/pki/pki-tomcat
2) Install CA with --force option

and 2) should complete successfully.

Comment 39 Jan Pazdziora 2018-11-06 07:17:44 UTC
Resp., even more specifically:

1) mkdir -p /data/etc/pki/pki-tomcat
2) ln -s /data/etc/pki/pki-tomcat /etc/pki/pki-tomcat
3) Install CA with --force option

and that 3) must pass and the /etc/pki/pki-tomcat much be unchanged, and everything stored under /data/etc/pki/pki-tomcat.

Comment 41 Christian Heimes 2018-12-12 13:56:42 UTC
You may not need to add a --force option at all. IMO it would be sensible to lift the restriction slightly to also acccept an empty directory as target. Replace the condition "if os.path.exists(directory): fail()" with "if os.path.exists(directory) and os.listdir(directory): fail()".