RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1845447 - pkispawn fails in FIPS mode: AJP connector has secretRequired="true" but no secret
Summary: pkispawn fails in FIPS mode: AJP connector has secretRequired="true" but no s...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: pki-core
Version: 8.3
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 8.3
Assignee: Alex Scheel
QA Contact: PKI QE
URL:
Whiteboard:
: 1845409 (view as bug list)
Depends On:
Blocks: 1793411 1842946 1849146 1849914
TreeView+ depends on / blocked
 
Reported: 2020-06-09 09:17 UTC by Christian Heimes
Modified: 2020-11-04 03:16 UTC (History)
9 users (show)

Fixed In Version: pki-core-10.6-8030020200625230535.5ff1562f
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1849146 (view as bug list)
Environment:
Last Closed: 2020-11-04 03:15:45 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Christian Heimes 2020-06-09 09:17:10 UTC
Description of problem:
IdM installation is failing in pkispawn call on FIPS enabled systems.


Version-Release number of selected component (if applicable):
ipa-server-4.8.6-2.module+el8.3.0+6878+7463afe6.x86_64
pki-servlet-engine-9.0.30-1.module+el8.3.0+6730+8f9c6254.noarch
pki-ca-10.9.0-0.1.module+el8.3.0+6811+1fa59e71.noarch
# cat /proc/sys/crypto/fips_enabled 
1

How reproducible:
always

Steps to Reproduce:
1. boot a system into FIPS mode
2. ipa-server-install

Actual results:
...
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/30]: configuring certificate server instance
Failed to configure CA instance: CalledProcessError(Command ['/usr/sbin/pkispawn', '-s', 'CA', '-f', '/tmp/tmpr6vsy6eo'] returned non-zero exit status 1: 'Notice: Trust flag u is set automatically if the private key is present.\nERROR: ConnectionError: (\'Connection aborted.\', OSError(0, \'Error\'))\n  File "/usr/lib/python3.6/site-packages/pki/server/pkispawn.py", line 569, in main\n    scriptlet.spawn(deployer)\n  File "/usr/lib/python3.6/site-packages/pki/server/deployment/scriptlets/configuration.py", line 925, in spawn\n    client.setupDatabase(database_setup_request)\n  File "/usr/lib/python3.6/site-packages/pki/system.py", line 430, in setupDatabase\n    headers)\n  File "/usr/lib/python3.6/site-packages/pki/client.py", line 50, in wrapper\n    return func(self, *args, **kwargs)\n  File "/usr/lib/python3.6/site-packages/pki/client.py", line 219, in post\n    params=params)\n  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 581, in post\n    return self.request(\'POST\', url, data=data, json=json, **kwargs)\n  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request\n    resp = self.send(prep, **send_kwargs)\n  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send\n    r = adapter.send(request, **kwargs)\n  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send\n    raise ConnectionError(err, request=request)\n\n')
See the installation logs and the following files/directories for more information:
  /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.
CA configuration failed.
The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information


Expected results:
No error

Additional info:
journald contains:
---
SEVERE: Failed to start component [Connector[AJP/1.3-8009]]
org.apache.catalina.LifecycleException: Protocol handler start failed
        at org.apache.catalina.connector.Connector.startInternal(Connector.java:1008)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:438)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:633)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:343)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:474)
Caused by: java.lang.IllegalArgumentException: The AJP Connector is configured with secretRequired="true" but the secret attribute is either null or "". This combination is not valid.
        at org.apache.coyote.ajp.AbstractAjpProtocol.start(AbstractAjpProtocol.java:264)
        at org.apache.catalina.connector.Connector.startInternal(Connector.java:1005)
        ... 12 more
---

pki-ca-spawn log just contains:
---
2020-06-09 03:17:10 ERROR: ConnectionError: ('Connection aborted.', OSError(0, 'Error'))
  File "/usr/lib/python3.6/site-packages/pki/server/pkispawn.py", line 569, in main
    scriptlet.spawn(deployer)
  File "/usr/lib/python3.6/site-packages/pki/server/deployment/scriptlets/configuration.py", line 924, in spawn
    client.setupDatabase(database_setup_request)
  File "/usr/lib/python3.6/site-packages/pki/system.py", line 430, in setupDatabase
    headers)
  File "/usr/lib/python3.6/site-packages/pki/client.py", line 50, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/pki/client.py", line 219, in post
    params=params)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
---

Comment 1 Kaleem 2020-06-09 09:20:40 UTC
*** Bug 1845409 has been marked as a duplicate of this bug. ***

Comment 2 Christian Heimes 2020-06-09 09:21:27 UTC
I could not find the string "secretRequired" anywhere in Dogtag or any XML configuration file. I *think* it's coming from the implicit default in the abstract AJP protocol class:

apache-tomcat-9.0.30.redhat-4-src/java/org/apache/coyote/ajp/AbstractAjpProtocol.java:    private boolean secretRequired = true;

The bug may be a regression in pki-servlet-engine.

Comment 3 Christian Heimes 2020-06-09 09:23:21 UTC
PS: /etc/pki/pki-tomcat/server.xml does not set a secret.

Comment 4 Alex Scheel 2020-06-09 12:06:12 UTC
Hey Christian,

My understanding was that IPA was supposed to take care of setting the secret for AJP-based installations. Perhaps this logic needs to be modified to install without AJP and then add the AJP connector afterwards -- including the secret?

I thought y'all had tested this in PR-CI... Perhaps we should switch the component to pki-servlet-engine and then needinfo Coty for any behavior changes between Fedora and RHEL? The difference in specs seems reasonable, other than that Fedora came from the upstream build and PKI came from JBoss's internal build tree. I can't seem to find that line in the upstream sources for 9.0.30...

Comment 5 Alex Scheel 2020-06-09 12:07:27 UTC
Hey Coty -- is the line Christian identified above (and resulting failure) something JWS-specific? Thanks!

Comment 6 Alexander Bokovoy 2020-06-09 12:10:35 UTC
Alex,

we do set up requireSecret but only after basic pkispawn completes. Before that there is no configuration file at all. So pkispawn fails because, it seems, the setting was enforced in tomcat as a default. This means pkispawn should generate own secret explicitly.

Comment 7 Christian Heimes 2020-06-09 12:12:27 UTC
Yes, we are going to take care afterwards. But first pkispawn must succeed. IPA cannot update the setting when Dogtag fails to install (sorry for pointing out the obvious) .

Comment 8 Alex Scheel 2020-06-09 12:38:01 UTC
(In reply to Alexander Bokovoy from comment #6)
> Alex,
> 
> we do set up requireSecret but only after basic pkispawn completes. Before
> that there is no configuration file at all. So pkispawn fails because, it
> seems, the setting was enforced in tomcat as a default. This means pkispawn
> should generate own secret explicitly.

Sorry Alexander, let me clarify:

I know you don't generate secret during pkispawn.

I know you don't specify AJP parameters during installation, but modify it afterwards.


My suggestion was:

 - To your configuration overrides passed to pkispawn (CA.cfg or whatever), add the line:
       pki_enable_proxy = false
   this will comment-out the AJP connector during installation:
       https://github.com/dogtagpki/pki/blob/master/base/server/tomcat-8.5/conf/server.xml#L191-L197
   (the rest of the values will be propagated as they currently are):
       https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/deployment/pkiparser.py#L754-L777
 - Run pkispawn
 - When there is no AJP connector, first look for a commented-out one:
   https://github.com/freeipa/freeipa/blob/master/ipaserver/install/dogtaginstance.py#L310-L313
 - Then set secret and restart CA.


My 2c, but this would ensure we always get a secured AJP connector and don't have to deal with a temporarily-insecure AJP connector. 

We could confirm with Endi that this approach would work if you want to take it.

Comment 10 Coty Sutherland 2020-06-09 15:47:36 UTC
(In reply to Alex Scheel from comment #5)
> Hey Coty -- is the line Christian identified above (and resulting failure)
> something JWS-specific? Thanks!

Hi Alex, I guess you could say that this functionality is JWS specific, but only because it's the result of me not modifying the behaviour of Tomcat like I did for Fedora and RHEL distros (or JWS 3.1, maintenance stream). We didn't want to introduce a change to defaults in a zstream or maintenance phase of any product so a patch was introduced to reset the default to it's previous value (false), to reset the default bind address from lo to 0.0.0.0, and a couple other things I can't recall at the moment. This was done in the Fedora package (except rawhide, it uses the new defaults and disables AJP by default) and RHEL 6 & 7 so that things would not break after updates. JWS 5.3 however did not revert the upstream defaults and therefore breaks the AJP connector if users don't modify their configurations accordingly.

For a fix, I would say that we can easily modify the default server.xml in the pki-servlet-engine package because you're the only ones using it :) However, JWS 5.3 doesn't have the AJP connector enabled by default (therefore neither does the pki-servlet-engine package), so it's being configured somewhere in your installation it seems. I think that's the place that needs updating, not the pki-servlet-engine package itself.

Comment 12 Alex Scheel 2020-06-09 20:54:01 UTC
\o Hey Kaleem,

Workaround for the time being until we get this fixed in isnap3:

Modify /usr/share/pki/server/conf/server.xml:

It should have contents like:

    <!-- Define an AJP 1.3 Connector on port [PKI_AJP_PORT] -->
[PKI_OPEN_AJP_PORT_COMMENT]
    <Connector port="[PKI_AJP_PORT]"
               protocol="AJP/1.3"
               redirectPort="[PKI_AJP_REDIRECT_PORT]"
               address="[PKI_AJP_HOST]" />
[PKI_CLOSE_AJP_PORT_COMMENT]


I _believe_ it should be sufficient to add an attribute for requiredSecret or secret here (depending on tomcat version used -- 9.0.30 or older for requiredSecret, 9.0.31+ for secret):

    <!-- Define an AJP 1.3 Connector on port [PKI_AJP_PORT] -->
[PKI_OPEN_AJP_PORT_COMMENT]
    <Connector port="[PKI_AJP_PORT]"
               protocol="AJP/1.3"
               redirectPort="[PKI_AJP_REDIRECT_PORT]"
               address="[PKI_AJP_HOST]" 
               requiredSecret="Secret.123" />
[PKI_CLOSE_AJP_PORT_COMMENT]


Untested, but this _should_ allow ipa-server-install to pick up this testing secret ("Secret.123") and put it in httpd config and let pkispawn succeed by providing a dummy secret.


HTH,

- Alex

Comment 13 Alex Scheel 2020-06-15 15:53:25 UTC
Checked-in upstream:

commit e2ee6e1e6a08cb128a7ce04f04961325eff96c0d
Author: Alexander Scheel <ascheel>
Date:   Tue Jun 9 13:51:21 2020 -0400

    Make pki_ajp_secret a random password by default
    
    Signed-off-by: Alexander Scheel <ascheel>

commit 1bd84062c95797572de2464c095f5ab4f55e03c7
Author: Alexander Scheel <ascheel>
Date:   Tue Jun 9 18:21:13 2020 -0400

    Add migration logic for 8.5 -> 9.0.31
    
    Signed-off-by: Alexander Scheel <ascheel>

commit 76820004f231bcc4291cb63699a89746bd78c8e7
Author: Alexander Scheel <ascheel>
Date:   Tue Jun 9 12:21:24 2020 -0400

    Introduce pki_ajp_secret configuration parameter
    
    Signed-off-by: Alexander Scheel <ascheel>


This was present in Fedora -a2 builds.

Comment 21 errata-xmlrpc 2020-11-04 03:15:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: pki-core:10.6 and pki-deps:10.6 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4847


Note You need to log in before you can comment on or make changes to this bug.