Bug 1481949

Created attachment 1313960 [details]
pki-destroy log

From triage:

The error happens in sslget - PKI tools and pkidestroy, moving to pki.

[20171025] - RHEL 7.5 / RHCS 9.3 pre-Alpha Offline Triage ==> 7.6

alee: this is old.  Is it still an issue?

I am still able to see the issue with the latest distro and the rpms mentioned below.

[root@master log]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.5 (Maipo)

ipa-server-4.5.4-10.el7_5.1.x86_64
389-ds-base-1.3.7.5-21.el7_5.x86_64
sssd-1.16.0-19.el7_5.1.x86_64
pki-server-10.5.1-11.el7.noarch
pki-ca-10.5.1-11.el7.noarch
krb5-server-1.15.1-19.el7.x86_64

---------------------
2018-05-03T13:19:53Z DEBUG stderr=
2018-05-03T13:19:59Z DEBUG Starting external process
2018-05-03T13:19:59Z DEBUG args=/bin/systemctl stop certmonger.service
2018-05-03T13:19:59Z DEBUG Process finished, return code=0
2018-05-03T13:19:59Z DEBUG stdout=
2018-05-03T13:19:59Z DEBUG stderr=
2018-05-03T13:19:59Z DEBUG Loading StateFile from '/var/lib/ipa/sysrestore/sysrestore.state'
2018-05-03T13:19:59Z DEBUG Unconfiguring CA
2018-05-03T13:19:59Z DEBUG Starting external process
2018-05-03T13:19:59Z DEBUG args=/usr/sbin/pkidestroy -i pki-tomcat -s CA
2018-05-03T13:20:00Z DEBUG Process finished, return code=0
2018-05-03T13:20:00Z DEBUG stdout=Log file: /var/log/pki/pki-ca-destroy.20180503185000.log
Loading deployment configuration from /var/lib/pki/pki-tomcat/ca/registry/ca/deployment.cfg.
WARNING: The 'pki_ssl_server_nickname' in [CA] has been deprecated. Use 'pki_sslserver_nickname' instead.
WARNING: The 'pki_ssl_server_subject_dn' in [CA] has been deprecated. Use 'pki_sslserver_subject_dn' instead.
Uninstalling CA from /var/lib/pki/pki-tomcat.

Uninstallation complete.

2018-05-03T13:20:00Z DEBUG stderr=pkidestroy  : WARNING  ....... this 'CA' entry will NOT be deleted from security domain 'IPA'!
pkidestroy  : WARNING  ....... security domain 'IPA' may be offline or unreachable!
pkidestroy  : ERROR    ....... subprocess.CalledProcessError:  Command '['/usr/bin/sslget', '-n', 'subsystemCert cert-pki-ca', '-p', '1Vu,Dr9Q!;oN5<x_fxq*QinokS{PTHZ]((:>1Hd!:', '-d', '/etc/pki/pki-tomcat/alias', '-e', 'name="/var/lib/pki/pki-tomcat"&type=CA&list=caList&host=master.testrelm.test&sport=443&ncsport=443&adminsport=443&agentsport=443&operation=remove', '-v', '-r', '/ca/agent/ca/updateDomainXML', 'master.testrelm.test:443']' returned non-zero exit status 6!
-----------------

Moved to RHEL 7.7.

Could I get some more information about how to reproduce this?


I started with these packages on F32:

 -  389-ds-base-1.4.3.15-1.fc32.x86_64
 -  freeipa-server-4.8.10-6.fc32.x86_64
 -  pki-base-10.10.0-2.fc32.noarch

I installed using this command:

# ipa-server-install -U -r "{{ realm|upper }}" -n "{{ domain }}" -p "{{ password }}" -a "{{ password }}" "--hostname={{ hostname }}.{{ domain }}" --setup-kra --setup-dns --auto-reverse "--forwarder={{ forwarder }}"

And then uninstalled:

# ipa-server-install --uninstall -U -v

And it appears to have worked.



Do I need any replicas or anything?

this report is missing the scenario details, what was the IPA topology, and the IPA PKI services.

an error like this can happen:
when the replica is a subordinate CA to some other Dogtag CA, within the same security domain, and the issuer of that replica with a CA or KRA is offline, so the security domain is not available, and the PKI registry cannot be updated,
or if /etc/httpd/conf.d/ipa-pki-proxy.conf is missing the location match for /ca/agent/ca/updateDomainXML
or if the TCP port 443 is incorrect
so the final destination on TCP 8443 cannot be made to reach the CA services.


pkidestroy could be more robust:

pkidestroy provided 2 warning messages but sslget had a serious network connection error that prevented the security domain update:
"
pkidestroy  : WARNING  ....... this 'CA' entry will NOT be deleted from security domain 'IPA'!
pkidestroy  : WARNING  ....... security domain 'IPA' may be offline or unreachable!
"

the remote socket connection failed in 
./base/native-tools/src/sslget/sslget.c
do_connect(
...
    prStatus = PR_Connect(tcp_sock, addr, PR_SecondsToInterval(3));
    if (prStatus != PR_SUCCESS) {
        errWarn("PR_Connect");
        if( tcp_sock != NULL ) {
            PR_Close(tcp_sock);
            tcp_sock = NULL;
        }
        exit(6);
    }
...

the NSPR call PR_Connect returns either PR_SUCCESS or PR_FAILURE.
sslget error code 6 is like PR_FAILURE.

should pkidestroy exit before un-installing a subsystem when there is no PR_SUCCESS from the sslget to access the security domain?

it seem like
./base/server/python/pki/server/deployment/pkihelper.py
    def deregister(self, install_token, critical_failure=False):

goes into

        else:
            output = self.update_domain_using_agent_port(
                typeval, secname, params, update_url, sechost, secagentport,
                critical_failure)

        if not output:
            if critical_failure:
                raise Exception("Cannot update domain using agent port")
            else:
                return


to call update_domain_using_agent_port()
which had the sslget error 6, and sends the 2 warnings

        except subprocess.CalledProcessError as exc:
            config.pki_log.warning(
                log.PKIHELPER_SECURITY_DOMAIN_UPDATE_FAILURE_2,
                typeval,
                secname,
                extra=config.PKI_INDENTATION_LEVEL_2)
            config.pki_log.warning(
                log.PKIHELPER_SECURITY_DOMAIN_UNREACHABLE_1,
                secname,
                extra=config.PKI_INDENTATION_LEVEL_2)
            config.pki_log.error(log.PKI_SUBPROCESS_ERROR_1, exc,
                                 extra=config.PKI_INDENTATION_LEVEL_2)
            if critical_failure:
                raise

        return None 

and then returns nothing to deregister, which returns 0 to pkidestroy, which returns 0 to ipa-server-install --uninstall

so the failed TCP connection with the sslget error code 6 was not really fully processed.

pkidestroy should be able to handle the non 0 error codes like 6, and not run all the scriptlets to un-install a sub system, until the PKI registry / security domain info is available, should exit early in /usr/lib/python3.6/site-packages/pki/server/deployment/scriptlets/initialization.py

that may lead to some partial / incomplete PKI susbsystem removal that may not be removed at the next attempt.

and "ipa-server-install --uninstall" should first verify the PKI registry is available if there is a PKI subsystem installed, before proceeding with an un-install.

But server uninstall won't fail anyways: the failure is treated as informational by PKI: 

https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/deployment/pkihelper.py#L2347

critical_failure is set to False by default; no callers override it. 

That means we should log the exception and continue on:

https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/deployment/pkihelper.py#L2414-L2415
https://github.com/dogtagpki/pki/blob/master/base/server/python/pki/server/deployment/scriptlets/initialization.py#L202-L217


AFAICT, this behavior has existed since the bug was originally filed. Indeed, the original bug just says "fix this error" -- but yet, it never _fails_ -- failure is logged so the admin can take appropriate action as required, but server uninstallation still succeeds.

This is completely a non-issue.

And, if force removal is what is desired, then IPA could pass --force to pkidestroy. Perhaps that's the best option?



My 2c, but this should be CLOSED -> NOTABUG.

Note that tools to manage security domains have existed for a while. See bz#1740697 about an RFE if there are any changes necessary, but as I read it, if IPA gets out of sync from PKI, there's nothing more we can do (at uninstallation time) --- we already tried to contact SD and it failed. We have had the tools to fix this for a while. These tools should be used in the appropriate cases, or if IPA doesn't like direct usage of PKI tools, they should wrap them as appropriate.


I don't think it is appropriate to stop removal over failure to contact SD. For one, we have no way of indicating to ipa-server-install --uninstall (prior to execution of pkidestory) that destroy might fail due to SD removal. If it does fail, we leave the customer with a broken/partial uninstall. For two, this is really up to IPA to decide (whether failure to remove from SD should be fatal) and if so, IPA should attempt to contact SD owner prior to removal. If they need help in this regard, we're happy to help.

We can't make the call on how fatal not updating the SD is. The CA is a black box. If we need to do something prior to uninstall then we'll do it.

IPA could call pki securitydomain-host-show prior to beginning the uninstall, for example, and fail if the connection fails and abort the entire uninstall.

Summarizing comments on #ipa...

Security Domain will never be contactable in an IPA scenario. This is because IPA shuts down all services prior to initiating uninstallation. Because IPA provisions the security domain with port 443 for access and turns off httpd prior to calling pkidestroy, pkidestroy will never be able to contact the security domain.

When ipa-server-install is called, and then pkidestroy is run manually, this failure doesn't manifest.

Thus re-assigning to IPA. PKI should provide a list of services which should still be enabled at the time it is shut down. Probably these include:

 - pki-tomcat
 - Directory server instance
 - httpd (for port 443 as configured in the SD config).


See https://github.com/freeipa/freeipa/blob/master/ipaserver/install/dogtaginstance.py#L820 for sd port configuration. 


So, SD likely hasn't ever been removed from IPA deploys and they get badly out of sync.

Christina, Jack, what's the consequence of this?

Alex, I agree with your assessment in https://bugzilla.redhat.com/show_bug.cgi?id=1740697#c6
Although I have not tried pki securitydomain-host myself, but if it works as advertised, and if it's an acceptable operation by IPA to manually fix the SD content, it might be sufficient for such corner cases.

The consequences of things out of sync in SD would be if one wishes to intall another instance in the SD, incorrect info would be provided.  E.g. an KRA that's already uninstalled would appear to be selectable.  Although I'm uncertain at what point the error would be discovered.

*** This bug has been marked as a duplicate of bug 1902173 ***