Bug 1498957

Summary:	pkidestroy does not work with nuxwdog
Product:	Red Hat Enterprise Linux 7	Reporter:	Roshni <rpattath>
Component:	pki-core	Assignee:	Jack Magne <jmagne>
Status:	CLOSED ERRATA	QA Contact:	Asha Akkiangady <aakkiang>
Severity:	urgent	Docs Contact:	Marc Muehlfeld <mmuehlfe>
Priority:	urgent
Version:	7.4	CC:	alee, jmagne, lmiksik, mharmsen, nkinder
Target Milestone:	rc	Keywords:	TestBlocker
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	pki-core-10.5.1-6.el7	Doc Type:	Bug Fix
Doc Text:	The pkidestroy utility now fully removes instances that are started by the pki-tomcatd-nuxwdog service Previously, the pkidestroy utility did not remove Certificate System instances that used the pki-tomcatd-nuxwdog service as a starting mechanism. As a consequence, administrators had to migrate pki-tomcatd-nuxwdog to the service without watchdog before using pkidestroy to fully remove an instance. The utility has been updated, and instances are correctly removed in the mentioned scenario. Note that if you manually removed the password file before running pkidestroy, the utility will ask for the password to update the security domain.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-04-10 17:01:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Roshni 2017-10-05 15:56:47 UTC

Description of problem:
pkidestroy does not work with nuxwdog

Version-Release number of selected component (if applicable):
pki-ca-10.4.1-15.el7_4.noarch

How reproducible:
always

Steps to Reproduce:
1. pkispawn CA
2. Enable nuxwdog as follows
cms.tokenList=<TOKEN_NAME>

# pki-server nuxwdog-enable
---------------------------
Nuxwdog enabled for system.

systemctl start pki-tomcatd-nuxwdog@<pki-ca>.service

3. pkidestroy -s CA -i <pki-ca>

Actual results:
pkidestroy is successful but seeign the following

[root@nocp1 ~]# ps -aef | grep pki-ca-
root      2689 28144  0 10:51 pts/0    00:00:00 grep --color=auto pki-ca-
dirsrv   17917     1  0 Oct04 ?        00:03:12 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-pki-ca-Oct5-LDAP -i /var/run/dirsrv/slapd-pki-ca-Oct5-LDAP.pid
root     18391     1  0 Oct04 ?        00:00:00 /bin/nuxwdog -f /etc/pki/pki-ca-Oct5/nuxwdog.conf
root     18392 18391  0 Oct04 ?        00:01:31 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -DRESTEASY_LIB=/usr/share/java/resteasy-base -Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/var/lib/pki/pki-ca-Oct5 -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-ca-Oct5/temp -Djava.util.logging.config.file=/var/lib/pki/pki-ca-Oct5/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.security.manager -Djava.security.policy==/var/lib/pki/pki-ca-Oct5/conf/catalina.policy org.apache.catalina.startup.Bootstrap start


Expected results:
pkidestroy should kill the related processes and cleanup the security domain

Additional info:

<alee> looks like there are two problems
<alee> 1. pkidestroy has not been modiied to get the password for the hsm in case the password.conf is not present
<alee> and so the entry is not removed from the security doamain
<alee> 2. its trying to stop pki-tomcatd  instead of pki-tomcatd-nuxwdog

<edewata> alee: I think this is also a problem: 3. pkidestroy ignores missing hsm password and just keeps going

Comment 6 Ade Lee 2018-01-03 19:08:19 UTC

commit 6716b82ecc38b23de81c8f0fe18863e1df4bfddb
Author: Ade Lee <alee>
Date:   Tue Jan 2 14:52:32 2018 -0500

    Allow prompting for token passwords if not present
    
    Change-Id: Ifa2e60424d713ebe15bf9aa92f1d5b7691b7e0ff

commit c7c907c07599ef1d9b52638c25153f7bd82de999
Author: Ade Lee <alee>
Date:   Tue Jan 2 13:38:40 2018 -0500

    Modified systemd invocations in pkispawn to handle nuxwdog
    
    The systemd invocations in pkispawn/pkidestroy did not account for
    nuxwdog enabled instances.  This patch allows pkispawn/pkidestroy to
    use the right service name if the nuxwdog service unit files exist.
    
    Also modified instance_layout deployment script to delete the right
    systemd link.
    
    Change-Id: I25eac0555aad022784d7728913ae4a335eab3463

commit e9b5fc7ef000abfd2cbdd6be6bfd4b2d015816a2
Author: Ade Lee <alee>
Date:   Tue Jan 2 13:24:23 2018 -0500

    Fix various PEP8 and pylint issues
    
    Change-Id: I8b2b52599ab6b2d4738b748f36598319f11477c7

Comment 8 Ade Lee 2018-01-26 16:03:29 UTC

Verification Step:

1. Create an instance.
2. Set it to be managed using nuxwdog (pki-server instance-nuxwodg-enable <instance_name>
3. Remove the password.conf file.
3. pkidestroy the instance/subsystem.  Instance should be removed correctly.  You should be prompted for the password of the instance where the subsystem cert is located.

You should try this for CA, KRA etc.

Comment 9 Roshni 2018-02-06 20:03:05 UTC

Using pki-ca-10.5.1-6.1.el7.noarch

The following are a few results I noticed, the password was prompted twice in both these cases:

[root@nocp1 ~]# pkidestroy -s TPS -i pki-tps-rpattath-Jan31
Log file: /var/log/pki/pki-tps-destroy.20180205142457.log
Loading deployment configuration from /var/lib/pki/pki-tps-rpattath-Jan31/tps/registry/tps/deployment.cfg.
Uninstalling TPS from /var/lib/pki/pki-tps-rpattath-Jan31.
Password for token NHSM-RPATTATH-SOFTCARD: 
pkidestroy  : WARNING  ....... Failed to update TPS connector for nocp1.idm.lab.eng.rdu2.redhat.com:23443
pkidestroy  : ERROR    ....... subprocess.CalledProcessError:  Command '['/bin/pki', '-p', '23443', '-h', 'nocp1.idm.lab.eng.rdu2.redhat.com', '-n', 'NHSM-RPATTATH-SOFTCARD:subsystemCert cert-pki-tps-rpattath-Jan31', '-P', 'https', '-d', '/etc/pki/pki-tps-rpattath-Jan31/alias', '-c', 'XXXXX', '-t', 'tks', 'tks-tpsconnector-del', '--host', 'nocp1.idm.lab.eng.rdu2.redhat.com', '--port', '25443']' returned non-zero exit status 255!
Password for token NHSM-RPATTATH-SOFTCARD: 
pkidestroy  : WARNING  ....... this 'TPS' entry will NOT be deleted from security domain 'Example-rhcs92-CA'!
pkidestroy  : WARNING  ....... security domain 'Example-rhcs92-CA' may be offline or unreachable!
pkidestroy  : ERROR    ....... subprocess.CalledProcessError:  Command '['/usr/bin/sslget', '-n', 'NHSM-RPATTATH-SOFTCARD:subsystemCert cert-pki-tps-rpattath-Jan31', '-p', 'XXXXX', '-d', '/etc/pki/pki-tps-rpattath-Jan31/alias', '-e', 'name="/var/lib/pki/pki-tps-rpattath-Jan31"&type=TPS&list=tpsList&host=nocp1.idm.lab.eng.rdu2.redhat.com&sport=25443&ncsport=25443&adminsport=25443&agentsport=25443&operation=remove', '-v', '-r', '/ca/agent/ca/updateDomainXML', 'nocp1.idm.lab.eng.rdu2.redhat.com:8443']' returned non-zero exit status 4!
pkidestroy  : WARNING  ....... File '/etc/pki/pki-tps-rpattath-Jan31/password.conf' is either missing or is NOT a regular file!


[root@nocp1 ~]# pkidestroy -s KRA -i pki-kra-rpattath-Jan31
Log file: /var/log/pki/pki-kra-destroy.20180205151038.log
Loading deployment configuration from /var/lib/pki/pki-kra-rpattath-Jan31/kra/registry/kra/deployment.cfg.
WARNING: The 'pki_ssl_server_key_algorithm' in [DEFAULT] has been deprecated. Use 'pki_sslserver_key_algorithm' instead.
WARNING: The 'pki_ssl_server_key_size' in [DEFAULT] has been deprecated. Use 'pki_sslserver_key_size' instead.
WARNING: The 'pki_ssl_server_key_type' in [DEFAULT] has been deprecated. Use 'pki_sslserver_key_type' instead.
WARNING: The 'pki_ssl_server_token' in [DEFAULT] has been deprecated. Use 'pki_sslserver_token' instead.
Uninstalling KRA from /var/lib/pki/pki-kra-rpattath-Jan31.
Password for token NHSM-RPATTATH-SOFTCARD: 
pkidestroy  : WARNING  ....... Failed to deregister KRA connector nocp1.idm.lab.eng.rdu2.redhat.com:31042 from CA nocp1.idm.lab.eng.rdu2.redhat.com:8443
Password for token NHSM-RPATTATH-SOFTCARD: 
pkidestroy  : WARNING  ....... this 'KRA' entry will NOT be deleted from security domain 'Example-rhcs92-CA'!
pkidestroy  : WARNING  ....... security domain 'Example-rhcs92-CA' may be offline or unreachable!
pkidestroy  : ERROR    ....... subprocess.CalledProcessError:  Command '['/usr/bin/sslget', '-n', 'NHSM-RPATTATH-SOFTCARD:subsystemCert cert-pki-kra-rpattath-Jan31', '-p', 'xxxxx', '-d', '/etc/pki/pki-kra-rpattath-Jan31/alias', '-e', 'name="/var/lib/pki/pki-kra-rpattath-Jan31"&type=KRA&list=kraList&host=nocp1.idm.lab.eng.rdu2.redhat.com&sport=31042&ncsport=31042&adminsport=31042&agentsport=31042&operation=remove', '-v', '-r', '/ca/agent/ca/updateDomainXML', 'nocp1.idm.lab.eng.rdu2.redhat.com:8443']' returned non-zero exit status 4!
pkidestroy  : WARNING  ....... File '/etc/pki/pki-kra-rpattath-Jan31/password.conf' is either missing or is NOT a regular file!

Uninstallation complete.


I tried running the connector-del cli manually and noticed the following:

Using HSM password for -c option

[root@auto-hv-01-guest05 ~]# pki -p 8443 -h auto-hv-01-guest05.idmqe.lab.eng.bos.redhat.com -n "NHSM6000-OCS:subsystemCert cert-pki-kra-rpattath-Feb2" -P https -d /etc/pki/pki-kra-rpattath-Feb2/alias -c xxxxx -t ca ca-kraconnector-del --host auto-hv-01-guest05.idmqe.lab.eng.bos.redhat.com --port 31042
WARNING: The -t option has been deprecated. Use pki ca command instead.
Error: Incorrect client security database password.

Using sec db password

[root@auto-hv-01-guest05 ~]# pki -p 8443 -h auto-hv-01-guest05.idmqe.lab.eng.bos.redhat.com -n "NHSM6000-OCS:subsystemCert cert-pki-kra-rpattath-Feb2" -P https -d /etc/pki/pki-kra-rpattath-Feb2/alias -c +ID5c5KdOH~P -t ca ca-kraconnector-del --host auto-hv-01-guest05.idmqe.lab.eng.bos.redhat.com --port 31042
WARNING: The -t option has been deprecated. Use pki ca command instead.
Enter password for NHSM6000-OCS

------------------------------------------------------------------------
Removed KRA host "auto-hv-01-guest05.idmqe.lab.eng.bos.redhat.com:31042"
------------------------------------------------------------------------

So there maybe be some more fixes needed for the instances that involve connectors.

Comment 11 Roshni 2018-02-15 19:44:35 UTC

[root@auto-hv-02-guest01 certdb]# rpm -qi pki-ca
Name        : pki-ca
Version     : 10.5.1
Release     : 7.el7
Architecture: noarch
Install Date: Wed 14 Feb 2018 05:50:27 PM EST
Group       : System Environment/Daemons
Size        : 2359899
License     : GPLv2
Signature   : RSA/SHA256, Tue 06 Feb 2018 02:32:49 AM EST, Key ID 199e2f91fd431d51
Source RPM  : pki-core-10.5.1-7.el7.src.rpm
Build Date  : Tue 06 Feb 2018 02:04:51 AM EST
Build Host  : ppc-035.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : http://pki.fedoraproject.org/
Summary     : Certificate System - Certificate Authority

pkidestroy of instances which has nuxwdog enabled are successful except that there is a connector delete issue for TPS and KRA https://bugzilla.redhat.com/show_bug.cgi?id=1545902

Comment 15 errata-xmlrpc 2018-04-10 17:01:25 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0925