1159011 – Trust setting not restored for CA cert with ipa-restore command

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1159011 - Trust setting not restored for CA cert with ipa-restore command

Summary: Trust setting not restored for CA cert with ipa-restore command

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	ipa
Sub Component:
Version:	7.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Jan Cholasta
QA Contact:	Namita Soman
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2014-10-30 16:55 UTC by Kaleem
Modified:	2015-03-05 10:14 UTC (History)
CC List:	8 users (show)
Fixed In Version:	ipa-4.1.0-5.el7
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2015-03-05 10:14:15 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
contents of /etc/httpd/alias (220.00 KB, application/x-tar) 2014-11-05 03:21 UTC, Kaleem	no flags	Details
ipa.p11-kit file attached along with console log (20.00 KB, application/x-tar) 2014-11-10 11:35 UTC, Kaleem	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2015:0442	0	normal	SHIPPED_LIVE	Moderate: ipa security, bug fix, and enhancement update	2015-03-05 14:50:39 UTC

Description Kaleem 2014-10-30 16:55:37 UTC

Description of problem:
Trust settings of CA cert not restored correctly in nss db. This observed while execution of ipa-restore which restores /etc/httpd/alias nss db which was backuped earlier and it causes httpd service restart failure. Consequently ipa service restart fails.

Version-Release number of selected component (if applicable):
[root@dhcp207-187 ~]# rpm -q ipa-server nss-tools nss
ipa-server-4.1.0-3.el7.x86_64
nss-tools-3.16.2-9.el7.x86_64
nss-3.16.2-9.el7.x86_64
[root@dhcp207-187 ~]# 

How reproducible:
Always.

Steps to Reproduce:
1. Please find the attached file (steps-with-console-output.txt) for detail.

Actual results:
Trust settings of CA cert not restored correctly.

Expected results:
Trust settings of CA cert should have been restored correctly

Additional info:

Comment 3 Kaleem 2014-10-30 16:59:42 UTC

======= Test machine =======

[root@dhcp207-187 alias]# certutil -L -d .

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Signing-Cert                                                 u,u,u
ipaCert                                                      u,u,u
Server-Cert                                                  u,u,u
TESTRELM.TEST IPA CA                                         ,,   
[root@dhcp207-187 alias]# rpm -q nss-tools
nss-tools-3.16.2-9.el7.x86_64

[root@dhcp207-187 alias]# sha1sum * | sha1sum
a6e1002f1f2da3e8ee822f32e796ef5cd5a326a7  -


======= My laptop =======

pviktori@dhcp-31-13:/tmp/tx/alias$ certutil -L -d .

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Signing-Cert                                                 u,u,u
ipaCert                                                      u,u,u
Server-Cert                                                  u,u,u
TESTRELM.TEST IPA CA                                         CT,C,C
pviktori@dhcp-31-13:/tmp/tx/alias$ rpm -q nss-tools
nss-tools-3.17.1-1.fc21.x86_64
pviktori@dhcp-31-13:/tmp/tx/alias$ sha1sum * | sha1sum
a6e1002f1f2da3e8ee822f32e796ef5cd5a326a7  -

Comment 5 Petr Viktorin (pviktori) 2014-11-04 15:16:38 UTC

Kaleem, could you also attach the contents of /etc/httpd/alias from the affected machine?

The database is the same (see the hashes above), but sometimes the trust flags are reported missing. It would be nice if NSS the folks could investigate.

Comment 6 Kaleem 2014-11-04 17:03:57 UTC

[root@dhcp207-187 alias]# certutil -L -d .

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Signing-Cert                                                 u,u,u
ipaCert                                                      u,u,u
Server-Cert                                                  u,u,u
TESTRELM.TEST IPA CA                                         ,,   
[root@dhcp207-187 alias]# rpm -q nss-tools
nss-tools-3.16.2-9.el7.x86_64
[root@dhcp207-187 alias]# sha1sum * | sha1sum
a6e1002f1f2da3e8ee822f32e796ef5cd5a326a7  -
[root@dhcp207-187 alias]#

Comment 7 Petr Viktorin (pviktori) 2014-11-04 17:24:02 UTC

I meant, add the contents of the /etc/httpd/alias directory as an attachment to the bug, so they can be examined.

Comment 8 Kaleem 2014-11-05 03:21:01 UTC

Created attachment 953904 [details]
contents of /etc/httpd/alias

Comment 10 Martin Kosek 2014-11-10 07:21:42 UTC

I tested the attachment and on both my F20 and my RHEL-7.x (nss-3.16.2-9.el7.x86_64) I get:

# certutil -L -d .

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

Signing-Cert                                                 u,u,u
ipaCert                                                      u,u,u
Server-Cert                                                  u,u,u
TESTRELM.TEST IPA CA                                         CT,C,C

Is this the right attachment or am I missing anything? Also CCing Kai in case he some idea what may be wrong.

BTW, I think the hashes demonstrated in Comment 3 will not work well as the dir points to libnssckbi.so which can change with NSS version.

Comment 11 Jan Cholasta 2014-11-10 07:52:12 UTC

Kaleem, could you please post the output of "certutil -L -d /etc/httpd/alias/ -h 'System Trust'" before and after the restore?

Comment 12 Kaleem 2014-11-10 09:14:24 UTC

[root@ibm-x3650m4-02-vm-03 ~]# certutil -L -d /etc/httpd/alias/ -h 'System Trust'

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

System Trust:TESTRELM.TEST IPA CA                            CT,C,C
[root@ibm-x3650m4-02-vm-03 ~]# ipa-restore /var/lib/ipa/backup/ipa-full-2014-11-10-03-53-26/
Directory Manager (existing master) password: 

Preparing restore from /var/lib/ipa/backup/ipa-full-2014-11-10-03-53-26/ on ibm-x3650m4-02-vm-03.testrelm.test
Restoring data will overwrite existing live data. Continue to restore? [no]: yes
Each master will individually need to be re-initialized or
re-created from this one. The replication agreements on
masters running IPA 3.1 or earlier will need to be manually
re-enabled. See the man page for details.
Disabling all replication.
Stopping IPA services
Restoring files
Restoring from userRoot in TESTRELM-TEST
Restoring from ipaca in TESTRELM-TEST
Starting IPA services
Command ''ipactl' 'start'' returned non-zero exit status 1
[root@ibm-x3650m4-02-vm-03 ~]# certutil -L -d /etc/httpd/alias/ -h 'System Trust'

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

System Trust:TESTRELM.TEST IPA CA                            ,,   
[root@ibm-x3650m4-02-vm-03 ~]# rpm -q ipa-server 389-ds-base nss
ipa-server-4.1.0-3.el7.x86_64
389-ds-base-1.3.3.1-6.el7.x86_64
nss-3.16.2-9.el7.x86_64
[root@ibm-x3650m4-02-vm-03 ~]#

Comment 13 Jan Cholasta 2014-11-10 10:29:38 UTC

OK, we found where the bug is, now we have to find out why it happens:

1) Do any of the files /etc/pki/ca-trust/source/ipa.p11-kit or /etc/pki/ca-trust/source/anchors/ipa-ca.crt exist before the restore? If they do, please attach them.

2) Do any of the files /etc/pki/ca-trust/source/ipa.p11-kit or /etc/pki/ca-trust/source/anchors/ipa-ca.crt exist after the restore? If they do, please attach them.

3) Does running update-ca-trust after the restore fix the problem?

Comment 14 Kaleem 2014-11-10 11:35:01 UTC

Created attachment 955782 [details]
ipa.p11-kit file attached along with console log

Comment 15 Jan Cholasta 2014-11-10 11:43:29 UTC

Could you please also try if running "update-ca-trust" fixes the problem, as requested in 3)?

Comment 16 Kaleem 2014-11-10 11:51:14 UTC

Sorry. forgot to include this as i ran this with earlier setup. Again ran with new setup also. update-ca-trust not fixing the problem.

[root@ibm-x3650m4-02-vm-03 ~]# update-ca-trust 
[root@ibm-x3650m4-02-vm-03 ~]# systemctl status httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled)
   Active: failed (Result: exit-code) since Mon 2014-11-10 06:22:52 EST; 23min ago
...
...
[root@ibm-x3650m4-02-vm-03 ~]#

[root@ibm-x3650m4-02-vm-03 ~]# systemctl restart httpd.service
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
[root@ibm-x3650m4-02-vm-03 ~]#

[root@ibm-x3650m4-02-vm-03 ~]# systemctl status httpd.service
httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled)
   Active: failed (Result: exit-code) since Mon 2014-11-10 06:46:53 EST; 1s ago
..
...
[root@ibm-x3650m4-02-vm-03 ~]#

[root@ibm-x3650m4-02-vm-03 ~]# tail -n 8 /var/log/httpd/error_log 
[Mon Nov 10 06:22:52.503472 2014] [core:notice] [pid 8342] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Mon Nov 10 06:22:52.504730 2014] [suexec:notice] [pid 8342] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Nov 10 06:22:52.745698 2014] [:error] [pid 8342] SSL Library Error: -8172 Certificate is signed by an untrusted issuer
[Mon Nov 10 06:22:52.745725 2014] [:error] [pid 8342] Unable to verify certificate 'Server-Cert'. Add "NSSEnforceValidCerts off" to nss.conf so the server can start until the problem can be resolved.
[Mon Nov 10 06:46:53.195510 2014] [core:notice] [pid 8412] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Mon Nov 10 06:46:53.196445 2014] [suexec:notice] [pid 8412] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Nov 10 06:46:53.371430 2014] [:error] [pid 8412] SSL Library Error: -8172 Certificate is signed by an untrusted issuer
[Mon Nov 10 06:46:53.371464 2014] [:error] [pid 8412] Unable to verify certificate 'Server-Cert'. Add "NSSEnforceValidCerts off" to nss.conf so the server can start until the problem can be resolved.
[root@ibm-x3650m4-02-vm-03 ~]#

Comment 17 Jan Cholasta 2014-11-10 14:30:15 UTC

The "TESTRELM.TEST IPA CA" cert in /etc/httpd/alias and /etc/pki/ca-trust/source/ipa.p11-kit do not match, which is why you are getting empty trust flags for it. This is because ipa.p11-kit is not backed up nor restored and thus wrong version of it is used after restore.

Comment 18 Jan Cholasta 2014-11-10 14:56:45 UTC

Upstream ticket:
https://fedorahosted.org/freeipa/ticket/4711

Comment 20 Petr Viktorin (pviktori) 2014-11-11 15:36:53 UTC

Fixed upstream
master:
https://fedorahosted.org/freeipa/changeset/2639997dfee43d66e94ef9b5441289816c465e7d
ipa-4-1:
https://fedorahosted.org/freeipa/changeset/7c2aad17da8bd5f50b9c1409f91c413bc454ce28

Comment 22 Kaleem 2014-11-24 11:24:43 UTC

Verified.

IPA version:
------------
[root@dhcp207-1 ~]# rpm -q ipa-server
ipa-server-4.1.0-7.el7.x86_64
[root@dhcp207-1 ~]#

[root@dhcp207-1 ~]# ipa-restore -p xxxxxxxx -U /var/lib/ipa/backup/ipa-full-2014-11-24-16-26-33/
Preparing restore from /var/lib/ipa/backup/ipa-full-2014-11-24-16-26-33/ on dhcp207-1.testrelm.test
Each master will individually need to be re-initialized or
re-created from this one. The replication agreements on
masters running IPA 3.1 or earlier will need to be manually
re-enabled. See the man page for details.
Disabling all replication.
Stopping IPA services
Systemwide CA database updated.
Restoring files
Systemwide CA database updated.
Restoring from userRoot in TESTRELM-TEST
Restoring from ipaca in TESTRELM-TEST
Starting IPA services
Restarting SSSD
The ipa-restore command was successful
[root@dhcp207-1 ~]#

Comment 24 errata-xmlrpc 2015-03-05 10:14:15 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0442.html

Note You need to log in before you can comment on or make changes to this bug.