Bug 1834664

Summary:	Upgrade to RHEL 8.2 causes smart card login to skip/fail due to failure in OCSP response verification
Product:	Red Hat Enterprise Linux 8	Reporter:	rmitra
Component:	sssd	Assignee:	Sumit Bose <sbose>
Status:	CLOSED NOTABUG	QA Contact:	sssd-qe <sssd-qe>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	8.2	CC:	arajendr, atikhono, esears, grajaiya, jhrozek, lslebodn, mzidek, pbrezina, sbose, spoore, tscherf
Target Milestone:	rc
Target Release:	8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-06-09 05:37:10 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description rmitra 2020-05-12 07:18:50 UTC

======================
Description of problem
======================

After upgrading to RHEL 8.2, IPA users are no longer able to login using a smart card, both for GDM and console login. This works fine on RHEL 8.1 before the upgrade.

On RHEL 8.2, when the username is entered in the login screen, it no longer prompts users for the PIN to unlock the certificate on the smart card.

Instead, it goes immediately to requesting the password. While users can still authenticate to IPA with their password, this violates security requirements  for customers enforcing PKI enabled login only.

The "p11_child" debug logs show failure in OCSP response verification for the smart card certificates, leading to smart card login getting skipped.


==============
Points to note
==============

a) The OCSP responder is reachable and functional, as confirmed by the fact that the smart card certificate successfully against the OCSP responder URI present in the certificate's OCSP extension. The following command returns success:

# openssl ocsp -issuer <issuer.pem> -cert <smartcard_cert.pem> -text -url <URI>


b) To further confirm this, we added "certificate_verification = no_ocsp" to /etc/sssd/sssd.conf and the smart card login starts working again, because OCSP verification is skipped.


c) The SSSD and PAM configurations are identical on both the working (RHEL 8.1) and non-working (RHEL 8.2) systems.


d) The smart card certificates are present in the nssdb and the issuer/signing certificate for the smart card is also present in the system trust store:

# certutil -L -d /etc/pki/nssdb/
# trust list | grep -B2 -A2 CA-51


============================================
Version-Release number of selected component
============================================

Non-working systems: RHEL 8.2
sssd-2.2.3-20

Working systems: RHEL 8.1
sssd-2.2.0-19


================
How reproducible
================

Upgrade an IPA client with smart card login configured from RHEL 8.1 to RHEL 8.2, smart card login gets skipped due to error in smart card certificate validation, caused by failure in OCSP response verification.


==================
Steps to Reproduce
==================

1. Setup an IPA client on RHEL 8.1 system.

2. Configure smart card login and verify that it works.

3. Upgrade to RHEL 8.2, smart card login stops working i.e. stops prompting for smart card PIN and directly skips to IPA password authentication.

4. Add the "certificate_verification = no_ocsp" option to /etc/sssd/sssd.conf and restart service, the smart card login starts working again, because OCSP verification is skipped.


Actual results
==============

Validation of smart card certificates fail due to OCSP response verification failure, as seen in the p11_child debug logs.


(Tue May  5 05:06:08 2020) [[sssd[p11_child[2556]]]] [read_certs] (0x4000): found cert[Certificate for PIV Authentication][/C=US/O=xxxxxxxx/OU=xxx/OU=xxx/OU=xxx/CN=xxxxx.xxxx.xxxxx.1233827793]
(Tue May  5 05:06:08 2020) [[sssd[p11_child[2556]]]] [do_ocsp] (0x4000): Using OCSP URL [http://xxxx.xxx.xxx].
(Tue May  5 05:06:08 2020) [[sssd[p11_child[2556]]]] [do_ocsp] (0x4000): Nonce in OCSP response is the same as the one used in the request.
(Tue May  5 05:06:08 2020) [[sssd[p11_child[2556]]]] [do_ocsp] (0x0020): OCSP_base_verify failed to verify OCSP response.
(Tue May  5 05:06:08 2020) [[sssd[p11_child[2556]]]] [do_verification] (0x0040): do_ocsp failed.
(Tue May  5 05:06:08 2020) [[sssd[p11_child[2556]]]] [read_certs] (0x0040): Certificate [Certificate for PIV Authentication][/C=US/O=xxxxxxxx/OU=xxx/OU=xxx/OU=xxx/CN=xxxxx.xxxx.xxxxx.1233827793] not valid, skipping.



Expected results
================
Smart card login should work on RHEL 8.2 in the same way as it does on RHEL 8.1, without having to skip OCSP verification.


Additional info
===============
Sensitive information has been redacted from the logs.
More debug logs coming soon.

Comment 1 Alexey Tikhonov 2020-05-12 07:42:57 UTC

(In reply to rmitra from comment #0)
> d) The smart card certificates are present in the nssdb and the
> issuer/signing certificate for the smart card is also present in the system
> trust store:
> 
> # certutil -L -d /etc/pki/nssdb/

As a note, SSSD for RHEL8 doesn't use NSS but OpenSSL (by default: /etc/sssd/pki/sssd_auth_ca_db.pem, see man sssd.conf "pam_cert_db_path")

But that doesn't explain why things were broken with an upgrade from 8.1 to 8.2.
Is machine run in a FIPS mode?

Comment 3 Sumit Bose 2020-05-12 08:59:58 UTC

Hi,

I have an idea what might be wrong. Would the customer like to test a test-build? If yes, please let me know which SSSD package version the customer is currently using.

bye,
Sumit

Comment 4 E. Sears 2020-05-12 09:28:52 UTC

(In reply to Sumit Bose from comment #3)
> Hi,
> 
> I have an idea what might be wrong. Would the customer like to test a
> test-build? If yes, please let me know which SSSD package version the
> customer is currently using.
> 
> bye,
> Sumit

I'd be happy to try out a test-build. 

Current version: sssd-2.2.3-20

Comment 5 rmitra 2020-05-13 01:23:24 UTC

(In reply to Alexey Tikhonov from comment #1)
> (In reply to rmitra from comment #0)
> > d) The smart card certificates are present in the nssdb and the
> > issuer/signing certificate for the smart card is also present in the system
> > trust store:
> > 
> > # certutil -L -d /etc/pki/nssdb/
> 
> As a note, SSSD for RHEL8 doesn't use NSS but OpenSSL (by default:
> /etc/sssd/pki/sssd_auth_ca_db.pem, see man sssd.conf "pam_cert_db_path")
> 
> But that doesn't explain why things were broken with an upgrade from 8.1 to
> 8.2.
> Is machine run in a FIPS mode?

Hi Alexey,

The RHEL 8.2 system in question has FIPS disabled (because they need MD5 for using Apache Maven), but they do have FIPS enabled on other systems.

Regards,
Ritu

Comment 10 rmitra 2020-05-13 02:08:05 UTC

(In reply to Alexey Tikhonov from comment #2)
> Can we have full p11_child debug log and sssd.conf please?
> 
> Feel free to make attachment private and/or delete sensitive information.

Added the following attachments:

1. Tar file having configurations + SSSD debug logs, includes sssd.conf and p11_child debug logs

2. Output of "/usr/libexec/sssd/p11_child --nssdb=/etc/pki/nssdb --pre -d 10 --debug-fd=1"

NOTE: this doesn't list the certificates because "--verify=no_verification" option is not used 

3. Output of "OPENSC_DEBUG=9 ssh -I /usr/lib64/pkcs11/opensc-pkcs11.so <user>@localhost" for both cases:

a) without "certificate_verification=no_ocsp" added in /etc/sssd/sssd.conf

b) with "certificate_verification=no_ocsp" added in /etc/sssd/sssd.conf

Comment 14 E. Sears 2020-05-22 10:45:11 UTC

Is anything need from me to try and resolve this? This is impacting the ability for all of our users to log in.

Comment 15 Sumit Bose 2020-06-09 05:37:10 UTC

Hi,

it turned out that the switch to sha256 as digest for OCSP was not received well by the used OCSP responder. Adding the sub-option 'ocsp_dgst=sha1' to the 'certificate_verification' in the [sssd] section of sssd.conf solved the issue, see man sssd.conf for details.

I close this ticket.

bye,
Sumit