RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1834664 - Upgrade to RHEL 8.2 causes smart card login to skip/fail due to failure in OCSP response verification
Summary: Upgrade to RHEL 8.2 causes smart card login to skip/fail due to failure in OC...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: sssd
Version: 8.2
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: 8.0
Assignee: Sumit Bose
QA Contact: sssd-qe
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-12 07:18 UTC by rmitra
Modified: 2023-12-15 17:52 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-06-09 05:37:10 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-42965 0 None None None 2023-09-07 23:10:32 UTC
Red Hat Knowledge Base (Solution) 5317101 0 None None None 2020-08-13 13:15:01 UTC

Description rmitra 2020-05-12 07:18:50 UTC 
======================
Description of problem
======================

After upgrading to RHEL 8.2, IPA users are no longer able to login using a smart card, both for GDM and console login. This works fine on RHEL 8.1 before the upgrade.

On RHEL 8.2, when the username is entered in the login screen, it no longer prompts users for the PIN to unlock the certificate on the smart card.

Instead, it goes immediately to requesting the password. While users can still authenticate to IPA with their password, this violates security requirements  for customers enforcing PKI enabled login only.

The "p11_child" debug logs show failure in OCSP response verification for the smart card certificates, leading to smart card login getting skipped.


==============
Points to note
==============

a) The OCSP responder is reachable and functional, as confirmed by the fact that the smart card certificate successfully against the OCSP responder URI present in the certificate's OCSP extension. The following command returns success:

# openssl ocsp -issuer <issuer.pem> -cert <smartcard_cert.pem> -text -url <URI>


b) To further confirm this, we added "certificate_verification = no_ocsp" to /etc/sssd/sssd.conf and the smart card login starts working again, because OCSP verification is skipped.


c) The SSSD and PAM configurations are identical on both the working (RHEL 8.1) and non-working (RHEL 8.2) systems.


d) The smart card certificates are present in the nssdb and the issuer/signing certificate for the smart card is also present in the system trust store:

# certutil -L -d /etc/pki/nssdb/
# trust list | grep -B2 -A2 CA-51


============================================
Version-Release number of selected component
============================================

Non-working systems: RHEL 8.2
sssd-2.2.3-20

Working systems: RHEL 8.1
sssd-2.2.0-19


================
How reproducible
================

Upgrade an IPA client with smart card login configured from RHEL 8.1 to RHEL 8.2, smart card login gets skipped due to error in smart card certificate validation, caused by failure in OCSP response verification.


==================
Steps to Reproduce
==================

1. Setup an IPA client on RHEL 8.1 system.

2. Configure smart card login and verify that it works.

3. Upgrade to RHEL 8.2, smart card login stops working i.e. stops prompting for smart card PIN and directly skips to IPA password authentication.

4. Add the "certificate_verification = no_ocsp" option to /etc/sssd/sssd.conf and restart service, the smart card login starts working again, because OCSP verification is skipped.


Actual results
==============

Validation of smart card certificates fail due to OCSP response verification failure, as seen in the p11_child debug logs.


(Tue May  5 05:06:08 2020) [[sssd[p11_child[2556]]]] [read_certs] (0x4000): found cert[Certificate for PIV Authentication][/C=US/O=xxxxxxxx/OU=xxx/OU=xxx/OU=xxx/CN=xxxxx.xxxx.xxxxx.1233827793]
(Tue May  5 05:06:08 2020) [[sssd[p11_child[2556]]]] [do_ocsp] (0x4000): Using OCSP URL [http://xxxx.xxx.xxx].
(Tue May  5 05:06:08 2020) [[sssd[p11_child[2556]]]] [do_ocsp] (0x4000): Nonce in OCSP response is the same as the one used in the request.
(Tue May  5 05:06:08 2020) [[sssd[p11_child[2556]]]] [do_ocsp] (0x0020): OCSP_base_verify failed to verify OCSP response.
(Tue May  5 05:06:08 2020) [[sssd[p11_child[2556]]]] [do_verification] (0x0040): do_ocsp failed.
(Tue May  5 05:06:08 2020) [[sssd[p11_child[2556]]]] [read_certs] (0x0040): Certificate [Certificate for PIV Authentication][/C=US/O=xxxxxxxx/OU=xxx/OU=xxx/OU=xxx/CN=xxxxx.xxxx.xxxxx.1233827793] not valid, skipping.



Expected results
================
Smart card login should work on RHEL 8.2 in the same way as it does on RHEL 8.1, without having to skip OCSP verification.


Additional info
===============
Sensitive information has been redacted from the logs.
More debug logs coming soon.
Comment 1 Alexey Tikhonov 2020-05-12 07:42:57 UTC 
(In reply to rmitra from comment #0)
> d) The smart card certificates are present in the nssdb and the
> issuer/signing certificate for the smart card is also present in the system
> trust store:
> 
> # certutil -L -d /etc/pki/nssdb/

As a note, SSSD for RHEL8 doesn't use NSS but OpenSSL (by default: /etc/sssd/pki/sssd_auth_ca_db.pem, see man sssd.conf "pam_cert_db_path")

But that doesn't explain why things were broken with an upgrade from 8.1 to 8.2.
Is machine run in a FIPS mode?
Comment 3 Sumit Bose 2020-05-12 08:59:58 UTC 
Hi,

I have an idea what might be wrong. Would the customer like to test a test-build? If yes, please let me know which SSSD package version the customer is currently using.

bye,
Sumit
Comment 4 E. Sears 2020-05-12 09:28:52 UTC 
(In reply to Sumit Bose from comment #3)
> Hi,
> 
> I have an idea what might be wrong. Would the customer like to test a
> test-build? If yes, please let me know which SSSD package version the
> customer is currently using.
> 
> bye,
> Sumit

I'd be happy to try out a test-build. 

Current version: sssd-2.2.3-20
Comment 5 rmitra 2020-05-13 01:23:24 UTC 
(In reply to Alexey Tikhonov from comment #1)
> (In reply to rmitra from comment #0)
> > d) The smart card certificates are present in the nssdb and the
> > issuer/signing certificate for the smart card is also present in the system
> > trust store:
> > 
> > # certutil -L -d /etc/pki/nssdb/
> 
> As a note, SSSD for RHEL8 doesn't use NSS but OpenSSL (by default:
> /etc/sssd/pki/sssd_auth_ca_db.pem, see man sssd.conf "pam_cert_db_path")
> 
> But that doesn't explain why things were broken with an upgrade from 8.1 to
> 8.2.
> Is machine run in a FIPS mode?

Hi Alexey,

The RHEL 8.2 system in question has FIPS disabled (because they need MD5 for using Apache Maven), but they do have FIPS enabled on other systems.

Regards,
Ritu
Comment 10 rmitra 2020-05-13 02:08:05 UTC 
(In reply to Alexey Tikhonov from comment #2)
> Can we have full p11_child debug log and sssd.conf please?
> 
> Feel free to make attachment private and/or delete sensitive information.

Added the following attachments:

1. Tar file having configurations + SSSD debug logs, includes sssd.conf and p11_child debug logs

2. Output of "/usr/libexec/sssd/p11_child --nssdb=/etc/pki/nssdb --pre -d 10 --debug-fd=1"

NOTE: this doesn't list the certificates because "--verify=no_verification" option is not used 

3. Output of "OPENSC_DEBUG=9 ssh -I /usr/lib64/pkcs11/opensc-pkcs11.so <user>@localhost" for both cases:

a) without "certificate_verification=no_ocsp" added in /etc/sssd/sssd.conf

b) with "certificate_verification=no_ocsp" added in /etc/sssd/sssd.conf
Comment 14 E. Sears 2020-05-22 10:45:11 UTC 
Is anything need from me to try and resolve this? This is impacting the ability for all of our users to log in.
Comment 15 Sumit Bose 2020-06-09 05:37:10 UTC 
Hi,

it turned out that the switch to sha256 as digest for OCSP was not received well by the used OCSP responder. Adding the sub-option 'ocsp_dgst=sha1' to the 'certificate_verification' in the [sssd] section of sssd.conf solved the issue, see man sssd.conf for details.

I close this ticket.

bye,
Sumit
Note You need to log in before you can comment on or make changes to this bug.