Bug 1641804

Summary: [RFE] Random Serial Number v3 Support
Product: Red Hat Enterprise Linux 9 Reporter: Rob Crittenden <rcritten>
Component: pki-coreAssignee: Endi Sukma Dewata <edewata>
Status: CLOSED ERRATA QA Contact: PKI QE <bugzilla-pkiqe>
Severity: high Docs Contact: Florian Delehaye <fdelehay>
Priority: urgent    
Version: 9.1CC: aakkiang, ckelley, dconsoli, edewata, ftweedal, jvilicic, mharmsen, pcech, radrao, skhandel
Target Milestone: rcKeywords: FutureFeature, Reopened, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pki-core-11.2.0-0.2.beta1.el9 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
: 2184509 (view as bug list) Environment:
Last Closed: 2022-11-15 10:13:02 UTC Type: Feature Request
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1851495, 2075154    
Bug Blocks: 747959, 2184509    

Description Rob Crittenden 2018-10-22 20:08:40 UTC
Description of problem:

In a recent customer escalation there were requests in ou=requests that didn't match with the serial numbers in the certificates that certmonger was trying to renew by serial number. I have two upstream reports of customers using CentOS 7.5 that look like the same problem.

I wrote a tool which attempts to find errors in the IPA cert tracking before they are show-stoppers. One test takes the certificate in the issued cert and looks up the value in cn=<SERIAL #>,ou=ca,ou=requests,o=ipaca and compares the value of the subject on the certmonger side to extdata-req--005fsubject--005fname in LDAP.

The output looks like:

Subject O=EXAMPLE.TEST,CN=CA Subsystem and template subject CN=CA Audit,O=EXAMPLE.TEST do not match for serial 57

My questions are:

1. Is my understanding wrong? Is it wrong to assume that cn=SERIAL will match up the the serial of an issued certificate?
2. If my understanding is not wrong is there a way to piece together how the LDAP data got into this state? IMHO this is a ticking time bomb if these are supposed to match. As an outsider it looks like database corruption.
3. Most importantly: what can be done to correct this in a user's system?

IMHO switching to renewing by CSR is just sweeping this under the rug.

Version-Release number of selected component (if applicable):

pki-ca-10.5.1-9
ipa-server-4.5.4-10.el7.centos.4.4

How reproducible:

Unknown. There are three known reports.

Additional info:

Upstream checking tool: https://github.com/rcritten/checkcerts in compare_requests()

Comment 2 Fraser Tweedale 2018-11-08 01:53:23 UTC
Unlike the entries under ou=certificateRepository, the CN of a request is
not related to the serial number of the issued cert.

Request objects have a 'serialno' attribute.  I will check whether that
is the serial number of the issued cert, which could be used for lookup
instead.

Alternatively, look up the certificate first, and the certificate entry
contains a reference to the request that produced it, e.g.:

  % ldapsearch -LLL -D cn=directory\ manager -w4me2Test
      -b ou=certificateRepository,ou=ca,o=ipaca '(cn=1)' metaInfo
  dn: cn=1,ou=certificateRepository,ou=ca,o=ipaca
  metaInfo: profileId:caCACert
  metaInfo: requestId:1            <-- this tells you the request ID.

Comment 3 Rob Crittenden 2018-11-08 03:17:21 UTC
This does not match the behavior seen elsewhere.

When re-issuing a cert certmonger renews by serial number:

GET /ca/ee/ca/profileSubmit?profileId=caServerCert&serial_num=5&renewal=true&xml=true&requestor_name=IPA

The returned certificate from dogtag was for a different hostname altogether. The hostname matched the value for extdata-req--005fsubject--005fname in:

dn: cn=5,ou=ca,ou=requests,o=ipaca

We had to manually reconfigure certmonger to not do renewal by serial number to work around this.

This was not a one-off.

So I'm ok with changing the query but it doesn't match what we were seeing in production.

Comment 5 Fraser Tweedale 2018-11-08 03:36:18 UTC
Thanks for the extra info.  I can't explain this and will need to investigate
further.  The problem might arise due to conflicting request and/or cert serialNo ranges on different replicas.  Certainly the request ID should always be looked up via the cert record.  The renewal code should not be using the serialNo as the request ID.  I will confirm that this is not the case.

Rob, next time you see a case like this (renewed cert has different serial number) please loop me in.  We will need to investigate whether there are
range conflicts and/or replication conflicts.

Comment 6 Rob Crittenden 2018-11-08 03:46:47 UTC
Will do. I'll see if the logs provided by users who have tried the tool any additional information.

Comment 7 Fraser Tweedale 2018-11-08 03:57:04 UTC
Further info:

The ProfileSubmitServlet reads the "serial_num" param and sets it into the
CertEnrolmentRequest data, before invoking the RenewalProcessor:

  https://github.com/dogtagpki/pki/blob/v10.6.7/base/server/cms/src/com/netscape/cms/servlet/profile/ProfileSubmitServlet.java#L269-L274

In RenewalProcessor the serial numbers gets read of the data:

  https://github.com/dogtagpki/pki/blob/v10.6.7/base/server/cms/src/com/netscape/cms/servlet/cert/RenewalProcessor.java#L113-L117

and used to look up the certificate record:

  https://github.com/dogtagpki/pki/blob/v10.6.7/base/server/cms/src/com/netscape/cms/servlet/cert/RenewalProcessor.java#L181

  
The certificate record is then used to look up the original request:

  https://github.com/dogtagpki/pki/blob/v10.6.7/base/server/cms/src/com/netscape/cms/servlet/cert/RenewalProcessor.java#L208

In the implementation of getOriginalRequest we see that the request ID is read
from the corresponding metaInfo field:

  https://github.com/dogtagpki/pki/blob/v10.6.7/base/server/cms/src/com/netscape/cms/servlet/processors/CAProcessor.java#L231-L248



Rob, I therefore suspicion that there is more to the story with the
cases we have seen (e.g. conflict request ranges).  The fact that the
serial number in question is so low (5) strengthens the suspicion.

In any case you should update the checker query to look up the request
record via the certificate record.

Comment 9 Dinesh Prasanth 2020-04-28 18:29:04 UTC
As per grooming meeting 20200428, this bug can be fixed by having a upgrade to database framework

Comment 28 errata-xmlrpc 2022-11-15 10:13:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (pki-core bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:8053