Bug 2001065

Summary: Candlepin CRL file continues to grow even though job to populate it is disabled
Product: Red Hat Satellite Reporter: Samson Wick <swick>
Component: InstallationAssignee: Jonathon Turel <jturel>
Status: CLOSED DUPLICATE QA Contact: Devendra Singh <desingh>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.9.0CC: ekohlvan, nmoumoul, wpinheir
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-09-28 14:09:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Samson Wick 2021-09-03 17:05:50 UTC 
Description of problem:
This is related to https://bugzilla.redhat.com/show_bug.cgi?id=1784341

The customer has historically experienced catastrophic failure of Satellite services due to the /var/lib/candlepin/candlepin-crl.crl file growing to an unmanageable size (i.e. 2GB or greater)

The customer upgraded to Satellite 6.8 approximately 6 months ago.  The bug ID and https://access.redhat.com/errata/RHSA-2020:4366 both indicate that as of 6.8 a change was made to /etc/candlepin/candlepin.conf to disable the "CertificateRevocationListTask" which populates the CRL file.  

We confirmed that their 6.8 version Satellite had this set according to the Errata and therefore the job should have been disabled.

For reference this is the setting present in candlepin.conf:

pinsetter.org.candlepin.pinsetter.tasks.CertificateRevocationListTask.schedule=0 0 0 1 1 ?

The existing crl file was approximately 750MB in size.

During the upgrade to Satellite 6.9.5 we removed the original CRL file and performed the workaround steps described in the BZ above to reset the CRL file.  This resulted in the creation of very small "candlepin-crl.crl" file which we observed over the following days.

We observed that something is still continuing to populate this file with additional entries at noon server local time each day.

The Satellite upgrade was performed on 8/31.

Here is a directory listing showing the modification time of the file at noon server local time:
[root@satellite-dev candlepin]# ls -l

total 38660

drwxr-xr-x. 6 tomcat tomcat       64 Dec 10  2020 activemq-artemis

-rw-r--r--. 1 tomcat tomcat   889508 Aug 26 12:00 candlepin-crl.BAK

-rw-r--r--. 1 tomcat tomcat     2196 Sep  2 12:00 candlepin-crl.crl

-rw-r--r--. 1 tomcat tomcat 38688917 Aug 16 12:00 candlepin-crl.crl.gz

-rw-r--r--. 1 root   root          0 Dec  7  2018 cpdb_done

-rw-r--r--. 1 root   root          0 Jun 24 23:39 cpdb_update_done

-rw-r--r--. 1 root   root          0 Dec  7  2018 cpinit_done

drwxr-xr-x. 6 tomcat tomcat       64 Apr 24  2019 hornetq



When we decode the file it shows that revocations are continuing to be recorded each day at 16:00 which corresponds with noon server local time in the file AND that they are increasing in number over time.

[root@satellite-dev candlepin]# openssl crl -text -noout -in candlepin-crl.crl

Certificate Revocation List (CRL):

        Version 2 (0x1)

    Signature Algorithm: sha256WithRSAEncryption

        Issuer: /C=US/ST=North Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=satellite-dev.dev.customer.com

        Last Update: Sep  2 16:00:00 2021 GMT

        Next Update: Sep  3 16:00:00 2021 GMT

        CRL extensions:

            X509v3 CRL Number:

                3

            X509v3 Authority Key Identifier:

                keyid:94:D8:05:54:9C:7B:59:F4:00:C4:E4:79:42:F1:45:9D:C6:03:38:CE

 

Revoked Certificates:

    Serial Number: 6E6BC87EBD9D68E3

        Revocation Date: Sep  1 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 364D6B92F58F43E3

        Revocation Date: Sep  1 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 40EAA4B9AF68A2CF

        Revocation Date: Sep  1 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

               Privilege Withdrawn

    Serial Number: 4D1E74B327F6469D

        Revocation Date: Sep  1 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 0441C659FE22F30F

        Revocation Date: Sep  1 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 1766D642499B57F7

        Revocation Date: Sep  1 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 5B63A7402E14CFD6

        Revocation Date: Sep  1 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 0A305E8C54D981FA

        Revocation Date: Sep  1 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 2BFE54F30F2B92AA

        Revocation Date: Sep  1 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 3CA65B88EC9E150B

        Revocation Date: Sep  1 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: AC6ED1007C16B7

        Revocation Date: Sep  1 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 65B911F0DD3B2F50

        Revocation Date: Sep  1 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 04D916B7C7FB2830

        Revocation Date: Sep  1 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 2BFE54F30F2B92AA

        Revocation Date: Sep  2 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 3CA65B88EC9E150B

        Revocation Date: Sep  2 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 65B911F0DD3B2F50

        Revocation Date: Sep  2 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 40EAA4B9AF68A2CF

        Revocation Date: Sep  2 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 4D1E74B327F6469D

        Revocation Date: Sep  2 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 0441C659FE22F30F

        Revocation Date: Sep  2 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 6E6BC87EBD9D68E3

        Revocation Date: Sep  2 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 5B63A7402E14CFD6

        Revocation Date: Sep  2 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 1766D642499B57F7

        Revocation Date: Sep  2 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 0A305E8C54D981FA

        Revocation Date: Sep  2 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: AC6ED1007C16B7

        Revocation Date: Sep  2 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 04D916B7C7FB2830

        Revocation Date: Sep  2 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Serial Number: 364D6B92F58F43E3

        Revocation Date: Sep  2 16:00:00 2021 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Privilege Withdrawn

    Signature Algorithm: sha256WithRSAEncryption

        56:5e:4c:7c:ba:f2:6f:bc:a6:1c:39:2e:01:d6:b9:e0:e7:bb:

         34:b7:43:68:ec:0f:13:b9:38:93:73:90:98:6f:ad:53:43:79:

         88:91:e1:40:d3:fb:01:4d:3d:26:cd:6d:86:61:e1:4b:5c:4b:

         dc:d2:99:10:27:98:ae:e0:67:46:2a:92:f4:eb:cc:50:0d:f3:

         04:d9:34:9a:b1:49:0b:2a:fc:32:90:a5:f4:03:82:07:b3:db:

         d4:50:78:97:6c:cf:fb:e6:93:29:41:5b:ea:a7:2d:b6:dc:e7:

         ab:55:37:e7:5a:62:98:f2:d3:a7:6d:4f:97:97:e2:87:41:f7:

         98:0c:78:b4:47:73:cd:78:87:d2:7b:cb:43:25:ea:1d:3d:9d:

         ce:61:49:ae:f5:5e:2d:51:b0:2c:11:b8:22:0c:61:31:ba:8d:

         bb:25:0b:1a:3e:4d:53:77:db:e7:de:14:01:4c:94:03:c7:2e:

         cc:f7:8d:7e:d4:ae:38:63:d6:1f:ef:c6:3d:78:54:43:03:63:

         fb:32:e2:99:31:b7:1c:4b:ab:c0:7b:03:a1:f7:fd:1d:a5:51:

         17:db:1b:2e:aa:b5:8a:3a:62:d7:3b:0c:f5:1a:ef:22:51:b2:

         ea:bc:e9:01:07:51:92:c0:ca:2e:dc:19:b0:0c:02:54:e2:77:

         76:4c:5c:73

[root@satellite-dev candlepin]# openssl crl -text -noout -in candlepin-crl.crl | grep "Sep  1" | wc -l

13

[root@satellite-dev candlepin]# openssl crl -text -noout -in candlepin-crl.crl | grep "Sep  2" | wc -l

14

Version-Release number of selected component (if applicable):
Satellite 6.9.5

How reproducible:
Happens automatically as part of Satellite function



Steps to Reproduce:
1.Truncate candlepin-crl.crl
2.Run Satellite
3.Observe that the file is growing in size and adding entries

Actual results:
candlepin-crl.crl file continues to be populated and grow in size

Expected results:
candlepin-crl.crl file does not continue to be populated or grow in size

Additional info:
The original BZ indicates that the candlepin-crl.crl file is not actually used by Satellite, and that the updated configuration should prevent it from being popuplated.  This was confirmed when I spoke with engineering and I was asked to open an additional BZ on this behavior.

This is a customer Satellite environment that I do not have arbitrary or continual access to.  If additional information or testing is needed I can put the engineer assigned directly in touch with the customer.
Comment 1 Nikos Moumoulidis 2021-09-07 14:16:12 UTC 
Satellite 6.8+ uses candlepin version 3.1+, in which the new Artemis-backed job system was introduced. Along with it the job-related configuration options (names and values) in candlepin.conf changed, which is why the old setting no longer works.

The old option
  pinsetter.org.candlepin.pinsetter.tasks.CertificateRevocationListTask.schedule
was replaced with:
  candlepin.async.jobs.CRLUpdateJob.schedule
   
The installer is responsible for managing the candlepin.conf file (and the default values in it), so I am changing the component to that. It looks like this has already been fixed upstream a few days ago, so it just needs to make it's way to the proper sat versions:  https://github.com/theforeman/puppet-candlepin/pull/205 (although it looks like with this schedule setting the job will still run, but only once a year on the 1st of January? it could be entirely disabled by setting 'candlepin.async.jobs.CRLUpdateJob.enabled=false')
Comment 2 Ewoud Kohl van Wijngaarden 2021-09-28 14:07:53 UTC 
Connecting redmine issue https://projects.theforeman.org/issues/33321 from this bug
Comment 3 Ewoud Kohl van Wijngaarden 2021-09-28 14:09:27 UTC 

*** This bug has been marked as a duplicate of bug 1996747 ***