Bug 1450938

Summary: 'Certcheck' does not remove Expired golden ticket entitlement from the system
Product: Red Hat Enterprise Linux 7 Reporter: Rehana <redakkan>
Component: subscription-managerAssignee: Jiri Hnidek <jhnidek>
Status: CLOSED DUPLICATE QA Contact: John Sefler <jsefler>
Severity: medium Docs Contact:
Priority: high    
Version: 7.4CC: jhnidek, khowell, redakkan, skallesh
Target Milestone: rcKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-22 11:03:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Rehana 2017-05-15 12:29:57 UTC
Description of problem:
Expired golden ticket certificate were not removed when certcheck was ran. Though a new valid golden ticket was attached , the expired one still exists in the /etc/pki/entitlement directory.

Version-Release number of selected component (if applicable):
# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 2.1.0-1
subscription management rules: 5.23
subscription-manager: 1.19.12-1.el7
python-rhsm: 1.19.6-1.el7


How reproducible:
always

Steps to Reproduce:
1.Set content access mode enabled on the org "snowwhite"
2.Register client and check if the golden ticket entitlement are attached along with other subscriptions 

[root@dhcp35-238 entitlement]# ll
total 48
-rw-------. 1 root root 1675 May 15  2017 1123175916477564753-key.pem
-rw-r--r--. 1 root root 2562 May 15  2017 1123175916477564753.pem
-rw-------. 1 root root 1675 May 15  2017 2562498625624795262-key.pem
-rw-r--r--. 1 root root 2878 May 15  2017 2562498625624795262.pem
-rw-------. 1 root root 1675 May 15  2017 5535330868310689431-key.pem
-rw-r--r--. 1 root root 2184 May 15  2017 5535330868310689431.pem
-rw-------. 1 root root 1675 May 15  2017 6664802491135239005-key.pem
-rw-r--r--. 1 root root 2473 May 15  2017 6664802491135239005.pem
-rw-------. 1 root root 1675 May 15  2017 8725765943167156017-key.pem
-rw-r--r--. 1 root root 2160 May 15  2017 8725765943167156017.pem
-rw-------. 1 root root 1675 May 15  2017 9178336430764673988-key.pem
-rw-r--r--. 1 root root 2574 May 15  2017 9178336430764673988.pem
[root@dhcp35-238 entitlement]# rct cc 2562498625624795262.pem --no-product --no-content

+-------------------------------------------+
	Entitlement Certificate
+-------------------------------------------+

Certificate:
	Path: 2562498625624795262.pem
	Version: 3.3
	Serial: 2562498625624795262
	Start Date: 2017-05-15 10:55:47+00:00
	End Date: 2018-05-15 10:55:47+00:00
	Pool ID: Not Available

Subject:
	CN: a909341d-d143-478c-b02b-c3cb1946f337
	O: snowwhite

Issuer:
	C: US
	CN: F21-candlepin.usersys.redhat.com
	L: Raleigh
Order:
	Name: Content Access
	Number: 
	SKU: content_access
	Contract: 
	Account: 
	Service Level: 
	Service Type: 
	Quantity: 
	Quantity Used: 1
	Socket Limit: 
	RAM Limit: 
	Core Limit: 
	Virt Only: False
	Stacking ID: 
	Warning Period: 0
	Provides Management: False

3.Now adjust the clock on both candlepin and client machine  to past "2018-05-15 ' date 

4. check the list consumed 

-rw-r--r--. 1 root root 2574 May 15  2017 9178336430764673988.pem
[root@dhcp35-238 entitlement]# subscription-manager list --consumed
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name:   Awesome OS OSTree
Provides:            Awesome OS OStree Bits
SKU:                 awesomeos-ostree
Contract:            0
Account:             12331131231
Serial:              8725765943167156017
Pool ID:             8ac6a3635c083b4e015c083d138804ef
Provides Management: No
Active:              False
Quantity Used:       1
Service Level:       
Service Type:        
Status Details:      Subscription is expired
Subscription Type:   
Starts:              05/15/2017
Ends:                05/15/2018
System Type:         Physical

Subscription Name:   Awesome OS Instance Based (Standard Support)
Provides:            Awesome OS Instance Server Bits
SKU:                 awesomeos-instancebased
Contract:            1
Account:             12331131231
Serial:              6664802491135239005
Pool ID:             8ac6a3635c083b4e015c083d1f23073a
Provides Management: No
Active:              False
Quantity Used:       1
Service Level:       Standard
Service Type:        L1-L3
Status Details:      Subscription is expired
Subscription Type:   
Starts:              05/15/2017
Ends:                05/16/2017
System Type:         Virtual

Subscription Name:   Awesome OS Server Bundled
Provides:            Load Balancing Bits
                     Awesome OS Server Bits
                     Clustering Bits
                     Shared Storage Bits
                     Large File Support Bits
                     Management Bits
SKU:                 awesomeos-server
Contract:            1
Account:             12331131231
Serial:              1123175916477564753
Pool ID:             8ac6a3635c083b4e015c083d1972061e
Provides Management: Yes
Active:              False
Quantity Used:       1
Service Level:       Premium
Service Type:        Level 3
Status Details:      Subscription is expired
Subscription Type:   
Starts:              05/15/2017
Ends:                05/15/2018
System Type:         Physical

Subscription Name:   Awesome OS Server Basic (dc-virt)
Provides:            Awesome OS Modifier Bits
                     Awesome OS Server Bits
SKU:                 awesomeos-server-basic-vdc
Contract:            0
Account:             12331131231
Serial:              9178336430764673988
Pool ID:             8ac6a3635c083b4e015c083d11de0495
Provides Management: No
Active:              False
Quantity Used:       1
Service Level:       Full-Service
Service Type:        Drive-Through
Status Details:      Subscription is expired
Subscription Type:   
Starts:              05/15/2017
Ends:                05/16/2017
System Type:         Virtual

Subscription Name:   Awesome OS Docker
Provides:            Awesome OS Docker Bits
SKU:                 awesomeos-docker
Contract:            1
Account:             12331131231
Serial:              5535330868310689431
Pool ID:             8ac6a3635c083b4e015c083d15b30565
Provides Management: No
Active:              False
Quantity Used:       1
Service Level:       
Service Type:        
Status Details:      Subscription is expired
Subscription Type:   
Starts:              05/15/2017
Ends:                05/15/2018
System Type:         Physical

^^ NOTICE all subscriptions are expired along with the golden ticket subscription

5.Wait for "certcheck' to run 

6.# tail -f /var/log/rhsm/rhsmcertd.log

Wed May 16 00:01:17 2018 [INFO] (Cert Check) Certificates updated.

7: All the expired certs are removed except the "golden ticket "subscription

[root@dhcp35-238 entitlement]# ll
total 16
-rw-------. 1 root root 1675 May 16 00:01 1752743130901370480-key.pem
-rw-r--r--. 1 root root 2878 May 16 00:01 1752743130901370480.pem
-rw-------. 1 root root 1675 May 15  2017 2562498625624795262-key.pem
-rw-r--r--. 1 root root 2878 May 15  2017 2562498625624795262.pem

[root@dhcp35-238 entitlement]# rct cc 2562498625624795262.pem --no-product --no-content

+-------------------------------------------+
	Entitlement Certificate
+-------------------------------------------+

Certificate:
	Path: 2562498625624795262.pem
	Version: 3.3
	Serial: 2562498625624795262
	Start Date: 2017-05-15 10:55:47+00:00
	End Date: 2018-05-15 10:55:47+00:00-->> Expired golden ticket subsscription
	Pool ID: Not Available

Subject:
	CN: a909341d-d143-478c-b02b-c3cb1946f337
	O: snowwhite

Issuer:
	C: US
	CN: F21-candlepin.usersys.redhat.com
	L: Raleigh


Order:
	Name: Content Access
	Number: 
	SKU: content_access
	Contract: 
	Account: 
	Service Level: 
	Service Type: 
	Quantity: 
	Quantity Used: 1
	Socket Limit: 
	RAM Limit: 
	Core Limit: 
	Virt Only: False
	Stacking ID: 
	Warning Period: 0
	Provides Management: False


[root@dhcp35-238 entitlement]# rct cc 1752743130901370480.pem --no-product --no-content

+-------------------------------------------+
	Entitlement Certificate
+-------------------------------------------+

Certificate:
	Path: 1752743130901370480.pem
	Version: 3.3
	Serial: 1752743130901370480
	Start Date: 2018-05-15 17:31:19+00:00
	End Date: 2019-05-15 17:31:19+00:00  --->> new subscription
	Pool ID: Not Available

Subject:
	CN: a909341d-d143-478c-b02b-c3cb1946f337
	O: snowwhite

Issuer:
	C: US
	CN: F21-candlepin.usersys.redhat.com
	L: Raleigh


Order:
	Name: Content Access
	Number: 
	SKU: content_access
	Contract: 
	Account: 
	Service Level: 
	Service Type: 
	Quantity: 
	Quantity Used: 1
	Socket Limit: 
	RAM Limit: 
	Core Limit: 
	Virt Only: False
	Stacking ID: 
	Warning Period: 0
	Provides Management: False


Actual results:
Certcheck process didnot remove the expired golden ticket subscriptions from the system

Expected results:
As per the approved test case , certcheck should remove the expired golden ticket subscription

Additional info:

Comment 1 Shwetha Kallesh 2017-05-15 12:49:44 UTC
Additional info from my observations:

Certificate remains on the system but repos are not available

[root@dhcp35-160 ~]# subscription-manager register --org snowwhite
Registering to: Shwetha-candlepin.usersys.redhat.com:8443/candlepin
Username: admin
Password: 
The system has been registered with ID: 15dee85a-6cec-4f1a-902b-caba420a0bd8 

[root@dhcp35-160 ~]# rct cc /etc/pki/entitlement/4074590224270831742.pem 

+-------------------------------------------+
	Entitlement Certificate
+-------------------------------------------+

Certificate:
	Path: /etc/pki/entitlement/4074590224270831742.pem
	Version: 3.3
	Serial: 4074590224270831742
	Start Date: 2017-05-15 10:30:02+00:00
	End Date: 2018-05-15 10:30:02+00:00
	Pool ID: Not Available

Subject:
	CN: 1f74def2-ad3e-4f00-91db-b3ddcbf288a3
	O: snowwhite

Issuer:
	C: US
	CN: Shwetha-candlepin.usersys.redhat.com
	L: Raleigh

Product:
	ID: content_access
	Name:  Content Access
	Version: 
	Arch: 
	Tags: 
	Brand Type: 
	Brand Name: 

Order:
	Name: Content Access
	Number: 
	SKU: content_access
	Contract: 
	Account: 
	Service Level: 
	Service Type: 
	Quantity: 
	Quantity Used: 1
	Socket Limit: 
	RAM Limit: 
	Core Limit: 
	Virt Only: False
	Stacking ID: 
	Warning Period: 0
	Provides Management: False

Content:
	Type: yum
	Name: awesomeos
	Label: awesomeos
	Vendor: Red Hat
	URL: /snowwhite/path/to/$basearch/$releasever/awesomeos
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-all
	Label: awesomeos-all
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/all
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: containerImage
	Name: awesomeos-docker-images
	Label: awesomeos-docker-images
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos-docker-images
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-i386
	Label: awesomeos-i386
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/i386
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-i386-only-content
	Label: awesomeos-i386-only-content
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/all
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: i386

Content:
	Type: yum
	Name: awesomeos-i686
	Label: awesomeos-i686
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/i686
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-ia64
	Label: awesomeos-ia64
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/ia64
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-ia64-only-content
	Label: awesomeos-ia64-only-content
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/all
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: ia64

Content:
	Type: ostree
	Name: awesomeos-ostree
	Label: awesomeos-ostree
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos-ostree
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-ppc
	Label: awesomeos-ppc
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/ppc
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-ppc-only-content
	Label: awesomeos-ppc-only-content
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/all
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: ppc

Content:
	Type: yum
	Name: awesomeos-ppc64
	Label: awesomeos-ppc64
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/ppc64
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-ppc64-only-content
	Label: awesomeos-ppc64-only-content
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/all
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: ppc64

Content:
	Type: yum
	Name: awesomeos-s390x
	Label: awesomeos-s390x
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/s390x
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-s390x-only-content
	Label: awesomeos-s390x-only-content
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/all
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: s390x

Content:
	Type: yum
	Name: awesomeos-x86
	Label: awesomeos-x86
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/$releasever/x86
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-x86_64
	Label: awesomeos-x86_64
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/x86_64
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 
[root@dhcp35-160 ~]# subscription-manager repos --list
+----------------------------------------------------------+
    Available Repositories in /etc/yum.repos.d/redhat.repo
+----------------------------------------------------------+
Repo ID:   awesomeos-ia64-only-content
Repo Name: awesomeos-ia64-only-content
Repo URL:  https://cdn.redhat.com/snowwhite/path/to/awesomeos/all
Enabled:   0

Repo ID:   awesomeos-i386-only-content
Repo Name: awesomeos-i386-only-content
Repo URL:  https://cdn.redhat.com/snowwhite/path/to/awesomeos/all
Enabled:   0

Repo ID:   snowy-content-label
Repo Name: snowy-content
Repo URL:  https://cdn.redhat.com/snowwhite/snowy/foo/path
Enabled:   0

Repo ID:   never-enabled-content
Repo Name: never-enabled-content
Repo URL:  https://cdn.redhat.com/snowwhite/foo/path/never
Enabled:   0

Repo ID:   awesomeos-all
Repo Name: awesomeos-all
Repo URL:  https://cdn.redhat.com/snowwhite/path/to/awesomeos/all
Enabled:   0

Repo ID:   awesomeos-ppc-only-content
Repo Name: awesomeos-ppc-only-content
Repo URL:  https://cdn.redhat.com/snowwhite/path/to/awesomeos/all
Enabled:   0


[root@dhcp35-160 ~]# date -s "2018-05-15 10:30:02+00:00"
Tue May 15 16:00:02 IST 2018

Date set on server:
[root@Shwetha-candlepin server]# date -s "2018-05-15 10:30:02+00:00"
Tue May 15 16:00:02 IST 2018

[root@dhcp35-160 ~]# ls /etc/pki/entitlement/4074590224270831742
4074590224270831742-key.pem  4074590224270831742.pem  

[root@dhcp35-160 ~]# rct cat-cert /etc/pki/entitlement/4074590224270831742.pem 

+-------------------------------------------+
	Entitlement Certificate
+-------------------------------------------+

Certificate:
	Path: /etc/pki/entitlement/4074590224270831742.pem
	Version: 3.3
	Serial: 4074590224270831742
	Start Date: 2017-05-15 10:30:02+00:00
	End Date: 2018-05-15 10:30:02+00:00
	Pool ID: Not Available

Subject:
	CN: 1f74def2-ad3e-4f00-91db-b3ddcbf288a3
	O: snowwhite

Issuer:
	C: US
	CN: Shwetha-candlepin.usersys.redhat.com
	L: Raleigh

Product:
	ID: content_access
	Name:  Content Access
	Version: 
	Arch: 
	Tags: 
	Brand Type: 
	Brand Name: 

Order:
	Name: Content Access
	Number: 
	SKU: content_access
	Contract: 
	Account: 
	Service Level: 
	Service Type: 
	Quantity: 
	Quantity Used: 1
	Socket Limit: 
	RAM Limit: 
	Core Limit: 
	Virt Only: False
	Stacking ID: 
	Warning Period: 0
	Provides Management: False

Content:
	Type: yum
	Name: awesomeos
	Label: awesomeos
	Vendor: Red Hat
	URL: /snowwhite/path/to/$basearch/$releasever/awesomeos
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-all
	Label: awesomeos-all
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/all
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: containerImage
	Name: awesomeos-docker-images
	Label: awesomeos-docker-images
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos-docker-images
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-i386
	Label: awesomeos-i386
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/i386
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-i386-only-content
	Label: awesomeos-i386-only-content
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/all
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: i386

Content:
	Type: yum
	Name: awesomeos-i686
	Label: awesomeos-i686
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/i686
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-ia64
	Label: awesomeos-ia64
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/ia64
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-ia64-only-content
	Label: awesomeos-ia64-only-content
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/all
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: ia64

Content:
	Type: ostree
	Name: awesomeos-ostree
	Label: awesomeos-ostree
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos-ostree
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 
	Required Tags: 
	Arches: 

Content:
	Type: yum
	Name: awesomeos-ppc
	Label: awesomeos-ppc
	Vendor: Red Hat
	URL: /snowwhite/path/to/awesomeos/ppc
	GPG: /path/to/awesomeos/gpg/
	Enabled: False
	Expires: 3600
	Required Tags: 
	Arches: 

    
[root@dhcp35-160 ~]# subscription-manager repos --list
This system has no repositories available through subscriptions.


[root@dhcp35-160 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: Unknown
subscription management rules: Unknown
subscription-manager: 1.20.0-1.git.5.37404a3.el7
python-rhsm: 1.20.0-1.git.5.37404a3.el7

Comment 2 Jiri Hnidek 2017-05-22 10:59:02 UTC
I can confirm, that this PR: https://github.com/candlepin/subscription-manager/pull/1630 fixes this bug too.

Comment 3 Jiri Hnidek 2017-05-22 11:03:08 UTC

*** This bug has been marked as a duplicate of bug 1450862 ***