Bug 1886772

Summary:	Subscription manager doesn't remove the SCA entitlement certificate when switching back to Entitlement mode
Product:	Red Hat Enterprise Linux 7	Reporter:	Hao Chang Yu <hyu>
Component:	subscription-manager	Assignee:	Chris Snyder <csnyder>
Status:	CLOSED ERRATA	QA Contact:	Red Hat subscription-manager QE Team <rhsm-qe>
Severity:	medium	Docs Contact:
Priority:	high
Version:	7.9	CC:	ahumbe, cdonnell, csnyder, jhnidek, jreznik, jsefler, nmoumoul, redakkan, rkarimpa, skallesh, wclark, yanpliu
Target Milestone:	rc	Keywords:	Triaged, ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	subscription-manager-1.24.48-1.el7_9	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1951057 (view as bug list)		Environment:
Last Closed:	2021-04-27 11:35:33 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Hao Chang Yu 2020-10-09 11:38:31 UTC

Description of problem:
If user switches back to Entitlement mode from the SCA mode, the subscription-manager doesn't remove the local SCA entitlement certificate.

Steps to Reproduce:
1) While SCA is disabled in the Satellite, check the client status.

# subscription-manager status
+-------------------------------------------+
System Status Details
+-------------------------------------------+
Overall Status: Current

2) Enable SCA in the Satellite

3) From the client, run refresh to download the SCA certificate

# subscription-manager refresh
1 local certificate has been deleted.
All local data refreshed

4) Check the client status

# subscription-manager status
+-------------------------------------------+
System Status Details
+-------------------------------------------+
Overall Status: Disabled
Content Access Mode is set to Organization/Environment Access. This host has access to content, regardless of subscription status.

5) Disable SCA in the Satellite

6) Check the client status again

# subscription-manager status
+-------------------------------------------+
System Status Details
+-------------------------------------------+
Overall Status: Current
Content Access Mode is set to Organization/Environment Access. This host has access to content, regardless of subscription status. <===== SCA message is still showing

# ls -lrt /etc/pki/entitlement/
4571984898862286599.pem
4571984898862286599-key.pem
3859574054200299686-key.pem
3859574054200299686.pem <======= SCA certificate file not deleted

# subscription-manager refresh
1 local certificate has been deleted.
All local data refreshed

# subscription-manager status
+-------------------------------------------+
System Status Details
+-------------------------------------------+
Overall Status: Current
Content Access Mode is set to Organization/Environment Access. This host has access to content, regardless of subscription status. <==== Symptom persist after refresh

Comment 2 Rehana 2020-10-09 14:56:30 UTC

Hi Hao, 

Can you please try again with subscription-manager refresh --force. We have changed the behaviour of subscription-manager refresh command in RHEL 79. 

If the above does not work either, can you please try deleting `/var/lib/rhsm/cache/content_access_mode.json` the file .  Please lets know your observations.

thanks,
Rehana

Comment 3 Hao Chang Yu 2020-10-12 05:25:42 UTC

Hi Rehana

Issue still persist after running refresh with "--force" and after deleting /var/lib/rhsm/cache/content_access_mode.json.


# subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Current
Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.

# subscription-manager refresh --force
1 local certificate has been deleted.
All local data refreshed

# ls -lrt
total 16
-rw-r--r--. 1 root root 3243 Oct 12 13:46 414348676818798724-key.pem  <===== SCA cert and key still not deleted
-rw-r--r--. 1 root root 2907 Oct 12 13:47 414348676818798724.pem
-rw-r--r--. 1 root root 3927 Oct 12 15:14 3333899503616676628.pem
-rw-r--r--. 1 root root 3243 Oct 12 15:14 3333899503616676628-key.pem

# subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Current
Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.


# rm /var/lib/rhsm/cache/content_access_mode.json 
rm: remove regular file ‘/var/lib/rhsm/cache/content_access_mode.json’? y

# ls -lrt /var/lib/rhsm/cache/
total 92
-rw-r--r--. 1 root root   164 Oct 12 12:32 installed_products.json
-rw-r--r--. 1 root root  1641 Oct 12 12:33 supported_resources.json
-rw-r--r--. 1 root root 54836 Oct 12 12:33 profile.json
-rw-r--r--. 1 root root     2 Oct 12 15:16 content_overrides.json
-rw-r--r--. 1 root root     2 Oct 12 15:16 written_overrides.json
-rw-r--r--. 1 root root     1 Oct 12 15:16 rhsm_icon.json
-rw-r--r--. 1 root root  5146 Oct 12 15:17 entitlement_status.json
-rw-r--r--. 1 root root    83 Oct 12 15:18 syspurpose.json
-rw-r--r--. 1 root root   287 Oct 12 15:18 syspurpose_compliance_status.json

# subscription-manager refresh --force
1 local certificate has been deleted.
All local data refreshed

# ls -lrt
total 16
-rw-r--r--. 1 root root 3243 Oct 12 13:46 414348676818798724-key.pem <===== SCA cert and key still not deleted
-rw-r--r--. 1 root root 2907 Oct 12 13:47 414348676818798724.pem
-rw-r--r--. 1 root root 3243 Oct 12 15:19 5244168280102514653-key.pem
-rw-r--r--. 1 root root 3927 Oct 12 15:19 5244168280102514653.pem

# subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Current
Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.

Comment 4 Craig Donnelly 2020-10-26 21:02:11 UTC

Hao,

Was the manifest on the Satellite refreshed after SCA was disabled in the customer portal for the manifest?

Please detail the entire process used.

Thanks.

Comment 5 Craig Donnelly 2020-10-28 15:32:18 UTC

Hao, please also provide the version of Satellite that is being used in this scenario.

Thus far, I have no reproduced your issue with the cert not being removed.

I have however found a separate issue with the cache not being properly cleared, leaving subscription-manager in a position of reporting that it is in SCA mode when it is not.

Comment 6 Hao Chang Yu 2020-10-29 07:04:49 UTC

(In reply to Craig Donnelly from comment #5)
> Hao, please also provide the version of Satellite that is being used in this
> scenario.

Hi Craig

It is Satelite 6.7.3.

> 
> Thus far, I have no reproduced your issue with the cert not being removed.

Make sure subscription-manager has downloaded the SCA cert by running "subscription-manager refresh --force" (Step 2 and 3 in comment #3) before switching back to entitlement mode.

> 
> I have however found a separate issue with the cache not being properly
> cleared, leaving subscription-manager in a position of reporting that it is
> in SCA mode when it is not.

Comment 7 Craig Donnelly 2020-11-03 16:48:09 UTC

Hao,

I attempted to reproduce this with your directions against Satellite 6.6.3 + 6.8 GA, with RHEL 7.9 (subscription-manager-1.24.42-1.el7.x86_64).

Everytime I refresh in any capacity after turning off SCA for the manifest a refreshing, the content access cert/entitlement is removed from the system.

The only error I am encountering is a failure to properly clean the cache for sub-man, which results in an incorrect response from `subscription-manager status` in regards to being in SCA state.

Do you have a reproduced environment available for this?

Comment 8 Jonathon Turel 2020-11-05 16:55:10 UTC

*** Bug 1882548 has been marked as a duplicate of this bug. ***

Comment 41 John Sefler 2021-04-15 18:06:52 UTC

[root@hp-z600-02 ~]# rpm -q subscription-manager
subscription-manager-1.24.48-1.el7_9.x86_64
[root@hp-z600-02 ~]# 
[root@hp-z600-02 ~]# rpm -q subscription-manager --changelog | head
* Thu Apr 15 2021 Christopher Snyder <csnyder> 1.24.48-1
- 1886772: check is_consumer_cert_key_valid (csnyder)

* Wed Apr 14 2021 Christopher Snyder <csnyder> 1.24.47-1
- 1886772: Clear content access mode cache on refresh (csnyder)

* Tue Apr 06 2021 Christopher Snyder <csnyder> 1.24.46-1
- 1896715: Set proper read permissions on certs (#2466) (wpoteat)
- 1935592: Fix getting releases, when SCA is used (jhnidek)

Comment 46 John Sefler 2021-04-16 13:52:48 UTC

Final VERIFICATION against build subscription-manager-1.24.48-1.el7_9 ...

[root@hp-z600-02 ~]# rpm -q subscription-manager
subscription-manager-1.24.48-1.el7_9.x86_64

[root@hp-z600-02 ~]# rpm -q subscription-manager --changelog | head
* Thu Apr 15 2021 Christopher Snyder <csnyder> 1.24.48-1
- 1886772: check is_consumer_cert_key_valid (csnyder)

* Wed Apr 14 2021 Christopher Snyder <csnyder> 1.24.47-1
- 1886772: Clear content access mode cache on refresh (csnyder)

* Tue Apr 06 2021 Christopher Snyder <csnyder> 1.24.46-1
- 1896715: Set proper read permissions on certs (#2466) (wpoteat)
- 1935592: Fix getting releases, when SCA is used (jhnidek)

[root@hp-z600-02 ~]# subscription-manager config --server.hostname=subscription.rhsm.stage.redhat.com
[root@hp-z600-02 ~]# 
[root@hp-z600-02 ~]# subscription-manager config --logging.default_log_level=DEBUG
[root@hp-z600-02 ~]# 
[root@hp-z600-02 ~]# curl --stderr /dev/null -X PUT -k -u jsefler_sca_testuser1:REDACTED  -d '{"contentAccessMode":"org_environment"}'  -H "Content-Type: application/json"  "https://subscription.rhsm.stage.redhat.com:443/candlepin/owners/13194530" | python -mjson.tool
{
    "autobindDisabled": false,
    "autobindHypervisorDisabled": false,
    "contentAccessMode": "org_environment",
    "contentAccessModeList": "entitlement,org_environment",
    "contentPrefix": null,
    "created": "2021-04-15T18:15:57+0000",
    "defaultServiceLevel": null,
    "displayName": "13194530",
    "href": "/owners/13194530",
    "id": "8a99f9aa78c68c380178d6bf30b76f9d",
    "key": "13194530",
    "lastRefreshed": "2021-04-15T18:18:59+0000",
    "logLevel": null,
    "parentOwner": null,
    "updated": "2021-04-15T18:30:54+0000",
    "upstreamConsumer": null
}
[root@hp-z600-02 ~]# 
[root@hp-z600-02 ~]# subscription-manager register
Registering to: subscription.rhsm.stage.redhat.com:443/subscription
Username: jsefler_sca_testuser1
Password: 
The system has been registered with ID: 4f1b1d7b-adbc-487d-9ce3-96b416fa4c60
The registered system name is: hp-z600-02.ml3.eng.bos.redhat.com
[root@hp-z600-02 ~]#
[root@hp-z600-02 ~]# truncate --size=0 /var/log/rhsm/rhsm.log 
[root@hp-z600-02 ~]#
[root@hp-z600-02 ~]# subscription-manager status; subscription-manager status; subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Disabled
Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.

System Purpose Status: Disabled

+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Disabled
Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.

System Purpose Status: Disabled

+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Disabled
Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.

System Purpose Status: Disabled

[root@hp-z600-02 ~]# egrep "GET.*/owner|cache/content_access_mode" /var/log/rhsm/rhsm.log 
2021-04-15 14:40:27,792 [DEBUG] subscription-manager:27826:MainThread @cache.py:890 - Trying to read ContentAccessModeCache from cache file /var/lib/rhsm/cache/content_access_mode.json
2021-04-15 14:40:27,792 [DEBUG] subscription-manager:27826:MainThread @cache.py:896 - Identity of system has changed. The cache file: /var/lib/rhsm/cache/content_access_mode.json is obsolete
2021-04-15 14:40:27,794 [DEBUG] subscription-manager:27826:MainThread @connection.py:572 - Making request: GET /subscription/consumers/4f1b1d7b-adbc-487d-9ce3-96b416fa4c60/owner
2021-04-15 14:40:28,305 [DEBUG] subscription-manager:27826:MainThread @connection.py:622 - Response: status=200, requestUuid=657f426e-cf0e-4ce3-ac9f-8636abfb712f, request="GET /subscription/consumers/4f1b1d7b-adbc-487d-9ce3-96b416fa4c60/owner"
2021-04-15 14:40:28,305 [DEBUG] subscription-manager:27826:MainThread @cache.py:119 - Wrote cache: /var/lib/rhsm/cache/content_access_mode.json
2021-04-15 14:40:30,889 [DEBUG] subscription-manager:27855:MainThread @cache.py:890 - Trying to read ContentAccessModeCache from cache file /var/lib/rhsm/cache/content_access_mode.json
2021-04-15 14:40:30,889 [DEBUG] subscription-manager:27855:MainThread @cache.py:900 - Data loaded from cache file: /var/lib/rhsm/cache/content_access_mode.json
2021-04-15 14:40:34,132 [DEBUG] subscription-manager:27873:MainThread @cache.py:890 - Trying to read ContentAccessModeCache from cache file /var/lib/rhsm/cache/content_access_mode.json
2021-04-15 14:40:34,133 [DEBUG] subscription-manager:27873:MainThread @cache.py:900 - Data loaded from cache file: /var/lib/rhsm/cache/content_access_mode.json
[root@hp-z600-02 ~]# 



VERIFIED: The logging above indicates that the "Identity of system has changed" causing cache/content_access_mode.json to be obsolete and a new GET from /owner was performed which provided new cache and the subsequent two calls to "subscription-manager status" read ContentAccessModeCache from cache.

NEXT: Let's change contentAccessMode back to "entitlement" at the server and verify the original bug comment 0


[root@hp-z600-02 ~]# ls /etc/pki/entitlement/
4858680105111917-key.pem  4858680105111917.pem
[root@hp-z600-02 ~]# 
[root@hp-z600-02 ~]# rct cat-cert /etc/pki/entitlement/4858680105111917.pem | grep "Product:" -A2
Product:
	ID: content_access
	Name:  Content Access
[root@hp-z600-02 ~]# 
[root@hp-z600-02 ~]# curl --stderr /dev/null -X PUT -k -u jsefler_sca_testuser1:REDACTED  -d '{"contentAccessMode":"entitlement"}'  -H "Content-Type: application/json"  "https://subscription.rhsm.stage.redhat.com:443/candlepin/owners/13194530" | python -mjson.tool
{
    "autobindDisabled": false,
    "autobindHypervisorDisabled": false,
    "contentAccessMode": "entitlement",
    "contentAccessModeList": "entitlement,org_environment",
    "contentPrefix": null,
    "created": "2021-04-15T18:15:57+0000",
    "defaultServiceLevel": null,
    "displayName": "13194530",
    "href": "/owners/13194530",
    "id": "8a99f9aa78c68c380178d6bf30b76f9d",
    "key": "13194530",
    "lastRefreshed": "2021-04-15T18:18:59+0000",
    "logLevel": null,
    "parentOwner": null,
    "updated": "2021-04-15T19:39:19+0000",
    "upstreamConsumer": null
}
[root@hp-z600-02 ~]# 
[root@hp-z600-02 ~]# truncate --size=0 /var/log/rhsm/rhsm.log
[root@hp-z600-02 ~]# 
[root@hp-z600-02 ~]# subscription-manager refresh
1 local certificate has been deleted.
All local data refreshed
[root@hp-z600-02 ~]# 
[root@hp-z600-02 ~]# ls /etc/pki/entitlement/
[root@hp-z600-02 ~]# 
[root@hp-z600-02 ~]# subscription-manager status; subscription-manager status; subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Invalid

Red Hat Enterprise Linux Server:
- Not supported by a valid subscription.

System Purpose Status: Not Specified

+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Invalid

Red Hat Enterprise Linux Server:
- Not supported by a valid subscription.

System Purpose Status: Not Specified

+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Invalid

Red Hat Enterprise Linux Server:
- Not supported by a valid subscription.

System Purpose Status: Not Specified

[root@hp-z600-02 ~]# egrep "GET.*/owner|cache/content_access_mode" /var/log/rhsm/rhsm.log
2021-04-15 15:41:19,434 [DEBUG] subscription-manager:32654:MainThread @cache.py:92 - Deleting cache: /var/lib/rhsm/cache/content_access_mode.json
2021-04-15 15:41:48,846 [DEBUG] subscription-manager:32701:MainThread @cache.py:890 - Trying to read ContentAccessModeCache from cache file /var/lib/rhsm/cache/content_access_mode.json
2021-04-15 15:41:48,846 [DEBUG] subscription-manager:32701:MainThread @cache.py:151 - Cache file /var/lib/rhsm/cache/content_access_mode.json does not exist
2021-04-15 15:41:48,848 [DEBUG] subscription-manager:32701:MainThread @connection.py:572 - Making request: GET /subscription/consumers/4f1b1d7b-adbc-487d-9ce3-96b416fa4c60/owner
2021-04-15 15:41:49,536 [DEBUG] subscription-manager:32701:MainThread @connection.py:622 - Response: status=200, requestUuid=a1ea1428-1b2c-46f7-92db-4b6d99a6c17d, request="GET /subscription/consumers/4f1b1d7b-adbc-487d-9ce3-96b416fa4c60/owner"
2021-04-15 15:41:49,537 [DEBUG] subscription-manager:32701:MainThread @cache.py:119 - Wrote cache: /var/lib/rhsm/cache/content_access_mode.json
2021-04-15 15:41:52,508 [DEBUG] subscription-manager:32732:MainThread @cache.py:890 - Trying to read ContentAccessModeCache from cache file /var/lib/rhsm/cache/content_access_mode.json
2021-04-15 15:41:52,509 [DEBUG] subscription-manager:32732:MainThread @cache.py:900 - Data loaded from cache file: /var/lib/rhsm/cache/content_access_mode.json
2021-04-15 15:41:55,351 [DEBUG] subscription-manager:32749:MainThread @cache.py:890 - Trying to read ContentAccessModeCache from cache file /var/lib/rhsm/cache/content_access_mode.json
2021-04-15 15:41:55,352 [DEBUG] subscription-manager:32749:MainThread @cache.py:900 - Data loaded from cache file: /var/lib/rhsm/cache/content_access_mode.json
[root@hp-z600-02 ~]# 


VERIFIED: After changing the contentAccessMode from "org_environment" mode to "entitlement" mode and calling "subscription-manager refresh", the ContentAccessModeCache is deleted and replaced by a new call to GET /owner which is used in subsequent calls to "subscription-manager status".

Moving to VERIFIED.

Comment 50 errata-xmlrpc 2021-04-27 11:35:33 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (subscription-manager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1394

Comment 51 Nikos Moumoulidis 2021-05-18 15:07:55 UTC

*** Bug 1960220 has been marked as a duplicate of this bug. ***