Bug 2095301

Summary: RFE: For dnf operations against Red Hat CDN, enable OCSP stapling verification [rhel-9.0.0.z]
Product: Red Hat Enterprise Linux 9 Reporter: RHEL Program Management Team <pgm-rhel-tools>
Component: subscription-managerAssignee: Pino Toscano <ptoscano>
Status: CLOSED ERRATA QA Contact: Red Hat subscription-manager QE Team <rhsm-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0CC: arpandey, candlepin-bugs, cdonnell, jpazdziora, jsefler, kanderso, redakkan, zpetrace
Target Milestone: rcKeywords: FutureFeature, Triaged, ZStream
Target Release: 9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: subscription-manager-1.29.26.1-1.el9_0 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: 2075455 Environment:
Last Closed: 2022-08-09 10:30:43 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2075455    
Bug Blocks:    

Comment 1 Zdenek Petracek 2022-06-21 13:01:30 UTC
Pre-verification:

Scenario1:Verify the "sslverifystatus" is added to each repos in /etc/yum.repos.d/redhat.repo when candlepin supports the "ssl_verify_status" capability	

Version:
[root@kvm-02-guest24 ~]# subscription-manager version
server type: This system is currently not registered.
subscription management server: 3.2.22-1
subscription management rules: 5.41
subscription-manager: 1.29.26+3.g56568472e-1.git.0.df26db4

copying certs:
[root@newcandlepin ~]# scp /home/candlepin/candlepin/generated_certs/3* kvm-02-guest24.rhts.eng.brq.redhat.com:/etc/pki/product/
The authenticity of host 'kvm-02-guest24.rhts.eng.brq.redhat.com (10.37.153.98)' can't be established.
ECDSA key fingerprint is SHA256:4ApbYwams2kpQrLoC22WgcLsApvLGlAkpccDJbCnU1Q.
ECDSA key fingerprint is MD5:0b:f0:5e:ec:e7:69:61:08:e1:d9:e8:ed:00:07:6a:4b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'kvm-02-guest24.rhts.eng.brq.redhat.com,10.37.153.98' (ECDSA) to the list of known hosts.
root.eng.brq.redhat.com's password: 
32060.pem                                                                                                                               100% 2090    11.2KB/s   00:00    
37060.pem                                                                                                                               100% 2078    11.5KB/s   00:00    
37062.pem                                                                                                                               100% 2098    12.7KB/s   00:00    
37065.pem                                                                                                                               100% 2082    11.2KB/s   00:00    
37067.pem                                                                                                                               100% 2090    12.7KB/s   00:00    
37068.pem                                                                                                                               100% 2094    12.7KB/s   00:00    
37069.pem                                                                                                                               100% 2082    12.6KB/s   00:00    
37070.pem                                                                                                                               100% 2090    12.7KB/s   00:00    
37080.pem                                                                                                                               100% 2078    12.6KB/s   00:00    
37090.pem                                                                                                                               100% 2074    12.6KB/s   00:00    
37091.pem                                                                                                                               100% 2074    12.5KB/s   00:00    
38070.pem                                                                                                                               100% 2074    12.6KB/s   00:00    
38072.pem                                                                                                                               100% 2061    12.5KB/s   00:00    

[root@newcandlepin ~]# scp /etc/candlepin/certs/candlepin-ca.crt kvm-02-guest24.rhts.eng.brq.redhat.com:/etc/rhsm/ca/candlepin-ca.pem
root.eng.brq.redhat.com's password: 
candlepin-ca.crt                                                                                                                        100% 2029    12.0KB/s   00:00    

registering to candlepin server:
[root@kvm-02-guest24 ~]# subscription-manager register
Registering to: 10.70.35.79:8443/candlepin
Username: admin
Password: 
Hint: User "admin" is member of following organizations: admin, snowwhite, donaldduck
Organization: snowwhite
The system has been registered with ID: f23d316c-2e91-4cb9-ba3e-1841b5659e68
The registered system name is: kvm-02-guest24.rhts.eng.brq.redhat.com

[root@kvm-02-guest24 ~]# subscription-manager attach --auto
Installed Product Current Status:
Product Name: Awesome OS Instance Server Bits
Status:       Subscribed

Product Name: Awesome OS Server Bits
Status:       Subscribed
.
.
.

[root@kvm-02-guest24 ~]# cat /etc/yum.repos.d/redhat.repo
#
# Certificate-Based Repositories
# Managed by (rhsm) subscription-manager
#
# *** This file is auto-generated.  Changes made here will be over-written. ***
# *** Use "subscription-manager repo-override --help" if you wish to make changes. ***
#
# If this file is empty and this system is subscribed consider
# a "yum repolist" to refresh available repos
#

[content-label-no-gpg-37060]
name = content-nogpg-37060
baseurl = https://cdn.redhat.com/foo/path/no_gpg/37060234
enabled = 0
gpgcheck = 0
sslverify = 1
sslcacert = /etc/rhsm/ca/redhat-uep.pem
sslclientkey = /etc/pki/entitlement/7691419403284580053-key.pem
sslclientcert = /etc/pki/entitlement/7691419403284580053.pem
sslverifystatus = 1
enabled_metadata = 0
^ sslverifystatus is set to 1 --> PASSED

Scenario2:Verify after revoking the OCSP stapling capabilities from candlepin, the redhat.repo file is refreshed and "sslverifystatus'is removed from the repo details section	
	
copying certs:
[root@newcandlepin ~]# scp /home/candlepin/candlepin/generated_certs/3* kvm-02-guest24.rhts.eng.brq.redhat.com:/etc/pki/product/
The authenticity of host 'kvm-02-guest24.rhts.eng.brq.redhat.com (10.37.153.98)' can't be established.
ECDSA key fingerprint is SHA256:4ApbYwams2kpQrLoC22WgcLsApvLGlAkpccDJbCnU1Q.
ECDSA key fingerprint is MD5:0b:f0:5e:ec:e7:69:61:08:e1:d9:e8:ed:00:07:6a:4b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'kvm-02-guest24.rhts.eng.brq.redhat.com,10.37.153.98' (ECDSA) to the list of known hosts.
root.eng.brq.redhat.com's password: 
32060.pem                                                                                                                               100% 2090    11.2KB/s   00:00    
37060.pem                                                                                                                               100% 2078    11.5KB/s   00:00    
37062.pem                                                                                                                               100% 2098    12.7KB/s   00:00    
37065.pem                                                                                                                               100% 2082    11.2KB/s   00:00    
37067.pem                                                                                                                               100% 2090    12.7KB/s   00:00    
37068.pem                                                                                                                               100% 2094    12.7KB/s   00:00    
37069.pem                                                                                                                               100% 2082    12.6KB/s   00:00    
37070.pem                                                                                                                               100% 2090    12.7KB/s   00:00    
37080.pem                                                                                                                               100% 2078    12.6KB/s   00:00    
37090.pem                                                                                                                               100% 2074    12.6KB/s   00:00    
37091.pem                                                                                                                               100% 2074    12.5KB/s   00:00    
38070.pem                                                                                                                               100% 2074    12.6KB/s   00:00    
38072.pem                                                                                                                               100% 2061    12.5KB/s   00:00    

[root@newcandlepin ~]# scp /etc/candlepin/certs/candlepin-ca.crt kvm-02-guest24.rhts.eng.brq.redhat.com:/etc/rhsm/ca/candlepin-ca.pem
root.eng.brq.redhat.com's password: 
candlepin-ca.crt                                                                                                                        100% 2029    12.0KB/s   00:00    

registering to candlepin server:
[root@kvm-02-guest24 ~]# subscription-manager register
Registering to: 10.70.35.79:8443/candlepin
Username: admin
Password: 
Hint: User "admin" is member of following organizations: admin, snowwhite, donaldduck
Organization: snowwhite
The system has been registered with ID: f23d316c-2e91-4cb9-ba3e-1841b5659e68
The registered system name is: kvm-02-guest24.rhts.eng.brq.redhat.com

[root@kvm-02-guest24 ~]# subscription-manager attach --auto
Installed Product Current Status:
Product Name: Awesome OS Instance Server Bits
Status:       Subscribed

Product Name: Awesome OS Server Bits
Status:       Subscribed
.
.
.

[root@kvm-02-guest24 ~]# cat /etc/yum.repos.d/redhat.repo
#
# Certificate-Based Repositories
# Managed by (rhsm) subscription-manager
#
# *** This file is auto-generated.  Changes made here will be over-written. ***
# *** Use "subscription-manager repo-override --help" if you wish to make changes. ***
#
# If this file is empty and this system is subscribed consider
# a "yum repolist" to refresh available repos
#

[content-label-no-gpg-37060]
name = content-nogpg-37060
baseurl = https://cdn.redhat.com/foo/path/no_gpg/37060234
enabled = 0
gpgcheck = 0
sslverify = 1
sslcacert = /etc/rhsm/ca/redhat-uep.pem
sslclientkey = /etc/pki/entitlement/7691419403284580053-key.pem
sslclientcert = /etc/pki/entitlement/7691419403284580053.pem
sslverifystatus = 1
enabled_metadata = 0

manually configuring repolib.py
[root@kvm-02-guest24 ~]# grep has_ssl_verify_status /usr/lib64/python3.9/site-packages/subscription_manager/repolib.py
        has_ssl_verify_status = self.get_consumer_auth_cp().has_capability("ssl_verify_status")
            if has_ssl_verify_status:
	has_ssl_verify_status = False

[root@kvm-02-guest24 ~]# yum repolist
.
.

[root@kvm-02-guest24 ~]# cat /etc/yum.repos.d/redhat.repo
#
# Certificate-Based Repositories
# Managed by (rhsm) subscription-manager
#
# *** This file is auto-generated.  Changes made here will be over-written. ***
# *** Use "subscription-manager repo-override --help" if you wish to make changes. ***
#
# If this file is empty and this system is subscribed consider
# a "yum repolist" to refresh available repos
#

[content-label-no-gpg-37060]
name = content-nogpg-37060
baseurl = https://cdn.redhat.com/foo/path/no_gpg/37060234
enabled = 0
gpgcheck = 0
sslverify = 1
sslcacert = /etc/rhsm/ca/redhat-uep.pem
sslclientkey = /etc/pki/entitlement/7691419403284580053-key.pem
sslclientcert = /etc/pki/entitlement/7691419403284580053.pem
sslverifystatus = 0
enabled_metadata = 0
.
.
.
^ sslverifystatus is set to 0 after manually configuring repolib.py --> PASSED

Scenario3:Verify the "sslverifystatus" is added to each repos and works with Proxy	

confoguring proxy:
[root@kvm-02-guest24 ~]# subscription-manager config --server.proxy_hostname=auto-services.usersys.redhat.com --server.proxy_port=3127 --server.proxy_user=redhat --server.proxy_password=redhat --server.proxy_scheme=https

Registering against stage:
[root@kvm-02-guest24 ~]# subscription-manager register
Registering to: subscription.rhsm.stage.redhat.com:443/subscription
Username: zpetracePH03
Password: 
The system has been registered with ID: f6c2c673-e78d-4656-9900-739f9a7a0338

[root@kvm-02-guest24 ~]# grep has_ssl_verify_status /usr/lib64/python3.9/site-packages/subscription_manager/repolib.py
        has_ssl_verify_status = self.get_consumer_auth_cp().has_capability("ssl_verify_status")
            if has_ssl_verify_status:
has_ssl_verify_status = True

[root@kvm-02-guest24 ~]# yum install zsh
Failed loading plugin "product-id": unexpected indent (repolib.py, line 521)
Failed loading plugin "subscription-manager": unexpected indent (repolib.py, line 521)
Last metadata expiration check: 0:31:45 ago on Tue 21 Jun 2022 02:22:35 PM CEST.
Dependencies resolved.
==========================================================================================================================================================================
 Package                             Architecture                           Version                                   Repository                                     Size
==========================================================================================================================================================================
Installing:
 zsh                                 x86_64                                 5.8-9.el9                                 beaker-BaseOS                                 3.2 M

Transaction Summary
.
.
.


[zpetracek@fedora ~]$  ssh root.redhat.com
The authenticity of host 'auto-services.usersys.redhat.com (10.8.30.63)' can't be established.
ED25519 key fingerprint is SHA256:oiv0PSJXlOdzfc1F8/mk82Gd+mfUukV58jPUf7O02HE.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'auto-services.usersys.redhat.com' (ED25519) to the list of known hosts.
root.redhat.com's password: 
client_global_hostkeys_private_confirm: server gave bad signature for RSA key 0: error in libcrypto
Activate the web console with: systemctl enable --now cockpit.socket

Last login: Tue Jun 21 08:05:59 2022 from 10.40.193.145
^ I was able to see the traffic --> PASSED

Comment 5 Archana Pandey 2022-07-26 10:45:58 UTC
Final verification on subscription-manager-1.29.26.1-1.el9_0.x86_64

 Beaker Test information:
                         HOSTNAME=sweetpig-19.4a2m.lab.eng.bos.redhat.com
                            JOBID=6839502
                         RECIPEID=12324957
                    RESULT_SERVER=
                           DISTRO=RHEL-9.0.0
                     ARCHITECTURE=x86_64

>> verifying presence of flag  'sslverifystatus = 1' in repo file when server supports the  ssl_verify_status capability -

[arpandey@ovpn-9-48 ~]$ curl --stderr /dev/null --insecure --user admin:admin --request GET 'https://archana-candlepin.usersys.redhat.com:8443/candlepin/status' | python -m json.tool
{
    "mode": "NORMAL",
    "modeReason": null,
    "modeChangeTime": null,
    "result": true,
    "version": "4.2.4",
    "release": "1",
    "standalone": false,
    "timeUTC": "2022-07-25T07:15:31-0400",
    "rulesSource": "default",
    "rulesVersion": "5.43",
    "managerCapabilities": [
        "instance_multiplier",
        "derived_product",
        "vcpu",
        "cert_v3",
        "hypervisors_heartbeat",
        "remove_by_pool_id",
        "syspurpose",
        "storage_band",
        "cores",
        "ssl_verify_status",
        "multi_environment",
        "hypervisors_async",
        "org_level_content_access",
        "guest_limit",
        "ram",
        "batch_bind"
    ],
    "keycloakRealm": null,
    "keycloakAuthUrl": null,
    "keycloakResource": null
}
[arpandey@ovpn-9-48 ~]$ 

[root@sweetpig-19 ~]# 
[root@sweetpig-19 ~]# subscription-manager version
server type: This system is currently not registered.
subscription management server: 4.2.4-1
subscription management rules: 5.43
subscription-manager: 1.29.26.1-1.el9_0
[root@sweetpig-19 ~]# 
[root@sweetpig-19 ~]# subscription-manager register --username ***** --password *****
Registering to: archana-candlepin.usersys.redhat.com:8443/candlepin
Hint: User "*****" is member of following organizations: snowwhite, admin
Organization: snowwhite
The system has been registered with ID: 7c873c51-e360-4a60-9b85-a671577d3b4f
The registered system name is: sweetpig-19.4a2m.lab.eng.bos.redhat.com
[root@sweetpig-19 ~]# 
[root@sweetpig-19 ~]# 
[root@sweetpig-19 ~]# subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Disabled
Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.

System Purpose Status: Disabled

[root@sweetpig-19 ~]# grep '^baseurl = https://cdn\.redhat\.com/' /etc/yum.repos.d/redhat.repo | wc -l
89
[root@sweetpig-19 ~]# grep '^sslverifystatus = 1' /etc/yum.repos.d/redhat.repo | wc -l
89                           << sslverifystatus flag is present in repo file
[root@sweetpig-19 ~]# 


----------------------------------------------------------------------------------------------------------------------


>> removing ssl_verify_status capability from server and then verifying removal of 'sslverifystatus = 1' flag from repo file 

steps to verify- 
[arpandey@ovpn-9-48 ~]$ curl --stderr /dev/null --insecure --user admin:admin --request GET 'https://archana-candlepin.usersys.redhat.com:8443/candlepin/status' | python -m json.tool
{
    "mode": "NORMAL",
    "modeReason": null,
    "modeChangeTime": null,
    "result": true,
    "version": "4.2.4",
    "release": "1",
    "standalone": false,
    "timeUTC": "2022-07-25T07:51:00-0400",
    "rulesSource": "default",
    "rulesVersion": "5.43",
    "managerCapabilities": [
        "instance_multiplier",
        "derived_product",
        "vcpu",
        "cert_v3",
        "hypervisors_heartbeat",
        "remove_by_pool_id",
        "syspurpose",
        "storage_band",
        "cores",
        "multi_environment",
        "hypervisors_async",
        "org_level_content_access",
        "guest_limit",
        "ram",
        "batch_bind"
    ],
    "keycloakRealm": null,
    "keycloakAuthUrl": null,
    "keycloakResource": null
}
[root@sweetpig-19 ~]# 
[root@sweetpig-19 ~]# subscription-manager register --username ****** --password ***** --org *****
Registering to: archana-candlepin.usersys.redhat.com:8443/candlepin
The system has been registered with ID: b0834227-3f7d-4661-9de7-a4c2d4b1d564
The registered system name is: sweetpig-19.4a2m.lab.eng.bos.redhat.com
[root@sweetpig-19 ~]# 
[root@sweetpig-19 ~]# subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Disabled
Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.

System Purpose Status: Unknown

>> Now verify that  ‘sslverifystatus =1’ flag is not present in repo file
[root@sweetpig-19 ~]# grep '^baseurl = https://cdn\.redhat\.com/' /etc/yum.repos.d/redhat.repo | wc -l
89
[root@sweetpig-19 ~]# grep '^sslverifystatus = 1' /etc/yum.repos.d/redhat.repo | wc -l
0                                         <<< sslverifystatus flag removed
[root@sweetpig-19 ~]# 

Based on above evidences, verified that 'sslverifystatus = 1' flag is getting added and removed in repo file as per the capability supported in server.

Verification : PASSED

Comment 9 errata-xmlrpc 2022-08-09 10:30:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (subscription-manager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:5944