Bug 2075455

Summary: RFE: For dnf operations against Red Hat CDN, enable OCSP stapling verification
Product: Red Hat Enterprise Linux 9 Reporter: Jan Pazdziora (Red Hat) <jpazdziora>
Component: subscription-managerAssignee: Pino Toscano <ptoscano>
Status: CLOSED ERRATA QA Contact: Red Hat subscription-manager QE Team <rhsm-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0CC: arpandey, candlepin-bugs, cdonnell, jpazdziora, jsefler, kanderso, redakkan, zpetrace
Target Milestone: rcKeywords: FutureFeature, Triaged, ZStream
Target Release: 9.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: subscription-manager-1.29.29-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2076522 2095301 (view as bug list) Environment:
Last Closed: 2022-11-15 11:19:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2076522, 2095301    

Description Jan Pazdziora (Red Hat) 2022-04-14 09:24:04 UTC
Description of problem:

The Red Hat's CDN (cdn.redhat.com) now supports OCSP stapling and dnf now has sslverifystatus option to enable Certificate Status Request on TLS connections when working with remote repositories, added in RHEL 8.5 via bug 1814383 and available in RHEL 9.

RHEL systems subscribed / configured to use Red Hat's CDN repositories should enable the sslverifystatus by default to harden the authenticity of those operations.

Version-Release number of selected component (if applicable):

subscription-manager-1.29.26-2.el9_0

How reproducible:

Deterministic.

Steps to Reproduce:

1. Subscribe the machine to the hosted (non-Satellite Candlepin) server:
   # subscription-manager register
2. Check how many repos are configured to use CDN:
   # grep '^baseurl = https://cdn\.redhat\.com/' /etc/yum.repos.d/redhat.repo | wc -l
3. Check how many repos have sslverifystatus enabled:
   # grep '^sslverifystatus = 1' /etc/yum.repos.d/redhat.repo | wc -l

Actual results:

2. 155
3. 0

Expected results:

2. 155
3. 155 (or in general the same as output of command 2)

Additional info:

When subscription manager registers / configures to use cdn.stage.redhat.com, sslverifystatus = 1 should be enabled as well.

On the contrary, for any other baseurl, sslverifystatus should not be enabled by default. So registering against Satellite (which typically would not have OCSP stapling enabled) should still result in a working setup.

To check that things work (for both the CDN hosted and Satellite cases), dnf operations like

# dnf install -y zsh
# dnf remove -y zsh

should be also tried. While doing so, make sure there aren't other repositories configured in /etc/yum.repos.d/ (like beaker-*.repo) which the installation would use by mistake

To verify that the option correctly gets enabled and takes effect in the CDN hosted case, it is possible to fake the cdn.redhat.com HTTPS end point:

# openssl req -new -keyout cdn-ca.key -x509 -nodes -subj "/CN=cdn.redhat.com" > cdn-ca.pem
# /usr/bin/cp cdn-ca.pem /etc/rhsm/ca/redhat-uep.pem
# echo 127.0.0.2 cdn.redhat.com >> /etc/hosts
In one terminal run
# openssl s_server -key cdn-ca.key -cert cdn-ca.pem -port 443
In other terminal run
# dnf install -y zsh

This should fail with
Updating Subscription Management repositories.

This system is registered with an entitlement server, but is not receiving updates. You can use subscription-manager to assign subscriptions.

Red Hat Enterprise Linux 9 for x86_64 - BaseOS  0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'rhel-9-for-x86_64-baseos-rpms':
  - Curl error (91): SSL server certificate status verification FAILED for https://cdn.redhat.com/content/dist/rhel9/9/x86_64/baseos/os/repodata/repomd.xml [No OCSP response received]
Error: Failed to download metadata for repo 'rhel-9-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Comment 4 Rehana 2022-04-28 13:24:26 UTC
//Requesting exception , as the plan to deliver the feature in RHEL 9.0 Zstream )

Comment 14 Jan Pazdziora (Red Hat) 2022-05-27 13:08:21 UTC
In the matrix of setups that you are testing, it might be useful to also check setups with --proxy when the client does not make connections directly to https://cdn.redhat.com/. Those should still continue working.

Comment 15 Rehana 2022-05-27 13:35:54 UTC
(In reply to Jan Pazdziora from comment #14)
> In the matrix of setups that you are testing, it might be useful to also
> check setups with --proxy when the client does not make connections directly
> to https://cdn.redhat.com/. Those should still continue working.

Thanks, sure we will test those scenarios and add the results soon.

Comment 25 Archana Pandey 2022-07-26 11:11:45 UTC
Final verification on an nightlybuild of RHEL 9.1 with subscription-manager-1.29.29-1.el9.x86_64

 Beaker Test information:
                         HOSTNAME=kvm-03-guest09.hv2.lab.eng.bos.redhat.com
                            JOBID=6853835
                         RECIPEID=12345831
                    RESULT_SERVER=
                           DISTRO=RHEL-9.1.0-20220721.1
                     ARCHITECTURE=x86_64

>> verifying presence of flag  'sslverifystatus = 1' in repo file when server supports the  ssl_verify_status capability -

[root@kvm-03-guest09 ~]# curl --stderr /dev/null --insecure --user *******:******** --request GET 'https://archana-candlepin.usersys.redhat.com:8443/candlepin/status' | python -m json.tool
{
    "mode": "NORMAL",
    "modeReason": null,
    "modeChangeTime": null,
    "result": true,
    "version": "4.2.4",
    "release": "1",
    "standalone": false,
    "timeUTC": "2022-07-26T06:55:34-0400",
    "rulesSource": "default",
    "rulesVersion": "5.43",
    "managerCapabilities": [
        "instance_multiplier",
        "derived_product",
        "vcpu",
        "cert_v3",
        "hypervisors_heartbeat",
        "remove_by_pool_id",
        "syspurpose",
        "storage_band",
        "cores",
        "ssl_verify_status",
        "multi_environment",
        "hypervisors_async",
        "org_level_content_access",
        "guest_limit",
        "ram",
        "batch_bind"
    ],
    "keycloakRealm": null,
    "keycloakAuthUrl": null,
    "keycloakResource": null
}

[root@kvm-03-guest09 ~]# subscription-manager version
server type: This system is currently not registered.
subscription management server: 4.2.4-1
subscription management rules: 5.43
subscription-manager: 1.29.29-1.el9
[root@kvm-03-guest09 ~]# 
[root@kvm-03-guest09 ~]# subscription-manager register 
Registering to: archana-candlepin.usersys.redhat.com:8443/candlepin
Username: ******
Password: 
Hint: User "*****" is member of following organizations: snowwhite, admin
Organization: snowwhite
The system has been registered with ID: 20a19beb-aad0-4d49-aae2-5e5b0d5ac385
The registered system name is: kvm-03-guest09.hv2.lab.eng.bos.redhat.com
[root@kvm-03-guest09 ~]#             
[root@kvm-03-guest09 ~]# subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Disabled
Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.

System Purpose Status: Disabled            

[root@kvm-03-guest09 ~]# grep '^baseurl = https://cdn\.redhat\.com/' /etc/yum.repos.d/redhat.repo | wc -l
89
[root@kvm-03-guest09 ~]# 
[root@kvm-03-guest09 ~]# grep '^sslverifystatus = 1' /etc/yum.repos.d/redhat.repo | wc -l
89                                     << slverifystatus = 1 flag present in repo when server supports this capability
[root@kvm-03-guest09 ~]# 

----------------------------------------------------------------------------------------------------------------------


>> removing ssl_verify_status capability from server and then verifying removal of 'sslverifystatus = 1' flag from repo file 

steps to verify- 
[root@kvm-03-guest09 ~]# curl --stderr /dev/null --insecure --user ****:###### --request GET 'https://archana-candlepin.usersys.redhat.com:8443/candlepin/status' | python -m json.tool
{
    "mode": "NORMAL",
    "modeReason": null,
    "modeChangeTime": null,
    "result": true,
    "version": "4.2.4",
    "release": "1",
    "standalone": false,
    "timeUTC": "2022-07-26T07:01:04-0400",
    "rulesSource": "default",
    "rulesVersion": "5.43",
    "managerCapabilities": [
        "instance_multiplier",
        "derived_product",
        "vcpu",
        "cert_v3",
        "hypervisors_heartbeat",
        "remove_by_pool_id",
        "syspurpose",
        "storage_band",
        "cores",
        "multi_environment",
        "hypervisors_async",
        "org_level_content_access",
        "guest_limit",
        "ram",
        "batch_bind"
    ],
    "keycloakRealm": null,
    "keycloakAuthUrl": null,
    "keycloakResource": null
}
[root@kvm-03-guest09 ~]# 
[root@kvm-03-guest09 ~]# 
[root@kvm-03-guest09 ~]# subscription-manager register
Registering to: archana-candlepin.usersys.redhat.com:8443/candlepin
Username: ******
Password: 
Hint: User "*****" is member of following organizations: admin, snowwhite
Organization: snowwhite
The system has been registered with ID: 823ed1a0-1587-4ae6-a5cf-61f893807533
The registered system name is: kvm-03-guest09.hv2.lab.eng.bos.redhat.com
[root@kvm-03-guest09 ~]#             
[root@kvm-03-guest09 ~]# subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Disabled
Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.

System Purpose Status: Disabled            

[root@kvm-03-guest09 ~]# 
[root@kvm-03-guest09 ~]# grep '^baseurl = https://cdn\.redhat\.com/' /etc/yum.repos.d/redhat.repo | wc -l
89
[root@kvm-03-guest09 ~]# 
[root@kvm-03-guest09 ~]# grep '^sslverifystatus = 1' /etc/yum.repos.d/redhat.repo | wc -l
0                                              <<<<<< 'sslverifystatus = 1' flag removed when server does not support this capability
[root@kvm-03-guest09 ~]# 
[root@kvm-03-guest09 ~]# 

Based on above evidences, verified that 'sslverifystatus = 1' flag is getting added and removed in repo file as per the capability supported in server.

Verification : PASSED

Comment 27 errata-xmlrpc 2022-11-15 11:19:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (subscription-manager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8341