Description of problem:

Getting 403 Forbidden error when running Yum commands.
-----------------------------------------------------------
# yum update
Loaded plugins: fastestmirror, product-id, subscription-manager
Loading mirror speeds from cached hostfile
redhat_test_product_custom-test-repo-1                                                                                                                                                            | 2.0 kB  00:00:00     
redhat_test_product_custom-test-repo-2                                                                                                                                                                | 1.9 kB  00:00:00     
https://satellite.example.com/pulp/content/redhat/Library/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden
Trying other mirror.
<snip>
<snip>
failure: repodata/repomd.xml from rhel-7-server-rpms: [Errno 256] No more mirrors to try.
https://satellite.example.com/pulp/content/redhat/Library/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden
-----------------------------------------------------------


No 403 Forbidden error if you only enable 1 repo
-----------------------------------------------------------
# subscription-manager repos --disable "*" --enable "rhel-7-server-rpms"
<snip>
# yum update
Loaded plugins: fastestmirror, product-id, subscription-manager
Loading mirror speeds from cached hostfile
rhel-7-server-rpms                                                                                                                                                                           
(1/2): rhel-7-server-rpms/7Server/x86_64/updateinfo                                                                                                                                              
(2/2): rhel-7-server-rpms/7Server/x86_64/primary      
<snip>                                                                                                                                        
No packages marked for update
-----------------------------------------------------------


This is because the client has installed a Redislabs third party application which shipped its own 'libcurl' library and will point the global library to use it instead.
-----------------------------------------------------------
# rpm -aq | grep redislabs
redislabs-6.2.10-96.rhel7.x86_64

# ldconfig -p -N -X | grep libcurl
	libcurl.so.4 (libc6,x86-64) => /opt/redislabs/lib/libcurl.so.4 <=======
	libcurl.so.4 (libc6,x86-64) => /lib64/libcurl.so.4
	libcurl.so (libc6,x86-64) => /opt/redislabs/lib/libcurl.so  <=========
	libcurl.so (libc6,x86-64) => /lib64/libcurl.so
-----------------------------------------------------------


This issue seems to be related to the SSL sessionid cache and only send the client certificate once.
-------------------------------------------------------------------
# URLGRABBER_DEBUG=1 yum update
*   Trying XX.XX.XX.XX...
* Connected to satellite.example.com (XX.XX.XX.XX) port 443 (#10)
* successfully set certificate verify locations:
*   CAfile: /etc/rhsm/ca/katello-server-ca.pem
  CApath: /etc/rhsm/ca/katello-server-ca.pem
* SSL re-using session ID   <====================================== This message is not printed in Redhat shipped libcurl
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
         <snip>
* 	 SSL certificate verify ok.
> GET /pulp/content/redhat/Library/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.10/os/repodata/repomd.xml HTTP/1.1
User-Agent: urlgrabber/3.10 yum/3.4.3
Host: satellite.example.com
-------------------------------------------------------------------


Version-Release number of selected component (if applicable):
This issue is only happening in Satellite 6.10. Satellite 6.9 doesn't have this issue.


Steps to Reproduce:
1. On any RHEL 7 client registered to a Satellite 6.10 (with SCA disabled), download the attached 'redislabs-6.2.10-96-rhel7-x86_64.tar' file. You can also download the file from 'https://app.redislabs.com/#/rlec-downloads'
2. Extract it and run the 'install.sh' to install the application.
3. Enable multiple repos. Must enable at least 1 custom repo and Redhat repo.
4. Run 'Yum' commands like 'yum repolist -v', 'yum updateinfo', 'yum update'

Actual results:
403 Forbidden error

Expected results:
No error


Additional info:
The client can workaround this error by disabling the SSL sessionid cache in the '/usr/lib/python2.7/site-packages/urlgrabber/grabber.py'.
-------------------------------------------------------------------
# cat disable_ssl_sessionid_cache.patch 
--- a/grabber.py	2019-08-24 07:12:38.000000000 +1000
+++ b/grabber.py	2022-04-09 21:38:45.472112081 +1000
@@ -1424,6 +1424,7 @@
         self.curl_obj.setopt(pycurl.FAILONERROR, True)
         self.curl_obj.setopt(pycurl.OPT_FILETIME, True)
         self.curl_obj.setopt(pycurl.FOLLOWLOCATION, True)
+        self.curl_obj.setopt(pycurl.SSL_SESSIONID_CACHE, False)
         
         if DEBUG and DEBUG.level <= 10:
             self.curl_obj.setopt(pycurl.VERBOSE, True)
-------------------------------------------------------------------


The issue can be solved on the server side by adding the following SSL settings in the httpd config to force sending the client certificate for authentication.
-------------------------------------------------------------------
Step:
1) In Satellite shell, open and edit "/etc/httpd/conf.d/05-foreman-ssl.conf" file.

2) Add the 'SSLRequireSSL' and 'SSLVerifyClient require' settings into the '<Location "/pulp/repos">' and '<Location "/pulp/content">' directives as below:

  <Location "/pulp/repos">
    RequestHeader unset X-CLIENT-CERT
    RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
    ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
    ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
    SSLRequireSSL    <==============  add this line
    SSLVerifyClient require  <============== add this line
  </Location>

<snip>

  <Location "/pulp/content">
    RequestHeader unset X-CLIENT-CERT
    RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
    ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
    ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
    SSLRequireSSL    <============= add this line
    SSLVerifyClient require  <============= add this line
  </Location>



3) Restart httpd

systemctl restart httpd
hammer ping


NOTE: For Capsule, we need to add the SSL settings to "/etc/httpd/conf.d/10-pulpcore-https.conf" instead.
-------------------------------------------------------------------


Since the issue can be solved by adding the above SSL settings into the httpd config. I suggest that we make it configurable by the Satellite installer.