Description of problem: Getting 403 Forbidden error when running Yum commands. ----------------------------------------------------------- # yum update Loaded plugins: fastestmirror, product-id, subscription-manager Loading mirror speeds from cached hostfile redhat_test_product_custom-test-repo-1 | 2.0 kB 00:00:00 redhat_test_product_custom-test-repo-2 | 1.9 kB 00:00:00 https://satellite.example.com/pulp/content/redhat/Library/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden Trying other mirror. <snip> <snip> failure: repodata/repomd.xml from rhel-7-server-rpms: [Errno 256] No more mirrors to try. https://satellite.example.com/pulp/content/redhat/Library/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden ----------------------------------------------------------- No 403 Forbidden error if you only enable 1 repo ----------------------------------------------------------- # subscription-manager repos --disable "*" --enable "rhel-7-server-rpms" <snip> # yum update Loaded plugins: fastestmirror, product-id, subscription-manager Loading mirror speeds from cached hostfile rhel-7-server-rpms (1/2): rhel-7-server-rpms/7Server/x86_64/updateinfo (2/2): rhel-7-server-rpms/7Server/x86_64/primary <snip> No packages marked for update ----------------------------------------------------------- This is because the client has installed a Redislabs third party application which shipped its own 'libcurl' library and will point the global library to use it instead. ----------------------------------------------------------- # rpm -aq | grep redislabs redislabs-6.2.10-96.rhel7.x86_64 # ldconfig -p -N -X | grep libcurl libcurl.so.4 (libc6,x86-64) => /opt/redislabs/lib/libcurl.so.4 <======= libcurl.so.4 (libc6,x86-64) => /lib64/libcurl.so.4 libcurl.so (libc6,x86-64) => /opt/redislabs/lib/libcurl.so <========= libcurl.so (libc6,x86-64) => /lib64/libcurl.so ----------------------------------------------------------- This issue seems to be related to the SSL sessionid cache and only send the client certificate once. ------------------------------------------------------------------- # URLGRABBER_DEBUG=1 yum update * Trying XX.XX.XX.XX... * Connected to satellite.example.com (XX.XX.XX.XX) port 443 (#10) * successfully set certificate verify locations: * CAfile: /etc/rhsm/ca/katello-server-ca.pem CApath: /etc/rhsm/ca/katello-server-ca.pem * SSL re-using session ID <====================================== This message is not printed in Redhat shipped libcurl * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * Server certificate: <snip> * SSL certificate verify ok. > GET /pulp/content/redhat/Library/content/dist/rhel/server/7/7Server/x86_64/sat-tools/6.10/os/repodata/repomd.xml HTTP/1.1 User-Agent: urlgrabber/3.10 yum/3.4.3 Host: satellite.example.com ------------------------------------------------------------------- Version-Release number of selected component (if applicable): This issue is only happening in Satellite 6.10. Satellite 6.9 doesn't have this issue. Steps to Reproduce: 1. On any RHEL 7 client registered to a Satellite 6.10 (with SCA disabled), download the attached 'redislabs-6.2.10-96-rhel7-x86_64.tar' file. You can also download the file from 'https://app.redislabs.com/#/rlec-downloads' 2. Extract it and run the 'install.sh' to install the application. 3. Enable multiple repos. Must enable at least 1 custom repo and Redhat repo. 4. Run 'Yum' commands like 'yum repolist -v', 'yum updateinfo', 'yum update' Actual results: 403 Forbidden error Expected results: No error Additional info: The client can workaround this error by disabling the SSL sessionid cache in the '/usr/lib/python2.7/site-packages/urlgrabber/grabber.py'. ------------------------------------------------------------------- # cat disable_ssl_sessionid_cache.patch --- a/grabber.py 2019-08-24 07:12:38.000000000 +1000 +++ b/grabber.py 2022-04-09 21:38:45.472112081 +1000 @@ -1424,6 +1424,7 @@ self.curl_obj.setopt(pycurl.FAILONERROR, True) self.curl_obj.setopt(pycurl.OPT_FILETIME, True) self.curl_obj.setopt(pycurl.FOLLOWLOCATION, True) + self.curl_obj.setopt(pycurl.SSL_SESSIONID_CACHE, False) if DEBUG and DEBUG.level <= 10: self.curl_obj.setopt(pycurl.VERBOSE, True) ------------------------------------------------------------------- The issue can be solved on the server side by adding the following SSL settings in the httpd config to force sending the client certificate for authentication. ------------------------------------------------------------------- Step: 1) In Satellite shell, open and edit "/etc/httpd/conf.d/05-foreman-ssl.conf" file. 2) Add the 'SSLRequireSSL' and 'SSLVerifyClient require' settings into the '<Location "/pulp/repos">' and '<Location "/pulp/content">' directives as below: <Location "/pulp/repos"> RequestHeader unset X-CLIENT-CERT RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600 ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content SSLRequireSSL <============== add this line SSLVerifyClient require <============== add this line </Location> <snip> <Location "/pulp/content"> RequestHeader unset X-CLIENT-CERT RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600 ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content SSLRequireSSL <============= add this line SSLVerifyClient require <============= add this line </Location> 3) Restart httpd systemctl restart httpd hammer ping NOTE: For Capsule, we need to add the SSL settings to "/etc/httpd/conf.d/10-pulpcore-https.conf" instead. ------------------------------------------------------------------- Since the issue can be solved by adding the above SSL settings into the httpd config. I suggest that we make it configurable by the Satellite installer.
*** This bug has been marked as a duplicate of bug 2088559 ***
Hi Eric I am reopening this bug as I don't see how it can be the duplicate of bug 2088559. They look totally different to me.
The changes suggested for the Apache config file in comment #0 to fix this -- those will get overwritten when a customer upgrades from 6.10 to 6.11, right?
Hello Bernie, Yes and that is the reason this BZ was opened so that we can make it configurable via installer or hiera which will then persist across upgrades. -- Sayan
Hello, When will the fix for this bug be part of the official Satellite releases?
The proposed solution is not a good long term solution and breaks other client workflows. I think we need to better understand why this thirdparty libcurl is required and why it breaks this workflow. Generally, a library like libcurl which affects RHEL's core systems shouldn't be changed without careful consideration. That being said, I downloaded the latest version of redislab and tested it against Satellite 6.13. I was unable to reproduce the issue. I'll leave this open for a little bit longer if someone can provide an updated reproducer otherwise I will close it.
If you are able to provide a reproducer with the latest bits from redislabs then please re-open this with additional information about what was used as specific as possible to help us debug.