Bug 2112015

Summary:	After deploying custom certs on Satellite, signed by a new CA, capsule can't fetch on-demand content
Product:	Red Hat Satellite	Reporter:	Joniel Pasqualetto <jpasqual>
Component:	Capsule - Content	Assignee:	Chris Roberts <chrobert>
Status:	CLOSED ERRATA	QA Contact:	Vladimír Sedmík <vsedmik>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.11.0	CC:	chrobert, damoore, paji, pcreech, saydas
Target Milestone:	6.12.0	Keywords:	Triaged, WorkAround
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	tfm-rubygem-katello-4.5.0.6-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-11-16 13:34:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Joniel Pasqualetto 2022-07-28 15:55:29 UTC

Description of problem:

After replacing custom certificates on Satellite by new ones, signed by a new CA (not previously trusted) capsule is unable to fetch on-demand content from Satellite and complains about self signed certificate in certificate chain.

Version-Release number of selected component (if applicable):

Satellite 6.11

How reproducible: 


Steps to Reproduce:
1. Have a Satellite + Capsule deployed ( using default self-signed certificates).
2. Ensure capsule is configured with on-demand download policy and some CV are synced to it
3. Deploy custom certs on Satellite + Capsule
4. Try to install, in a client registered to the capsule, any content not yet locally downloaded. You should see errors like this on the capsule logs:

~~~
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: Giving up download_wrapper(...) after 5 tries (aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host bombsat611-rhel7.usersys.redhat.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)')])
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 5 tries (aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host bombsat611-rhel7.usersys.redhat.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)')])
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: [2022-07-28 15:36:37 +0000] [2666] [ERROR] Error handling request
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: Traceback (most recent call last):
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/connector.py", line 986, in _wrap_create_connection
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_events.py", line 1050, in create_connection
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: transport, protocol = await self._create_connection_transport(
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: await waiter
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/sslproto.py", line 529, in data_received
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: ssldata, appdata = self._sslpipe.feed_ssldata(data)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: self._sslobj.do_handshake()
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/ssl.py", line 944, in do_handshake
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: self._sslobj.do_handshake()
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: The above exception was the direct cause of the following exception:
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: Traceback (most recent call last):
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: resp = await request_handler(request)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/web_app.py", line 504, in _handle
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: resp = await handler(request)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/web_middlewares.py", line 117, in impl
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await handler(request)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/authentication.py", line 41, in authenticate
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await handler(request)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 173, in cached_function
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: response = await self.make_entry(key, bk, func, args, self.default_expires_ttl)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/cache/cache.py", line 202, in make_entry
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: response = await handler(*args)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/handler.py", line 203, in stream_content
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await self._match_and_stream(path, request)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/handler.py", line 514, in _match_and_stream
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await self._stream_content_artifact(
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/handler.py", line 656, in _stream_content_artifact
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: response = await self._stream_remote_artifact(request, response, remote_artifact)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/content/handler.py", line 868, in _stream_remote_artifact
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: download_result = await downloader.run()
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/http.py", line 271, in run
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await download_wrapper()
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/backoff/_async.py", line 133, in retry
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: ret = await target(*args, **kwargs)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/download/http.py", line 256, in download_wrapper
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: return await self._run(extra_data=extra_data)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/downloaders.py", line 114, in _run
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: async with self.session.get(
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/client.py", line 1138, in __aenter__
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: self._resp = await self._coro
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/client.py", line 535, in _request
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: conn = await self._connector.connect(
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/connector.py", line 542, in connect
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: proto = await self._create_connection(req, traces, timeout)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/connector.py", line 907, in _create_connection
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: _, proto = await self._create_direct_connection(req, traces, timeout)
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: raise last_exc
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: transp, proto = await self._wrap_create_connection(
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: raise ClientConnectorCertificateError(req.connection_key, exc) from exc
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host bombsat611-rhel7.usersys.redhat.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)')]
Jul 28 11:36:37 bombcap611-rhel7 pulpcore-content: [28/Jul/2022:15:36:27 +0000] "GET /pulp/content/Default_Organization/Dev/Test1/content/dist/rhel/server/7/7Server/x86_64/ansible/2.9/os/Packages/s/sshpass-1.06-2.el7.x86_64.rpm HTTP/1.1" 500 244 "-" "urlgrabber/3.10 yum/3.4.3"
~~~

Actual results: 

Fail to download content. Return HTTP 500 to client.


Expected results:

Successfully download the content.

Additional info:

The problem is that the ca_cert on the remote (inside pulp3) didn't get updated after changing the certificate.

Running an optimized sync of the capsule fix it and content can be downloaded again.

Comment 7 Vladimír Sedmík 2022-09-02 12:01:07 UTC

Verified in 6.12.0 snap 9.0:
1) The issue was reproduced using steps from comment #0.
2) Refreshed the repositories:
[root@satellite ~]# foreman-rake katello:refresh_repos
Repos are being refreshed in the background.
3) After the task finished, installation of the content (still missing on the on_demand capsule side) succeeded on the content host. No errors observed in the logs anymore.

Comment 11 errata-xmlrpc 2022-11-16 13:34:51 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506