Bug 2258309
| Summary: | Capsule sync fails with error "cadata should be an ASCII string or a bytes-like object" when cacert has a non-ascii charecter | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Sayan Das <saydas> |
| Component: | Pulp | Assignee: | satellite6-bugs <satellite6-bugs> |
| Status: | CLOSED MIGRATED | QA Contact: | Satellite QE Team <sat-qe-bz-list> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 6.14.0 | CC: | aruzicka, dalley, ehelms, jpasqual, rlavi |
| Target Milestone: | Unspecified | Keywords: | MigratedToJIRA, Triaged |
| Target Release: | Unused | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-06-06 16:40:59 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I confirm the following workaround\resolution for the time being: * Fix the CA certificate in satellite and reapply it on satellite ( using --certs-update-server --certs-update-server-ca ) * On Capsule, run the following to unset the certificate-related fields in the Pulp remotes : # PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell << EOF from pulpcore.app.models import Remote for r in Remote.objects.all(): r.ca_cert=None r.client_cert=None r.client_key=None r.save() EOF * From Satellite, Initiate a new Capsule sync that should re-populate the correct certificate data for each repository and sync them. Both Optimized and Complete sync can fix the issue at this stage. >* From Satellite, Initiate a new Capsule sync that should re-populate the correct certificate data for each repository and sync them. Both Optimized and Complete sync can fix the issue at this stage. Would it not re-populate the certificate data even without the step to wipe it from the remotes? I would think that so long as it changes it would get updated, right? In fact it might even be updated even if it doesn't change - it's one theory for why there are so many update tasks on capsules / smart proxy hosts https://community.theforeman.org/t/smartproxy-database-size/36257/10?u=dralley But maybe I'm wrong here. If the appropriate resolution (per comment #2) is katello-certs-check, then what is the correct component for this BZ? Or should we handle it in both places? To corroborate the traceback, the SSLContext object we use (from the Python standard library) says the following in its docs: "The cadata object, if present, is either an ASCII string of one or more PEM-encoded certificates or a bytes-like object of DER-encoded certificates. Like with capath extra lines around PEM-encoded certificates are ignored but at least one certificate must be present." So I concur that we need to substitute, strip or reject any non-ascii characters. (In reply to Daniel Alley from comment #5) > >* From Satellite, Initiate a new Capsule sync that should re-populate the correct certificate data for each repository and sync them. Both Optimized and Complete sync can fix the issue at this stage. > > Would it not re-populate the certificate data even without the step to wipe > it from the remotes? I would think that so long as it changes it would get > updated, right? In fact it might even be updated even if it doesn't change > - it's one theory for why there are so many update tasks on capsules / smart > proxy hosts > https://community.theforeman.org/t/smartproxy-database-size/36257/ > 10?u=dralley > > But maybe I'm wrong here. If I recall correctly When this issue happens: --> Optimized Sync would finish normally --> When we do Complete Sync, It complains about the same error about cadata having ASCII stuff The only way to get the Complete Sync working is to ensure that the ca_cert object has been nullified for all remotes. We don't necessarily need to delete all remotes but just nullification of the cert concert stored for each is good enough. > > If the appropriate resolution (per comment #2) is katello-certs-check, then > what is the correct component for this BZ? Or should we handle it in both > places? > > To corroborate the traceback, the SSLContext object we use (from the Python > standard library) says the following in its docs: > > "The cadata object, if present, is either an ASCII string of one or more > PEM-encoded certificates or a bytes-like object of DER-encoded certificates. > Like with capath extra lines around PEM-encoded certificates are ignored but > at least one certificate must be present." > > So I concur that we need to substitute, strip or reject any non-ascii > characters. That's what my initial suggestion was. It's better to handle it gracefully at the product level than relying on katello-certs-check i.e. don't use the file as it is but extract the cert content from it and then use it. Hello all
Talking with Sayan about this BZ and [1], both BZs have as result very similar issues. We were thinking that maybe we could sanitize the CA bundle (inside satellite-installer) before actually deploying it.
This sanitation would be simple to do and would avoid the consequences described here and on [1].
Example of how to sanitize the file "bundle_file_provided.pem":
~~~
awk -v cmd='openssl x509' ' /BEGIN/{close(cmd)};{print | cmd}' bundle_file_provided.pem > sanitized_bundle_file.pem
~~~
Any thoughts?
[1]: https://bugzilla.redhat.com/show_bug.cgi?id=2259026
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "SAT-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |
Description of problem: Sometimes when end-users obtain their CA and Signed certificates for satellites and capsules, often those certs can have some comments mentioned before each certificate block. If any such comment or any of the content of the CA bundle contains a non-ascii character, Then despite the Satellite server functioning just fine, The capsule server will never be able to sync content from the satellite. Version-Release number of selected component (if applicable): Any Version of Satellite 6 on Pulp3 ( Tested on Satellite 6.14 \ Reported on 6.14 + 6.11 ) How reproducible: Always and easily Steps to Reproduce: 1. Install a Satellite and Capsule server with default certs 2. Create/Obtain SSL and CA bundle certificates for satellite and capsule 3. Add a comment at the very top of the CA bundle cert with a non-ascii charecter e.g. something like # saydas Intermédiaire CA 4. Now install those certs in both satellite and capsule. 5. Add Library lifecycle to capsule server for content syncing. 6. Import manifest in satellite and then Enable and Sync some repos in Satellite server. 7. Observe the auto-sync triggered for the capsule server or else manually trigger one. Actual results: Sync fails for every repo on the capsule server ~~~ Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: pulp [27024eb5-c48c-4d8a-be6f-c47f1d633911]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 381b7d4b-57e8-46b1-b585-ced536b3a6ea Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: pulp [27024eb5-c48c-4d8a-be6f-c47f1d633911]: pulp_rpm.app.tasks.synchronizing:INFO: Synchronizing: repository=54d852d4-2c81-4776-bb4c-8a5bdbe53166 remote=54d852d4-2c81-4776-bb4c-8a5bdbe53166 Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: pulp [27024eb5-c48c-4d8a-be6f-c47f1d633911]: pulpcore.tasking.pulpcore_worker:INFO: Task 381b7d4b-57e8-46b1-b585-ced536b3a6ea failed (cadata should be an ASCII string or a bytes-like object) Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: pulp [27024eb5-c48c-4d8a-be6f-c47f1d633911]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 460, in execute_task Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: result = func(*args, **kwargs) Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: File "/usr/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 482, in synchronize Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: remote_url = fetch_remote_url(remote, url) Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: File "/usr/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 285, in fetch_remote_url Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: get_repomd_file(remote, normalized_remote_url) Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: File "/usr/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 240, in get_repomd_file Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: downloader = remote.get_downloader(url=urlpath_sanitize(url, "repodata/repomd.xml")) Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: File "/usr/lib/python3.9/site-packages/pulp_rpm/app/models/repository.py", line 106, in get_downloader Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: return super().get_downloader(remote_artifact=remote_artifact, url=url, **kwargs) Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: File "/usr/lib/python3.9/site-packages/pulpcore/app/models/repository.py", line 476, in get_downloader Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: download_factory = self.download_factory Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: File "/usr/lib/python3.9/site-packages/pulp_rpm/app/models/repository.py", line 74, in download_factory Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: self._download_factory = DownloaderFactory( Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: File "/usr/lib/python3.9/site-packages/pulpcore/download/factory.py", line 78, in __init__ Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: self._session = self._make_aiohttp_session_from_remote() Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: File "/usr/lib/python3.9/site-packages/pulpcore/download/factory.py", line 109, in _make_aiohttp_session_from_remote Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: sslcontext = ssl.create_default_context(cadata=self._remote.ca_cert) Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: File "/usr/lib64/python3.9/ssl.py", line 746, in create_default_context Jan 14 13:22:03 saydas-capsule pulpcore-worker-1[38266]: context.load_verify_locations(cafile, capath, cadata) ~~~ Reason is well explained in the traceback i.e. the ca_cert contains a non-ascii charecter. From pulpcore DB: pulpcore=# select distinct ca_cert from core_remote; ca_cert ------------------------------------------------------------------ # saydas Intermédiaire CA + -----BEGIN CERTIFICATE----- + MIIF3DCCA8SgAwIBAgIBATANBgkqhkiG9w0BAQsFADCBgTELMAkGA1UEBhMCSU4x+ CzAJBgNVBAgTAldCMQwwCgYDVQQHEwNLT0wxDzANBgNVBAoTBlJlZEhhdDELMAkG+ A1UECxMCWEUxFzAVBgNVBAMTDnNheWRhcy5wbnEuY3NiMSAwHgYJKoZIhvcNAQkB+ FhFzYXlkYXNAcmVkaGF0LmNvbTAeFw0yMzA0MjUxMTUwMDNaFw0zMDA3MjcxMTUw+ MDNaMHkxCzAJBgNVBAYTAklOMQswCQYDVQQIEwJXQjEPMA0GA1UEChMGUmVkSGF0+ ... .. .. output snipped .. here "é" is the non-ascii character creating the problem. Expected results: Either katello-certs-check itself will be able to detect the non-ascii character and then inform end-user Or, Satellite\Capsule\Pulp\Katello would only use the content of valid certificate blocks in a file ( ignoring any comments inbetween ). Additional info: Fix is rather simple here i.e. * On capsule, clear all the remote objects: # PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell << EOF from pulpcore.app.models import Remote Remote.objects.all().delete() EOF * On Satellite, Remove the ascii charecter from the CA bundle and re-apply the certs with "--certs-update-server --certs-update-server-ca" flag. Re-deploy the same CA on capsule as well via capsule-certs-generate and satellite-installer. * Perform a "Complete Sync" of the capsule server to ReCreate the Pulp Remotes with correct certs.