Description of problem: Trying to sync a repo with URL "https://cdn.fwupd.org/downloads/" fails with this error: ~~ Error: A file located at the url https://cdn.fwupd.org/downloads/firmware.xml.gz failed validation due to checksum. ~~ Version-Release number of selected component (if applicable): Red Hat Satellite 6.12.3 ( or any satellite version using Pulp 3 ) How reproducible: 100% Steps to Reproduce: 1. Build an affected version of satellite 2. Follow till "hammer repository synchronize" step from https://access.redhat.com/articles/5215571 Actual results: # hammer repository synchronize --product "fwupdates" --name "fwupdates" --organization RedHat [..............................................................................................................................................................................................................................] [100%] No content added. Total steps: 1015/7893 -------------------------------- Associating Content: 0/0 Downloading Artifacts: 11/11 Downloading Metadata: 1/1 Parsing Metadata Lines: 1003/7881 Error: A file located at the url https://cdn.fwupd.org/downloads/firmware.xml.gz failed validation due to checksum. Pulp messages: Jul 20 15:04:38 satellite612 pulpcore-api[869944]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:38 +0000] "GET /pulp/api/v3/status HTTP/1.1" 301 0 "-" "rest-client/2.0.2 (linux x86_64) ruby/2.7.6p219" Jul 20 15:04:38 satellite612 pulpcore-api[869944]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:38 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 1452 "-" "rest-client/2.0.2 (linux x86_64) ruby/2.7.6p219" Jul 20 15:04:38 satellite612 pulpcore-api[869946]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:38 +0000] "PATCH /pulp/api/v3/remotes/file/file/81de8399-10a0-4955-93ab-4ab716290c2a/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/1.10.0/ruby" Jul 20 15:04:39 satellite612 pulpcore-worker-1[1325370]: pulp [1470469e-04dc-446e-9544-87c328e01605]: pulpcore.tasking.pulpcore_worker:INFO: Starting task d5a5ac56-2d5a-4c06-9f8f-c55b204f2fd0 Jul 20 15:04:39 satellite612 pulpcore-worker-1[1325370]: pulp [1470469e-04dc-446e-9544-87c328e01605]: pulpcore.tasking.pulpcore_worker:INFO: Task completed d5a5ac56-2d5a-4c06-9f8f-c55b204f2fd0 Jul 20 15:04:39 satellite612 pulpcore-api[869944]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:39 +0000] "GET /pulp/api/v3/tasks/d5a5ac56-2d5a-4c06-9f8f-c55b204f2fd0/ HTTP/1.1" 200 608 "-" "OpenAPI-Generator/3.18.5/ruby" Jul 20 15:04:39 satellite612 pulpcore-api[869945]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:39 +0000] "POST /pulp/api/v3/repositories/file/file/a13d212e-23dd-486e-8b46-48a471ea6bdc/sync/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/1.10.0/ruby" Jul 20 15:04:39 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: pulpcore.tasking.pulpcore_worker:INFO: Starting task c816c293-7879-4796-b271-d6ec059471d5 Jul 20 15:04:39 satellite612 pulpcore-api[869944]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:39 +0000] "GET /pulp/api/v3/tasks/c816c293-7879-4796-b271-d6ec059471d5/ HTTP/1.1" 200 675 "-" "OpenAPI-Generator/3.18.5/ruby" Jul 20 15:04:39 satellite612 pulpcore-api[869946]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:39 +0000] "GET /pulp/api/v3/tasks/c816c293-7879-4796-b271-d6ec059471d5/ HTTP/1.1" 200 1165 "-" "OpenAPI-Generator/3.18.5/ruby" Jul 20 15:04:40 satellite612 pulpcore-worker-2[1325374]: Backing off download_wrapper(...) for 0.9s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-05151-testing.xml.gz failed validation due to checksum.) Jul 20 15:04:40 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: backoff:INFO: Backing off download_wrapper(...) for 0.9s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-05151-testing.xml.gz failed validation due to checksum.) Jul 20 15:04:40 satellite612 pulpcore-worker-2[1325374]: Backing off download_wrapper(...) for 0.0s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-testing.xml.gz failed validation due to checksum.) Jul 20 15:04:40 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: backoff:INFO: Backing off download_wrapper(...) for 0.0s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-testing.xml.gz failed validation due to checksum.) Jul 20 15:04:40 satellite612 pulpcore-worker-2[1325374]: Backing off download_wrapper(...) for 1.0s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-06639-stable.xml.gz failed validation due to checksum.) Jul 20 15:04:40 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: backoff:INFO: Backing off download_wrapper(...) for 1.0s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-06639-stable.xml.gz failed validation due to checksum.) Jul 20 15:04:40 satellite612 pulpcore-worker-2[1325374]: Backing off download_wrapper(...) for 0.9s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware.xml.gz failed validation due to checksum.) Jul 20 15:04:40 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: backoff:INFO: Backing off download_wrapper(...) for 0.9s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware.xml.gz failed validation due to checksum.) Jul 20 15:04:40 satellite612 pulpcore-api[869944]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:40 +0000] "GET /pulp/api/v3/tasks/c816c293-7879-4796-b271-d6ec059471d5/ HTTP/1.1" 200 1165 "-" "OpenAPI-Generator/3.18.5/ruby" Jul 20 15:04:40 satellite612 pulpcore-worker-2[1325374]: Backing off download_wrapper(...) for 0.2s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-testing.xml.gz failed validation due to checksum.) Jul 20 15:04:40 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: backoff:INFO: Backing off download_wrapper(...) for 0.2s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-testing.xml.gz failed validation due to checksum.) Jul 20 15:04:41 satellite612 pulpcore-worker-2[1325374]: Backing off download_wrapper(...) for 2.0s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-05151-testing.xml.gz failed validation due to checksum.) Jul 20 15:04:41 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: backoff:INFO: Backing off download_wrapper(...) for 2.0s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-05151-testing.xml.gz failed validation due to checksum.) Jul 20 15:04:41 satellite612 pulpcore-worker-2[1325374]: Backing off download_wrapper(...) for 3.3s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-testing.xml.gz failed validation due to checksum.) Jul 20 15:04:41 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: backoff:INFO: Backing off download_wrapper(...) for 3.3s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-testing.xml.gz failed validation due to checksum.) Jul 20 15:04:41 satellite612 pulpcore-api[869944]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:41 +0000] "GET /pulp/api/v3/tasks/c816c293-7879-4796-b271-d6ec059471d5/ HTTP/1.1" 200 1165 "-" "OpenAPI-Generator/3.18.5/ruby" Jul 20 15:04:41 satellite612 pulpcore-worker-2[1325374]: Backing off download_wrapper(...) for 1.8s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-06639-stable.xml.gz failed validation due to checksum.) Jul 20 15:04:41 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: backoff:INFO: Backing off download_wrapper(...) for 1.8s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware-06639-stable.xml.gz failed validation due to checksum.) Jul 20 15:04:41 satellite612 pulpcore-api[869945]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:41 +0000] "GET /pulp/api/v3/tasks/c816c293-7879-4796-b271-d6ec059471d5/ HTTP/1.1" 200 1165 "-" "OpenAPI-Generator/3.18.5/ruby" Jul 20 15:04:41 satellite612 pulpcore-worker-2[1325374]: Backing off download_wrapper(...) for 0.2s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware.xml.gz failed validation due to checksum.) Jul 20 15:04:41 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: backoff:INFO: Backing off download_wrapper(...) for 0.2s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware.xml.gz failed validation due to checksum.) Jul 20 15:04:42 satellite612 pulpcore-api[869946]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:42 +0000] "GET /pulp/api/v3/tasks/c816c293-7879-4796-b271-d6ec059471d5/ HTTP/1.1" 200 1166 "-" "OpenAPI-Generator/3.18.5/ruby" Jul 20 15:04:42 satellite612 pulpcore-worker-2[1325374]: Backing off download_wrapper(...) for 0.3s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware.xml.gz failed validation due to checksum.) Jul 20 15:04:42 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: backoff:INFO: Backing off download_wrapper(...) for 0.3s (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware.xml.gz failed validation due to checksum.) Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: Giving up download_wrapper(...) after 4 tries (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware.xml.gz failed validation due to checksum.) Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: backoff:ERROR: Giving up download_wrapper(...) after 4 tries (pulpcore.exceptions.validation.DigestValidationError: A file located at the url https://cdn.fwupd.org/downloads/firmware.xml.gz failed validation due to checksum.) Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: pulpcore.tasking.pulpcore_worker:INFO: Task c816c293-7879-4796-b271-d6ec059471d5 failed (A file located at the url https://cdn.fwupd.org/downloads/firmware.xml.gz failed validation due to checksum.) Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: pulp [1470469e-04dc-446e-9544-87c328e01605]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 410, in _perform_task Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: result = func(*args, **kwargs) Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/pulp_file/app/tasks/synchronizing.py", line 51, in synchronize Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: rv = dv.create() Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: loop.run_until_complete(pipeline) Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: return future.result() Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: await asyncio.gather(*futures) Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__ Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: await self.run() Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 183, in run Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: pb.done += task.result() # download_count Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 209, in _handle_content_unit Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: await asyncio.gather(*downloaders_for_content) Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/models.py", line 89, in download Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: download_result = await downloader.run(extra_data=self.extra_data) Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 273, in run Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: return await download_wrapper() Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/backoff/_async.py", line 133, in retry Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: ret = await target(*args, **kwargs) Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 258, in download_wrapper Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: return await self._run(extra_data=extra_data) Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 295, in _run Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: to_return = await self._handle_response(response) Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 209, in _handle_response Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: await self.finalize() Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/pulpcore/download/base.py", line 173, in finalize Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: self.validate_digests() Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: File "/usr/lib/python3.9/site-packages/pulpcore/download/base.py", line 223, in validate_digests Jul 20 15:04:43 satellite612 pulpcore-worker-2[1325374]: raise DigestValidationError(self.url) Jul 20 15:04:43 satellite612 pulpcore-api[869944]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:43 +0000] "GET /pulp/api/v3/tasks/c816c293-7879-4796-b271-d6ec059471d5/ HTTP/1.1" 200 3573 "-" "OpenAPI-Generator/3.18.5/ruby" Jul 20 15:04:43 satellite612 pulpcore-api[869945]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:43 +0000] "GET /pulp/api/v3/distributions/file/file/?base_path=RedHat%2FLibrary%2Fcustom%2Ffwupdates%2Ffwupdates HTTP/1.1" 200 530 "-" "OpenAPI-Generator/1.10.0/ruby" Jul 20 15:04:43 satellite612 pulpcore-api[869946]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:43 +0000] "GET /pulp/api/v3/publications/file/file/73f6b013-88e0-4ab4-8d5d-802b56313ec7/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/1.10.0/ruby" Jul 20 15:04:44 satellite612 pulpcore-api[869945]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:44 +0000] "PATCH /pulp/api/v3/distributions/file/file/db5119ea-d1ee-43ab-b2d5-cb53c4d6d500/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/1.10.0/ruby" Jul 20 15:04:44 satellite612 pulpcore-worker-2[1325391]: pulp [1470469e-04dc-446e-9544-87c328e01605]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 6e8a8514-40cf-47bf-a472-57506edbfd26 Jul 20 15:04:44 satellite612 pulpcore-api[869944]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:44 +0000] "GET /pulp/api/v3/tasks/6e8a8514-40cf-47bf-a472-57506edbfd26/ HTTP/1.1" 200 535 "-" "OpenAPI-Generator/3.18.5/ruby" Jul 20 15:04:44 satellite612 pulpcore-worker-2[1325391]: pulp [1470469e-04dc-446e-9544-87c328e01605]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 6e8a8514-40cf-47bf-a472-57506edbfd26 Jul 20 15:04:44 satellite612 pulpcore-api[869944]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:44 +0000] "GET /pulp/api/v3/tasks/6e8a8514-40cf-47bf-a472-57506edbfd26/ HTTP/1.1" 200 562 "-" "OpenAPI-Generator/3.18.5/ruby" Jul 20 15:04:44 satellite612 pulpcore-api[869945]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:44 +0000] "GET /pulp/api/v3/publications/file/file/73f6b013-88e0-4ab4-8d5d-802b56313ec7/ HTTP/1.1" 200 453 "-" "OpenAPI-Generator/1.10.0/ruby" Jul 20 15:04:44 satellite612 pulpcore-api[869944]: pulp [1470469e-04dc-446e-9544-87c328e01605]: - - [20/Jul/2023:09:34:44 +0000] "PATCH /pulp/api/v3/distributions/file/file/db5119ea-d1ee-43ab-b2d5-cb53c4d6d500/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/1.10.0/ruby" Jul 20 15:04:45 satellite612 pulpcore-worker-2[1325397]: pulp [1470469e-04dc-446e-9544-87c328e01605]: pulpcore.tasking.pulpcore_worker:INFO: Starting task 28580d5a-2316-407f-b7f6-d076aa1fefe0 Jul 20 15:04:45 satellite612 pulpcore-worker-2[1325397]: pulp [1470469e-04dc-446e-9544-87c328e01605]: pulpcore.tasking.pulpcore_worker:INFO: Task completed 28580d5a-2316-407f-b7f6-d076aa1fefe0 Expected results: No such errors. Additional info: Relevant upstream issues: Checksum validation error at Satellite repo sync · Issue #5970 · fwupd/fwupd https://github.com/fwupd/fwupd/issues/5970 Issue #9224: Pulp 3.14 - can't sync a repository because of a checksum - RPM Support - Pulp https://pulp.plan.io/issues/9224 Story #3907: As a plugin writer, the HttpDownloader provides auto_decompress=True like aiohttp - Pulp https://pulp.plan.io/issues/3907 Respect Accept-Encoding for already compressed content · Issue #1309 · pulp/pulp-smash https://github.com/pulp/pulp-smash/issues/1309 Apache headers : # curl -sI https://cdn.fwupd.org/downloads/firmware.xml.gz | grep content content-type: text/xml; charset=utf-8 content-encoding: gzip content-disposition: inline; filename=firmware.xml.gz content-length: 919098 So i am assuming "content-encoding: gzip" is the issue here, causing pulp(aiohttp) to extract the file and then comparing the actual checksum of the file with the checksum of the extracted file.