Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2262110

Summary:	pulpcore-content does not validate content of downloaded on-demand artifacts before delivering them to clients
Product:	Red Hat Satellite	Reporter:	Joniel Pasqualetto <jpasqual>
Component:	Pulp	Assignee:	satellite6-bugs <satellite6-bugs>
Status:	CLOSED MIGRATED	QA Contact:	Satellite QE Team <sat-qe-bz-list>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.13.6	CC:	dalley, dkliban, ggainey, rchan, rlavi
Target Milestone:	Unspecified	Keywords:	MigratedToJIRA, Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2024-06-06 17:01:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Joniel Pasqualetto 2024-01-31 16:22:27 UTC

Description of problem:

When using download on-demand, if an artifact upstream changed between the moment the sync happened and the first time a user requests it, pulpcore-content will download the new file (which won't have the same checksum) and deliver it to the user without verifying if it is a match. While saving the downloaded file, a checksum verification is done and the artifact is not saved (because of mismatching checksum), but for the final  user's eyes nothing wrong happened. Regardless of no errors being shown to the user, they receive a bad artifact.

Version-Release number of selected component (if applicable):

Observed in the wild on Satellite 6.11. 
Reproduced internally on 6.11 and 6.13 (didn't test 6.12 and 6.14)

How reproducible:

Always when circumstances met. 

Steps to Reproduce:

1. Create a local repository from where pulp will sync. Ensure you will be able to make changes on the repository content.

Example of a small repository, easy to create on localhost:

~~~
mkdir /var/www/html/pub/repo           
cd /var/www/html/pub/repo
curl -O https://repos.fedorapeople.org/pulp/pulp/demo_repos/zoo/bear-4.1-1.noarch.rpm
createrepo .
~~~

2. On Satellite, create a product and a repository that will sync from http://satellite.example.com/pub/repo/ and sync it:

~~~
hammer product create --name TestProduct --organization-id 1
hammer repository create --url http://localhost/pub/repo/ --name ChangingRepo --product TestProduct --mirroring-policy mirror_complete --organization-id 1 --download-policy on_demand --content-type yum
hammer repository synchronize --name ChangingRepo --product TestProduct --organization-id 1
~~~

3. Change the content of the rpmfile bear-4.1-1.noarch.rpm from the upstream (on localhost) repository. Ensure the checksum of the file changed!!

~~~
$ sha256sum /var/www/html/pub/repo/bear-4.1-1.noarch.rpm 
7a831f9f90bf4d21027572cb503d20b702de8e8785b02c0397445c2e481d81b3  /var/www/html/pub/repo/bear-4.1-1.noarch.rpm    <=== original checksum 

$ echo "CORRUPTED" >> /var/www/html/pub/repo/bear-4.1-1.noarch.rpm                                           <=== forcing a corruption of the file

$ sha256sum /var/www/html/pub/repo/bear-4.1-1.noarch.rpm 
1469d4fc47073fffbecf8e68419adf804de3c9c3645719779778fd0d634fca0e  /var/www/html/pub/repo/bear-4.1-1.noarch.rpm   <==== corrupted checksum
~~~

4. Try downloading the file from the repository created on Satellite:

# hammer repository info --name ChangingRepo --product TestProduct --organization-id 1 --fields "Published At"
Published At: https://satellite.example.com/pulp/content/DefaultOrganization/Library/custom/TestProduct/ChangingRepo/

# curl -O https://satellite.example.com/pulp/content/DefaultOrganization/Library/custom/TestProduct/ChangingRepo/bear-4.1-1.noarch.rpm
# echo $?     <=== note that command returns no error

# sha256sum bear-4.1-1.noarch.rpm         <====== checksum is something else

Actual results:

Client receives a bad file and no errors.

Pulpcore-content logs errors when saving the artifact:

~~~
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: Giving up download_wrapper(...) after 5 tries (pulpcore.exceptions.validation.DigestValidationError: A file located at the url http://localhost/pub/repo/bear-4.1-1.noarch.rpm failed validation due to checksum. Expected '7a831f9f90bf4d21027572cb503d20b702de8e8785b02c0397445c2e481d81b3', Actual '1469d4fc47073fffbecf8e68419adf804de3c9c3645719779778fd0d634fca0e')
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: pulp [None]: backoff:ERROR: Giving up download_wrapper(...) after 5 tries (pulpcore.exceptions.validation.DigestValidationError: A file located at the url http://localhost/pub/repo/bear-4.1-1.noarch.rpm failed validation due to checksum. Expected '7a831f9f90bf4d21027572cb503d20b702de8e8785b02c0397445c2e481d81b3', Actual '1469d4fc47073fffbecf8e68419adf804de3c9c3645719779778fd0d634fca0e')
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: [2024-01-31 16:10:08 +0000] [1141] [ERROR] Error handling request
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: Traceback (most recent call last):
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib64/python3.9/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     resp = await request_handler(request)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib64/python3.9/site-packages/aiohttp/web_app.py", line 504, in _handle
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     resp = await handler(request)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib64/python3.9/site-packages/aiohttp/web_middlewares.py", line 117, in impl
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     return await handler(request)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulpcore/content/authentication.py", line 41, in authenticate
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     return await handler(request)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 339, in cached_function
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     response = await self.make_entry(
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulpcore/cache/cache.py", line 378, in make_entry
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     response = await handler(*args, **kwargs)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulpcore/content/handler.py", line 237, in stream_content
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     return await self._match_and_stream(path, request)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulpcore/content/handler.py", line 544, in _match_and_stream
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     return await self._stream_content_artifact(
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulpcore/content/handler.py", line 693, in _stream_content_artifact
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     response = await self._stream_remote_artifact(request, response, remote_artifact)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulpcore/content/handler.py", line 943, in _stream_remote_artifact
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     download_result = await downloader.run()
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 273, in run
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     return await download_wrapper()
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/backoff/_async.py", line 151, in retry
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     ret = await target(*args, **kwargs)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 258, in download_wrapper
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     return await self._run(extra_data=extra_data)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulp_rpm/app/downloaders.py", line 118, in _run
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     to_return = await self._handle_response(response)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 209, in _handle_response
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     await self.finalize()
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulpcore/content/handler.py", line 934, in finalize
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     await original_finalize()
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulpcore/download/base.py", line 171, in finalize
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     self.validate_digests()
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:   File "/usr/lib/python3.9/site-packages/pulpcore/download/base.py", line 223, in validate_digests
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]:     raise DigestValidationError(actual_digest, expected_digest, url=self.url)
Jan 31 11:10:08 satellite.example.com pulpcore-content[1141]: pulpcore.exceptions.validation.DigestValidationError: A file located at the url http://localhost/pub/repo/bear-4.1-1.noarch.rpm failed validation due to checksum. Expected '7a831f9f90bf4d21027572cb503d20b702de8e8785b02c0397445c2e481d81b3', Actual '1469d4fc47073fffbecf8e68419adf804de3c9c3645719779778fd0d634fca0e'
~~~

Expected results:

Client would receive an error instead of the bad file that does not match the metadata stored on the DB.

Additional info:

Steps described here are artificial (in order to facilitate reproducing it) but there are real use cases being hit by customers.

I personally observed this behavior on Satellite 6.11 and 6.13 (didn't tested other versions but I suppose they behave the same)

Comment 1 Daniel Alley 2024-02-01 03:43:22 UTC

Joniel, what do you think is the correct outcome here? There is no HTTP status code for bad checksum, so I think returning status code 410 or 404 might be the best we can do - the original file is very likely gone forever (such is the risk of using on-demand).  Any customer that can't tolerate any possibility of such files disappearing probably needs to use immediate mode.

Maybe 410 makes the most sense?  https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/410

Comment 2 Joniel Pasqualetto 2024-02-01 14:17:41 UTC

Hello Daniel

I think 404 makes more sense. The fact that the downloaded artifact is not a match for the checksum we expect may simply mean the file somehow got corrupted upstream and not necessarily is gone forever.

On pulp2, when on-demand packages couldn't be retrieved for some reason, I remember getting 404 on the client.

Comment 3 Eric Helms 2024-06-06 17:01:16 UTC

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "SAT-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.