Description of problem: If the satellite\capsule servers are missing artifacts related to both rpm and docker type repos, Then The Repair API, when executed, may not be able to repair all the rpm data and stop it's action if it fails to process any docker manifest-related artifacts. Version-Release number of selected component (if applicable): satellite-6.13.0-6.el8sat.noarch katello-4.7.0-1.el8sat.noarch python39-pulpcore-3.21.5-1.1.el8pc.noarch python39-pulp-rpm-3.18.11-1.el8pc.noarch How reproducible: Always Steps to Reproduce: 1. Complete The reproducer from https://bugzilla.redhat.com/show_bug.cgi?id=2173671 2. Enable Red Hat Satellite Client 6 for RHEL 8 x86_64 (RPMs) 3. Change the download policy to immediate and sync the repo 4. Ensure that we have two repos enabled now in satellite, yum as well as docker type 5. Delete any artifacts related to the satellite client repo 6. Run the Repair API Actual results: Step 3: Added Rpms: 16, Errata: 3 Total steps: 65/65 -------------------------------- Associating Content: 20/20 Downloading Artifacts: 16/16 Downloading Metadata Files: 9/9 Parsed Advisories: 3/3 Parsed Comps: 1/1 Parsed Packages: 16/16 Skipping Packages: 0/0 Un-Associating Content: 0/0 Step 4: # hammer repository list --fields "Id,Name,Content Type" ---|---------------------------------------------------|------------- ID | NAME | CONTENT TYPE ---|---------------------------------------------------|------------- 2 | Red Hat Satellite Client 6 for RHEL 8 x86_64 RPMs | yum 1 | rhel8/toolbox | docker ---|---------------------------------------------------|------------- Step 5: # echo "COPY (select base_path from core_distribution where base_path like '%client%') TO STDOUT;" | su - postgres -c "psql pulpcore" RedHat/Library/content/dist/layered/rhel8/x86_64/sat-client/6/os # echo "COPY(select cpa.relative_path,ca.file from core_distribution as core_d RIGHT JOIN core_publication as cp ON core_d.publication_id=cp.pulp_id LEFT JOIN core_publishedartifact as cpa ON cp.pulp_id=cpa.publication_id LEFT JOIN core_contentartifact as core_ca ON core_ca.pulp_id=cpa.content_artifact_id LEFT JOIN core_artifact as ca ON ca.pulp_id=core_ca.artifact_id where core_d.base_path = 'RedHat/Library/content/dist/layered/rhel8/x86_64/sat-client/6/os' order by core_d.base_path) TO STDOUT WITH CSV" | su - postgres -c "psql pulpcore" > /tmp/filename_to_artifact_mapping.txt # for map in $(cat /tmp/filename_to_artifact_mapping.txt) do FILENAME=$(echo $map | awk -F',' '{print $1}') FULLPATH="/var/lib/pulp/media/$(echo $map | cut -d, -f2)" if [ $FULLPATH == "/var/lib/pulp/media/" ] then STAT="Not-Downloaded" else STAT=$(test -e $FULLPATH && echo "PRESENT" || echo "ABSENT") fi echo "$FILENAME,$STAT" done | column -s, -t Packages/f/foreman_ygg_worker-0.1.1-2.el8sat.x86_64.rpm PRESENT Packages/g/gofer-2.12.5-7.el8sat.noarch.rpm PRESENT Packages/g/gofer-2.12.5-8.el8sat.noarch.rpm PRESENT Packages/k/katello-agent-3.5.7-3.el8sat.noarch.rpm PRESENT Packages/k/katello-host-tools-3.5.7-3.el8sat.noarch.rpm PRESENT Packages/k/katello-host-tools-tracer-3.5.7-3.el8sat.noarch.rpm PRESENT Packages/k/katello-pull-transport-migrate-1.0.2-1.el8sat.noarch.rpm PRESENT Packages/p/puppet-agent-7.16.0-2.el8sat.x86_64.rpm PRESENT Packages/p/python3-gofer-2.12.5-7.el8sat.noarch.rpm PRESENT Packages/p/python3-gofer-2.12.5-8.el8sat.noarch.rpm PRESENT Packages/p/python3-gofer-proton-2.12.5-7.el8sat.noarch.rpm PRESENT Packages/p/python3-gofer-proton-2.12.5-8.el8sat.noarch.rpm PRESENT Packages/p/python3-qpid-proton-0.33.0-4.el8.x86_64.rpm PRESENT Packages/q/qpid-proton-c-0.33.0-4.el8.x86_64.rpm PRESENT Packages/r/rubygem-foreman_scap_client-0.5.0-1.el8sat.noarch.rpm PRESENT Packages/y/yggdrasil-0.2.0-2.el8sat.x86_64.rpm PRESENT repodata/2ec7ea4c19077fc0f495944e5ed4a8bb3dc9af45bfe6634dbb7aabc2cb41fb6d-updateinfo.xml.gz PRESENT repodata/37e6338d623013fd1285fd13033596e2e36c55f0db8b6635f8422a079fb98568-primary.xml.gz PRESENT repodata/4badd0660333ce1828baa560d2208b1439e969de67d9848301894614310f221a-filelists.sqlite.bz2 PRESENT repodata/77895819bddc7dcd7cb8ad334cfe33aee2f00c0fd03ab663914ce280944e4ff2-comps.xml PRESENT repodata/9b0bd210bfc9887bf17a2bf26e986285a9ab3190cec96b5ec5972df56c546713-filelists.xml.gz PRESENT repodata/c4cc5c111ae7df9225eea3f1ac0bb6dd44706aeb643b9b89495d1bb16cdf0fd9-other.xml.gz PRESENT repodata/e735df44940193e25ea4e467630ef8a4e94e55d4b4d37e320d463dc3f5ca1669-other.sqlite.bz2 PRESENT repodata/fdddabeef43c982760a1c94724e98f9314974fe7a0b67e13c12af5726e48025f-primary.sqlite.bz2 PRESENT repodata/repomd.xml PRESENT # for map in $(cat /tmp/filename_to_artifact_mapping.txt) do FULLPATH="/var/lib/pulp/media/$(echo $map | cut -d, -f2)" if [ $FULLPATH == "/var/lib/pulp/media/" ] then echo '' else rm -f $FULLPATH fi done # for map in $(cat /tmp/filename_to_artifact_mapping.txt) do FILENAME=$(echo $map | awk -F',' '{print $1}') FULLPATH="/var/lib/pulp/media/$(echo $map | cut -d, -f2)" if [ $FULLPATH == "/var/lib/pulp/media/" ] then STAT="Not-Downloaded" else STAT=$(test -e $FULLPATH && echo "PRESENT" || echo "ABSENT") fi echo "$FILENAME,$STAT" done | column -s, -t Packages/f/foreman_ygg_worker-0.1.1-2.el8sat.x86_64.rpm ABSENT Packages/g/gofer-2.12.5-7.el8sat.noarch.rpm ABSENT Packages/g/gofer-2.12.5-8.el8sat.noarch.rpm ABSENT Packages/k/katello-agent-3.5.7-3.el8sat.noarch.rpm ABSENT Packages/k/katello-host-tools-3.5.7-3.el8sat.noarch.rpm ABSENT Packages/k/katello-host-tools-tracer-3.5.7-3.el8sat.noarch.rpm ABSENT Packages/k/katello-pull-transport-migrate-1.0.2-1.el8sat.noarch.rpm ABSENT Packages/p/puppet-agent-7.16.0-2.el8sat.x86_64.rpm ABSENT Packages/p/python3-gofer-2.12.5-7.el8sat.noarch.rpm ABSENT Packages/p/python3-gofer-2.12.5-8.el8sat.noarch.rpm ABSENT Packages/p/python3-gofer-proton-2.12.5-7.el8sat.noarch.rpm ABSENT Packages/p/python3-gofer-proton-2.12.5-8.el8sat.noarch.rpm ABSENT Packages/p/python3-qpid-proton-0.33.0-4.el8.x86_64.rpm ABSENT Packages/q/qpid-proton-c-0.33.0-4.el8.x86_64.rpm ABSENT Packages/r/rubygem-foreman_scap_client-0.5.0-1.el8sat.noarch.rpm ABSENT Packages/y/yggdrasil-0.2.0-2.el8sat.x86_64.rpm ABSENT repodata/2ec7ea4c19077fc0f495944e5ed4a8bb3dc9af45bfe6634dbb7aabc2cb41fb6d-updateinfo.xml.gz ABSENT repodata/37e6338d623013fd1285fd13033596e2e36c55f0db8b6635f8422a079fb98568-primary.xml.gz ABSENT repodata/4badd0660333ce1828baa560d2208b1439e969de67d9848301894614310f221a-filelists.sqlite.bz2 ABSENT repodata/77895819bddc7dcd7cb8ad334cfe33aee2f00c0fd03ab663914ce280944e4ff2-comps.xml ABSENT repodata/9b0bd210bfc9887bf17a2bf26e986285a9ab3190cec96b5ec5972df56c546713-filelists.xml.gz ABSENT repodata/c4cc5c111ae7df9225eea3f1ac0bb6dd44706aeb643b9b89495d1bb16cdf0fd9-other.xml.gz ABSENT repodata/e735df44940193e25ea4e467630ef8a4e94e55d4b4d37e320d463dc3f5ca1669-other.sqlite.bz2 ABSENT repodata/fdddabeef43c982760a1c94724e98f9314974fe7a0b67e13c12af5726e48025f-primary.sqlite.bz2 ABSENT repodata/repomd.xml ABSENT # file /var/lib/pulp/media/artifact/*/* /var/lib/pulp/media/artifact/1c/b61ea996355add02b1426ed4c1780ea75ce0c04c5d1107c025c3fbd7d8bcae: gzip compressed data, from Unix, original size 167 /var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text /var/lib/pulp/media/artifact/5d/88a016637781e6d3eeb22c15c788ce1cc572e2dbb7bad4935d37a775c252d9: gzip compressed data, original size 169271808 /var/lib/pulp/media/artifact/64/554374912209545e580ebabc0c025363b6982d1fbffb00c50edfeea18e79b9: gzip compressed data, original size 212109824 /var/lib/pulp/media/artifact/6d/2cf55f7908bf17bd221297c8cff9750fad3187c3eb102d1c3f72a208efbf0b: gzip compressed data, original size 235748864 /var/lib/pulp/media/artifact/70/2f3a622329ec2d54c1bf3c13fa094c422d48abaad700a2e055c3b77bea15c0: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/95/a4415d859d7120efb6b3cf964c07bebbff9a5275ca673e6e74a97bcbfb2a5f: gzip compressed data, from Unix, original size 125 /var/lib/pulp/media/artifact/a5/d805d6872a1693a518e623b95182f115915c37671f70c1b13eb136e930bfe4: gzip compressed data, original size 204123648 /var/lib/pulp/media/artifact/ac/88d64d7b415280e8c1221725a8455917b99711b754a5981d3460fa9fdf40f9: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/c0/cace4d6bd977ad29096c46467682b30b7ef65aff60fe7f7238096f9003d898: gzip compressed data, from Unix, original size 59 /var/lib/pulp/media/artifact/cb/920667676480deb7ebd62198e571e416462118d58b9c280b413fec31745e4e: XML 1.0 document, ASCII text /var/lib/pulp/media/artifact/d0/898311a8acc605593edbc1f37171f42aab926fffd820a0f0460e062a4f6717: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/d5/2583c79ff512b040d97ca0ce70b61b9c496e674b789f8857efbcf93727e4b0: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/ef/3e20691954c3d1318ec3071a982da339f4ed76967ded668b795c9e070aaab6: gzip compressed data, from Unix, original size 121 Step 6: # curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X POST https://`hostname -f`/pulp/api/v3/repair/ | json_reformat % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 67 100 67 0 0 683 0 --:--:-- --:--:-- --:--:-- 683 { "task": "/pulp/api/v3/tasks/783ec953-8902-4d1d-9566-1ed8f7a5a00f/" } # curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X GET https://`hostname -f`/pulp/api/v3/tasks/783ec953-8902-4d1d-9566-1ed8f7a5a00f/ | json_reformat % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2860 100 2860 0 0 42058 0 --:--:-- --:--:-- --:--:-- 42058 { "pulp_href": "/pulp/api/v3/tasks/783ec953-8902-4d1d-9566-1ed8f7a5a00f/", "pulp_created": "2023-02-27T17:00:38.933371Z", "state": "failed", "name": "pulpcore.app.tasks.repository.repair_all_artifacts", "logging_cid": "3997124f3767438e86884df2e6d3519d", "started_at": "2023-02-27T17:00:38.970659Z", "finished_at": "2023-02-27T17:00:40.625673Z", "error": { "traceback": " File \"/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 452, in _perform_task\n result = func(*args, **kwargs)\n File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 192, in repair_all_artifacts\n loop.run_until_complete(_repair_artifacts_for_content(verify_checksums=verify_checksums))\n File \"/usr/lib64/python3.9/asyncio/base_events.py\", line 647, in run_until_complete\n return future.result()\n File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 148, in _repair_artifacts_for_content\n await asyncio.gather(*done) # Clean up tasks\n File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 77, in _repair_ca\n dl_result = await downloader.run()\n File \"/usr/lib/python3.9/site-packages/pulpcore/download/http.py\", line 273, in run\n return await download_wrapper()\n File \"/usr/lib/python3.9/site-packages/backoff/_async.py\", line 151, in retry\n ret = await target(*args, **kwargs)\n File \"/usr/lib/python3.9/site-packages/pulpcore/download/http.py\", line 258, in download_wrapper\n return await self._run(extra_data=extra_data)\n File \"/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py\", line 81, in _run\n return await self._run(handle_401=False, extra_data=extra_data)\n File \"/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py\", line 66, in _run\n response.raise_for_status()\n File \"/usr/lib64/python3.9/site-packages/aiohttp/client_reqrep.py\", line 1004, in raise_for_status\n raise ClientResponseError(\n", "description": "404, message='Not Found', url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?_auth_=exp=1677520840~hmac=4c1a924c0c8b5af741a5025eaa730f8b5fca99f539f5004d65aeb6bc136c3db4')" }, "worker": "/pulp/api/v3/workers/60781c7e-a9fd-4386-8ba6-6c7d4785a9a1/", "parent_task": null, "child_tasks": [ ], "task_group": null, "progress_reports": [ { "message": "Repair corrupted units", "code": "repair.repaired", "state": "failed", "total": null, "done": 0, "suffix": null }, { "message": "Identify corrupted units", "code": "repair.corrupted", "state": "failed", "total": null, "done": 0, "suffix": null }, { "message": "Identify missing units", "code": "repair.missing", "state": "failed", "total": null, "done": 6, "suffix": null } ], "created_resources": [ ], "reserved_resources_record": [ ] } # file /var/lib/pulp/media/artifact/*/* /var/lib/pulp/media/artifact/1c/b61ea996355add02b1426ed4c1780ea75ce0c04c5d1107c025c3fbd7d8bcae: gzip compressed data, from Unix, original size 167 /var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text /var/lib/pulp/media/artifact/5d/88a016637781e6d3eeb22c15c788ce1cc572e2dbb7bad4935d37a775c252d9: gzip compressed data, original size 169271808 /var/lib/pulp/media/artifact/64/554374912209545e580ebabc0c025363b6982d1fbffb00c50edfeea18e79b9: gzip compressed data, original size 212109824 /var/lib/pulp/media/artifact/6d/2cf55f7908bf17bd221297c8cff9750fad3187c3eb102d1c3f72a208efbf0b: gzip compressed data, original size 235748864 /var/lib/pulp/media/artifact/70/2f3a622329ec2d54c1bf3c13fa094c422d48abaad700a2e055c3b77bea15c0: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/95/a4415d859d7120efb6b3cf964c07bebbff9a5275ca673e6e74a97bcbfb2a5f: gzip compressed data, from Unix, original size 125 /var/lib/pulp/media/artifact/a5/d805d6872a1693a518e623b95182f115915c37671f70c1b13eb136e930bfe4: gzip compressed data, original size 204123648 /var/lib/pulp/media/artifact/ac/88d64d7b415280e8c1221725a8455917b99711b754a5981d3460fa9fdf40f9: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/c0/cace4d6bd977ad29096c46467682b30b7ef65aff60fe7f7238096f9003d898: gzip compressed data, from Unix, original size 59 /var/lib/pulp/media/artifact/cb/920667676480deb7ebd62198e571e416462118d58b9c280b413fec31745e4e: XML 1.0 document, ASCII text /var/lib/pulp/media/artifact/d0/898311a8acc605593edbc1f37171f42aab926fffd820a0f0460e062a4f6717: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/d5/2583c79ff512b040d97ca0ce70b61b9c496e674b789f8857efbcf93727e4b0: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/ef/3e20691954c3d1318ec3071a982da339f4ed76967ded668b795c9e070aaab6: gzip compressed data, from Unix, original size 121 Expected results: The Repair API should be able to skip the failed units and proceed fixing the other RPM-type units that It can. Additional info: * This is not a huge blocker for satellite as satellite ( via katello ) offers an easy way to run repair API per repo . * But when it comes to Capsules, That is not the case and hence the Repair action needs to be executed for all repos and that will fails at the very beginning ( under the specific condition as mentioned earlier )
Is this a regression from 6.12? Thanks!
Hello, The behavior is the same in 6.12 as well. So no regression. I was not able to test it during 6.12 GA properly and hence tested it during 6.13. The "Repair API" of pulp is something really important for Satellite to have working properly. This is the key to get back all downloadable pulp artifacts after a backup without pulp data was restored or the pulp filesystem was corrupted and later recreated.
The Pulp upstream bug status is at closed. Updating the external tracker on this bug.
Failed QA As per steps performed in comment 8
Requesting needsinfo from upstream developer dkliban, ggainey because the 'FailedQA' flag is set.
This fix was primary focused to cover ``raise ClientResponseError(\n", "description": "404, message='Not Found', url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?_auth_=exp=1677520840~hmac=4c1a924c0c8b5af741a5025eaa730f8b5fca99f539f5004d65aeb6bc136c3db4')"`` as stated in the BZ report. Verification steps fail with ``raise TimeoutException(self.url)\n", "description": "Request timed out for https://registry.redhat.io/v2/rhel8/toolbox/manifests/sha256:629545c824752b6d75f7f22a73b6c7fcba9413b6017f2f67f58494a958bcd4ce. Increasing the total_timeout value on the remote might help."`` which is a different error and not covered by this fix. We need agree on the definition of done this BZ request, because it was not intended to catch all the errors that might come from the server. What is the preferred way to proceed?
(In reply to Ina Panova from comment #11) > This fix was primary focused to cover ``raise ClientResponseError(\n", > "description": "404, message='Not Found', > url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/ > multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256: > 0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?_auth_=exp=1 > 677520840~hmac=4c1a924c0c8b5af741a5025eaa730f8b5fca99f539f5004d65aeb6bc136c3d > b4')"`` as stated in the BZ report. > > Verification steps fail with ``raise TimeoutException(self.url)\n", > "description": "Request timed out for > https://registry.redhat.io/v2/rhel8/toolbox/manifests/sha256: > 629545c824752b6d75f7f22a73b6c7fcba9413b6017f2f67f58494a958bcd4ce. Increasing > the total_timeout value on the remote might help."`` which is a different > error and not covered by this fix. > > We need agree on the definition of done this BZ request, because it was not > intended to catch all the errors that might come from the server. > What is the preferred way to proceed? Hello, Definitely, 404 was the error that I received initially because of the way it was reproduced. From that point of view, ClientResponseError was handled via https://github.com/pulp/pulpcore/commit/72f1902bd9d5b4d4237297b3c68a617682f4b5f7 and it's fine. But that is not the only error that someone can receive. If Satellite\Pulp's remote is associated with a proxy and that proxy is limiting connections or misconfigured someway, Then TimeoutException would be a common thing to occur as well. If Firewall is not properly configured that may even result in "Connection Reset" but I don't know how Pulp will show that message. ( but this is not very important at this moment ). Anyways, The point is, No matter what the failure is, The Repair API, should not halt but continue processing rest of the units that it has identified as corrupted or missing and that has always been the goal of this BZ ( if you see the "Expected Results" section in Description ). My request would be to handle the TimeoutException in the same way as ClientResponseError was done as these two are the commonly seen errors when it comes to pulp trying to download something from upstream or another pulp instance. -- Sayan
Sayan, thanks for clarifying the scope. I have filed a new upstream request https://github.com/pulp/pulpcore/issues/4111
Verified Version Tested: Satellite 6.14.0 Snap 9