Description of problem: Repair API of Pulp3 ( Verify Content Checksum of Katello ) cannot get back\fix docker manifests Version-Release number of selected component (if applicable): satellite-6.13.0-6.el8sat.noarch katello-4.7.0-1.el8sat.noarch python39-pulpcore-3.21.5-1.1.el8pc.noarch python39-pulp-rpm-3.18.11-1.el8pc.noarch How reproducible: Always Steps to Reproduce: 1. Install a Satellite 6.13 and import a manifest with some valid subs 2. Use the repo_discovery feature to discover the "rhel8/toolbox" repo and create it inside the Containers product 3. Go to Satellite UI Content --> Products --> Containers --> Repositories --> rhel8/toolbox and add "8.7-5" in the "Include Tags" fields and sync the repo. ( It will only sync that tag now ) 4. Delete the artifacts # rm -rf /var/lib/pulp/media/artifact/* 5. Go to Satellite UI Content --> Products --> Containers --> Repositories --> rhel8/toolbox --> Select Action dropdown --> Sync the repo. 6. Go to Satellite UI Content --> Products --> Containers --> Repositories --> rhel8/toolbox --> Select Action dropdown --> Verify Content Checksum Or else just invoke the Repair API # curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X POST https://`hostname -f`/pulp/api/v3/repair/ | json_reformat Actual results: Step 3: Sync status from Sat UI: Added Docker tags: 1 Total steps: 32/32 -------------------------------- Associating Content: 18/18 Downloading Artifacts: 12/12 Downloading tag list: 1/1 Processing Tags: 1/1 Un-Associating Content: 0/0 # du -sh /var/lib/pulp/media/artifact/ 556M /var/lib/pulp/media/artifact/ # file /var/lib/pulp/media/artifact/*/* /var/lib/pulp/media/artifact/01/65232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276: ASCII text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text /var/lib/pulp/media/artifact/5d/88a016637781e6d3eeb22c15c788ce1cc572e2dbb7bad4935d37a775c252d9: gzip compressed data, original size 169271808 /var/lib/pulp/media/artifact/62/9545c824752b6d75f7f22a73b6c7fcba9413b6017f2f67f58494a958bcd4ce: ASCII text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/64/554374912209545e580ebabc0c025363b6982d1fbffb00c50edfeea18e79b9: gzip compressed data, original size 212109824 /var/lib/pulp/media/artifact/6d/2cf55f7908bf17bd221297c8cff9750fad3187c3eb102d1c3f72a208efbf0b: gzip compressed data, original size 235748864 /var/lib/pulp/media/artifact/70/2f3a622329ec2d54c1bf3c13fa094c422d48abaad700a2e055c3b77bea15c0: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/7f/060f344d579611663e84ea94d89d9372d7431e3e56e11a19e704207d5576e9: ASCII text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/9a/394c72313c9bcac63dc8980018582d90ece7ae65d03940e6d7e9b8d4cb884d: gzip compressed data, original size 264029696 /var/lib/pulp/media/artifact/a5/d805d6872a1693a518e623b95182f115915c37671f70c1b13eb136e930bfe4: gzip compressed data, original size 204123648 /var/lib/pulp/media/artifact/ac/88d64d7b415280e8c1221725a8455917b99711b754a5981d3460fa9fdf40f9: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/b1/45dc70f6c036069d549762f39055162cac12b414e9e502fd0d4b3331a0f5eb: gzip compressed data, original size 171049472 /var/lib/pulp/media/artifact/b9/2727ef7443d73bb77323e5a969bf02187286fccf53930e83dc82be60a36529: gzip compressed data, original size 214139392 /var/lib/pulp/media/artifact/c6/25ce0354e96c43d4bfca9cb0a3ed8373b0707980f152582f731e61808df9bd: ASCII text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/d0/898311a8acc605593edbc1f37171f42aab926fffd820a0f0460e062a4f6717: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/d4/d59e4eb38ff5eda9a9f6255575abc64501adb78153f5788bf580b1034d4f8d: gzip compressed data, original size 186984960 /var/lib/pulp/media/artifact/d5/2583c79ff512b040d97ca0ce70b61b9c496e674b789f8857efbcf93727e4b0: UTF-8 Unicode text, with very long lines, with no line terminators Step 4: # rm -rf /var/lib/pulp/media/artifact/* # du -sh /var/lib/pulp/media/artifact/ 0 /var/lib/pulp/media/artifact/ Step 5: Sync fails but able to get some data in pulp: # file /var/lib/pulp/media/artifact/*/* /var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text Step 6: # curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X POST https://`hostname -f`/pulp/api/v3/repair/ | json_reformat % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 67 100 67 0 0 881 0 --:--:-- --:--:-- --:--:-- 881 { "task": "/pulp/api/v3/tasks/bff6cd3c-c7cb-43c1-a717-ec68512e10fa/" } # curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X GET https://`hostname -f`/pulp/api/v3/tasks/bff6cd3c-c7cb-43c1-a717-ec68512e10fa/ | json_reformat % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2860 100 2860 0 0 46885 0 --:--:-- --:--:-- --:--:-- 46885 { "pulp_href": "/pulp/api/v3/tasks/bff6cd3c-c7cb-43c1-a717-ec68512e10fa/", "pulp_created": "2023-02-27T15:30:24.465374Z", "state": "failed", "name": "pulpcore.app.tasks.repository.repair_all_artifacts", "logging_cid": "15f07cd0ac6146968f91e6b229189b3f", "started_at": "2023-02-27T15:30:24.508613Z", "finished_at": "2023-02-27T15:30:26.289194Z", "error": { "traceback": " File \"/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 452, in _perform_task\n result = func(*args, **kwargs)\n File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 192, in repair_all_artifacts\n loop.run_until_complete(_repair_artifacts_for_content(verify_checksums=verify_checksums))\n File \"/usr/lib64/python3.9/asyncio/base_events.py\", line 647, in run_until_complete\n return future.result()\n File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 148, in _repair_artifacts_for_content\n await asyncio.gather(*done) # Clean up tasks\n File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 77, in _repair_ca\n dl_result = await downloader.run()\n File \"/usr/lib/python3.9/site-packages/pulpcore/download/http.py\", line 273, in run\n return await download_wrapper()\n File \"/usr/lib/python3.9/site-packages/backoff/_async.py\", line 151, in retry\n ret = await target(*args, **kwargs)\n File \"/usr/lib/python3.9/site-packages/pulpcore/download/http.py\", line 258, in download_wrapper\n return await self._run(extra_data=extra_data)\n File \"/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py\", line 81, in _run\n return await self._run(handle_401=False, extra_data=extra_data)\n File \"/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py\", line 66, in _run\n response.raise_for_status()\n File \"/usr/lib64/python3.9/site-packages/aiohttp/client_reqrep.py\", line 1004, in raise_for_status\n raise ClientResponseError(\n", "description": "404, message='Not Found', url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?_auth_=exp=1677515426~hmac=8110d87e5268564dd809419bef770002c546ec040bf0f13e8733d4ae9ded8c6d')" }, "worker": "/pulp/api/v3/workers/854914bf-6e9d-4bd9-8c84-d13557f44e82/", "parent_task": null, "child_tasks": [ ], "task_group": null, "progress_reports": [ { "message": "Repair corrupted units", "code": "repair.repaired", "state": "failed", "total": null, "done": 0, "suffix": null }, { "message": "Identify corrupted units", "code": "repair.corrupted", "state": "failed", "total": null, "done": 0, "suffix": null }, { "message": "Identify missing units", "code": "repair.missing", "state": "failed", "total": null, "done": 6, "suffix": null } ], "created_resources": [ ], "reserved_resources_record": [ ] } Trace from logs: Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: pulp [15f07cd0ac6146968f91e6b229189b3f]: pulp_container.app.downloaders:INFO: Updating bearer token Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?_auth_=exp=1677515426~hmac=8110d87e5268564dd809419bef770002c546ec040bf0f13e8733d4ae9ded8c6d')) Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: pulp [15f07cd0ac6146968f91e6b229189b3f]: backoff:ERROR: Giving up download_wrapper(...) after 1 tries (aiohttp.client_exceptions.ClientResponseError: 404, message='Not Found', url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?_auth_=exp=1677515426~hmac=8110d87e5268564dd809419bef770002c546ec040bf0f13e8733d4ae9ded8c6d')) Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: pulp [15f07cd0ac6146968f91e6b229189b3f]: pulpcore.tasking.pulpcore_worker:INFO: Task bff6cd3c-c7cb-43c1-a717-ec68512e10fa failed (404, message='Not Found', url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?_auth_=exp=1677515426~hmac=8110d87e5268564dd809419bef770002c546ec040bf0f13e8733d4ae9ded8c6d')) Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: pulp [15f07cd0ac6146968f91e6b229189b3f]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 452, in _perform_task Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: result = func(*args, **kwargs) Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py", line 192, in repair_all_artifacts Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: loop.run_until_complete(_repair_artifacts_for_content(verify_checksums=verify_checksums)) Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: return future.result() Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py", line 148, in _repair_artifacts_for_content Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: await asyncio.gather(*done) # Clean up tasks Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py", line 77, in _repair_ca Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: dl_result = await downloader.run() Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 273, in run Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: return await download_wrapper() Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/backoff/_async.py", line 151, in retry Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: ret = await target(*args, **kwargs) Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulpcore/download/http.py", line 258, in download_wrapper Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: return await self._run(extra_data=extra_data) Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py", line 81, in _run Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: return await self._run(handle_401=False, extra_data=extra_data) Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py", line 66, in _run Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: response.raise_for_status() Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: File "/usr/lib64/python3.9/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status Feb 27 21:00:26 satellite613 pulpcore-worker-1[75377]: raise ClientResponseError( Filesystem state: # file /var/lib/pulp/media/artifact/*/* /var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text /var/lib/pulp/media/artifact/5d/88a016637781e6d3eeb22c15c788ce1cc572e2dbb7bad4935d37a775c252d9: gzip compressed data, original size 169271808 /var/lib/pulp/media/artifact/64/554374912209545e580ebabc0c025363b6982d1fbffb00c50edfeea18e79b9: gzip compressed data, original size 212109824 /var/lib/pulp/media/artifact/6d/2cf55f7908bf17bd221297c8cff9750fad3187c3eb102d1c3f72a208efbf0b: gzip compressed data, original size 235748864 /var/lib/pulp/media/artifact/70/2f3a622329ec2d54c1bf3c13fa094c422d48abaad700a2e055c3b77bea15c0: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/a5/d805d6872a1693a518e623b95182f115915c37671f70c1b13eb136e930bfe4: gzip compressed data, original size 204123648 /var/lib/pulp/media/artifact/ac/88d64d7b415280e8c1221725a8455917b99711b754a5981d3460fa9fdf40f9: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/d0/898311a8acc605593edbc1f37171f42aab926fffd820a0f0460e062a4f6717: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/d5/2583c79ff512b040d97ca0ce70b61b9c496e674b789f8857efbcf93727e4b0: UTF-8 Unicode text, with very long lines, with no line terminators Expected results: The Repair API should be able to gracefully get back all the syncable data ( even for container\docker type repos ) Additional info: * Reproducer details shared in the next private comment * It has been noticed that, if this API fails to fix something, Then it immediately ends the task instead of checking for the other fixable objects\units and trying to fix\redownload them.
Is this a regression from 6.12? Thanks!
Not a regression for sure. My answer remains same as https://bugzilla.redhat.com/show_bug.cgi?id=2173692#c3
Verified Satellite & package version: ---------------------------- satellite-installer-6.14.0.5-1.el8sat.noarch satellite-maintain-0.0.2-1.el8sat.noarch satellite-6.14.0-2.el8sat.noarch satellite-cli-6.14.0-2.el8sat.noarch python39-pulp-container-2.14.6-1.el8pc.noarch python39-pulp-container-2.14.6-1.el8pc.noarch Steps performed: ---------------- 1. Create product (name- Containers) and repository (type- docker)- # hammer product create --name Containers --organization "Default Organization" # hammer repository create --name "rhel8/toolbox" --docker-upstream-name "rhel8/toolbox" --content-type docker --url https://registry.redhat.io --upstream-username <kerberos username> --upstream-password <kerberos password> --docker-tags-whitelist "8.7-5" --download-policy immediate --product Containers --organization "Default Organization" 2. Sync status from Sat UI- Added Docker tags: 1 Total steps: 32/32 -------------------------------- Associating Content: 18/18 Downloading Artifacts: 12/12 Downloading tag list: 1/1 Processing Tags: 1/1 Un-Associating Content: 0/0 3. Delete the artifacts- # du -sh /var/lib/pulp/media/artifact/ 556M /var/lib/pulp/media/artifact/ # rm -rf /var/lib/pulp/media/artifact/* 4. Go to Satellite UI Content --> Products --> Containers --> Repositories --> rhel8/toolbox --> Select Action dropdown --> Sync the repo Sync failed but was able to get some data in pulp # file /var/lib/pulp/media/artifact/*/* /var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text 5. Go to Satellite UI Content --> Products --> Containers --> Repositories --> rhel8/toolbox --> Select Action dropdown --> Verify Content Checksum Verify Content Checksum from Satellite web-UI or Invoke the Repair API # curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X POST https://`hostname -f`/pulp/api/v3/repair/ | json_reformat % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 67 100 67 0 0 1488 0 --:--:-- --:--:-- --:--:-- 1488 { "task": "/pulp/api/v3/tasks/60247332-91bd-4ad6-8225-dccf328c2678/" } # curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X GET https://`hostname -f`/pulp/api/v3/tasks/60247332-91bd-4ad6-8225-dccf328c2678/ | json_reformat % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 870 100 870 0 0 21219 0 --:--:-- --:--:-- --:--:-- 21219 { "pulp_href": "/pulp/api/v3/tasks/60247332-91bd-4ad6-8225-dccf328c2678/", "pulp_created": "2023-07-06T10:16:46.862202Z", "state": "running", "name": "pulpcore.app.tasks.repository.repair_all_artifacts", "logging_cid": "d79f26f447774f7eb33eef48f3552ddc", "started_at": "2023-07-06T10:16:46.888009Z", "finished_at": null, "error": null, "worker": "/pulp/api/v3/workers/cdf1d423-79a1-42f0-84a3-321e41801e93/", "parent_task": null, "child_tasks": [ ], "task_group": null, "progress_reports": [ { "message": "Identify missing units", "code": "repair.missing", "state": "running", "total": null, "done": 16, "suffix": null }, { "message": "Identify corrupted units", "code": "repair.corrupted", "state": "running", "total": null, "done": 0, "suffix": null }, { "message": "Repair corrupted units", "code": "repair.repaired", "state": "running", "total": null, "done": 11, "suffix": null } ], "created_resources": [ ], "reserved_resources_record": [ ] } Observation: ------------ Repair API is able to download files from Upstream # file /var/lib/pulp/media/artifact/*/* /var/lib/pulp/media/artifact/01/65232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276: ASCII text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text /var/lib/pulp/media/artifact/5d/88a016637781e6d3eeb22c15c788ce1cc572e2dbb7bad4935d37a775c252d9: gzip compressed data, original size 169271808 /var/lib/pulp/media/artifact/62/9545c824752b6d75f7f22a73b6c7fcba9413b6017f2f67f58494a958bcd4ce: ASCII text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/64/554374912209545e580ebabc0c025363b6982d1fbffb00c50edfeea18e79b9: gzip compressed data, original size 212109824 /var/lib/pulp/media/artifact/6d/2cf55f7908bf17bd221297c8cff9750fad3187c3eb102d1c3f72a208efbf0b: gzip compressed data, original size 235748864 /var/lib/pulp/media/artifact/70/2f3a622329ec2d54c1bf3c13fa094c422d48abaad700a2e055c3b77bea15c0: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/7f/060f344d579611663e84ea94d89d9372d7431e3e56e11a19e704207d5576e9: ASCII text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/9a/394c72313c9bcac63dc8980018582d90ece7ae65d03940e6d7e9b8d4cb884d: gzip compressed data, original size 264029696 /var/lib/pulp/media/artifact/a5/d805d6872a1693a518e623b95182f115915c37671f70c1b13eb136e930bfe4: gzip compressed data, original size 204123648 /var/lib/pulp/media/artifact/ac/88d64d7b415280e8c1221725a8455917b99711b754a5981d3460fa9fdf40f9: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/b1/45dc70f6c036069d549762f39055162cac12b414e9e502fd0d4b3331a0f5eb: gzip compressed data, original size 171049472 /var/lib/pulp/media/artifact/b9/2727ef7443d73bb77323e5a969bf02187286fccf53930e83dc82be60a36529: gzip compressed data, original size 214139392 /var/lib/pulp/media/artifact/c6/25ce0354e96c43d4bfca9cb0a3ed8373b0707980f152582f731e61808df9bd: ASCII text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/d0/898311a8acc605593edbc1f37171f42aab926fffd820a0f0460e062a4f6717: UTF-8 Unicode text, with very long lines, with no line terminators /var/lib/pulp/media/artifact/d4/d59e4eb38ff5eda9a9f6255575abc64501adb78153f5788bf580b1034d4f8d: gzip compressed data, original size 186984960 /var/lib/pulp/media/artifact/d5/2583c79ff512b040d97ca0ce70b61b9c496e674b789f8857efbcf93727e4b0: UTF-8 Unicode text, with very long lines, with no line terminators