Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2173692 - [Pulp-3] The Repair API fails to perform any further repair if it has encountered errors while processning any docker manifest
Summary: [Pulp-3] The Repair API fails to perform any further repair if it has encount...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.13.0
Hardware: All
OS: All
medium
high
Target Milestone: 6.14.0
Assignee: satellite6-bugs
QA Contact: visawant
URL:
Whiteboard:
: 2240519 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-02-27 17:10 UTC by Sayan Das
Modified: 2024-03-08 04:25 UTC (History)
11 users (show)

Fixed In Version: pulpcore-3.21.12, pulpcore-3.22.9, pulpcore-3.23.10, pulpcore-3.18.22
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2218980 (view as bug list)
Environment:
Last Closed: 2023-11-08 14:18:33 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github pulp pulpcore issues 3611 0 None closed Pulp repair functionality should try to continue when encountering a 404 during the repair process 2023-05-25 13:06:55 UTC
Github pulp pulpcore issues 4111 0 None closed Repair API should be tolerant of any failure during the repair process 2023-07-21 20:46:28 UTC
Red Hat Issue Tracker SAT-16327 0 None None None 2023-03-06 14:22:57 UTC
Red Hat Product Errata RHSA-2023:6818 0 None None None 2023-11-08 14:18:48 UTC

Description Sayan Das 2023-02-27 17:10:48 UTC 
Description of problem:

If the satellite\capsule servers are missing artifacts related to both rpm and docker type repos, Then The Repair API, when executed, may not be able to repair all the rpm data and stop it's action if it fails to process any docker manifest-related artifacts.  

Version-Release number of selected component (if applicable):

satellite-6.13.0-6.el8sat.noarch
katello-4.7.0-1.el8sat.noarch
python39-pulpcore-3.21.5-1.1.el8pc.noarch
python39-pulp-rpm-3.18.11-1.el8pc.noarch

How reproducible:

Always

Steps to Reproduce:
1. Complete The reproducer from https://bugzilla.redhat.com/show_bug.cgi?id=2173671 

2. Enable Red Hat Satellite Client 6 for RHEL 8 x86_64 (RPMs) 

3. Change the download policy to immediate and sync the repo

4. Ensure that we have two repos enabled now in satellite, yum as well as docker type

5. Delete any artifacts related to the satellite client repo

6. Run the Repair API 


Actual results:


Step 3:

Added Rpms: 16, Errata: 3
Total steps: 65/65
--------------------------------
Associating Content: 20/20
Downloading Artifacts: 16/16
Downloading Metadata Files: 9/9
Parsed Advisories: 3/3
Parsed Comps: 1/1
Parsed Packages: 16/16
Skipping Packages: 0/0
Un-Associating Content: 0/0

Step 4: 

# hammer repository list --fields "Id,Name,Content Type"
---|---------------------------------------------------|-------------
ID | NAME                                              | CONTENT TYPE
---|---------------------------------------------------|-------------
2  | Red Hat Satellite Client 6 for RHEL 8 x86_64 RPMs | yum         
1  | rhel8/toolbox                                     | docker      
---|---------------------------------------------------|-------------

Step 5:

# echo "COPY (select base_path from core_distribution where base_path like '%client%') TO STDOUT;" | su - postgres -c "psql pulpcore"
RedHat/Library/content/dist/layered/rhel8/x86_64/sat-client/6/os

# echo "COPY(select cpa.relative_path,ca.file from core_distribution as core_d RIGHT JOIN core_publication as cp ON core_d.publication_id=cp.pulp_id LEFT JOIN core_publishedartifact as cpa ON  cp.pulp_id=cpa.publication_id LEFT JOIN core_contentartifact as core_ca ON core_ca.pulp_id=cpa.content_artifact_id LEFT JOIN core_artifact as ca ON ca.pulp_id=core_ca.artifact_id where core_d.base_path = 'RedHat/Library/content/dist/layered/rhel8/x86_64/sat-client/6/os' order by core_d.base_path) TO STDOUT WITH CSV" | su - postgres -c "psql pulpcore" > /tmp/filename_to_artifact_mapping.txt


# for map in $(cat /tmp/filename_to_artifact_mapping.txt)
  do
  FILENAME=$(echo $map | awk -F',' '{print $1}')
  FULLPATH="/var/lib/pulp/media/$(echo $map | cut -d, -f2)"
  if [ $FULLPATH == "/var/lib/pulp/media/" ]
  then
  STAT="Not-Downloaded"
  else
  STAT=$(test -e $FULLPATH && echo "PRESENT" || echo "ABSENT")
  fi
  echo "$FILENAME,$STAT"
  done | column -s, -t

Packages/f/foreman_ygg_worker-0.1.1-2.el8sat.x86_64.rpm                                         PRESENT
Packages/g/gofer-2.12.5-7.el8sat.noarch.rpm                                                     PRESENT
Packages/g/gofer-2.12.5-8.el8sat.noarch.rpm                                                     PRESENT
Packages/k/katello-agent-3.5.7-3.el8sat.noarch.rpm                                              PRESENT
Packages/k/katello-host-tools-3.5.7-3.el8sat.noarch.rpm                                         PRESENT
Packages/k/katello-host-tools-tracer-3.5.7-3.el8sat.noarch.rpm                                  PRESENT
Packages/k/katello-pull-transport-migrate-1.0.2-1.el8sat.noarch.rpm                             PRESENT
Packages/p/puppet-agent-7.16.0-2.el8sat.x86_64.rpm                                              PRESENT
Packages/p/python3-gofer-2.12.5-7.el8sat.noarch.rpm                                             PRESENT
Packages/p/python3-gofer-2.12.5-8.el8sat.noarch.rpm                                             PRESENT
Packages/p/python3-gofer-proton-2.12.5-7.el8sat.noarch.rpm                                      PRESENT
Packages/p/python3-gofer-proton-2.12.5-8.el8sat.noarch.rpm                                      PRESENT
Packages/p/python3-qpid-proton-0.33.0-4.el8.x86_64.rpm                                          PRESENT
Packages/q/qpid-proton-c-0.33.0-4.el8.x86_64.rpm                                                PRESENT
Packages/r/rubygem-foreman_scap_client-0.5.0-1.el8sat.noarch.rpm                                PRESENT
Packages/y/yggdrasil-0.2.0-2.el8sat.x86_64.rpm                                                  PRESENT
repodata/2ec7ea4c19077fc0f495944e5ed4a8bb3dc9af45bfe6634dbb7aabc2cb41fb6d-updateinfo.xml.gz     PRESENT
repodata/37e6338d623013fd1285fd13033596e2e36c55f0db8b6635f8422a079fb98568-primary.xml.gz        PRESENT
repodata/4badd0660333ce1828baa560d2208b1439e969de67d9848301894614310f221a-filelists.sqlite.bz2  PRESENT
repodata/77895819bddc7dcd7cb8ad334cfe33aee2f00c0fd03ab663914ce280944e4ff2-comps.xml             PRESENT
repodata/9b0bd210bfc9887bf17a2bf26e986285a9ab3190cec96b5ec5972df56c546713-filelists.xml.gz      PRESENT
repodata/c4cc5c111ae7df9225eea3f1ac0bb6dd44706aeb643b9b89495d1bb16cdf0fd9-other.xml.gz          PRESENT
repodata/e735df44940193e25ea4e467630ef8a4e94e55d4b4d37e320d463dc3f5ca1669-other.sqlite.bz2      PRESENT
repodata/fdddabeef43c982760a1c94724e98f9314974fe7a0b67e13c12af5726e48025f-primary.sqlite.bz2    PRESENT
repodata/repomd.xml                                                                             PRESENT


# for map in $(cat /tmp/filename_to_artifact_mapping.txt)
  do
  FULLPATH="/var/lib/pulp/media/$(echo $map | cut -d, -f2)"
  if [ $FULLPATH == "/var/lib/pulp/media/" ]
  then
  echo ''
  else
  rm -f $FULLPATH
  fi
  done

# for map in $(cat /tmp/filename_to_artifact_mapping.txt)
  do
  FILENAME=$(echo $map | awk -F',' '{print $1}')
  FULLPATH="/var/lib/pulp/media/$(echo $map | cut -d, -f2)"
  if [ $FULLPATH == "/var/lib/pulp/media/" ]
  then
  STAT="Not-Downloaded"
  else
  STAT=$(test -e $FULLPATH && echo "PRESENT" || echo "ABSENT")
  fi
  echo "$FILENAME,$STAT"
  done | column -s, -t  
  
Packages/f/foreman_ygg_worker-0.1.1-2.el8sat.x86_64.rpm                                         ABSENT
Packages/g/gofer-2.12.5-7.el8sat.noarch.rpm                                                     ABSENT
Packages/g/gofer-2.12.5-8.el8sat.noarch.rpm                                                     ABSENT
Packages/k/katello-agent-3.5.7-3.el8sat.noarch.rpm                                              ABSENT
Packages/k/katello-host-tools-3.5.7-3.el8sat.noarch.rpm                                         ABSENT
Packages/k/katello-host-tools-tracer-3.5.7-3.el8sat.noarch.rpm                                  ABSENT
Packages/k/katello-pull-transport-migrate-1.0.2-1.el8sat.noarch.rpm                             ABSENT
Packages/p/puppet-agent-7.16.0-2.el8sat.x86_64.rpm                                              ABSENT
Packages/p/python3-gofer-2.12.5-7.el8sat.noarch.rpm                                             ABSENT
Packages/p/python3-gofer-2.12.5-8.el8sat.noarch.rpm                                             ABSENT
Packages/p/python3-gofer-proton-2.12.5-7.el8sat.noarch.rpm                                      ABSENT
Packages/p/python3-gofer-proton-2.12.5-8.el8sat.noarch.rpm                                      ABSENT
Packages/p/python3-qpid-proton-0.33.0-4.el8.x86_64.rpm                                          ABSENT
Packages/q/qpid-proton-c-0.33.0-4.el8.x86_64.rpm                                                ABSENT
Packages/r/rubygem-foreman_scap_client-0.5.0-1.el8sat.noarch.rpm                                ABSENT
Packages/y/yggdrasil-0.2.0-2.el8sat.x86_64.rpm                                                  ABSENT
repodata/2ec7ea4c19077fc0f495944e5ed4a8bb3dc9af45bfe6634dbb7aabc2cb41fb6d-updateinfo.xml.gz     ABSENT
repodata/37e6338d623013fd1285fd13033596e2e36c55f0db8b6635f8422a079fb98568-primary.xml.gz        ABSENT
repodata/4badd0660333ce1828baa560d2208b1439e969de67d9848301894614310f221a-filelists.sqlite.bz2  ABSENT
repodata/77895819bddc7dcd7cb8ad334cfe33aee2f00c0fd03ab663914ce280944e4ff2-comps.xml             ABSENT
repodata/9b0bd210bfc9887bf17a2bf26e986285a9ab3190cec96b5ec5972df56c546713-filelists.xml.gz      ABSENT
repodata/c4cc5c111ae7df9225eea3f1ac0bb6dd44706aeb643b9b89495d1bb16cdf0fd9-other.xml.gz          ABSENT
repodata/e735df44940193e25ea4e467630ef8a4e94e55d4b4d37e320d463dc3f5ca1669-other.sqlite.bz2      ABSENT
repodata/fdddabeef43c982760a1c94724e98f9314974fe7a0b67e13c12af5726e48025f-primary.sqlite.bz2    ABSENT
repodata/repomd.xml                                                                             ABSENT
  

# file /var/lib/pulp/media/artifact/*/*
/var/lib/pulp/media/artifact/1c/b61ea996355add02b1426ed4c1780ea75ce0c04c5d1107c025c3fbd7d8bcae: gzip compressed data, from Unix, original size 167
/var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text
/var/lib/pulp/media/artifact/5d/88a016637781e6d3eeb22c15c788ce1cc572e2dbb7bad4935d37a775c252d9: gzip compressed data, original size 169271808
/var/lib/pulp/media/artifact/64/554374912209545e580ebabc0c025363b6982d1fbffb00c50edfeea18e79b9: gzip compressed data, original size 212109824
/var/lib/pulp/media/artifact/6d/2cf55f7908bf17bd221297c8cff9750fad3187c3eb102d1c3f72a208efbf0b: gzip compressed data, original size 235748864
/var/lib/pulp/media/artifact/70/2f3a622329ec2d54c1bf3c13fa094c422d48abaad700a2e055c3b77bea15c0: UTF-8 Unicode text, with very long lines, with no line terminators
/var/lib/pulp/media/artifact/95/a4415d859d7120efb6b3cf964c07bebbff9a5275ca673e6e74a97bcbfb2a5f: gzip compressed data, from Unix, original size 125
/var/lib/pulp/media/artifact/a5/d805d6872a1693a518e623b95182f115915c37671f70c1b13eb136e930bfe4: gzip compressed data, original size 204123648
/var/lib/pulp/media/artifact/ac/88d64d7b415280e8c1221725a8455917b99711b754a5981d3460fa9fdf40f9: UTF-8 Unicode text, with very long lines, with no line terminators
/var/lib/pulp/media/artifact/c0/cace4d6bd977ad29096c46467682b30b7ef65aff60fe7f7238096f9003d898: gzip compressed data, from Unix, original size 59
/var/lib/pulp/media/artifact/cb/920667676480deb7ebd62198e571e416462118d58b9c280b413fec31745e4e: XML 1.0 document, ASCII text
/var/lib/pulp/media/artifact/d0/898311a8acc605593edbc1f37171f42aab926fffd820a0f0460e062a4f6717: UTF-8 Unicode text, with very long lines, with no line terminators
/var/lib/pulp/media/artifact/d5/2583c79ff512b040d97ca0ce70b61b9c496e674b789f8857efbcf93727e4b0: UTF-8 Unicode text, with very long lines, with no line terminators
/var/lib/pulp/media/artifact/ef/3e20691954c3d1318ec3071a982da339f4ed76967ded668b795c9e070aaab6: gzip compressed data, from Unix, original size 121



Step 6:

# curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X POST https://`hostname -f`/pulp/api/v3/repair/ | json_reformat
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    67  100    67    0     0    683      0 --:--:-- --:--:-- --:--:--   683
{
    "task": "/pulp/api/v3/tasks/783ec953-8902-4d1d-9566-1ed8f7a5a00f/"
}
# curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem -H "Content-Type: application/json" -X GET https://`hostname -f`/pulp/api/v3/tasks/783ec953-8902-4d1d-9566-1ed8f7a5a00f/ | json_reformat
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2860  100  2860    0     0  42058      0 --:--:-- --:--:-- --:--:-- 42058
{
    "pulp_href": "/pulp/api/v3/tasks/783ec953-8902-4d1d-9566-1ed8f7a5a00f/",
    "pulp_created": "2023-02-27T17:00:38.933371Z",
    "state": "failed",
    "name": "pulpcore.app.tasks.repository.repair_all_artifacts",
    "logging_cid": "3997124f3767438e86884df2e6d3519d",
    "started_at": "2023-02-27T17:00:38.970659Z",
    "finished_at": "2023-02-27T17:00:40.625673Z",
    "error": {
        "traceback": "  File \"/usr/lib/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py\", line 452, in _perform_task\n    result = func(*args, **kwargs)\n  File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 192, in repair_all_artifacts\n    loop.run_until_complete(_repair_artifacts_for_content(verify_checksums=verify_checksums))\n  File \"/usr/lib64/python3.9/asyncio/base_events.py\", line 647, in run_until_complete\n    return future.result()\n  File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 148, in _repair_artifacts_for_content\n    await asyncio.gather(*done)  # Clean up tasks\n  File \"/usr/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py\", line 77, in _repair_ca\n    dl_result = await downloader.run()\n  File \"/usr/lib/python3.9/site-packages/pulpcore/download/http.py\", line 273, in run\n    return await download_wrapper()\n  File \"/usr/lib/python3.9/site-packages/backoff/_async.py\", line 151, in retry\n    ret = await target(*args, **kwargs)\n  File \"/usr/lib/python3.9/site-packages/pulpcore/download/http.py\", line 258, in download_wrapper\n    return await self._run(extra_data=extra_data)\n  File \"/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py\", line 81, in _run\n    return await self._run(handle_401=False, extra_data=extra_data)\n  File \"/usr/lib/python3.9/site-packages/pulp_container/app/downloaders.py\", line 66, in _run\n    response.raise_for_status()\n  File \"/usr/lib64/python3.9/site-packages/aiohttp/client_reqrep.py\", line 1004, in raise_for_status\n    raise ClientResponseError(\n",
        "description": "404, message='Not Found', url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?_auth_=exp=1677520840~hmac=4c1a924c0c8b5af741a5025eaa730f8b5fca99f539f5004d65aeb6bc136c3db4')"
    },
    "worker": "/pulp/api/v3/workers/60781c7e-a9fd-4386-8ba6-6c7d4785a9a1/",
    "parent_task": null,
    "child_tasks": [

    ],
    "task_group": null,
    "progress_reports": [
        {
            "message": "Repair corrupted units",
            "code": "repair.repaired",
            "state": "failed",
            "total": null,
            "done": 0,
            "suffix": null
        },
        {
            "message": "Identify corrupted units",
            "code": "repair.corrupted",
            "state": "failed",
            "total": null,
            "done": 0,
            "suffix": null
        },
        {
            "message": "Identify missing units",
            "code": "repair.missing",
            "state": "failed",
            "total": null,
            "done": 6,
            "suffix": null
        }
    ],
    "created_resources": [

    ],
    "reserved_resources_record": [

    ]
}


# file /var/lib/pulp/media/artifact/*/*
/var/lib/pulp/media/artifact/1c/b61ea996355add02b1426ed4c1780ea75ce0c04c5d1107c025c3fbd7d8bcae: gzip compressed data, from Unix, original size 167
/var/lib/pulp/media/artifact/48/fd57131d1a3ccda35d72ca802a44ced559941d9fe5ee160dbe3df0414e7cb3: ASCII text
/var/lib/pulp/media/artifact/5d/88a016637781e6d3eeb22c15c788ce1cc572e2dbb7bad4935d37a775c252d9: gzip compressed data, original size 169271808
/var/lib/pulp/media/artifact/64/554374912209545e580ebabc0c025363b6982d1fbffb00c50edfeea18e79b9: gzip compressed data, original size 212109824
/var/lib/pulp/media/artifact/6d/2cf55f7908bf17bd221297c8cff9750fad3187c3eb102d1c3f72a208efbf0b: gzip compressed data, original size 235748864
/var/lib/pulp/media/artifact/70/2f3a622329ec2d54c1bf3c13fa094c422d48abaad700a2e055c3b77bea15c0: UTF-8 Unicode text, with very long lines, with no line terminators
/var/lib/pulp/media/artifact/95/a4415d859d7120efb6b3cf964c07bebbff9a5275ca673e6e74a97bcbfb2a5f: gzip compressed data, from Unix, original size 125
/var/lib/pulp/media/artifact/a5/d805d6872a1693a518e623b95182f115915c37671f70c1b13eb136e930bfe4: gzip compressed data, original size 204123648
/var/lib/pulp/media/artifact/ac/88d64d7b415280e8c1221725a8455917b99711b754a5981d3460fa9fdf40f9: UTF-8 Unicode text, with very long lines, with no line terminators
/var/lib/pulp/media/artifact/c0/cace4d6bd977ad29096c46467682b30b7ef65aff60fe7f7238096f9003d898: gzip compressed data, from Unix, original size 59
/var/lib/pulp/media/artifact/cb/920667676480deb7ebd62198e571e416462118d58b9c280b413fec31745e4e: XML 1.0 document, ASCII text
/var/lib/pulp/media/artifact/d0/898311a8acc605593edbc1f37171f42aab926fffd820a0f0460e062a4f6717: UTF-8 Unicode text, with very long lines, with no line terminators
/var/lib/pulp/media/artifact/d5/2583c79ff512b040d97ca0ce70b61b9c496e674b789f8857efbcf93727e4b0: UTF-8 Unicode text, with very long lines, with no line terminators
/var/lib/pulp/media/artifact/ef/3e20691954c3d1318ec3071a982da339f4ed76967ded668b795c9e070aaab6: gzip compressed data, from Unix, original size 121




Expected results:

The Repair API should be able to skip the failed units and proceed fixing the other RPM-type units that It can. 


Additional info:

* This is not a huge blocker for satellite as satellite ( via katello ) offers an easy way to run repair API per repo .

* But when it comes to Capsules, That is not the case and hence the Repair action needs to be executed for all repos and that will fails at the very beginning ( under the specific condition as mentioned earlier )
Comment 2 Brad Buckingham 2023-03-02 15:30:43 UTC 
Is this a regression from 6.12?  Thanks!
Comment 3 Sayan Das 2023-03-02 15:37:52 UTC 
Hello,

The behavior is the same in 6.12 as well. So no regression. I was not able to test it during 6.12 GA properly and hence tested it during 6.13. 

The "Repair API" of pulp is something really important for Satellite to have working properly. This is the key to get back all downloadable pulp artifacts after a backup without pulp data was restored or the pulp filesystem was corrupted and later recreated.
Comment 4 Robin Chan 2023-05-25 13:06:56 UTC 
The Pulp upstream bug status is at closed. Updating the external tracker on this bug.
Comment 9 visawant 2023-07-13 09:01:55 UTC 
Failed QA
As per steps performed in comment 8
Comment 10 Robin Chan 2023-07-13 09:05:24 UTC 
Requesting needsinfo from upstream developer dkliban, ggainey because the 'FailedQA' flag is set.
Comment 11 Ina Panova 2023-07-17 16:28:25 UTC 
This fix was primary focused to cover ``raise ClientResponseError(\n",
        "description": "404, message='Not Found', url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?_auth_=exp=1677520840~hmac=4c1a924c0c8b5af741a5025eaa730f8b5fca99f539f5004d65aeb6bc136c3db4')"`` as stated in the BZ report. 

Verification steps fail with ``raise TimeoutException(self.url)\n",
    "description": "Request timed out for https://registry.redhat.io/v2/rhel8/toolbox/manifests/sha256:629545c824752b6d75f7f22a73b6c7fcba9413b6017f2f67f58494a958bcd4ce. Increasing the total_timeout value on the remote might help."`` which is a different error and not covered by this fix.

We need agree on the definition of done this BZ request, because it was not intended to catch all the errors that might come from the server.
What is the preferred way to proceed?
Comment 12 Robin Chan 2023-07-17 17:05:06 UTC 
Requesting needsinfo from upstream developer dkliban, ggainey because the 'FailedQA' flag is set.
Comment 13 Sayan Das 2023-07-21 06:13:23 UTC 
(In reply to Ina Panova from comment #11)
> This fix was primary focused to cover ``raise ClientResponseError(\n",
>         "description": "404, message='Not Found',
> url=URL('https://registry.redhat.io/containers/content/dist/containers/rhel8/
> multiarch/containers/redhat-rhel8-toolbox/manifests/1/sha256:
> 0165232f07e945f1c1831eebb8c42e5695f80d47d7c3d787af20752c89333276?_auth_=exp=1
> 677520840~hmac=4c1a924c0c8b5af741a5025eaa730f8b5fca99f539f5004d65aeb6bc136c3d
> b4')"`` as stated in the BZ report. 
> 
> Verification steps fail with ``raise TimeoutException(self.url)\n",
>     "description": "Request timed out for
> https://registry.redhat.io/v2/rhel8/toolbox/manifests/sha256:
> 629545c824752b6d75f7f22a73b6c7fcba9413b6017f2f67f58494a958bcd4ce. Increasing
> the total_timeout value on the remote might help."`` which is a different
> error and not covered by this fix.
> 
> We need agree on the definition of done this BZ request, because it was not
> intended to catch all the errors that might come from the server.
> What is the preferred way to proceed?

Hello,

Definitely, 404 was the error that I received initially because of the way it was reproduced. From that point of view, ClientResponseError was handled via https://github.com/pulp/pulpcore/commit/72f1902bd9d5b4d4237297b3c68a617682f4b5f7 and it's fine. But that is not the only error that someone can receive. 

If Satellite\Pulp's remote is associated with a proxy and that proxy is limiting connections or misconfigured someway, Then TimeoutException would be a common thing to occur as well. 

If Firewall is not properly configured that may even result in "Connection Reset" but I don't know how Pulp will show that message. ( but this is not very important at this moment ).

Anyways, The point is, No matter what the failure is, The Repair API, should not halt but continue processing rest of the units that it has identified as corrupted or missing and that has always been the goal of this BZ ( if you see the "Expected Results" section in Description ).

My request would be to handle the TimeoutException in the same way as ClientResponseError was done as these two are the commonly seen errors when it comes to pulp trying to download something from upstream or another pulp instance. 


-- Sayan
Comment 14 Ina Panova 2023-07-21 10:42:41 UTC 
Sayan, thanks for clarifying the scope. I have filed a new upstream request https://github.com/pulp/pulpcore/issues/4111
Comment 17 visawant 2023-07-31 07:36:13 UTC 
Verified

Version Tested: Satellite 6.14.0 Snap 9
Comment 19 Daniel Alley 2023-10-13 16:40:40 UTC 
*** Bug 2240519 has been marked as a duplicate of this bug. ***
Comment 22 errata-xmlrpc 2023-11-08 14:18:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.14 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6818
Comment 23 Red Hat Bugzilla 2024-03-08 04:25:25 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days
Note You need to log in before you can comment on or make changes to this bug.