Bug 1903046 - Retry for connection reset by peer doesn't work when syncing docker repository
Summary: Retry for connection reset by peer doesn't work when syncing docker repository
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.8.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: 6.9.0
Assignee: satellite6-bugs
QA Contact: Lai
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-01 08:14 UTC by Hao Chang Yu
Modified: 2022-07-21 21:49 UTC (History)
7 users (show)

Fixed In Version: python-nectar-1.6.3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1903208 (view as bug list)
Environment:
Last Closed: 2021-04-21 13:24:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 7911 0 Normal MODIFIED Unable to sync large openshift docker repos 2020-12-03 21:05:14 UTC
Red Hat Product Errata RHSA-2021:1313 0 None None None 2021-04-21 13:24:44 UTC

Description Hao Chang Yu 2020-12-01 08:14:17 UTC
Description of problem:
This is the regression of bug 1822560. 

The retry doesn't work because the following lines are checking the wrong element in the dictionary.

https://github.com/pulp/nectar/blob/master/nectar/downloaders/threaded.py#L348
https://github.com/pulp/nectar/blob/master/nectar/downloaders/threaded.py#L378

# The alue of e.args is:
(104, 'Connection reset by peer')


# retry only if there's indication of connection reset
if nretry < DEFAULT_GENERIC_TRIES - 1 and e.args[1] == errno.ECONNRESET:  <==== Error code should be in e.args[0]
    continue

Comment 1 pulp-infra@redhat.com 2020-12-02 12:08:06 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 2 pulp-infra@redhat.com 2020-12-02 12:08:08 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 3 Ina Panova 2020-12-02 13:56:55 UTC
To reproduce the issue, set the max_downloads to 30. This will lead to the sync failure.

Steps to verify the BZ
======================

1. Install the new nectar build 1.6.3
2. Set the log_level to DEBUG in the /etc/pulp/server.conf
3. Create a custom docker repo for openshift3/mediawiki-apb on https://registry.redhat.io
3. Include the correct login credentials to access that repo
4. Sync repo

Expected result:
1. Repo sync should complete
2. In the logs look for "Connection reset. Retrying to connect to {url}" https://github.com/pulp/nectar/pull/75/files#diff-3461af5a57eeec99959f2195d150f5464dc133d972d16b15652d0642976c04d6R349

If you do not see those debug statements it means that whether server did not send the RST OR the patch still has some gaps that have not been covered.
Successful repo sync AND presence of the mentioned above debug statement is the key of the BZ verification.

Comment 5 pulp-infra@redhat.com 2020-12-03 21:05:15 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2020-12-03 22:06:55 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 7 Lai 2021-01-05 18:39:11 UTC
Steps to retest

1. Install the new nectar build 1.6.3
2. Set the log_level to DEBUG in the /etc/pulp/server.conf
3. Create a custom docker repo for openshift3/mediawiki-apb on https://registry.redhat.io
3. Include the correct login credentials to access that repo
4. Sync repo
5. Check logs for "Connection reset message."

Expected result:
4. Repo sync successfully
5. In the logs look for "Connection reset. Retrying to connect to {url}" https://github.com/pulp/nectar/pull/<#>/file#diff-<name>

Actual:
4. Repo sync successfully
5. Log doesn't display "connection reset. Retrying to connect to..."

Cannot see "connection reset.  Retrying to connect to..." in log was because the condition cannot be reproduce 100%.  Regardless, this is verify since it's not negatively affecting syncing, but should be noted if the problem does present itself in the future.

Verified that it's in python-nectar-1.6.4-1.el7sat.noarch

Verified in 6.9.0_07

Comment 10 errata-xmlrpc 2021-04-21 13:24:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.9 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1313


Note You need to log in before you can comment on or make changes to this bug.