Bug 1903046

Summary: Retry for connection reset by peer doesn't work when syncing docker repository
Product: Red Hat Satellite Reporter: Hao Chang Yu <hyu>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Lai <ltran>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.8.0CC: bmbouter, egolov, ggainey, ipanova, mmccune, rchan, ttereshc
Target Milestone: 6.9.0Keywords: Regression, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-nectar-1.6.3 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1903208 (view as bug list) Environment:
Last Closed: 2021-04-21 13:24:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Hao Chang Yu 2020-12-01 08:14:17 UTC
Description of problem:
This is the regression of bug 1822560. 

The retry doesn't work because the following lines are checking the wrong element in the dictionary.

https://github.com/pulp/nectar/blob/master/nectar/downloaders/threaded.py#L348
https://github.com/pulp/nectar/blob/master/nectar/downloaders/threaded.py#L378

# The alue of e.args is:
(104, 'Connection reset by peer')


# retry only if there's indication of connection reset
if nretry < DEFAULT_GENERIC_TRIES - 1 and e.args[1] == errno.ECONNRESET:  <==== Error code should be in e.args[0]
    continue

Comment 1 pulp-infra@redhat.com 2020-12-02 12:08:06 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 2 pulp-infra@redhat.com 2020-12-02 12:08:08 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 3 Ina Panova 2020-12-02 13:56:55 UTC
To reproduce the issue, set the max_downloads to 30. This will lead to the sync failure.

Steps to verify the BZ
======================

1. Install the new nectar build 1.6.3
2. Set the log_level to DEBUG in the /etc/pulp/server.conf
3. Create a custom docker repo for openshift3/mediawiki-apb on https://registry.redhat.io
3. Include the correct login credentials to access that repo
4. Sync repo

Expected result:
1. Repo sync should complete
2. In the logs look for "Connection reset. Retrying to connect to {url}" https://github.com/pulp/nectar/pull/75/files#diff-3461af5a57eeec99959f2195d150f5464dc133d972d16b15652d0642976c04d6R349

If you do not see those debug statements it means that whether server did not send the RST OR the patch still has some gaps that have not been covered.
Successful repo sync AND presence of the mentioned above debug statement is the key of the BZ verification.

Comment 5 pulp-infra@redhat.com 2020-12-03 21:05:15 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2020-12-03 22:06:55 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 7 Lai 2021-01-05 18:39:11 UTC
Steps to retest

1. Install the new nectar build 1.6.3
2. Set the log_level to DEBUG in the /etc/pulp/server.conf
3. Create a custom docker repo for openshift3/mediawiki-apb on https://registry.redhat.io
3. Include the correct login credentials to access that repo
4. Sync repo
5. Check logs for "Connection reset message."

Expected result:
4. Repo sync successfully
5. In the logs look for "Connection reset. Retrying to connect to {url}" https://github.com/pulp/nectar/pull/<#>/file#diff-<name>

Actual:
4. Repo sync successfully
5. Log doesn't display "connection reset. Retrying to connect to..."

Cannot see "connection reset.  Retrying to connect to..." in log was because the condition cannot be reproduce 100%.  Regardless, this is verify since it's not negatively affecting syncing, but should be noted if the problem does present itself in the future.

Verified that it's in python-nectar-1.6.4-1.el7sat.noarch

Verified in 6.9.0_07

Comment 10 errata-xmlrpc 2021-04-21 13:24:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.9 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1313