Bug 1002741

Summary: Sync of Fedora content never recovers from interruption
Product: Red Hat Satellite Reporter: Travis Gummels <tgummels>
Component: RepositoriesAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED NEXTRELEASE QA Contact: Katello QA List <katello-qa-list>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.0.0CC: bbuckingham, cwelton, mmccune, xdmoon
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-28 20:16:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Travis Gummels 2013-08-29 20:44:40 UTC
Description of problem:

Sync of Fedora content never recovers from interruption.  It seems that whenever the sync gets interrupted by network connectivity then it gets into some confused state.

Here is an entry from the /var/log/pulp/ginder.log  (note: log entry timestamps) The download url is confusing because there is no such url on the server side of the mirror. This sync started Wednesday and it's still going on today, Thursday, in these odd requeueing state.

     Furthermore, one can't seem to login to the web ui anymore after this happens.

=================================================================================
2013-08-27 13:25:34,080 grinder.YumInfo:INFO:

     delta rpm added {
                                     'downloadurl':
'http://mirror.pnl.gov/fedora/linux/releases/19/Everything/x86_64/os/drpms/telepathy-glib-devel-0.20.1-2.fc19_0.20.2-1.fc19.i686.drpm',
                                     'checksumtype': 'sha256',
                                     'sequence':
'telepathy-glib-devel-0.20.1-2.fc19-783cf18c7fd56c04ed835bfcdf9cc6c18c50',
                                     'relativepath':
'drpms/telepathy-glib-devel-0.20.1-2.fc19_0.20.2-1.fc19.i686.drpm',
                                     'epoch': '0',
                                     'fileName':
'drpms/telepathy-glib-devel-0.20.1-2.fc19_0.20.2-1.fc19.i686.drpm',
                                     'new_package':
'telepathy-glib-devel-0:0.20.2-1.fc19.i686',
                                     'item_type': 'delta_rpm',
                                     'savepath':
u'/var/lib/pulp/working/repos/ACME_Corporation-fedora-19-x86_64-pnl/importers/yum_importer/ACME_Corporation-fedora-19-x86_64-pnl',
                                     'version': '0.20.1',
                                     'checksum':
'8144027d9629baba3811206c5289daea243546eae3051b0bcf5ddbfd6d76ee02',
                                     'release': '2.fc19',
                                     'pkgpath': './',
                                     'size': 110548L
     }

2013-08-27 13:25:41,843 grinder.YumInfo:INFO: 15953 delta rpms have been
marked to be fetched

=================================================================================
2013-08-27 14:45:20,481 grinder.ParallelFetch:INFO: 4 threads are
active. 11876 items left to be fetched
2013-08-27 14:45:20,482 grinder.ParallelFetch:INFO:

     Requeueing: {
                                     'new_package':
'libwbclient-2:4.0.6-3.fc19.i686',
                                     'checksumtype': 'sha256',
                                     'sequence':
'libwbclient-2:4.0.6-2.fc19-c24100161b1bc9f4a52c3ebdc5f1335f30',
                                     'relativepath':
'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
                                     'savepath':
u'/var/lib/pulp/working/repos/ACME_Corporation-fedora-19-x86_64-pnl/importers/yum_importer/ACME_Corporation-fedora-19-x86_64-pnl',
                                     'fileName':
'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
                                     'downloadurl':
'http://mirror.pnl.gov/fedora/linux/releases/19/Everything/x86_64/os/drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
                                     'item_type': 'delta_rpm',
                                     'epoch': '2',
                                     'version': '4.0.6',
                                     'checksum':
'3802fd63dab195ebe134af6c13d757abc63f66e1d784ca5f15cfb641476770ff',
                                     'release': '2.fc19',
                                     'filename':
'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
                                     'pkgpath':
'/var/lib/pulp/content/drpm/.',
                                     'size': 59892L
     }

=================================================================================
2013-08-28 09:42:07,880 grinder.ParallelFetch:INFO: 4 threads are
active. 11876 items left to be fetched

=================================================================================

The packages keep requeueing over and over again. See below.
======================================
grep libwbclient-2:4.0.6-3.fc19.i686 /var/log/pulp/grinder.log*

grinder.log.1:

2013-08-28 08:57:37,820 grinder.ParallelFetch:INFO: Requeueing:
{'new_package': 'libwbclient-2:4.0.6-3.fc19.i686', 'checksumtype':
'sha256', 'sequence':
'libwbclient-2:4.0.6-2.fc19-c24100161b1bc9f4a52c3ebdc5f1335f30',
'relativepath': 'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'savepath':
u'/var/lib/pulp/working/repos/ACME_Corporation-fedora-19-x86_64-pnl/importers/yum_importer/ACME_Corporation-fedora-19-x86_64-pnl',
'fileName': 'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'downloadurl':
'http://mirror.pnl.gov/fedora/linux/releases/19/Everything/x86_64/os/drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'item_type': 'delta_rpm', 'epoch': '2', 'version': '4.0.6', 'checksum':
'3802fd63dab195ebe134af6c13d757abc63f66e1d784ca5f15cfb641476770ff',
'release': '2.fc19', 'filename':
'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm', 'pkgpath':
'/var/lib/pulp/content/drpm/.', 'size': 59892L}

grinder.log.2:

2013-08-28 07:18:27,346 grinder.ParallelFetch:INFO: Requeueing:
{'new_package': 'libwbclient-2:4.0.6-3.fc19.i686', 'checksumtype':
'sha256', 'sequence':
'libwbclient-2:4.0.6-2.fc19-c24100161b1bc9f4a52c3ebdc5f1335f30',
'relativepath': 'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'savepath':
u'/var/lib/pulp/working/repos/ACME_Corporation-fedora-19-x86_64-pnl/importers/yum_importer/ACME_Corporation-fedora-19-x86_64-pnl',
'fileName': 'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'downloadurl':
'http://mirror.pnl.gov/fedora/linux/releases/19/Everything/x86_64/os/drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'item_type': 'delta_rpm', 'epoch': '2', 'version': '4.0.6', 'checksum':
'3802fd63dab195ebe134af6c13d757abc63f66e1d784ca5f15cfb641476770ff',
'release': '2.fc19', 'filename':
'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm', 'pkgpath':
'/var/lib/pulp/content/drpm/.', 'size': 59892L}

grinder.log.3:

2013-08-28 05:39:17,046 grinder.ParallelFetch:INFO: Requeueing:
{'new_package': 'libwbclient-2:4.0.6-3.fc19.i686', 'checksumtype':
'sha256', 'sequence':
'libwbclient-2:4.0.6-2.fc19-c24100161b1bc9f4a52c3ebdc5f1335f30',
'relativepath': 'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'savepath':
u'/var/lib/pulp/working/repos/ACME_Corporation-fedora-19-x86_64-pnl/importers/yum_importer/ACME_Corporation-fedora-19-x86_64-pnl',
'fileName': 'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'downloadurl':
'http://mirror.pnl.gov/fedora/linux/releases/19/Everything/x86_64/os/drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'item_type': 'delta_rpm', 'epoch': '2', 'version': '4.0.6', 'checksum':
'3802fd63dab195ebe134af6c13d757abc63f66e1d784ca5f15cfb641476770ff',
'release': '2.fc19', 'filename':
'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm', 'pkgpath':
'/var/lib/pulp/content/drpm/.', 'size': 59892L}

Version-Release number of selected component (if applicable):
Satellite 6 MDP 1
Fedora 19

How reproducible:
100%

Steps to Reproduce:
See bz 1002728

Actual results:
Fails to recover from interruption

Expected results:
Recovers gracefully

Additional info:

Comment 1 RHEL Program Management 2013-09-17 04:17:02 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 6 Brad Buckingham 2016-04-28 20:16:26 UTC
This bug report is quite old; however, we have attempted to reproduce the issue raised (on Satellite 6.2 Beta) and have been unsuccessful in doing so.  Since the logic involved has been fully redesigned since the original report, I am going to close the bug.  If the issue is re-encountered, please feel free to re-open the bug, provide latest details and attach the output of the 'foreman-debug'.   Thanks!