Description of problem:

Sync of Fedora content never recovers from interruption.  It seems that whenever the sync gets interrupted by network connectivity then it gets into some confused state.

Here is an entry from the /var/log/pulp/ginder.log  (note: log entry timestamps) The download url is confusing because there is no such url on the server side of the mirror. This sync started Wednesday and it's still going on today, Thursday, in these odd requeueing state.

     Furthermore, one can't seem to login to the web ui anymore after this happens.

=================================================================================
2013-08-27 13:25:34,080 grinder.YumInfo:INFO:

     delta rpm added {
                                     'downloadurl':
'http://mirror.pnl.gov/fedora/linux/releases/19/Everything/x86_64/os/drpms/telepathy-glib-devel-0.20.1-2.fc19_0.20.2-1.fc19.i686.drpm',
                                     'checksumtype': 'sha256',
                                     'sequence':
'telepathy-glib-devel-0.20.1-2.fc19-783cf18c7fd56c04ed835bfcdf9cc6c18c50',
                                     'relativepath':
'drpms/telepathy-glib-devel-0.20.1-2.fc19_0.20.2-1.fc19.i686.drpm',
                                     'epoch': '0',
                                     'fileName':
'drpms/telepathy-glib-devel-0.20.1-2.fc19_0.20.2-1.fc19.i686.drpm',
                                     'new_package':
'telepathy-glib-devel-0:0.20.2-1.fc19.i686',
                                     'item_type': 'delta_rpm',
                                     'savepath':
u'/var/lib/pulp/working/repos/ACME_Corporation-fedora-19-x86_64-pnl/importers/yum_importer/ACME_Corporation-fedora-19-x86_64-pnl',
                                     'version': '0.20.1',
                                     'checksum':
'8144027d9629baba3811206c5289daea243546eae3051b0bcf5ddbfd6d76ee02',
                                     'release': '2.fc19',
                                     'pkgpath': './',
                                     'size': 110548L
     }

2013-08-27 13:25:41,843 grinder.YumInfo:INFO: 15953 delta rpms have been
marked to be fetched

=================================================================================
2013-08-27 14:45:20,481 grinder.ParallelFetch:INFO: 4 threads are
active. 11876 items left to be fetched
2013-08-27 14:45:20,482 grinder.ParallelFetch:INFO:

     Requeueing: {
                                     'new_package':
'libwbclient-2:4.0.6-3.fc19.i686',
                                     'checksumtype': 'sha256',
                                     'sequence':
'libwbclient-2:4.0.6-2.fc19-c24100161b1bc9f4a52c3ebdc5f1335f30',
                                     'relativepath':
'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
                                     'savepath':
u'/var/lib/pulp/working/repos/ACME_Corporation-fedora-19-x86_64-pnl/importers/yum_importer/ACME_Corporation-fedora-19-x86_64-pnl',
                                     'fileName':
'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
                                     'downloadurl':
'http://mirror.pnl.gov/fedora/linux/releases/19/Everything/x86_64/os/drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
                                     'item_type': 'delta_rpm',
                                     'epoch': '2',
                                     'version': '4.0.6',
                                     'checksum':
'3802fd63dab195ebe134af6c13d757abc63f66e1d784ca5f15cfb641476770ff',
                                     'release': '2.fc19',
                                     'filename':
'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
                                     'pkgpath':
'/var/lib/pulp/content/drpm/.',
                                     'size': 59892L
     }

=================================================================================
2013-08-28 09:42:07,880 grinder.ParallelFetch:INFO: 4 threads are
active. 11876 items left to be fetched

=================================================================================

The packages keep requeueing over and over again. See below.
======================================
grep libwbclient-2:4.0.6-3.fc19.i686 /var/log/pulp/grinder.log*

grinder.log.1:

2013-08-28 08:57:37,820 grinder.ParallelFetch:INFO: Requeueing:
{'new_package': 'libwbclient-2:4.0.6-3.fc19.i686', 'checksumtype':
'sha256', 'sequence':
'libwbclient-2:4.0.6-2.fc19-c24100161b1bc9f4a52c3ebdc5f1335f30',
'relativepath': 'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'savepath':
u'/var/lib/pulp/working/repos/ACME_Corporation-fedora-19-x86_64-pnl/importers/yum_importer/ACME_Corporation-fedora-19-x86_64-pnl',
'fileName': 'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'downloadurl':
'http://mirror.pnl.gov/fedora/linux/releases/19/Everything/x86_64/os/drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'item_type': 'delta_rpm', 'epoch': '2', 'version': '4.0.6', 'checksum':
'3802fd63dab195ebe134af6c13d757abc63f66e1d784ca5f15cfb641476770ff',
'release': '2.fc19', 'filename':
'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm', 'pkgpath':
'/var/lib/pulp/content/drpm/.', 'size': 59892L}

grinder.log.2:

2013-08-28 07:18:27,346 grinder.ParallelFetch:INFO: Requeueing:
{'new_package': 'libwbclient-2:4.0.6-3.fc19.i686', 'checksumtype':
'sha256', 'sequence':
'libwbclient-2:4.0.6-2.fc19-c24100161b1bc9f4a52c3ebdc5f1335f30',
'relativepath': 'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'savepath':
u'/var/lib/pulp/working/repos/ACME_Corporation-fedora-19-x86_64-pnl/importers/yum_importer/ACME_Corporation-fedora-19-x86_64-pnl',
'fileName': 'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'downloadurl':
'http://mirror.pnl.gov/fedora/linux/releases/19/Everything/x86_64/os/drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'item_type': 'delta_rpm', 'epoch': '2', 'version': '4.0.6', 'checksum':
'3802fd63dab195ebe134af6c13d757abc63f66e1d784ca5f15cfb641476770ff',
'release': '2.fc19', 'filename':
'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm', 'pkgpath':
'/var/lib/pulp/content/drpm/.', 'size': 59892L}

grinder.log.3:

2013-08-28 05:39:17,046 grinder.ParallelFetch:INFO: Requeueing:
{'new_package': 'libwbclient-2:4.0.6-3.fc19.i686', 'checksumtype':
'sha256', 'sequence':
'libwbclient-2:4.0.6-2.fc19-c24100161b1bc9f4a52c3ebdc5f1335f30',
'relativepath': 'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'savepath':
u'/var/lib/pulp/working/repos/ACME_Corporation-fedora-19-x86_64-pnl/importers/yum_importer/ACME_Corporation-fedora-19-x86_64-pnl',
'fileName': 'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'downloadurl':
'http://mirror.pnl.gov/fedora/linux/releases/19/Everything/x86_64/os/drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm',
'item_type': 'delta_rpm', 'epoch': '2', 'version': '4.0.6', 'checksum':
'3802fd63dab195ebe134af6c13d757abc63f66e1d784ca5f15cfb641476770ff',
'release': '2.fc19', 'filename':
'drpms/libwbclient-4.0.6-2.fc19_4.0.6-3.fc19.i686.drpm', 'pkgpath':
'/var/lib/pulp/content/drpm/.', 'size': 59892L}

Version-Release number of selected component (if applicable):
Satellite 6 MDP 1
Fedora 19

How reproducible:
100%

Steps to Reproduce:
See bz 1002728

Actual results:
Fails to recover from interruption

Expected results:
Recovers gracefully

Additional info: