Bug 798007

Created attachment 566125 [details]
Sync errors sorted by type and time (/var/log/katello/production.log)

Description of problem:

After importing a custom manifest generated from access.qa.redhat.com, I perform a provider sync.  Due to some errors in uploading content to the CDN, some of the package URLs result in incorrect permissions (e.g 403 Forbidden).

It seems that the synchronization never completes (or takes *HOURS*) when 403's occur.  I'm not sure if katello should be failing the synchronization sooner, or if it is retrying an expected number of times.  Something seems inefficient in the handling of these errors, but I'm not clear where the fault is.

Version-Release number of selected component (if applicable):
 * candlepin-0.5.22-1.el6.src.rpm
 * katello-0.1.300-1.el6.src.rpm
 * katello-certs-tools-1.0.2-2.el6.src.rpm
 * katello-cli-0.1.100-2.el6.src.rpm
 * katello-configure-0.1.100-7.el6.src.rpm
 * katello-httpd-ssl-key-pair-1.0-1.src.rpm
 * katello-qpid-broker-key-pair-1.0-1.src.rpm
 * katello-selinux-0.1.7-1.el6.src.rpm
 * katello-trusted-ssl-cert-1.0-1.src.rpm
 * pulp-0.0.267-2.el6.src.rpm

How reproducible:
* 3 of 3 attempts have highlighted this problem

Steps to Reproduce:
1. ks provider update --name "Red Hat" --url http://cdn.rcm-qa.redhat.com
2. curl -o /tmp/manifest.zip "http://file.rdu.redhat.com/jlaska/manifests_accessqa.zip"
3. ks provider import_manifest --name "Red Hat" --file /tmp/manifest.zip
4. ks provider synchronize --name "Red Hat"
  
Actual results:

Progress: [##################################################] 100.0%

Expected results:

I'd expect the sync to complete, and report errors.  Instead the sync remains at 100.00% and does not exit until the 403 permission problems are resolved on the server.

Additional info:

 * How many times should katello/pulp retry downloading a rpm before moving on?  the attached re-parse of /var/log/katello/production.log  seems to indicate a high amount of retry.