Issue is that after a sync fails looking in the logs may not easily yield the source of the error.  

The underlying cause is that syncs happen in parallel, if an error happens in one thread, it will re-try the download, if that still fails after retry counts have been exhausted the item is marked as an error and the status queue is notified.    Currently we only increment a counter stating that an error occurred.  When the error occurs we will log the error, but it's possible that thousands of packages are sync'd later and have updated the log, so the errors could be deep in the logs, or possibly even rotated out.

This BZ is asking that we add fine grained error reporting returned at the end of a sync.  When an error occurs, create a dictionary of "item information" and the string exception message, possibly even a traceback if appropriate.  Add this error information to the status queue.

At the end of a sync the error information should be available by querying the status of the task. 

This will allow the CLI or API to see exactly what items failed to be synced and why.  No need to hunt through the log.


To test:
1) Setup a repo that will contain network errors on fetching certain packages, be sure the package metadata is present in repodata so grinder will attempt to fetch the repo and fail.
2) Sync the repo and note that the CLI displays # errors occurred.
3) Verify that detailed error information is present:
 a) item that failed is identified
 b) exception string is present

Below is an example of what output looks like when there are errors now.
Note that we limit the number of errors to display per item type to 5.  If it's better to display all the errors from the CLI output, it's an easy change.


$ sudo pulp-admin repo sync --id local_vt_errors -F
Sync for repository local_vt_errors started
Sync: Finished
0/6403 new items downloaded
6403/6403 existing items processed

Item Details: 
Tree_Files: 4/6
Rpms: 303/306
Drpms: 6083/6091
Warning: 13 errors occurred
2 Tree_File Error(s):
	[Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/images/file3.img'
	[Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/images/file1.img'
3 Rpm Error(s):
	[Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/xen-devel-3.0.3-105.el5_5.2.i386.rpm'
	[Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/Virtualization-de-DE-5.2-9.noarch.rpm'
	[Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/xen-devel-3.0.3-64.el5_2.9.i386.rpm'
8 Drpm Error(s):
	[Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/drpms/znc-extra-0.093-2.svn2101.fc14_0.097-2.svn2214.fc14.x86_64.drpm'
	[Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/drpms/webkitgtk-1.3.6-1.fc14_1.3.10-1.fc14.x86_64.drpm'
	[Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/drpms/pmd-4.2.5-7.fc14_4.2.5-8.fc14.noarch.drpm'
	[Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/drpms/boost-program-options-1.44.0-1.fc14_1.44.0-4.fc14.i686.drpm'
	[Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/drpms/kdegames-libs-4.5.3-3.fc14_4.5.4-1.fc14.x86_64.drpm'
	... 3 more error(s) occured.  See server logs for all errors.

Below is an example with yum syncs

$ sudo pulp-admin repo sync --id yum_error_test -F
Sync for repository yum_error_test started
Sync: Finished
0/3 new items downloaded
3/3 existing items processed

Item Details: 
Rpms: 2/3
Warning: 1 errors occurred
1 Rpm Error(s):
	HTTP status code of 403 received for http://jmatthews.fedorapeople.org/repo_with_bad_read_perms/pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm


Grinder 0.78 introduced the needed changes along with the below pulp commits:

http://git.fedorahosted.org/git/?p=pulp.git;a=commitdiff;h=75b797551862aedaf076342850acb331356f7355

http://git.fedorahosted.org/git/?p=pulp.git;a=commitdiff;h=b4e96dbbf388f6511136cffbcd2ff34fef6b455d

Fixed in 0.136.

verified
[root@preethi 8a4]# rpm -q pulp
pulp-0.0.172-1.fc14.noarch

for yum repo


[root@preethi ~]# pulp-admin repo sync --id=bad-repo -F
Sync for repository bad-repo started
Sync: Finished
0/3 new items downloaded
3/3 existing items processed

Item Details: 
Rpms: 0/3
Warning: 3 errors occurred
3 Rpm Error(s):
	HTTP status code of 403 received for http://jmatthews.fedorapeople.org/repo_with_bad_read_perms//pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm
	HTTP status code of 403 received for http://jmatthews.fedorapeople.org/repo_with_bad_read_perms//pulp-test-package-0.3.1-1.fc11.x86_64.rpm
	HTTP status code of 403 received for http://jmatthews.fedorapeople.org/repo_with_bad_read_perms//pulp-test-package-0.2.1-1.fc11.x86_64.rpm

for local repo

[root@preethi 8a4]# pulp-admin repo sync --id=local -F
Sync for repository local started
Sync: Finished
0/4 new items downloaded
4/4 existing items processed

Item Details: 
Rpms: 3/4
Warning: 1 errors occurred
1  Error(s):
	[Errno 13] Permission denied: '/var/lib/pulp/repos/repos/pulp/pulp/demo_repos/test_bandwidth_repo/pulp-large_1mb_test-packageB-0.1.1-1.fc14.noarch.rpm'

Closing with Community Release 15

pulp-0.0.223-4.

Closing with Community Release 15

pulp-0.0.223-4.