Bug 752288

Summary: Progress Reporting jumps from 99% to 88% when syncing with distribution files present
Product: [Retired] Pulp Reporter: John Matthews <jmatthew>
Component: user-experienceAssignee: John Matthews <jmatthew>
Status: CLOSED CURRENTRELEASE QA Contact: Preethi Thomas <pthomas>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 1.0.0CC: skarmark
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-02-24 20:12:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description John Matthews 2011-11-09 03:40:17 UTC
Description of problem:
When syncing rhel6_x86_64 I saw progress reach 99% then drop to 88%, this happened because distribution files were marked as 0 bytes and we updated the progress tracker with the accurate size after the transfer for each file started.

Issue is distribution metadata doesn't have file size info.
We build up progress info from known sizes, when we come to the piece of fetching distribution files we have them recorded as 0 bytes, once the transfer begins we get the correct size from pycurl and update the progress info, hence it drops down in percentage since there is more work to do.

Proposal is when we are adding an item of 0 bytes, we perform a remote pycurl operation to determine the size.  

Sample of logs below

2011-11-08 22:32:16,482 20621:140358906271488: grinder.BaseFetch:INFO: activeobject:160 Fetching Unknown bytes: initrd.img from https://cdn.redhat.com/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/images/pxeboot/
initrd.img
2011-11-08 22:32:17,538 20621:140358906271488: grinder.ProgressTracker:WARNING: ProgressTracker:126 Modifying size information for <https://cdn.redhat.com/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/images/pxeb
oot/initrd.img>. original size = <0>, new size = <30063052.0>, diff_in_size = <30063052.0>
2011-11-08 22:32:17,580 20621:140358895781632: grinder.BaseFetch:INFO: activeobject:160 Create a link in repo directory for the package at /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os//Packa
ges/libgweather-2.28.0-4.el6.i686.rpm to ../../../../../../../../../../../packages/libgweather/2.28.0/4.el6/i686/c46/libgweather-2.28.0-4.el6.i686.rpm
2011-11-08 22:32:17,582 20621:140358895781632: grinder.ParallelFetch:INFO: ParallelFetch:138 4 threads are active. 9 items left to be fetched
2011-11-08 22:32:17,681 20621:140358895781632: grinder.BaseFetch:INFO: activeobject:160 Fetching Unknown bytes: vmlinuz from https://cdn.redhat.com/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/images/pxeboot/vml
inuz
2011-11-08 22:32:18,597 20621:140358916761344: grinder.BaseFetch:INFO: activeobject:160 Create a link in repo directory for the package at /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os//Packages/libsmi-0.4.8-4.el6.i686.rpm to ../../../../../../../../../../../packages/libsmi/0.4.8/4.el6/i686/cdb/libsmi-0.4.8-4.el6.i686.rpm
2011-11-08 22:32:18,598 20621:140358916761344: grinder.ParallelFetch:INFO: ParallelFetch:138 4 threads are active. 8 items left to be fetched
2011-11-08 22:32:19,619 20621:140358916761344: grinder.BaseFetch:INFO: activeobject:160 Fetching Unknown bytes: boot.iso from https://cdn.redhat.com/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/images/boot.iso
2011-11-08 22:32:19,645 20621:140358895781632: grinder.ProgressTracker:WARNING: ProgressTracker:126 Modifying size information for <https://cdn.redhat.com/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/images/pxeboot/vmlinuz>. original size = <0>, new size = <3791744.0>, diff_in_size = <3791744.0>
2011-11-08 22:32:20,584 20621:140358916761344: grinder.ProgressTracker:WARNING: ProgressTracker:126 Modifying size information for <https://cdn.redhat.com/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/images/boot.iso>. original size = <0>, new size = <232761344.0>, diff_in_size = <232761344.0>

Comment 1 John Matthews 2011-12-13 16:34:09 UTC
Issue is that distribution metadata lacks file size.  To avoid introducing a risky change into grinder I opted to move the handling of distribution files to the front of the queue.  This means that there will be some fluctuation of progress reporting in the beginning of the sync, yet it will be right in the beginning and not overly noticeable to a user.  

Diff:
http://git.fedorahosted.org/git/?p=grinder.git;a=commitdiff;h=092e47960f18b08e5aa158893ba73e4b601752a3


QE:  Sync a repo with distribution files and use the foreground repo sync option.
Look at the size reported and the number of kickstart files synced.  You should see kickstart files begin to download immediately and the file size may change in the beginning of the operation but once kickstart trees are complete and rpms are only being downloaded total download size does not change.

Will be in grinder 0.133

Comment 2 Jeff Ortel 2011-12-15 20:18:20 UTC
build: 0.255

Comment 3 Preethi Thomas 2011-12-19 19:31:15 UTC
verified

[root@preethi ~]# pulp-admin repo sync --id=f15 -FSync for repository f15 started
Sync: Finished
2941/2941 new items downloaded
0/2941 existing items processed

Item Details: 
Tree Files: 6/6
RPMs: 2935/2935

Comment 4 Preethi Thomas 2012-02-24 20:12:40 UTC
Pulp v1.0 is released
Closed Current Release.