Hide Forgot
Description of problem: When syncing rhel6_x86_64 I saw progress reach 99% then drop to 88%, this happened because distribution files were marked as 0 bytes and we updated the progress tracker with the accurate size after the transfer for each file started. Issue is distribution metadata doesn't have file size info. We build up progress info from known sizes, when we come to the piece of fetching distribution files we have them recorded as 0 bytes, once the transfer begins we get the correct size from pycurl and update the progress info, hence it drops down in percentage since there is more work to do. Proposal is when we are adding an item of 0 bytes, we perform a remote pycurl operation to determine the size. Sample of logs below 2011-11-08 22:32:16,482 20621:140358906271488: grinder.BaseFetch:INFO: activeobject:160 Fetching Unknown bytes: initrd.img from https://cdn.redhat.com/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/images/pxeboot/ initrd.img 2011-11-08 22:32:17,538 20621:140358906271488: grinder.ProgressTracker:WARNING: ProgressTracker:126 Modifying size information for <https://cdn.redhat.com/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/images/pxeb oot/initrd.img>. original size = <0>, new size = <30063052.0>, diff_in_size = <30063052.0> 2011-11-08 22:32:17,580 20621:140358895781632: grinder.BaseFetch:INFO: activeobject:160 Create a link in repo directory for the package at /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os//Packa ges/libgweather-2.28.0-4.el6.i686.rpm to ../../../../../../../../../../../packages/libgweather/2.28.0/4.el6/i686/c46/libgweather-2.28.0-4.el6.i686.rpm 2011-11-08 22:32:17,582 20621:140358895781632: grinder.ParallelFetch:INFO: ParallelFetch:138 4 threads are active. 9 items left to be fetched 2011-11-08 22:32:17,681 20621:140358895781632: grinder.BaseFetch:INFO: activeobject:160 Fetching Unknown bytes: vmlinuz from https://cdn.redhat.com/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/images/pxeboot/vml inuz 2011-11-08 22:32:18,597 20621:140358916761344: grinder.BaseFetch:INFO: activeobject:160 Create a link in repo directory for the package at /var/lib/pulp//repos/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os//Packages/libsmi-0.4.8-4.el6.i686.rpm to ../../../../../../../../../../../packages/libsmi/0.4.8/4.el6/i686/cdb/libsmi-0.4.8-4.el6.i686.rpm 2011-11-08 22:32:18,598 20621:140358916761344: grinder.ParallelFetch:INFO: ParallelFetch:138 4 threads are active. 8 items left to be fetched 2011-11-08 22:32:19,619 20621:140358916761344: grinder.BaseFetch:INFO: activeobject:160 Fetching Unknown bytes: boot.iso from https://cdn.redhat.com/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/images/boot.iso 2011-11-08 22:32:19,645 20621:140358895781632: grinder.ProgressTracker:WARNING: ProgressTracker:126 Modifying size information for <https://cdn.redhat.com/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/images/pxeboot/vmlinuz>. original size = <0>, new size = <3791744.0>, diff_in_size = <3791744.0> 2011-11-08 22:32:20,584 20621:140358916761344: grinder.ProgressTracker:WARNING: ProgressTracker:126 Modifying size information for <https://cdn.redhat.com/content/dist/rhel/rhui/server-6/releases/6Server/x86_64/os/images/boot.iso>. original size = <0>, new size = <232761344.0>, diff_in_size = <232761344.0>
Issue is that distribution metadata lacks file size. To avoid introducing a risky change into grinder I opted to move the handling of distribution files to the front of the queue. This means that there will be some fluctuation of progress reporting in the beginning of the sync, yet it will be right in the beginning and not overly noticeable to a user. Diff: http://git.fedorahosted.org/git/?p=grinder.git;a=commitdiff;h=092e47960f18b08e5aa158893ba73e4b601752a3 QE: Sync a repo with distribution files and use the foreground repo sync option. Look at the size reported and the number of kickstart files synced. You should see kickstart files begin to download immediately and the file size may change in the beginning of the operation but once kickstart trees are complete and rpms are only being downloaded total download size does not change. Will be in grinder 0.133
build: 0.255
verified [root@preethi ~]# pulp-admin repo sync --id=f15 -FSync for repository f15 started Sync: Finished 2941/2941 new items downloaded 0/2941 existing items processed Item Details: Tree Files: 6/6 RPMs: 2935/2935
Pulp v1.0 is released Closed Current Release.