Bug 799512

Seems like a few more jumps were needed before the % complete was *increasing*

== Actual results ==

Output starts at ...

Progress: [##################################################] 100.0%
Progress: [                                                  ] 1.1%
Progress: [                                                  ] 0.5%
Progress: [                                                  ] 0.7%
Progress: [                                                  ] 1.4%
Progress: [#                                                 ] 3.3%
...

Fixed in commit 81f3eac56e17521a27cbd63380e8f5929cf04a16

In case that the task consist of more subtasks (usually product or provider
synchronization) the problem was we calculated the progress from
finished/unfinished files reported by pulp but we didn't know the
total number of files at the beginning. So the total number of files was
changing every time a new sync task changed from pending to running.

After this change the progress for tasks consisting from more subtasks is
calculated from this number of finished/unfinished tasks. For single repo
synchronization it stays the same. For product or provider synchronization the
progress bar won't be theoretically so accurate (the progress bar will grow the
same amount regardless small or large repo). But since we can't get the size of
repo before the synchronization started it's better. For more details the user
can to to UI (sync management).

Verified:

* candlepin-0.5.26-1.el6.noarch
* candlepin-tomcat6-0.5.26-1.el6.noarch
* katello-0.1.304-1.el6.noarch
* katello-all-0.1.304-1.el6.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.0.4-1.el6.noarch
* katello-cli-0.1.105-1.el6.noarch
* katello-cli-common-0.1.105-1.el6.noarch
* katello-common-0.1.304-1.el6.noarch
* katello-configure-0.1.106-1.el6.noarch
* katello-glue-candlepin-0.1.304-1.el6.noarch
* katello-glue-foreman-0.1.304-1.el6.noarch
* katello-glue-pulp-0.1.304-1.el6.noarch
* katello-qpid-broker-key-pair-1.0-1.noarch
* katello-qpid-client-key-pair-1.0-1.noarch
* katello-selinux-0.1.9-1.el6.noarch
* pulp-1.0.0-5.el6.noarch
* pulp-common-1.0.0-5.el6.noarch
* pulp-selinux-server-1.0.0-5.el6.noarch

On 3 different systems, located in 2 sites (BOS and RDU), I've observed a strange behavior with the katello CLI progress meter during a 'provider sync' from cdn.rcm-qa.redhat.com.

On all 3 different systems, using the following packages:
 * katello-0.1.306-1.el6.src.rpm
 * katello-candlepin-cert-key-pair-1.0-1.src.rpm
 * katello-certs-tools-1.0.6-1.el6.src.rpm
 * katello-cli-0.1.107-1.el6.src.rpm
 * katello-configure-0.1.106-1.el6.src.rpm
 * katello-qpid-broker-key-pair-1.0-1.src.rpm
 * katello-qpid-client-key-pair-1.0-1.src.rpm
 * katello-selinux-0.1.9-1.el6.src.rpm
 * pulp-1.0.0-7.el6.src.rpm

I've observed that during a provider sync, the progress meter starts with a consistent rate of increase.  However, after a short time, the progress meter stops.

Two examples are shown below:

> # Stuck at 25% for 2 hours and 30 minutes
> katello -u admin -p admin provider synchronize --name "Red Hat"
> Progress: [############                                      ] 25.0%

> # Stuck at 12% for 3 hours and 50 minutes
> katello -u admin -p admin provider synchronize --name "Red Hat"
> Progress: [######                                            ] 12.5%

I can confirm that content *is* syncing by `tail`ing /var/log/pulp/{grinder,pulp}.log.  However, for some reason the progress meter stops.

I am attempting to sync the following repositories from the CDN:

> # katello -u admin -p admin repo list -v --environment Library | grep Name
> Name:          Red Hat CloudForms Cloud Engine Beta RPMs x86_64 6Server
> Name:          Red Hat CloudForms System Engine RPMs x86_64 6Server
> Name:          Red Hat CloudForms System Engine Beta RPMs x86_64 6Server
> Name:          Red Hat CloudForms Cloud Engine RPMs x86_64 6Server
> Name:          Red Hat Enterprise Linux 5 Server RPMs x86_64 5Server
> Name:          Red Hat CloudForms Tools for RHEL 6 Beta RPMs i386 6Server
> Name:          Red Hat CloudForms Tools for RHEL 6 Beta RPMs x86_64 6Server
> Name:          Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server
> Name:          Red Hat CloudForms Tools for RHEL 6 RPMs x86_64 6Server
> Name:          Red Hat CloudForms Tools for RHEL 5 RPMs x86_64 5Server
> Name:          Red Hat CloudForms Tools for RHEL 5 Beta RPMs x86_64 5Server
> Name:          Red Hat Enterprise Linux 5 Server RPMs i386 5Server
> Name:          Red Hat CloudForms Tools for RHEL 5 RPMs i386 5Server
> Name:          Red Hat CloudForms Tools for RHEL 5 Beta RPMs i386 5Server
> Name:          Red Hat Enterprise Linux 6 Server RPMs i386 6Server
> Name:          Red Hat CloudForms Tools for RHEL 6 RPMs i386 6Server

Could there be a new problem related to the progress meter, introduced by this change?

inecas: Do you have any thoughts on comment#8, could this be a problem introduced by this bug fix?

Yeah. I've tried to say that this might be the issue in comment https://bugzilla.redhat.com/show_bug.cgi?id=799512#c4. So the issue you're seeing is when you synchronize 4 repositories: one small and 3 large. After the first is synchronized you see the 25% very soon. But then another repositories start syncing that take much more time.

The only way how to deal with that came to my mind would be to create multiple progress-bars for each repo sync. However I'm afraid it won't be possible without depending on ncurses. This doesn't have to be wrong but having less dependencies possible for CLI is not a bad idea.

(In reply to comment #11)
> The only way how to deal with that came to my mind would be to create multiple
> progress-bars for each repo sync. However I'm afraid it won't be possible
> without depending on ncurses. This doesn't have to be wrong but having less
> dependencies possible for CLI is not a bad idea.

Hmm, good thinking.  I think multiple progress bars is a really good idea (probably post v1).  Additionally, I don't think this provides too much detail.  In fact, I think I'm more familiar with this approach from other tools, and I think it would be much more informative for the user.  If it's possible, I definitely support the idea :)

I could envision this working much the same way 'yum' emits download progress when syncing repodata or downloading packages.  Should I open this up as a new bug against post-v1?

getting rid of 6.0.0 version since that doesn't exist