Bug 435475 - RFE: [Performance] yum update slower as more packages go into each minor release, esp. bad on IA64 hw
RFE: [Performance] yum update slower as more packages go into each minor rele...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: rpm (Show other bugs)
5.2
ia64 Linux
low Severity low
: rc
: ---
Assigned To: Panu Matilainen
Petr Sklenar
:
Depends On:
Blocks: RHEL5u2_relnotes
  Show dependency treegraph
 
Reported: 2008-02-29 11:15 EST by Alexander Todorov
Modified: 2010-02-23 07:23 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
This had a 5.2 release note that needs to be removed for 5.3 - the text was: (ia64) When upgrading a large number of packages with yum update, the update process will take longer to finish than previous versions.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-09-02 07:41:01 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Alexander Todorov 2008-02-29 11:15:03 EST
Description of problem:
This is a performance issue and most probably not a real bug.
Doing updates from 5.1->5.2 takes very long time compared to previous releases.
This is only on ia64

Version-Release number of selected component (if applicable):
yum-3.2.8-7.el5

How reproducible:
100% on selected hardware

Steps to Reproduce:
1. Install RHEL5.1 Server, all repos enabled, @everything
2. configure yum repos for RHEL 5.2
3. yum update
  
Actual results:
update takes about 7hrs.

Expected results:
update is as fast as possible

Additional info:
5.0 -> 5.1 updates take about 4hrs on the same hardware but for 5.1->5.2 case we
have more packages in the transaction.
Same results with `yum update` vs. `yum update yum` on that hardware.

James Antill provided me with a debugging yum package and says:
james: as from the numbers I saw it's not a real bug ... we are mostly linear in
time, as the package set is expanding
Comment 2 Don Domingo 2008-03-02 19:18:51 EST
just to verify: no workarounds for this? 
Comment 3 Alexander Todorov 2008-03-03 10:21:32 EST
Don,
a proposed trick is

1) yum update yum
2) rpm --rebuilddb
3) yum update

but I haven't seen much difference. I'm performing tests once again right now.
Will post the results and logs later today.
Comment 4 James Antill 2008-03-03 10:48:01 EST
 So the first point is that this only happens when you have _large_ amounts of
packages, so if you don't have @everything ... things are much better.

 If the timings suggest that it is doing more work per. pkg as the pkg count
gets higher, then you could try (I would run a yum update -y too, after this
finishes):

http://people.redhat.com/jantill/yum/commands/yum-iter-update.py

...or something like it (it basically tries to install the updates one at a
time), and a supported core feature to do something like that might be making
it's way upstream for later release.

 But my quick numbers suggested that wasn't the case.
Comment 5 Don Domingo 2008-03-04 18:37:53 EST
thanks James, Alex. adding to "Known Issues" (although i may move this to
"Installation-Related Notes" soon):

<quote>
(ia64) When upgrading a large number of packages with yum update, the update
process may take a very long time to finish.
</quote>

please advise if there's any definitive way to ensure that the problem is
alleviated significantly, at the very least; i will document upon your verification.
Comment 6 Alexander Todorov 2008-03-05 04:52:19 EST
Don,
as of now I'm not aware of any way to alleviate this problem signifficantly. See
James' quote in comment #0. 

I propose the following minor edit because the speed depends on the actual
hardware used, some machines are faster than others.

<quote>
(ia64) When upgrading a large number of packages with yum update, the update
process may take a very long time to finish on some hardware.
</quote>
Comment 7 Alexander Todorov 2008-03-05 09:33:10 EST
Bug #71184 has some nice ideas on how to minimize install time. Some of them may
apply here as well.
Comment 8 Denise Dumas 2008-03-06 17:39:54 EST
Here is the preferred release note text. We're not planning a yum code change
for 5.2, so I'd also like to move this to 5.3. 

(ia64) When upgrading, due to the larger number of packages in the
update, the update process will take longer to finish than previous
versions.
Comment 9 Don Domingo 2008-03-06 17:59:29 EST
thanks Denise, edited release note as requested.
Comment 10 Don Domingo 2008-04-01 22:16:54 EDT
Hi,
the RHEL5.2 release notes will be dropped to translation on April 15, 2008, at
which point no further additions or revisions will be entertained.

a mockup of the RHEL5.2 release notes can be viewed at the following link:
http://intranet.corp.redhat.com/ic/intranet/RHEL5u2relnotesmockup.html

please use the aforementioned link to verify if your bugzilla is already in the
release notes (if it needs to be). each item in the release notes contains a
link to its original bug; as such, you can search through the release notes by
bug number.

Cheers,
Don
Comment 13 Denise Dumas 2008-05-14 15:53:54 EDT
Changing summary to better reflect the topic
Comment 14 James Antill 2008-05-14 16:02:42 EDT
 AIUI it's just package releated ... so changing summary to that.

 Also is it possible to get some feedback on the Fed-9/upstream releases we are
doing? As if we still need to do a significant amount of work I'd like to start
that very soon ... I'm hoping that we'll be close enough in 3.2.15, but
confirmation of that would be good.
Comment 15 RHEL Product and Program Management 2008-06-02 16:16:12 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 18 James Antill 2008-07-16 10:50:59 EDT
 The needinfo was for someone testing a pre. Fed-9 version of YUM.
 I can say that all the tests I've done the yum that'll be in 5.3 will be faster
at updates and use less memory. Obviously it still goes up relative to the
number of packages installed/available ... and noone in Fedora uses ia64, AFAIK :)

 So, anyway, I'm going to devel ack this ... as I think we're pretty good now.
As always, if you could do any testing while we can still fix things that'd be
great (3.2.17++ should be going into the 5.3 branch soon, so it should be easily
available this week or so).
Comment 19 Ryan Lerch 2008-08-06 20:43:52 EDT
Tracking this bug for the Red Hat Enterprise Linux 5.3 Release Notes.
Comment 20 Ryan Lerch 2008-08-06 20:43:52 EDT
Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.
Comment 21 James Antill 2008-08-06 23:52:21 EDT
 To anyone testing this ... can you try the packages in the repo at:

http://people.redhat.com/jantill/53/10/

...there isn't an ia64 arch, but every package is noarch ... so you should be able to install it over 3.2.8 just fine (ie. from GA do update yum, then install packages from the above repo.)
 Some rough numbers on where we are relative to the older yum's would be great.
Comment 22 Alexander Todorov 2008-08-07 08:50:23 EDT
(In reply to comment #21)
>  Some rough numbers on where we are relative to the older yum's would be great.

I've kicked some tests and hopefully we'll have results by tomorrow. The tests are:
5.1 -> 5.2, 5.2 -> 5.3 nightly, 5.1 -> 5.2 (with yum updated) and 5.2 -> 5.3 nightly (with yum updated)
Comment 23 Alexander Todorov 2008-08-08 02:54:08 EDT
(In reply to comment #21)
>  To anyone testing this ... can you try the packages in the repo at:
> 
> http://people.redhat.com/jantill/53/10/
> 

James,
can you generate a repodata/ directory under noarch please?

Some results from previous releases on the same hardware:
RHEL 5.1 -> 5.2, @everything, 06:25:44 hrs
RHEL 5.2 -> 5.3 nightly - too few updated packages so far, will use 5.1 -> 5.2 for testing
Comment 24 James Antill 2008-08-08 09:07:18 EDT
 Ok, I generated the repodata under noarch and made ppc and ia64 basearch directories.
Comment 26 James Antill 2008-08-08 09:50:42 EDT
 Just rpm -e yum-skip-broken will fix it, I'll put a conflict into the specfile.
Comment 27 Alexander Todorov 2008-08-11 10:22:03 EDT
(In reply to comment #23)
> RHEL 5.1 -> 5.2, @everything, 06:25:44 hrs

RHEL 5.1 -> 5.2, @everything/yum update yum/, yum update -> 06:16:07 hrs
Comment 28 James Antill 2008-08-11 10:36:26 EDT
 Ok, so almost 10 minutes better ... which isn't much, however I'm now wondering how much time yum is taking and how much rpm.

 I assume you are basically doing "time yum update -y" ... is it possible for you to do "time (echo n | yum update)". Which will test the time for everything upto the download and rpm-transaction.

 Also, more generally, is it possible to get access to the machine you are running this on before the "yum update" is done?
 I can get ia64 boxes from RHTS, but they don't have everything ... and a 5.1 => 5.2 update is on the order of 8-12 minutes (8 for 3.2.17, 12 for 3.0.1).
Comment 29 Alexander Todorov 2008-08-12 02:50:43 EDT
(In reply to comment #28)
>  Ok, so almost 10 minutes better ... which isn't much, however I'm now
> wondering how much time yum is taking and how much rpm.
> 
>  I assume you are basically doing "time yum update -y" ... is it possible for
> you to do "time (echo n | yum update)". Which will test the time for everything
> upto the download and rpm-transaction.
> 


I need to clarify my results. The total time of about 6hrs reported is from the initial install to the end of the upgrade. Roughly half of that time is the install only. I'll perform more precise timing as described above. 

>  Also, more generally, is it possible to get access to the machine you are
> running this on before the "yum update" is done?
>  I can get ia64 boxes from RHTS, but they don't have everything ... and a 5.1
> => 5.2 update is on the order of 8-12 minutes (8 for 3.2.17, 12 for 3.0.1).

I can provide you with access to the machine in question.
Comment 30 Alexander Todorov 2008-08-12 10:09:21 EDT
New timings for only 'yum update', without actually installing anything:

5.1 -> 5.2: ()
real	5m45.746s
user	1m12.660s
sys	0m7.612s

NOTE:
yum tracebacks with (I guess it can't wait so long to read the 'n' character)

Install     33 Package(s)         
Update     742 Package(s)         
Remove       4 Package(s)         

Total download size: 1.4 G
Is this ok [y/N]: Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 180, in main
    base.doTransaction()
  File "/usr/share/yum-cli/cli.py", line 390, in doTransaction
    if not self.userconfirm():
  File "/usr/share/yum-cli/output.py", line 123, in userconfirm
    choice = raw_input('Is this ok [y/N]: ')
EOFError: EOF when reading a line


5.1 -> 5.2 (yum updated):
real	2m10.889s
user	0m22.272s
sys	0m1.772s


We're 2/3 times faster with the new yum version but what is really slow is the package download/install/remove process. It's sequential if I'm not wrong. 

Bug 71184 has some nice ideas how to speed up things. It's been open for years and the reporter has been active providing information but the bug was CLOSED/WONTFIX.

One thing to start with is parallel downloads of packages to utilize bandwidth. One thread will download the largest packages while another will start downloading the smallest ones (making more connections) and they meet in the middle.

James,
please let me know how you feel about the ideas in bug 71184 and I can start filing them as separate items which can go either to RHEL or Fedora.

Thanks,
Alexander
Comment 31 James Antill 2008-08-12 11:13:41 EDT
> We're 2/3 times faster with the new yum version but what is really slow is the
> package download/install/remove process. It's sequential if I'm not wrong. 

 Ok, yeh. I hadn't realized the yum working parts was so fast already ... was the first test with 3.2.8 or 3.0.l?

 Post depsolving there's work we can do in yum, but I'd guess that most of the work actually needs to be done in rpm. Yum stages are roughly:

1. init - read configs. etc.

2. Run command
  . Possibly download any metadata

3. If command is doing install/remove/update/etc. do a depsolving run
  . Possibly download any metadata

4. Do package downloads
  . We basically do "for pkg in pkgs: download(pkg)"
  . This is also not non-piplined, so each pkg add the latency of the request response.

5. Call "run rpm transaction", this is all inside rpm.


...AIUI from your last post you are getting to #4 in ~2m, and then taking over 3hrs for #4 and #5.
 From yum itself we have no control over #5, although Panu ffesti etc. are working on speeding it up ... but I'd imagine that's going to be a RHEL6 thing.
 We have #4 on the TODO list, to make it do parallel/piplined downloading etc. ... but I don't think we can guarantee that for RHEL5 (part of the process is changing to use curl/pycurl as the backend instead of urllib/httplib).

 We also have on the TODO list to split a large transaction into N smaller ones, and then run each (serially at first, but parallelizing some of it might be possible). This _might_ help you, and we can likely get this into 5.5.

 Looking at BZ 71184 the only thing that looks relevant to yum is the parallel/pipelined download.
Comment 32 James Antill 2008-08-12 11:16:44 EDT
 As a quick test can you do:

rm -rf /var/cache/yum (yum clean has problems with RHN)
time yum update --downloadonly

...that shouldn't have changed much between 3.2.8 and 3.2.17+, so feel free to only test the later one. This'll give us the data on how much is the downloading and how much is rpm.
 Can you also paste the total line, at the end of the download (this is probably fine on it's own if you have it from a previous run).
Comment 33 Alexander Todorov 2008-08-13 03:37:17 EDT
(In reply to comment #31)
>  Ok, yeh. I hadn't realized the yum working parts was so fast already ... was
> the first test with 3.2.8 or 3.0.l?
> 

The slower one was 3.0.1 the faster one 3.2.17

>  Post depsolving there's work we can do in yum, but I'd guess that most of the
> work actually needs to be done in rpm. Yum stages are roughly:
> 
> 1. init - read configs. etc.
> 
> 2. Run command
>   . Possibly download any metadata
> 
> 3. If command is doing install/remove/update/etc. do a depsolving run
>   . Possibly download any metadata
> 
> 4. Do package downloads
>   . We basically do "for pkg in pkgs: download(pkg)"
>   . This is also not non-piplined, so each pkg add the latency of the request
> response.
> 
> 5. Call "run rpm transaction", this is all inside rpm.
> 
> 
> ...AIUI from your last post you are getting to #4 in ~2m, and then taking over
> 3hrs for #4 and #5.

Indeed #4 takes not so long as I use a local http server but pipelined downloads will speed things up.
Comment 34 Alexander Todorov 2008-08-13 03:44:27 EDT
(In reply to comment #32)
>  As a quick test can you do:
> 
> rm -rf /var/cache/yum (yum clean has problems with RHN)
> time yum update --downloadonly
> 
> ...that shouldn't have changed much between 3.2.8 and 3.2.17+, so feel free to
> only test the later one. This'll give us the data on how much is the
> downloading and how much is rpm.
>  Can you also paste the total line, at the end of the download (this is
> probably fine on it's own if you have it from a previous run).

I'd propose something else. If you can provide me with a patched yum package that will do some more verbose logging and print timestamps at different stages I'll be glad to run the test again. 

I have no data how much the download takes from a previous run and this data is not likely to be very correct. The environment used is a shared one and the server/network may be idle or overloaded depending on other machines in the environment. I can run multiple tests (e.g. 10) and get an average of the results if you want.
Comment 37 Denise Dumas 2008-09-30 09:48:37 EDT
We could really close this since yum improved significantly, but this has now become the package installation performance tracker. I'm moving to 5.4/rpm so we dont lose this, but the release note can disappear.
Denise
Comment 38 Denise Dumas 2008-09-30 09:48:37 EDT
Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1,2 +1,4 @@
+This had a 5.2 release note that needs to be removed for 5.3 - the text was:
+
 (ia64)   
 When upgrading a large number of packages with yum update, the update process will take longer to finish than previous versions.
Comment 39 Ryan Lerch 2008-10-12 18:48:36 EDT
Thanks Denise!

I have removed this note from the 5.3 release notes.
Comment 45 errata-xmlrpc 2009-09-02 07:41:01 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-1371.html

Note You need to log in before you can comment on or make changes to this bug.