Bug 737584 - RHUA 2.0 is much slower sync'ing repos than 1.2
Summary: RHUA 2.0 is much slower sync'ing repos than 1.2
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Update Infrastructure for Cloud Providers
Classification: Red Hat
Component: RHUA
Version: 2.0.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: James Slagle
QA Contact: mkovacik
URL:
Whiteboard:
Depends On:
Blocks: tracker-rhui-2.0.1
TreeView+ depends on / blocked
 
Reported: 2011-09-12 16:01 UTC by James Slagle
Modified: 2017-03-01 22:06 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-03-01 22:06:01 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0367 normal SHIPPED_LIVE Red Hat Update Infrastructure 3.0 Release 2017-03-02 03:05:22 UTC

Description James Slagle 2011-09-12 16:01:19 UTC
RHUA 2.0 is much slower sync'ing repos and CDS's than 1.2.  

1.2 syncs 1 repo at a time with 10 download threads.
2.0 syncs 4 repos at a time with 4 download threads each.

Here are some preliminary numbers I gathered:
eu-west 2.0 initial all repo sync: 22 hours
eu-west 2.0 initial all repo sync: 5 hours

For ap-northeast and ap-southeast, I don't have the exact numbers, but after 7 hours, 1.2 was had pulled down twice as much content as 2.0 had.  It was about 90 GB vs 45 GB.

Comment 1 James Slagle 2011-10-03 20:25:04 UTC
fixed in grinder in commit 493f87bc20609a1a1b83d56b25d69295f3fd14ad

Comment 2 James Slagle 2011-10-03 20:27:28 UTC
To QA this bug, you can verify that you don't see a large number of errors in the grinder log that say "<package> size mismtach, read X byes, was expecting Y bytes".

One or two of those errors might be ok, just b/c occasionally download errors do happen.  But you shouldn't see many occurrences of this error.

Comment 3 wes hayutin 2011-10-17 20:01:55 UTC
set tracker bug. 746803

Comment 5 Sachin Ghai 2011-10-21 10:51:57 UTC
Verified this defect with the following RHUI ISO:

RHEL-6.1-RHUI-2.0.1-20111017.0-Server-x86_64-DVD1.iso


I've not seen the error messages mentioned in comment2 related to package size mismatch. However following error messages are found.

Most of them are related to 

- md5sum mismatch 
- Response 404 fetching treeinfo

[root@dhcp201-152 ~]# cat /var/log/pulp/grinder.log | grep ERROR
2011-10-21 03:01:44,072 14335:139933873743616: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/5/5Server/x86_64/os/treeinfo.
2011-10-21 03:32:43,012 14335:139933146658560: grinder.BaseFetch:ERROR: activeobject:169 boot.iso md5sum mismatch, read md5sum of: d7cf7e76408914f6fa4f92cc09b409fcf864d5155c694fbf1675e56c3b5a9b29 expected md5sum of 9a4980bc273da004e2160f2e0e81f1e159b67bb8cb6783d947f5ca3574b77b19
2011-10-21 03:32:43,094 14335:139933146658560: grinder.BaseFetch:ERROR: activeobject:169 Retrying fetch of: boot.iso with 1 retry attempts left.
2011-10-21 03:37:10,988 14335:139933054400256: grinder.BaseFetch:ERROR: activeobject:169 boot.iso md5sum mismatch, read md5sum of: 91f1ff57787b5885561095a217eed2ef06a438a9f56a5c3aed24551b1afa06be expected md5sum of 6a4ded381bda9a591f009ec48ba8f53ef1223370b4add89028247418fc290d00
2011-10-21 03:37:11,047 14335:139933054400256: grinder.BaseFetch:ERROR: activeobject:169 Retrying fetch of: boot.iso with 1 retry attempts left.
2011-10-21 04:01:19,667 14335:139933146658560: grinder.BaseFetch:ERROR: activeobject:169 boot.iso md5sum mismatch, read md5sum of: d7cf7e76408914f6fa4f92cc09b409fcf864d5155c694fbf1675e56c3b5a9b29 expected md5sum of 9a4980bc273da004e2160f2e0e81f1e159b67bb8cb6783d947f5ca3574b77b19
2011-10-21 04:01:19,690 14335:139933146658560: grinder.BaseFetch:ERROR: activeobject:169 Retrying fetch of: boot.iso with 0 retry attempts left.
2011-10-21 04:19:43,235 14335:139933054400256: grinder.BaseFetch:ERROR: activeobject:169 boot.iso md5sum mismatch, read md5sum of: 91f1ff57787b5885561095a217eed2ef06a438a9f56a5c3aed24551b1afa06be expected md5sum of 6a4ded381bda9a591f009ec48ba8f53ef1223370b4add89028247418fc290d00
2011-10-21 04:19:43,273 14335:139933054400256: grinder.BaseFetch:ERROR: activeobject:169 Retrying fetch of: boot.iso with 0 retry attempts left.
2011-10-21 04:52:49,464 14335:139933146658560: grinder.BaseFetch:ERROR: activeobject:169 boot.iso md5sum mismatch, read md5sum of: d7cf7e76408914f6fa4f92cc09b409fcf864d5155c694fbf1675e56c3b5a9b29 expected md5sum of 9a4980bc273da004e2160f2e0e81f1e159b67bb8cb6783d947f5ca3574b77b19
2011-10-21 05:16:41,054 14335:139933054400256: grinder.BaseFetch:ERROR: activeobject:169 boot.iso md5sum mismatch, read md5sum of: 91f1ff57787b5885561095a217eed2ef06a438a9f56a5c3aed24551b1afa06be expected md5sum of 6a4ded381bda9a591f009ec48ba8f53ef1223370b4add89028247418fc290d00
2011-10-21 15:39:46,666 17449:139750331623168: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.0/i386/rhui/2.0/os/.treeinfo.
2011-10-21 15:39:49,807 17449:139750115636992: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.0/x86_64/rhui/2.0/os/.treeinfo.
2011-10-21 15:39:49,849 17449:139750105147136: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.1/i386/rhui/2.0/os/.treeinfo.
2011-10-21 15:39:49,962 17449:139750094657280: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.1/x86_64/rhui/2.0/os/.treeinfo.
2011-10-21 15:39:54,431 17449:139750331623168: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.0/i386/rhui/2.0/os/treeinfo.
2011-10-21 15:39:56,403 17449:139750105147136: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.1/i386/rhui/2.0/os/treeinfo.
2011-10-21 15:39:56,506 17449:139750094657280: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.1/x86_64/rhui/2.0/os/treeinfo.
2011-10-21 15:39:56,552 17449:139750115636992: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.0/x86_64/rhui/2.0/os/treeinfo.
2011-10-21 15:40:25,696 17449:139750115636992: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6Server/x86_64/rhui/2.0/os/.treeinfo.
2011-10-21 15:40:25,738 17449:139750105147136: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6Server/i386/rhui/2.0/os/.treeinfo.
2011-10-21 15:40:31,772 17449:139750115636992: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6Server/x86_64/rhui/2.0/os/treeinfo.
2011-10-21 15:40:32,218 17449:139750105147136: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6Server/i386/rhui/2.0/os/treeinfo.



If these errors are ignorable, I'll move this to verified state. Please confirm

Comment 6 James Slagle 2011-10-21 15:20:06 UTC
The 404 on treeinfo files is normal.  Many of these files are missing from the CDN.

As for the md5sum mismatch on the boot.iso files, I'm not sure.  If a .treeinfo file was found, it's possible it's not correct, hence the mismatch.

Can you attach the full grinder logs to the issue so I can look?

Comment 7 James Slagle 2011-10-21 15:34:18 UTC
The md5sum mismatch does appear to be a problem with content on the CDN.

In https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.1/i386/os/treeinfo the sha256sum for boot.iso is listed as 9a4980bc273da004e2160f2e0e81f1e159b67bb8cb6783d947f5ca3574b77b19

However if you download that file, it's actually:
/home/jslagle/Downloads $ sha256sum boot.iso
d7cf7e76408914f6fa4f92cc09b409fcf864d5155c694fbf1675e56c3b5a9b29  boot.iso

You can see this exact error reported in grinder.log.2 on the qa RHUA 10.65.201.152.

Going to bump this back to ON_QA since this is a content problem.  I've reported it to jgreguske in rel-eng.

Comment 8 Sachin Ghai 2011-10-22 03:39:15 UTC
As per comment7, this is not RHUI related issue and specific to contents. 

Since the reported issue related to grinder log that say "<package> size mismtach, read X byes, was expecting Y bytes", is no longer reproducible now. So moving this to verified.

Comment 9 Sachin Ghai 2011-10-27 11:38:20 UTC
This issue "<package> size mismtach, read X byes, was expecting Y
bytes" is reproducible with new RHUI ISO while syncing CDS nodes from RHUA.

RHEL-6.1-RHUI-2.0.1-20111024.0-Server-x86_64-DVD1.iso

this is from /var/log/gofer/agent.log:
===========================================
---
-10-27 16:47:38,830 [ERROR][Thread-204] __logchild() @ activeobject.py:169 - systemtap-runtime-1.2-11.el6_0.i686.rpm size mismatch, read: 406 bytes, was expecting 76008 bytes
2011-10-27 16:47:38,855 [ERROR][Thread-206] __logchild() @ activeobject.py:169 - systemtap-runtime-1.2-9.el6.i686.rpm size mismatch, read: 403 bytes, was expecting 75888 bytes
2011-10-27 16:47:38,862 [ERROR][Thread-205] __logchild() @ activeobject.py:169 - systemtap-runtime-1.4-6.el6.i686.rpm size mismatch, read: 403 bytes, was expecting 89728 bytes
2011-10-27 16:47:38,926 [ERROR][Thread-208] __logchild() @ activeobject.py:169 - systemtap-sdt-devel-1.2-9.el6.i686.rpm size mismatch, read: 405 bytes, was expecting 13584 bytes
2011-10-27 16:47:38,929 [ERROR][Thread-209] __logchild() @ activeobject.py:169 - systemtap-sdt-devel-1.4-6.el6.i686.rpm size mismatch, read: 405 bytes, was expecting 45108 bytes
2011-10-27 16:47:38,941 [ERROR][Thread-207] __logchild() @ activeobject.py:169 - systemtap-sdt-devel-1.2-11.el6_0.i686.rpm size mismatch, read: 408 bytes, was expecting 13676 bytes
-----

Comment 10 James Slagle 2011-10-27 16:09:08 UTC
These size mismatch errors are a result of 404 errors that get saved as the rpm package filename.  The 404 errors are a result of the packages not being on the RHUA since I enabled the remove_old_packages flag.  That caused the issue since the packages are still referenced in the repo metadata and they will 404 for both CDS's and clients.

For now, we decided to just toggle that flag back to false.

committed to pulp rhui branch: 0fd8cccafd9df947ff2e95d0a28d3c1d6b8e2951

You can verify that remove_old_packages is set to false in /etc/pulp/pulp.conf, and you should no longer see these errors in the logs on the CDS.

Comment 11 Sachin Ghai 2011-10-28 08:06:50 UTC
Verified with RHUI Gold iso having version:

RHEL-6.1-RHUI-2.0.1-20111027.1-Server-x86_64-DVD1.iso

[root@dhcp201-180 ~]# cat /etc/pulp/pulp.conf | grep remove_old_packages
remove_old_packages: false

[root@dhcp201-180 ~]# rpm -qa | grep pulp
pulp-client-0.0.214-8.el6.noarch
pulp-0.0.214-8.el6.noarch
pulp-common-0.0.214-8.el6.noarch
[root@dhcp201-180 ~]# 

Now remove_old_packages is set to false in /etc/pulp/pulp.conf

Comment 12 Sachin Ghai 2011-10-28 11:06:58 UTC
CDS sync is finished successfully.

-= CDS Synchronization Status =-

Last Refreshed: 17:01:22
(updated every 5 seconds, ctrl+c to exit)


cds136 ...................................................... [  UP  ]
cds187 ...................................................... [  UP  ]


Next Sync                    Last Sync                    Last Result         
------------------------------------------------------------------------------
cds136
10-28-2011 21:24             10-28-2011 16:51             Success    

cds187
10-28-2011 21:25             10-28-2011 16:51             Success    


                                  Connected: dhcp201-180.englab.pnq.redhat.com
------------------------------------------------------------------------------
^Crhui (sync) => 



No errors reported related to <package> size mismatch

Comment 16 errata-xmlrpc 2017-03-01 22:06:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0367


Note You need to log in before you can comment on or make changes to this bug.