RHUA 2.0 is much slower sync'ing repos and CDS's than 1.2. 1.2 syncs 1 repo at a time with 10 download threads. 2.0 syncs 4 repos at a time with 4 download threads each. Here are some preliminary numbers I gathered: eu-west 2.0 initial all repo sync: 22 hours eu-west 2.0 initial all repo sync: 5 hours For ap-northeast and ap-southeast, I don't have the exact numbers, but after 7 hours, 1.2 was had pulled down twice as much content as 2.0 had. It was about 90 GB vs 45 GB.
fixed in grinder in commit 493f87bc20609a1a1b83d56b25d69295f3fd14ad
To QA this bug, you can verify that you don't see a large number of errors in the grinder log that say "<package> size mismtach, read X byes, was expecting Y bytes". One or two of those errors might be ok, just b/c occasionally download errors do happen. But you shouldn't see many occurrences of this error.
set tracker bug. 746803
Verified this defect with the following RHUI ISO: RHEL-6.1-RHUI-2.0.1-20111017.0-Server-x86_64-DVD1.iso I've not seen the error messages mentioned in comment2 related to package size mismatch. However following error messages are found. Most of them are related to - md5sum mismatch - Response 404 fetching treeinfo [root@dhcp201-152 ~]# cat /var/log/pulp/grinder.log | grep ERROR 2011-10-21 03:01:44,072 14335:139933873743616: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/5/5Server/x86_64/os/treeinfo. 2011-10-21 03:32:43,012 14335:139933146658560: grinder.BaseFetch:ERROR: activeobject:169 boot.iso md5sum mismatch, read md5sum of: d7cf7e76408914f6fa4f92cc09b409fcf864d5155c694fbf1675e56c3b5a9b29 expected md5sum of 9a4980bc273da004e2160f2e0e81f1e159b67bb8cb6783d947f5ca3574b77b19 2011-10-21 03:32:43,094 14335:139933146658560: grinder.BaseFetch:ERROR: activeobject:169 Retrying fetch of: boot.iso with 1 retry attempts left. 2011-10-21 03:37:10,988 14335:139933054400256: grinder.BaseFetch:ERROR: activeobject:169 boot.iso md5sum mismatch, read md5sum of: 91f1ff57787b5885561095a217eed2ef06a438a9f56a5c3aed24551b1afa06be expected md5sum of 6a4ded381bda9a591f009ec48ba8f53ef1223370b4add89028247418fc290d00 2011-10-21 03:37:11,047 14335:139933054400256: grinder.BaseFetch:ERROR: activeobject:169 Retrying fetch of: boot.iso with 1 retry attempts left. 2011-10-21 04:01:19,667 14335:139933146658560: grinder.BaseFetch:ERROR: activeobject:169 boot.iso md5sum mismatch, read md5sum of: d7cf7e76408914f6fa4f92cc09b409fcf864d5155c694fbf1675e56c3b5a9b29 expected md5sum of 9a4980bc273da004e2160f2e0e81f1e159b67bb8cb6783d947f5ca3574b77b19 2011-10-21 04:01:19,690 14335:139933146658560: grinder.BaseFetch:ERROR: activeobject:169 Retrying fetch of: boot.iso with 0 retry attempts left. 2011-10-21 04:19:43,235 14335:139933054400256: grinder.BaseFetch:ERROR: activeobject:169 boot.iso md5sum mismatch, read md5sum of: 91f1ff57787b5885561095a217eed2ef06a438a9f56a5c3aed24551b1afa06be expected md5sum of 6a4ded381bda9a591f009ec48ba8f53ef1223370b4add89028247418fc290d00 2011-10-21 04:19:43,273 14335:139933054400256: grinder.BaseFetch:ERROR: activeobject:169 Retrying fetch of: boot.iso with 0 retry attempts left. 2011-10-21 04:52:49,464 14335:139933146658560: grinder.BaseFetch:ERROR: activeobject:169 boot.iso md5sum mismatch, read md5sum of: d7cf7e76408914f6fa4f92cc09b409fcf864d5155c694fbf1675e56c3b5a9b29 expected md5sum of 9a4980bc273da004e2160f2e0e81f1e159b67bb8cb6783d947f5ca3574b77b19 2011-10-21 05:16:41,054 14335:139933054400256: grinder.BaseFetch:ERROR: activeobject:169 boot.iso md5sum mismatch, read md5sum of: 91f1ff57787b5885561095a217eed2ef06a438a9f56a5c3aed24551b1afa06be expected md5sum of 6a4ded381bda9a591f009ec48ba8f53ef1223370b4add89028247418fc290d00 2011-10-21 15:39:46,666 17449:139750331623168: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.0/i386/rhui/2.0/os/.treeinfo. 2011-10-21 15:39:49,807 17449:139750115636992: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.0/x86_64/rhui/2.0/os/.treeinfo. 2011-10-21 15:39:49,849 17449:139750105147136: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.1/i386/rhui/2.0/os/.treeinfo. 2011-10-21 15:39:49,962 17449:139750094657280: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.1/x86_64/rhui/2.0/os/.treeinfo. 2011-10-21 15:39:54,431 17449:139750331623168: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.0/i386/rhui/2.0/os/treeinfo. 2011-10-21 15:39:56,403 17449:139750105147136: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.1/i386/rhui/2.0/os/treeinfo. 2011-10-21 15:39:56,506 17449:139750094657280: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.1/x86_64/rhui/2.0/os/treeinfo. 2011-10-21 15:39:56,552 17449:139750115636992: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.0/x86_64/rhui/2.0/os/treeinfo. 2011-10-21 15:40:25,696 17449:139750115636992: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6Server/x86_64/rhui/2.0/os/.treeinfo. 2011-10-21 15:40:25,738 17449:139750105147136: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6Server/i386/rhui/2.0/os/.treeinfo. 2011-10-21 15:40:31,772 17449:139750115636992: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6Server/x86_64/rhui/2.0/os/treeinfo. 2011-10-21 15:40:32,218 17449:139750105147136: grinder.BaseFetch:WARNING: BaseFetch:271 ERROR: Response = 404 fetching https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6Server/i386/rhui/2.0/os/treeinfo. If these errors are ignorable, I'll move this to verified state. Please confirm
The 404 on treeinfo files is normal. Many of these files are missing from the CDN. As for the md5sum mismatch on the boot.iso files, I'm not sure. If a .treeinfo file was found, it's possible it's not correct, hence the mismatch. Can you attach the full grinder logs to the issue so I can look?
The md5sum mismatch does appear to be a problem with content on the CDN. In https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.1/i386/os/treeinfo the sha256sum for boot.iso is listed as 9a4980bc273da004e2160f2e0e81f1e159b67bb8cb6783d947f5ca3574b77b19 However if you download that file, it's actually: /home/jslagle/Downloads $ sha256sum boot.iso d7cf7e76408914f6fa4f92cc09b409fcf864d5155c694fbf1675e56c3b5a9b29 boot.iso You can see this exact error reported in grinder.log.2 on the qa RHUA 10.65.201.152. Going to bump this back to ON_QA since this is a content problem. I've reported it to jgreguske in rel-eng.
As per comment7, this is not RHUI related issue and specific to contents. Since the reported issue related to grinder log that say "<package> size mismtach, read X byes, was expecting Y bytes", is no longer reproducible now. So moving this to verified.
This issue "<package> size mismtach, read X byes, was expecting Y bytes" is reproducible with new RHUI ISO while syncing CDS nodes from RHUA. RHEL-6.1-RHUI-2.0.1-20111024.0-Server-x86_64-DVD1.iso this is from /var/log/gofer/agent.log: =========================================== --- -10-27 16:47:38,830 [ERROR][Thread-204] __logchild() @ activeobject.py:169 - systemtap-runtime-1.2-11.el6_0.i686.rpm size mismatch, read: 406 bytes, was expecting 76008 bytes 2011-10-27 16:47:38,855 [ERROR][Thread-206] __logchild() @ activeobject.py:169 - systemtap-runtime-1.2-9.el6.i686.rpm size mismatch, read: 403 bytes, was expecting 75888 bytes 2011-10-27 16:47:38,862 [ERROR][Thread-205] __logchild() @ activeobject.py:169 - systemtap-runtime-1.4-6.el6.i686.rpm size mismatch, read: 403 bytes, was expecting 89728 bytes 2011-10-27 16:47:38,926 [ERROR][Thread-208] __logchild() @ activeobject.py:169 - systemtap-sdt-devel-1.2-9.el6.i686.rpm size mismatch, read: 405 bytes, was expecting 13584 bytes 2011-10-27 16:47:38,929 [ERROR][Thread-209] __logchild() @ activeobject.py:169 - systemtap-sdt-devel-1.4-6.el6.i686.rpm size mismatch, read: 405 bytes, was expecting 45108 bytes 2011-10-27 16:47:38,941 [ERROR][Thread-207] __logchild() @ activeobject.py:169 - systemtap-sdt-devel-1.2-11.el6_0.i686.rpm size mismatch, read: 408 bytes, was expecting 13676 bytes -----
These size mismatch errors are a result of 404 errors that get saved as the rpm package filename. The 404 errors are a result of the packages not being on the RHUA since I enabled the remove_old_packages flag. That caused the issue since the packages are still referenced in the repo metadata and they will 404 for both CDS's and clients. For now, we decided to just toggle that flag back to false. committed to pulp rhui branch: 0fd8cccafd9df947ff2e95d0a28d3c1d6b8e2951 You can verify that remove_old_packages is set to false in /etc/pulp/pulp.conf, and you should no longer see these errors in the logs on the CDS.
Verified with RHUI Gold iso having version: RHEL-6.1-RHUI-2.0.1-20111027.1-Server-x86_64-DVD1.iso [root@dhcp201-180 ~]# cat /etc/pulp/pulp.conf | grep remove_old_packages remove_old_packages: false [root@dhcp201-180 ~]# rpm -qa | grep pulp pulp-client-0.0.214-8.el6.noarch pulp-0.0.214-8.el6.noarch pulp-common-0.0.214-8.el6.noarch [root@dhcp201-180 ~]# Now remove_old_packages is set to false in /etc/pulp/pulp.conf
CDS sync is finished successfully. -= CDS Synchronization Status =- Last Refreshed: 17:01:22 (updated every 5 seconds, ctrl+c to exit) cds136 ...................................................... [ UP ] cds187 ...................................................... [ UP ] Next Sync Last Sync Last Result ------------------------------------------------------------------------------ cds136 10-28-2011 21:24 10-28-2011 16:51 Success cds187 10-28-2011 21:25 10-28-2011 16:51 Success Connected: dhcp201-180.englab.pnq.redhat.com ------------------------------------------------------------------------------ ^Crhui (sync) => No errors reported related to <package> size mismatch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0367