Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1002728 - Sync of Fedora content takes an inordinate amount of time to complete.
Summary: Sync of Fedora content takes an inordinate amount of time to complete.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Repositories
Version: 6.0.0
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: Unspecified
Assignee: Justin Sherrill
QA Contact: jcallaha
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-08-29 19:48 UTC by Travis Gummels
Modified: 2019-09-25 21:14 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-27 11:08:46 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Travis Gummels 2013-08-29 19:48:10 UTC 
Description of problem:

Sync of Fedora content takes an inordinate amount of time to complete.

Version-Release number of selected component (if applicable):

Satellite 6 MDP 1
Fedora 19

How reproducible:
100%

Steps to Reproduce:
1.  Downloaded iso from the Red Hat channels:

        Satellite-6.0.1-RHEL-6-20130807.0-Satellite-x86_64-dvd1.iso

2.  Installed software as described in the following document section 2.2 command:   ./install_packages

        https://access.redhat.com/site/documentation/en-US/Red_Hat_Satellite/6.0/html/Installation_Guide/Installing_Red_Hat_Satellite_with_an_ISO_Image.html

        (a)  GENERATED ERROR and installation stopped

                This script will install the katello packages on the current machine.
                    - Ensuring we are in an expected directory.
                    - Creating a Repository File
                    - Checking to see if Katello is already installed.
                    - katello-foreman-all is not yet installed, installing it.
                warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID f21541eb: NOKEY

                Public key for ruby193-rubygem-audited-activerecord-3.0.0-7.el6_4.noarch.rpm is not installed
                Error while executing command: 'yum install -y katello-foreman-all'

        (b)  RESOLVED ERROR WITH

                yum install -y --nogpgcheck katello-foreman-all

3.  Configured Katello:

        katello-configure --user-pass=mypass

4.  Authenticated to web user interface:   https://rhn3/katello/

5.  Added with UI:  Content /Repositories /Custom Content Repositories

         (a) Created Provider:

            Name: Fedora
            Description: Fedora Distributions

        (b)  Added Product:

            Name: Fedora 18 x64
            Label: Fedora_18_x64
            Description:
            GPG Key: None defined for this Organization

        (c)  Added Repository

            Name:  yocum
            Label: yocum
            URL:  http://mirrors.yocum.org/fedora/releases/18/Everything/x86_64/os/
            Publish via http: No
            GPG Key:  None defined for this Organization

6.  Initiated sync with UI:   Content /Sync Managment /Sync Status

        (a) Within product, Fedora 18 x64,  selected repository:  yocum
        (b) Started sync by pressing button:  'Synchronize Now'

7.  Sync Status:  NOTE: The sync status never completes when I go to  Content/ Sync Management/ Sync Status

    The sync begins and after a while it goes into this loop that never seems to end.  Currently, the sync time is at  22HRS.

    The grinder log entries that keep recycling for many hours non-stop
    --------------------------------------------------------------------------------------------------------

    2013-08-23 13:23:18,480 grinder.ParallelFetch:INFO: 4 threads are active. 9990 items left to be fetched
    2013-08-23 13:23:18,480 grinder.ParallelFetch:INFO: Requeueing: {'new_package': 'texlive-tkz-linknodes-1:1.0c.svn22833-8.fc18.noarch', 'checksumtype': 'sha256', 'sequence': 'texlive-tkz-linknodes-1:1.0c.svn22833-6.fc18-7fa162e5ec16bfd930cc9f517470dd9b30', 'relativepath': 'drpms/texlive-tkz-linknodes-1.0c.svn22833-6.fc18_1.0c.svn22833-8.fc18.noarch.drpm', 'savepath': u'/var/lib/pulp/working/repos/ACME_Corporation-Fedora_18_x64-yocum/importers/yum_importer/ACME_Corporation-Fedora_18_x64-yocum', 'fileName': 'drpms/texlive-tkz-linknodes-1.0c.svn22833-6.fc18_1.0c.svn22833-8.fc18.noarch.drpm', 'downloadurl': 'http://mirrors.yocum.org/fedora/releases/18/Everything/x86_64/os//drpms/texlive-tkz-linknodes-1.0c.svn22833-6.fc18_1.0c.svn22833-8.fc18.noarch.drpm', 'item_type': 'delta_rpm', 'epoch': '1', 'version': '1.0c.svn22833', 'checksum': '259fc403e8e0e2fea1af27406711c0dd94621afdbb9d6ef9eb5f7a2687242561', 'release': '6.fc18', 'filename': 'drpms/texlive-tkz-linknodes-1.0c.svn22833-6.fc18_1.0c.svn22833-8.fc18.noarch.drpm', 'pkgpath': '/var/lib/pulp/content/drpm/.', 'size': 13312L}

    Running Processes:  There is a total of four
    -------------------------------------------
    apache   25679 25656  0 Aug22 ?        00:00:54 /usr/bin/python /usr/lib/python2.6/site-packages/grinder/activeobject.pyc /usr/lib64/python26.zip /usr/lib64/python2.6 /usr/lib64/python2.6/plat-linux2 /usr/lib64/python2.6/lib-tk /usr/lib64/python2.6/lib-old /usr/lib64/python2.6/lib-dynload /usr/lib64/python2.6/site-packages /usr/lib64/python2.6/site-packages/gst-0.10 /usr/lib64/python2.6/site-packages/gtk-2.0 /usr/lib64/python2.6/site-packages/webkit-1.0 /usr/lib/python2.6/site-packages /usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info /usr/lib/pulp/plugins/distributors /usr/lib/pulp/plugins/importers /usr/lib/pulp/plugins/profilers

Actual results:
Sync never completes

Expected results:
Sync completes

Additional info:
Comment 1 Travis Gummels 2013-08-30 02:13:05 UTC 
From: "Miguel"

Travis, 

Thanks. I have some more info. Last night I deleted all Custom Content Repositories and the like but at first I couldn't delete them. The web user interface would not respond to the request ( Remove repository/ Remove Product / Remove Provider ). I later found some left over stuff here ( /var/lib/pulp/content/drpm/drpms ) from the interrupted sync process. When I deleted the left over stuff, it then allowed me to manage custom content once again. 

ls /var/lib/pulp/content/drpm/drpms 
----------------------------------------------- 

caribou-devel-0.4.4.2-5.fc18_0.4.4.2-6.fc18.x86_64.drpm.lock opensm-libs-3.3.13-3.fc18_3.3.15-3.fc18.i686.drpm.lock 
gallery2-slideshow-2.3.2-4.fc18_2.3.2-5.fc18.noarch.drpm.lock polkit-0.107-3.fc18_0.107-4.fc18.x86_64.drpm.lock 
isns-utils-0.91-8.fc18_0.93-1.fc18.x86_64.drpm.lock stormbaancoureur-2.1.5-9.fc18_2.1.6-1.fc18.x86_64.drpm.lock 
kdenetwork-kget-libs-4.9.3-1.fc18_4.9.4-2.fc18.x86_64.drpm.lock texlive-bibtex8-doc-3.71.svn18835-6.fc18_3.71.svn18835-8.fc18.noarch.drpm.lock 
LabPlot-doc-1.6.0.2-12.fc18_1.6.0.3-1.fc18.x86_64.drpm.lock texlive-greekdates-doc-1.0.svn15878-6.fc18_1.0.svn15878-8.fc18.noarch.drpm.lock 

Anyway, I then recreated the custom content once again for Fedora 18 and 19. But this time I specified an ftp url ( ftp://mirror.anl.gov/pub/fedora/linux/releases/18/Everything/x86_64/os ) for the repository. I figure that way the firewall will do less with ftp then with http. 

The sync went really fast approaching almost 90% in about 1.5 hours then it bogged down once again. It went really really fast, if it only had completed. 
Some of the errors from the sync with the ftp url below: 

/var/log/pulp/grinder.log 
------------------------------- 

* Normal type sync entry: 

2013-08-29 13:20:04,331 grinder.ParallelFetch:INFO: 4 threads are active. 15959 items left to be fetched 
2013-08-29 13:20:04,682 grinder.BaseFetch: INFO: Symlink missing in repo directory. Creating link /var/lib/pulp/working/repos/ACME_Corporation-Fedora_v19-Base_x86_64/importers/yum_importer/ACME_Corporation-Fedora_v19-Base_x86_64/Packages/o/ocaml-lacaml-devel-7.0.3-2.fc19.i686.rpm to ../../../../../../../../content/rpm/./ocaml-lacaml-devel/7.0.3/2.fc19/i686/84 
ebb69f731f929b0ee42f9c6b1d5e3ca627893cfc17133c74381414c87dfe96/ocaml-lacaml-devel-7.0.3-2.fc19.i686.rpm 
2013-08-29 13:20:04,682 grinder.ParallelFetch:INFO: 4 threads are active. 15958 items left to be fetched 
2013-08-29 13:20:04,723 grinder.BaseFetch:INFO: Symlink missing in repo directory. Creating link /var/lib/pulp/working/repos/ACME_Corporation-Fedora_v19-Base_x86_64/importers/yum_i 
mporter/ACME_Corporation-Fedora_v19-Base_x86_64/Packages/p/python-ldaptor-tools-0.0.44-2.20120515git29a06fa.fc19.noarch.rpm to ../../../../../../../../content/rpm/./python-ldaptor- 
tools/0.0.44/2.20120515git29a06fa.fc19/noarch/60dec2944ab07693500df4eecded0eadd6be3e3dfe5c05d9e61089c3a064dacd/python-ldaptor-tools-0.0.44-2.20120515git29a06fa.fc19.noarch.rpm 
2013-08-29 13:20:04,723 grinder.ParallelFetch:INFO: 4 threads are active. 15957 items left to be fetched 
2013-08-29 13:20:04,774 grinder.BaseFetch:INFO: Symlink missing in repo directory. Creating link /var/lib/pulp/working/repos/ACME_Corporation-Fedora_v19-Base_x86_64/importers/yum_i 
mporter/ACME_Corporation-Fedora_v19-Base_x86_64/Packages/a/antlr-tool-2.7.7-26.fc19.noarch.rpm to ../../../../../../../../content/rpm/./antlr-tool/2.7.7/26.fc19/noarch/ce485c8a6214 
a4c2954434ab549c4d8989a192b43847f3bb19b607ef6477411d/antlr-tool-2.7.7-26.fc19.noarch.rpm 
2013-08-29 13:20:04,774 grinder.ParallelFetch:INFO: 4 threads are active. 15956 items left to be fetched 
2013-08-29 13:20:04,879 grinder.BaseFetch:INFO: Symlink missing in repo directory. Creating link /var/lib/pulp/working/repos/ACME_Corporation-Fedora_v19-Base_x86_64/importers/yum_i 
mporter/ACME_Corporation-Fedora_v19-Base_x86_64/Packages/q/qt3-3.3.8b-47.fc19.x86_64.rpm to ../../../../../../../../content/rpm/./qt3/3.3.8b/47.fc19/x86_64/29a0ba7a2d4ebc57cfb26c6c 
4ab49bfbf73a7e7789f0dcdcfbc221c1117a6a59/qt3-3.3.8b-47.fc19.x86_64.rpm 
2013-08-29 13:20:04,880 grinder.ParallelFetch:INFO: 4 threads are active. 15955 items left to be fetched 
. 
. 
. 
. 


* Once it errors out 

2013-08-29 13:20:07,863 grinder.BaseFetch:INFO: Fetching 15740 bytes: drpms/texlive-tocloft-svn20084.2.3e-20.fc19_svn20084.2.3e-22.fc19.noarch.drpm from ftp://mirror.anl.gov/pub/fedora/linux/releases/19/Everything/x86_64/os//drpms/texlive-tocloft-svn20084.2.3e-20.fc19_svn20084.2.3e-22.fc19.noarch.drpm 
2013-08-29 13:20:10,770 grinder.BaseFetch:INFO: Cleanup /var/lib/pulp/content/drpm/./drpms/mrpt-gui-0.9.6-6.fc19_1.0.0-2.fc19.i686.drpm.part 
2013-08-29 13:20:10,770 grinder.BaseFetch:INFO: Cleanup /var/lib/pulp/content/drpm/./drpms/mrpt-gui-0.9.6-6.fc19_1.0.0-2.fc19.i686.drpm 
2013-08-29 13:20:10,921 grinder.BaseFetch:INFO: Cleanup /var/lib/pulp/content/drpm/./drpms/rubygem-simple_form-2.0.3-2.fc19_2.0.3-3.fc19.noarch.drpm.part 
2013-08-29 13:20:10,921 grinder.BaseFetch:INFO: Cleanup /var/lib/pulp/content/drpm/./drpms/rubygem-simple_form-2.0.3-2.fc19_2.0.3-3.fc19.noarch.drpm 
2013-08-29 13:20:10,924 grinder.BaseFetch:INFO: Cleanup /var/lib/pulp/content/drpm/./drpms/libtidy-devel-0.99.0-27.20091203.fc19_0.99.0-28.20091203.fc19.x86_64.drpm.part 
2013-08-29 13:20:10,924 grinder.BaseFetch:INFO: Cleanup /var/lib/pulp/content/drpm/./drpms/libtidy-devel-0.99.0-27.20091203.fc19_0.99.0-28.20091203.fc19.x86_64.drpm 
2013-08-29 13:20:10,943 grinder.BaseFetch:INFO: Cleanup /var/lib/pulp/content/drpm/./drpms/texlive-tocloft-svn20084.2.3e-20.fc19_svn20084.2.3e-22.fc19.noarch.drpm.part 
2013-08-29 13:20:10,943 grinder.BaseFetch:INFO: Cleanup /var/lib/pulp/content/drpm/./drpms/texlive-tocloft-svn20084.2.3e-20.fc19_svn20084.2.3e-22.fc19.noarch.drpm 
2013-08-29 13:20:10,967 grinder.BaseFetch:ERROR: Caught exception<(9, 'Server denied you to change to the given directory')> in fetch(drpms/mrpt-gui-0.9.6-6.fc19_1.0.0-2.fc19.i686.drpm, ftp://mirror.anl.gov/pub/fedora/linux/releases/19/Everything/x86_64/os//drpms/mrpt-gui-0.9.6-6.fc19_1.0.0-2.fc19.i686.drpm ) 
2013-08-29 13:20:10,968 grinder.BaseFetch:ERROR: Traceback (most recent call last): 
File "/usr/lib/python2.6/site-packages/grinder/BaseFetch.py", line 294, in fetch 
curl.perform() 
error: (9, 'Server denied you to change to the given directory') 

2013-08-29 13:20:10,968 grinder.BaseFetch:ERROR: Caught exception<(9, 'Server denied you to change to the given directory')> in fetch(drpms/texlive-tocloft-svn20084.2.3e-20.fc19_svn20084.2.3e-22.fc19.noarch.drpm, ftp://mirror.anl.gov/pub/fedora/linux/releases/19/Everything/x86_64/os//drpms/texlive-tocloft-svn20084.2.3e-20.fc19_svn20084.2.3e-22.fc19.noarch.drpm ) 
2013-08-29 13:20:10,969 grinder.BaseFetch:ERROR: Retrying fetch of: drpms/mrpt-gui-0.9.6-6.fc19_1.0.0-2.fc19.i686.drpm with 1 retry attempts left. 
2013-08-29 13:20:10,969 grinder.BaseFetch:ERROR: Caught exception<(9, 'Server denied you to change to the given directory')> in fetch(drpms/rubygem-simple_form-2.0.3-2.fc19_2.0.3-3 
.fc19.noarch.drpm, ftp://mirror.anl.gov/pub/fedora/linux/releases/19/Everything/x86_64/os//drpms/rubygem-simple_form-2.0.3-2.fc19_2.0.3-3.fc19.noarch.drpm ) 
2013-08-29 13:20:10,969 grinder.BaseFetch:ERROR: Caught exception<(9, 'Server denied you to change to the given directory')> in fetch(drpms/libtidy-devel-0.99.0-27.20091203.fc19_0.99.0-28.20091203.fc19.x86_64.drpm, ftp://mirror.anl.gov/pub/fedora/linux/releases/19/Everything/x86_64/os//drpms/libtidy-devel-0.99.0-27.20091203.fc19_0.99.0-28.20091203.fc19.x86_64.drpm ) 

. 
. 
. 
. 
. 

2013-08-29 13:38:13,313 grinder.ParallelFetch:ERROR: Traceback (most recent call last): 
File "/usr/lib/python2.6/site-packages/grinder/ParallelFetch.py", line 312, in run 
result = self.fetcher.fetchItem(itemInfo) 
File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 82, in __call__ 
return self.object(self, *args, **kwargs) 
File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 269, in __call__ 
return self.__call(method, args, kwargs) 
File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 245, in __call 
return self.__rmi(method.name, args, kwargs) 
File "/usr/lib/python2.6/site-packages/grinder/activeobject.py", line 137, in __rmi 
raise ex 
error: (9, 'Server denied you to change to the given directory') 

. 
. 
. 
. 
. 
. 

2013-08-29 14:10:16,228 grinder.ParallelFetch:INFO: 4 threads are active. 11892 items left to be fetched 
2013-08-29 14:10:16,228 grinder.ParallelFetch:INFO: Requeueing: {'new_package': 'texlive-extsizes-doc-2:svn17263.1.4a-22.fc19.noarch', 'checksumtype': 'sha256', 'sequence': 'texlive-extsizes-doc-2:svn17263.1.4a-20.fc19-1fd7d020312c5db02b688c28eabf1b3060', 'relativepath': 'drpms/texlive-extsizes-doc-svn17263.1.4a-20.fc19_svn17263.1.4a-22.fc19.noarch.drpm', 'savepath': u'/var/lib/pulp/working/repos/ACME_Corporation-Fedora_v19-Base_x86_64/importers/yum_importer/ACME_Corporation-Fedora_v19-Base_x86_64', 'fileName': 'drpms/texlive-extsizes-doc-svn17263.1.4a-20.fc19_svn17263.1.4a-22.fc19.noarch.drpm', 'downloadurl': ' ftp://mirror.anl.gov/pub/fedora/linux/releases/19/Everything/x86_64/os//drpms/texlive-extsizes-doc-svn17263.1.4a-20.fc19_svn17263.1.4a-22.fc19.noarch.drpm ', 'item_type': 'delta_rpm', 'epoch': '2', 'version': 'svn17263.1.4a', 'checksum': '5932e00dad6a4173f350b3912cfebc2967dfee5d5c8337b8d9e985b5d5d21dab', 'release': '20.fc19', 'filename': 'drpms/texlive-extsizes-doc-svn17263.1.4a-20.fc19_svn17263.1.4a-22.fc19.noarch.drpm', 'pkgpath': '/var/lib/pulp/content/drpm/.', 'size': 15204L} 

. 
. 
. 
.
Comment 2 RHEL Program Management 2013-09-17 04:17:09 UTC 
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.
Comment 7 Justin Sherrill 2016-03-17 14:26:02 UTC 
I don't believe this is an issue any longer.  A ton of the underlying technology in pulp has changed and I believe it is much faster.  A quick test for me showed it syncing with the new on_demand mode in under an hour.  

I will move to on_qa for validation.
Comment 8 jcallaha 2016-05-20 23:48:52 UTC 
Justin, sync'ing the repo on-demand wouldn't satisfy this BZ since the packages aren't truly pulled in during the sync stage. However, I've tested this with Satellite 6.2 Beta Snap 12 and was able to successfully sync the entire repo in an hour! With that said, I am marking this as Verified.

New packages: 46074 (46.1 GB)
Duration: 1 Hour
Comment 9 Bryan Kearney 2016-07-27 11:08:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501
Note You need to log in before you can comment on or make changes to this bug.