Description of problem: When I try to mirror a centos mirror with pulp, should I include the trailing '/' in the URI, pulp finds itself unable to download the netinstall image: Individual package errors encountered during sync: Package: CentOS-6.4-x86_64-netinstall.iso Error: HTTP status code of 404 received for http://yum.singlehop.com/CentOS/6.4/os/x86_64//../../isos/x86_64/CentOS-6.4-x86_ 64-netinstall.iso It behaves properly without said trailing slash. Version-Release number of selected component (if applicable): 2.1.1-1 How reproducible: imminently Steps to Reproduce: 1. Mirror a Centos repo, including a trailing '/' 2. Sync the repo 3. Read the error message Actual results: Error message, no download of netinstall iso Expected results: No error message, download of netinstall iso; Additional info:
Hi Jim! The CentOS repository does not publish an ISO manifest that I can find. Pulp looks for a file called PULP_MANIFEST at the given feed URL. For example, if your feed URL was http://redhat.com/isos, Pulp will attempt to download http://redhat.com/isos/PULP_MANIFEST. This manifest tells Pulp which ISOs are available in the feed. I attempted to look for a manifest on singlehop.com, but I'm not seeing one: http://yum.singlehop.com/CentOS/6.4/isos/x86_64/ What exact commands did you use to create and configure the Pulp repository that you are experiencing this problem with?
Also, for all the times I referenced "Pulp" above, you should substitute "The Pulp ISO plugins".
pulp-admin rpm repo create --repo-id="magic" --feed="http://yum.singlehop.com/CentOS/6.4/os/x86_64/" As opposed to pulp-admin rpm repo create --repo-id="magic" --feed="http://yum.singlehop.com/CentOS/6.4/os/x86_64" A 'pulp-admin rpm repo sync --repo-id="magic"' will work on the second, but not the first.
To be clear, those are different URLs, and pulp (or any other http client) should not be expected to treat them the same. One of the easiest ways to illustrate this is when it comes to joining a relative path to a base path. See here for the rules (especially noting the part about excluding characters to the right of the right-most "/"): http://tools.ietf.org/html/rfc3986#section-5.2.3 Joining "http://yum.singlehop.com/CentOS/6.4/os/x86_64/" with "repodata/repomd.xml" gets you "http://yum.singlehop.com/CentOS/6.4/os/x86_64/repodata/repomd.xml", which is what you want. Joining "http://yum.singlehop.com/CentOS/6.4/os/x86_64" with "repodata/repomd.xml" gets you "http://yum.singlehop.com/CentOS/6.4/os/repodata/repomd.xml, which is missing the "x86_64" segment. Here is the same exercise done with python's standard library implementation: https://dpaste.de/t7O3z/ Sadly, this suggests that if pulp was going to be justified in failing on one of these forms, it was failing on the wrong one. All of that said, either URL will work correctly with pulp's new importer. Should a user mistakenly leave off the trailing "/", pulp will recognize that error and assume the user meant a trailing "/" to be present before attempting any joins. QE: this can be verified by finding any repository that includes a distribution, creating it in pulp with and without the trailing "/" on the feed, and doing a sync.
[root@cloud-qe-1-vm-1 ~]# pulp-admin rpm repo create --repo-id="magic" --feed="http://yum.singlehop.com/CentOS/6.4/os/x86_64/" Successfully created repository [magic] [root@cloud-qe-1-vm-1 ~]# pulp-admin rpm repo sync run --repo-id magic +----------------------------------------------------------------------+ Synchronizing Repository [magic] +----------------------------------------------------------------------+ This command may be exited by pressing ctrl+c without affecting the actual operation on the server. Downloading metadata... [-] ... completed Downloading repository content... [==================================================] 100% RPMs: 6381/6381 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 4/4 items ... completed Importing errata... [-] ... completed Importing package groups/categories... [/] ... completed Publishing packages... [==================================================] 100% Packages: 6381/6381 items ... completed Publishing distributions... ... failed Generating metadata [-] ... completed Publishing repository over HTTPS [-] ... completed [root@cloud-qe-1-vm-1 ~]# pulp-admin rpm repo list +----------------------------------------------------------------------+ RPM Repositories +----------------------------------------------------------------------+ Id: fedora-updates Display Name: fedora-updates Description: None Content Unit Counts: Id: magic Display Name: magic Description: None Content Unit Counts: Distribution: 1 Package Category: 14 Package Group: 210 Rpm: 6381 [root@cloud-qe-1-vm-1 ~]#
[root@cloud-qe-14 ~]# pulp-admin rpm repo create --repo-id rhel6-2 --feed https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.2/x86_64/os --feed-ca-cert CDN/cdn.redhat.com-chain.crt --feed-cert CDN/1359391926_4512.crt --feed-key CDN/1359391926_4512.key Successfully created repository [rhel6-2] C[root@cloud-qe-14 ~]# pulp-admin rpm repo sync run --repo-id rhel6-2 +----------------------------------------------------------------------+ Synchronizing Repository [rhel6-2] +----------------------------------------------------------------------+ A sync task is already in progress for this repository. Its progress will be tracked below. This command may be exited by pressing ctrl+c without affecting the actual operation on the server. Downloading metadata... [-] ... completed Downloading repository content... [==================================================] 100% RPMs: 7281/7281 items Delta RPMs: 0/0 items ... completed Downloading distribution files... [==================================================] 100% Distributions: 6/6 items ... completed Importing errata... [-] ... completed Importing package groups/categories... [|] ... completed Publishing packages... [==================================================] 100% Packages: 7281/7281 items ... completed Publishing distributions... [==================================================] 100% Distributions: 6/6 items ... completed Generating metadata [/] ... completed Publishing repository over HTTPS [-] ... completed [root@cloud-qe-14 ~]#
Verified [root@cloud-qe-14 ~]# rpm -qa pulp-server pulp-server-2.2.0-0.24.beta.el6.noarch [root@cloud-qe-14 ~]# see above 2 comments. Repos that have distribution that has feed with or without ending '/' can be synced
2.2 released http://repos.fedorapeople.org/repos/pulp/pulp/stable/2.2/