Bug 973767 - Pulp does not properly handle feeds ending in '/'
Pulp does not properly handle feeds ending in '/'
Status: CLOSED CURRENTRELEASE
Product: Pulp
Classification: Community
Component: user-experience (Show other bugs)
2.1.1
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 2.2.0
Assigned To: Michael Hrivnak
Preethi Thomas
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-12 12:21 EDT by Jim
Modified: 2013-09-10 11:45 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-10 11:45:20 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jim 2013-06-12 12:21:47 EDT
Description of problem:

When I try to mirror a centos mirror with pulp, should I include the trailing '/' in the URI, pulp finds itself unable to download the netinstall image:

Individual package errors encountered during sync:

Package: CentOS-6.4-x86_64-netinstall.iso
Error:   HTTP status code of 404                                                                                                                                                          
received for                                                                                                                                                                              
http://yum.singlehop.com/CentOS/6.4/os/x86_64//../../isos/x86_64/CentOS-6.4-x86_                                                                                                          
64-netinstall.iso  

It behaves properly without said trailing slash.

Version-Release number of selected component (if applicable):

2.1.1-1

How reproducible:

imminently

Steps to Reproduce:
1. Mirror a Centos repo, including a trailing '/'
2. Sync the repo
3. Read the error message

Actual results:

Error message, no download of netinstall iso

Expected results:

No error message, download of netinstall iso;

Additional info:
Comment 1 Randy Barlow 2013-06-21 10:32:02 EDT
Hi Jim!

The CentOS repository does not publish an ISO manifest that I can find. Pulp looks for a file called PULP_MANIFEST at the given feed URL. For example, if your feed URL was http://redhat.com/isos, Pulp will attempt to download http://redhat.com/isos/PULP_MANIFEST. This manifest tells Pulp which ISOs are available in the feed.

I attempted to look for a manifest on singlehop.com, but I'm not seeing one:

http://yum.singlehop.com/CentOS/6.4/isos/x86_64/

What exact commands did you use to create and configure the Pulp repository that you are experiencing this problem with?
Comment 2 Randy Barlow 2013-06-21 10:34:17 EDT
Also, for all the times I referenced "Pulp" above, you should substitute "The Pulp ISO plugins".
Comment 3 Jim 2013-06-21 11:33:54 EDT
pulp-admin rpm repo create --repo-id="magic" --feed="http://yum.singlehop.com/CentOS/6.4/os/x86_64/"

As opposed to 

pulp-admin rpm repo create --repo-id="magic" --feed="http://yum.singlehop.com/CentOS/6.4/os/x86_64"


A 'pulp-admin rpm repo sync --repo-id="magic"' will work on the second, but not the first.
Comment 4 Michael Hrivnak 2013-07-30 17:53:07 EDT
To be clear, those are different URLs, and pulp (or any other http client) should not be expected to treat them the same. One of the easiest ways to illustrate this is when it comes to joining a relative path to a base path.

See here for the rules (especially noting the part about excluding characters to the right of the right-most "/"): http://tools.ietf.org/html/rfc3986#section-5.2.3

Joining "http://yum.singlehop.com/CentOS/6.4/os/x86_64/" with "repodata/repomd.xml" gets you "http://yum.singlehop.com/CentOS/6.4/os/x86_64/repodata/repomd.xml", which is what you want.

Joining "http://yum.singlehop.com/CentOS/6.4/os/x86_64" with "repodata/repomd.xml" gets you "http://yum.singlehop.com/CentOS/6.4/os/repodata/repomd.xml, which is missing the "x86_64" segment.

Here is the same exercise done with python's standard library implementation: https://dpaste.de/t7O3z/

Sadly, this suggests that if pulp was going to be justified in failing on one of these forms, it was failing on the wrong one.

All of that said, either URL will work correctly with pulp's new importer. Should a user mistakenly leave off the trailing "/", pulp will recognize that error and assume the user meant a trailing "/" to be present before attempting any joins.

QE: this can be verified by finding any repository that includes a distribution, creating it in pulp with and without the trailing "/" on the feed, and doing a sync.
Comment 5 Preethi Thomas 2013-08-07 08:58:43 EDT
[root@cloud-qe-1-vm-1 ~]# pulp-admin rpm repo create --repo-id="magic" --feed="http://yum.singlehop.com/CentOS/6.4/os/x86_64/"
Successfully created repository [magic]


[root@cloud-qe-1-vm-1 ~]# pulp-admin rpm repo sync run  --repo-id magic
+----------------------------------------------------------------------+
                    Synchronizing Repository [magic]
+----------------------------------------------------------------------+

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[-]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       6381/6381 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 4/4 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[/]
... completed

Publishing packages...
[==================================================] 100%
Packages: 6381/6381 items
... completed

Publishing distributions...
... failed
Generating metadata
[-]
... completed

Publishing repository over HTTPS
[-]
... completed

[root@cloud-qe-1-vm-1 ~]# pulp-admin rpm repo list
+----------------------------------------------------------------------+
                            RPM Repositories
+----------------------------------------------------------------------+

Id:                  fedora-updates
Display Name:        fedora-updates
Description:         None
Content Unit Counts: 

Id:                  magic
Display Name:        magic
Description:         None
Content Unit Counts: 
  Distribution:     1
  Package Category: 14
  Package Group:    210
  Rpm:              6381


[root@cloud-qe-1-vm-1 ~]#
Comment 6 Preethi Thomas 2013-08-07 13:28:17 EDT
[root@cloud-qe-14 ~]#  pulp-admin rpm repo create --repo-id rhel6-2 --feed https://cdn.redhat.com/content/dist/rhel/rhui/server/6/6.2/x86_64/os --feed-ca-cert CDN/cdn.redhat.com-chain.crt --feed-cert CDN/1359391926_4512.crt  --feed-key CDN/1359391926_4512.key
Successfully created repository [rhel6-2]

C[root@cloud-qe-14 ~]# pulp-admin rpm repo sync run --repo-id rhel6-2
+----------------------------------------------------------------------+
                   Synchronizing Repository [rhel6-2]
+----------------------------------------------------------------------+

A sync task is already in progress for this repository. Its progress will be
tracked below.

This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.

Downloading metadata...
[-]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       7281/7281 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 6/6 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[|]
... completed

Publishing packages...
[==================================================] 100%
Packages: 7281/7281 items
... completed

Publishing distributions...
[==================================================] 100%
Distributions: 6/6 items
... completed

Generating metadata
[/]
... completed

Publishing repository over HTTPS
[-]
... completed

[root@cloud-qe-14 ~]#
Comment 7 Preethi Thomas 2013-08-07 13:30:09 EDT
Verified
[root@cloud-qe-14 ~]# rpm -qa pulp-server
pulp-server-2.2.0-0.24.beta.el6.noarch
[root@cloud-qe-14 ~]# 

see above 2 comments. Repos that have distribution that has feed with or without ending '/' can be synced
Comment 8 Preethi Thomas 2013-09-10 11:45:20 EDT
2.2 released
http://repos.fedorapeople.org/repos/pulp/pulp/stable/2.2/

Note You need to log in before you can comment on or make changes to this bug.