Bug 1331863

Created redmine issue http://projects.theforeman.org/issues/15816 from this bug

Upstream bug component is WebUI

Upstream bug component is Repositories

Tested against Pulp 2.8.7 and it works. I think this was probably one of the lazy sync bugs that pulp fixed in 2.8.7.

# ll
total 216828
lrwxrwxrwx. 1 foreman-proxy root                 40 Oct 12 09:32 fdi-image-rhel_7-img -> foreman-discovery-image-3.1.1-17.iso-img
lrwxrwxrwx. 1 foreman-proxy root                 44 Oct 12 09:32 fdi-image-rhel_7-vmlinuz -> foreman-discovery-image-3.1.1-17.iso-vmlinuz
-rw-r--r--. 1 foreman-proxy root          216873508 Oct 12 09:32 foreman-discovery-image-3.1.1-17.iso-img
-rw-r--r--. 1 foreman-proxy root            5154912 Oct 12 09:32 foreman-discovery-image-3.1.1-17.iso-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy         0 Oct 13 02:12 RedHat-5.11-x86_64-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy         0 Oct 13 02:12 RedHat-5.11-x86_64-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy         0 Oct 13 04:09 RedHat-7.2-x86_64-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy         0 Oct 13 04:09 RedHat-7.2-x86_64-vmlinuz


Tested with Sat6.2.3 SNAP2.

Please note, initrd and vmlinuz are of size zero bytes for both RHEL5 and RHEL7, when we set download_policy as "on_demand" and try to sync kickstart content.

This was intentionally tested using a completely fresh setup, without syncing any content prior to this. 

NOTE: Syncing even once in an ORG using "immediate" may populate the vmlinuz and initrd.img files.

Things found during the investigation on reproducer:

Log:

Oct 18 09:53:58 qe-testing-rhel7 pulp_streamer: pulp.streamer.server:INFO: ** STREAMER DOWNLOADING: https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm
Oct 18 09:53:58 qe-testing-rhel7 pulp_streamer: [HTTPChannel,0,127.0.0.1] /usr/lib/python2.7/site-packages/pulp/server/managers/content/catalog.py:107: exceptions.DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead.
Oct 18 09:53:58 qe-testing-rhel7 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Oct 18 09:53:59 qe-testing-rhel7 pulp_streamer: [-] 127.0.0.1 - - [18/Oct/2016:13:53:58 +0000] "GET /var/lib/pulp/content/units/rpm/b0/30d4d287ec902b350adcc5a720700519f995fda880a0f1166d190443617d4a/rpm-python-4.4.2.3-34.el5.x86_64.rpm HTTP/1.1" 404 - "-" "urlgrabber/3.1.0 yum/3.2.22"

Catalog:

> db.lazy_content_catalog.find({path:'/var/lib/pulp/content/units/rpm/b0/30d4d287ec902b350adcc5a720700519f995fda880a0f1166d190443617d4a/rpm-python-4.4.2.3-34.el5.x86_64.rpm'}).pretty()
{
	"_id" : ObjectId("57fefc761daf670e21935f4d"),
	"_ns" : "lazy_content_catalog",
	"path" : "/var/lib/pulp/content/units/rpm/b0/30d4d287ec902b350adcc5a720700519f995fda880a0f1166d190443617d4a/rpm-python-4.4.2.3-34.el5.x86_64.rpm",
	"importer_id" : "57fefbe01daf670e478e3e27",
	"unit_id" : "38bf08ac-5c0b-4b6b-878a-0cb0f6695467",
	"unit_type_id" : "rpm",
	"url" : "https://cdn.redhat.com/content/dist/rhel/server/5/5.11/x86_64/kickstart/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm",
	"checksum" : "4fb47fca71ff8ec650ebf7903c2894819e306563",
	"checksum_algorithm" : "sha1",
	"revision" : 1,
	"data" : {
		
	}
}
{
	"_id" : ObjectId("57fefda81daf670e1f7a1eb8"),
	"_ns" : "lazy_content_catalog",
	"path" : "/var/lib/pulp/content/units/rpm/b0/30d4d287ec902b350adcc5a720700519f995fda880a0f1166d190443617d4a/rpm-python-4.4.2.3-34.el5.x86_64.rpm",
	"importer_id" : "57fefb481daf670e46ff9b4d",
	"unit_id" : "38bf08ac-5c0b-4b6b-878a-0cb0f6695467",
	"unit_type_id" : "rpm",
	"url" : "https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm",
	"checksum" : "4fb47fca71ff8ec650ebf7903c2894819e306563",
	"checksum_algorithm" : "sha1",
	"revision" : 1,
	"data" : {
		
	}
}

The 1st catalog entry URL exists on the CDN: 

https://cdn.redhat.com/content/dist/rhel/server/5/5.11/x86_64/kickstart/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm

The 2nd catalog entry URL does *not* exist on the CDN:

https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm does not exist on the CDN and 

Should be (based on browsing the CDN):

https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/Packages/rpm-python-4.4.2.3-34.el5.x86_64.rpm 

Still digging.

I found the problem.  In the RPM (pulp) importer, the logic for adding catalog entries for units that already exist is uses the "download_url()" which for rpm is the stored "relativepath" from the existing unit to create the download URL. 

The content unit in the reproducer was created with a relativepath=Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm during the sync from https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/.  The catalog entry with url=https://cdn.redhat.com/content/dist/rhel/server/5/5.11/x86_64/kickstart/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm was created at the same time 2016-10-13T03:16:06.000Z.

Then, when the 2nd repository with feed=https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/ sync'd, it found the existing unit and used the baseurl=https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/ and existing unit relativepath=Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm to create the catalog entry with the incorrect url=https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm at a time of: 2016-10-13T03:21:12.000Z.  Instead of using the ref (url) defined in the primary.xml.

This is only a problem when a content unit (rpm) is available from two upstream repositories but not at the same relative path within the baseurl.  In this case the rpms were under Server/ in one upstream YUM repository and under Packages/ in another.

Fixing this will require some code refactoring and flow changes in the YUM importer to accommodate creating catalog entries for existing content units using information in the primary.xml.

The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Patched upstream and needs to be cherry picked downstream.

commit: ecef550a05ed71d054c29cd403d0560a01fa102f

The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

This bug causes pulp-rpm to create invalid catalog entries.  Any satellite customer using lazy downloading already as an unsupported feature MUST re-create any repository with an on_demand or background downloading policy.  This applies to QE as well.  Please test with fresh (newly created) repositories.

Does this only occur with lazy repositories that have already been sync'd? I would assume so, since pulp doesn't do much until the sync is kicked off for the first time.

(In reply to jcallaha from comment #19)
> Does this only occur with lazy repositories that have already been sync'd? I
> would assume so, since pulp doesn't do much until the sync is kicked off for
> the first time.

Correct.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2108

The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.