Bug 1331863 - Lazy Sync: RHEL5 content update/install results in 404
Summary: Lazy Sync: RHEL5 content update/install results in 404
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Repositories
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent vote
Target Milestone: Unspecified
Assignee: David Davis
QA Contact: Jitendra Yejare
URL:
Whiteboard:
Depends On: 1317047 1386671
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-29 20:45 UTC by Mike McCune
Modified: 2019-09-25 21:29 UTC (History)
15 users (show)

Fixed In Version: pulp-rpm-2.8.7.3-1
Doc Type: Bug Fix
Doc Text:
This fix ensures that the we can use the Lazy Sync feature with RHEL5 clients.
Clone Of:
: 1386672 (view as bug list)
Environment:
Last Closed: 2016-10-26 12:26:10 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Pulp Redmine 2354 High CLOSED - CURRENTRELEASE Incorrect URL on lazy catalog entries created for existing RPM content units 2016-12-02 16:01:40 UTC
Red Hat Product Errata RHBA-2016:2108 normal SHIPPED_LIVE Satellite 6.2.3 Async Bug Release 2016-10-26 16:21:52 UTC

Description Mike McCune 2016-04-29 20:45:23 UTC
1) Lazy sync (on_demand) RHEL5 repositories (5Server, Kickstart 5.11 and Sat6 Beta Tools)

2) Setup a host group for this OS

3) Provision a host

The provision works fine but after completion login to the host and run:

yum clean all
yum repolist

This generates:

# yum repolist -v
Loading "package_upload" plugin
Loading "product-id" plugin
Not loading "rhnplugin" plugin, as it is disabled
Loading "security" plugin
Loading "subscription-manager" plugin
Updating Subscription Management repositories.
Config time: 0.315
Yum Version: 3.2.22
https://sat-r220-03.lab.eng.rdu2.redhat.com/pulp/repos/Default_Organization/Library/content/dist/rhel/server/5/%24releasever/x86_64/os/repodata/repomd.xml: [Errno 14] HTTP Error 404: NOT FOUND

Comment 3 Brad Buckingham 2016-07-24 16:12:44 UTC
Created redmine issue http://projects.theforeman.org/issues/15816 from this bug

Comment 4 Bryan Kearney 2016-07-24 18:15:10 UTC
Upstream bug component is WebUI

Comment 5 Bryan Kearney 2016-07-25 14:15:19 UTC
Upstream bug component is Repositories

Comment 6 David Davis 2016-09-01 14:57:26 UTC
Tested against Pulp 2.8.7 and it works. I think this was probably one of the lazy sync bugs that pulp fixed in 2.8.7.

Comment 8 Kedar Bidarkar 2016-10-13 14:07:33 UTC
# ll
total 216828
lrwxrwxrwx. 1 foreman-proxy root                 40 Oct 12 09:32 fdi-image-rhel_7-img -> foreman-discovery-image-3.1.1-17.iso-img
lrwxrwxrwx. 1 foreman-proxy root                 44 Oct 12 09:32 fdi-image-rhel_7-vmlinuz -> foreman-discovery-image-3.1.1-17.iso-vmlinuz
-rw-r--r--. 1 foreman-proxy root          216873508 Oct 12 09:32 foreman-discovery-image-3.1.1-17.iso-img
-rw-r--r--. 1 foreman-proxy root            5154912 Oct 12 09:32 foreman-discovery-image-3.1.1-17.iso-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy         0 Oct 13 02:12 RedHat-5.11-x86_64-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy         0 Oct 13 02:12 RedHat-5.11-x86_64-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy         0 Oct 13 04:09 RedHat-7.2-x86_64-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy         0 Oct 13 04:09 RedHat-7.2-x86_64-vmlinuz


Tested with Sat6.2.3 SNAP2.

Please note, initrd and vmlinuz are of size zero bytes for both RHEL5 and RHEL7, when we set download_policy as "on_demand" and try to sync kickstart content.

This was intentionally tested using a completely fresh setup, without syncing any content prior to this. 

NOTE: Syncing even once in an ORG using "immediate" may populate the vmlinuz and initrd.img files.

Comment 9 Jeff Ortel 2016-10-18 18:42:24 UTC
Things found during the investigation on reproducer:

Log:

Oct 18 09:53:58 qe-testing-rhel7 pulp_streamer: pulp.streamer.server:INFO: ** STREAMER DOWNLOADING: https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm
Oct 18 09:53:58 qe-testing-rhel7 pulp_streamer: [HTTPChannel,0,127.0.0.1] /usr/lib/python2.7/site-packages/pulp/server/managers/content/catalog.py:107: exceptions.DeprecationWarning: remove is deprecated. Use delete_one or delete_many instead.
Oct 18 09:53:58 qe-testing-rhel7 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
Oct 18 09:53:59 qe-testing-rhel7 pulp_streamer: [-] 127.0.0.1 - - [18/Oct/2016:13:53:58 +0000] "GET /var/lib/pulp/content/units/rpm/b0/30d4d287ec902b350adcc5a720700519f995fda880a0f1166d190443617d4a/rpm-python-4.4.2.3-34.el5.x86_64.rpm HTTP/1.1" 404 - "-" "urlgrabber/3.1.0 yum/3.2.22"

Catalog:

> db.lazy_content_catalog.find({path:'/var/lib/pulp/content/units/rpm/b0/30d4d287ec902b350adcc5a720700519f995fda880a0f1166d190443617d4a/rpm-python-4.4.2.3-34.el5.x86_64.rpm'}).pretty()
{
	"_id" : ObjectId("57fefc761daf670e21935f4d"),
	"_ns" : "lazy_content_catalog",
	"path" : "/var/lib/pulp/content/units/rpm/b0/30d4d287ec902b350adcc5a720700519f995fda880a0f1166d190443617d4a/rpm-python-4.4.2.3-34.el5.x86_64.rpm",
	"importer_id" : "57fefbe01daf670e478e3e27",
	"unit_id" : "38bf08ac-5c0b-4b6b-878a-0cb0f6695467",
	"unit_type_id" : "rpm",
	"url" : "https://cdn.redhat.com/content/dist/rhel/server/5/5.11/x86_64/kickstart/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm",
	"checksum" : "4fb47fca71ff8ec650ebf7903c2894819e306563",
	"checksum_algorithm" : "sha1",
	"revision" : 1,
	"data" : {
		
	}
}
{
	"_id" : ObjectId("57fefda81daf670e1f7a1eb8"),
	"_ns" : "lazy_content_catalog",
	"path" : "/var/lib/pulp/content/units/rpm/b0/30d4d287ec902b350adcc5a720700519f995fda880a0f1166d190443617d4a/rpm-python-4.4.2.3-34.el5.x86_64.rpm",
	"importer_id" : "57fefb481daf670e46ff9b4d",
	"unit_id" : "38bf08ac-5c0b-4b6b-878a-0cb0f6695467",
	"unit_type_id" : "rpm",
	"url" : "https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm",
	"checksum" : "4fb47fca71ff8ec650ebf7903c2894819e306563",
	"checksum_algorithm" : "sha1",
	"revision" : 1,
	"data" : {
		
	}
}

The 1st catalog entry URL exists on the CDN: 

https://cdn.redhat.com/content/dist/rhel/server/5/5.11/x86_64/kickstart/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm

The 2nd catalog entry URL does *not* exist on the CDN:

https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm does not exist on the CDN and 

Should be (based on browsing the CDN):

https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/Packages/rpm-python-4.4.2.3-34.el5.x86_64.rpm 

Still digging.

Comment 11 Jeff Ortel 2016-10-18 22:48:16 UTC
I found the problem.  In the RPM (pulp) importer, the logic for adding catalog entries for units that already exist is uses the "download_url()" which for rpm is the stored "relativepath" from the existing unit to create the download URL. 

The content unit in the reproducer was created with a relativepath=Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm during the sync from https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/.  The catalog entry with url=https://cdn.redhat.com/content/dist/rhel/server/5/5.11/x86_64/kickstart/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm was created at the same time 2016-10-13T03:16:06.000Z.

Then, when the 2nd repository with feed=https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/ sync'd, it found the existing unit and used the baseurl=https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/ and existing unit relativepath=Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm to create the catalog entry with the incorrect url=https://cdn.redhat.com/content/dist/rhel/server/5/5Server/x86_64/os/Server/rpm-python-4.4.2.3-34.el5.x86_64.rpm at a time of: 2016-10-13T03:21:12.000Z.  Instead of using the ref (url) defined in the primary.xml.

This is only a problem when a content unit (rpm) is available from two upstream repositories but not at the same relative path within the baseurl.  In this case the rpms were under Server/ in one upstream YUM repository and under Packages/ in another.

Fixing this will require some code refactoring and flow changes in the YUM importer to accommodate creating catalog entries for existing content units using information in the primary.xml.

Comment 13 pulp-infra@redhat.com 2016-10-19 16:30:54 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 14 pulp-infra@redhat.com 2016-10-19 16:31:00 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 15 pulp-infra@redhat.com 2016-10-19 20:30:52 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 16 Jeff Ortel 2016-10-21 15:45:26 UTC
Patched upstream and needs to be cherry picked downstream.

commit: ecef550a05ed71d054c29cd403d0560a01fa102f

Comment 17 pulp-infra@redhat.com 2016-10-21 16:01:07 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 18 Jeff Ortel 2016-10-21 16:26:45 UTC
This bug causes pulp-rpm to create invalid catalog entries.  Any satellite customer using lazy downloading already as an unsupported feature MUST re-create any repository with an on_demand or background downloading policy.  This applies to QE as well.  Please test with fresh (newly created) repositories.

Comment 19 jcallaha 2016-10-21 19:29:31 UTC
Does this only occur with lazy repositories that have already been sync'd? I would assume so, since pulp doesn't do much until the sync is kicked off for the first time.

Comment 20 Jeff Ortel 2016-10-24 13:26:08 UTC
(In reply to jcallaha from comment #19)
> Does this only occur with lazy repositories that have already been sync'd? I
> would assume so, since pulp doesn't do much until the sync is kicked off for
> the first time.

Correct.

Comment 23 errata-xmlrpc 2016-10-26 12:26:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2108

Comment 24 pulp-infra@redhat.com 2016-11-21 22:31:40 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 25 pulp-infra@redhat.com 2016-12-02 16:01:41 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.


Note You need to log in before you can comment on or make changes to this bug.