Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1094498

Summary:	Re-download packages that don't exist on disk during synchronization
Product:	[Retired] Pulp	Reporter:	Mike McCune <mmccune>
Component:	rpm-support	Assignee:	Sayli Karmarkar <skarmark>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Preethi Thomas <pthomas>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	2.4 Beta	CC:	cperry, mhrivnak, mmccune, pthomas, skarmark
Target Milestone:	---	Keywords:	Triaged
Target Release:	2.4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-08-09 06:55:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	950743

Description Mike McCune 2014-05-05 20:20:35 UTC

In cases where packages or other types of content are no longer available in /var/lib/pulp/content there should be a way to re-download these packages from an feed if they are no longer available on disk.

Historically there is a very common use case that you can take a restored database of Satellite/Spacewalk and resync all the content from the remote sources to restore the state of the synced content.  

With Pulp 2.X you can no longer do this because if an RPM is missing from /var/lib/pulp/content there is no simple way to force a resync as the check to see if it should re-download a package only checks to see if it exists in the database.

Myself and others made the case on IRC to advocate at least checking the disk location to see if the file should be re-downloaded:

https://gist.githubusercontent.com/mccun934/b930430d2044420c6355/raw/gistfile1.txt

Without this ability to re-create /var/lib/pulp/content Katello will have no way to correct itself in this situation:

 * We don't expose our users a way to un-associate content from a repository and determine what orphans should be deleted
 
 * If a subset of the content is missing, calculating what is missing isn't possible so there is no way to determine what should be orphaned and re-synced

This bug requests that we add a check to the download process to see if the target piece of content exists in /var/lib/pulp/content/ and re-download if it is missing.

Comment 1 Michael Hrivnak 2014-05-08 18:23:01 UTC

Is it ok/valuable to only do this for rpms right now? Doing this for every content type would require major changes to how pulp stores files. Those are changes we would like to make anyway, but can't do for this release.

The alternative would be a lot of work to add type-specific logic in the workflow that syncs each and every content type.

Comment 2 Sayli Karmarkar 2014-05-22 22:53:17 UTC

https://github.com/pulp/pulp_rpm/pull/500/

Comment 3 Randy Barlow 2014-05-29 18:07:30 UTC

This was fixed in pulp-2.4.0-0.19.beta.

Comment 4 Preethi Thomas 2014-06-05 14:42:00 UTC

verified
[root@hp-dl380pgen8-01 ~]# rpm -qa pulp-server
pulp-server-2.4.0-0.19.beta.fc20.noarch
[root@hp-dl380pgen8-01 ~]# 

[root@hp-dl380pgen8-01 ~]# rm -f /var/lib/pulp/content/rpm/pulp-test-package/0.2.1/1.fc11/x86_64/4dbde07b4a8eab57e42ed0c9203083f1d61e0b13935d1a569193ed8efc9ecfd7/pulp-test-package-0.2.1-1.fc11.x86_64.rpm 
[root@hp-dl380pgen8-01 ~]# 
[root@hp-dl380pgen8-01 ~]# 
[root@hp-dl380pgen8-01 ~]# pulp-admin rpm repo sync run --repo-id pulp+----------------------------------------------------------------------+
                    Synchronizing Repository [pulp]
+----------------------------------------------------------------------+

This command may be exited via ctrl+c without affecting the request.


Downloading metadata...
[|]
... completed

Downloading repository content...
[==================================================] 100%
RPMs:       1/1 items
Delta RPMs: 0/0 items

... completed

Downloading distribution files...
[==================================================] 100%
Distributions: 3/3 items
... completed

Importing errata...
[-]
... completed

Importing package groups/categories...
[-]
... completed


Task Succeeded



Initializing repo metadata
[-]
... completed

Publishing Distribution files
[-]
... completed

Publishing RPMs
[==================================================] 100%
3 of 3 items
... completed

Publishing Delta RPMs
[-]
... completed

Publishing Errata
[==================================================] 100%
52 of 52 items
... completed

Publishing Comps file
[==================================================] 100%
5 of 5 items
... completed

Publishing Metadata.
[-]
... completed

Closing repo metadata
[-]
... completed

Publishing files to web
[-]
... completed

Writing Listings File
[-]
... completed


Task Succeeded


[root@hp-dl380pgen8-01 ~]# ls -l /var/lib/pulp/content/rpm/pulp-test-package/0.2.1/1.fc11/x86_64/4dbde07b4a8eab57e42ed0c9203083f1d61e0b13935d1a569193ed8efc9ecfd7/
total 4
-rw-r--r--. 1 apache apache 2216 Jun  5 10:38 pulp-test-package-0.2.1-1.fc11.x86_64.rpm
[root@hp-dl380pgen8-01 ~]#

Comment 5 Randy Barlow 2014-08-09 06:55:51 UTC

This has been fixed in Pulp 2.4.0-1.

Comment 6 Michael Hrivnak 2016-03-29 16:20:43 UTC

I don't think we need additional info anymore.