Bug 1410357 - [RFE] Provide option to only keep newest packages on capsule
Summary: [RFE] Provide option to only keep newest packages on capsule
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Capsule - Content
Version: 6.2.4
Hardware: x86_64
OS: Linux
high
medium vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-05 09:47 UTC by Edu Alcaniz
Modified: 2019-08-12 16:13 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-10 20:11:31 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Foreman Issue Tracker 21395 None None None 2017-10-19 13:56:33 UTC

Description Edu Alcaniz 2017-01-05 09:47:10 UTC
Description of problem:
New feature request. 
Possibility to clean up repositories on capsules to clean up old packages that it is not need it. 

'Is there an easy way to cleanup repositories? I mean without delete them and rebuild content views, etc. It should be great if I could remove all packages and let satellite download them again only "on demand".'

Unfortunately, this is not possible and I'm not aware of any RFE requesting this functionality.


Version-Release number of selected component (if applicable):

Satellite 6.2.4
How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Brad Buckingham 2017-01-05 15:46:12 UTC
When you mention deleting old packages that are not needed, can you provide an example?

Is this a scenario, where an existing repository is synced, it brings in a new version of a package and the old one remains?

If so, does the user have 'Mirror on Sync' set on the repositories?  If so, that should remove the old versions.

Comment 4 Edu Alcaniz 2017-01-09 08:02:18 UTC
it seems that we found 

Satellite: server1 "du -sh /var/lib/pulp/content"
129G    /var/lib/pulp/content

Capsule: capsule1 "du -sh /var/lib/pulp/content"
118G    /var/lib/pulp/content

just provisioned a new Satellite Capsule capsule1 and our default_download_policy is on_demand.

Disk usage is 130G in /var/lib, after first synchronization with no hosts associated to this capsule. After that we have self associated this capsule (no package update has been done).

Is that the expected behavior of "lazy sync"?

Comment 5 Michael Hrivnak 2017-01-09 19:39:40 UTC
Since there is a lot of other data stored in /var/lib/ (such as databases), it's hard to compare the new capsule's disk usage at /var/lib/ to the old capsule's usage at /var/lib/pulp/content/

I'm moving this to the "Capsule Content" component for now, since I think the request is for new or different behavior with regards to managing content in Pulp. If there's a new Pulp feature to implement, I'm happy to talk about that, but this looks more like Katello's domain for now.

Comment 6 Edu Alcaniz 2017-01-11 10:19:13 UTC
Hi, this affects on Satellite server too

Comment 7 Carlos Peón 2017-01-11 10:28:42 UTC
> When you mention deleting old packages that are not needed, can you provide an example?

This applies to rolling releases 6Server, 7Server:

sssd-1.11.2-65.el7.x86_64 	
sssd-1.11.2-68.el7_0.5.x86_64 	
sssd-1.11.2-68.el7_0.6.x86_64 	
sssd-1.12.2-58.el7.x86_64 	
sssd-1.12.2-58.el7_1.6.x86_64 	
sssd-1.12.2-58.el7_1.14.x86_64 	
sssd-1.12.2-58.el7_1.17.x86_64 	
sssd-1.12.2-58.el7_1.18.x86_64 	
sssd-1.13.0-40.el7.x86_64 	
sssd-1.13.0-40.el7_2.1.x86_64 	
sssd-1.13.0-40.el7_2.2.x86_64 	
sssd-1.13.0-40.el7_2.4.x86_64 	
sssd-1.13.0-40.el7_2.9.x86_64 	
sssd-1.13.0-40.el7_2.12.x86_64 	
sssd-1.14.0-43.el7.x86_64 	
sssd-1.14.0-43.el7_3.4.x86_64 	 

Usually you only need the latest revision of packages, and over time it tends to store obsolete packages.

Comment 8 Brad Buckingham 2017-01-11 13:25:59 UTC
Hi Edu and carlospec, 

Regarding comment 7, do the repositories containing sssd have the 'mirror_on_sync' setting enabled? The goal of that feature is to have the Satellite repository mirror exactly the content of the source repository.  So, if the source repository removes the older versions and the repository is re-synced, they will no longer exist on the Satellite either.

Comment 9 Brad Buckingham 2017-01-11 13:27:33 UTC
Justin, would the solution for bug 1380799 address the observation provided in comment 4?

Comment 10 Carlos Peón 2017-01-11 14:20:06 UTC
Hello Brad,

Regarding comment #8, mirror_on_sync is enabled. 

Note that all versions should be available (you could need to reinstall or downgrade to an old version) but they needn't be downloaded if you use "on demand" policy. 

Imo the key is that, once a package is downloaded, it's stored "forever". But in a few weeks the probability that you need it again will be low because you should use newer versions.

Comment 11 Justin Sherrill 2017-01-11 14:31:07 UTC
Brad,  I dont' think it would.  The only thing i can of would be to have some sort of auto-expire mode that removes packages after they haven't been used for some amount of time.  This would require some changes in pulp and katello.

Its sounding more and more like a squid proxy :)

Comment 12 Brad Buckingham 2017-01-11 15:10:59 UTC
Thanks for the feedback all.

The bugzilla that I referenced in comment 9 (bug 1380799) should help reduce some of the space/resource utilization on the capsule.  That feature essentially allows for the capsule to be configured for 'on_demand', similar to the behavior on Satellite.  That feature is not currently available in 6.2.z; however, it looks like it should be coming in 6.3.

That said, bug 1380799 does not address a scenario where content has been synced, but is no longer used and possibly obsolete.  Is it correct to assume that that behavior is is requested?  (e.g. A means to define a policy for content expiration or obsoleting, which will result in that content being deleted from the filesystem when met).

Comment 13 Carlos Peón 2017-01-11 15:14:42 UTC
> A means to define a policy for content expiration or obsoleting, which will result in that content being deleted from the filesystem when met

Right, that would be great.

Comment 14 Michael Hrivnak 2017-01-11 18:01:34 UTC
(In reply to Justin Sherrill from comment #11)
> Brad,  I dont' think it would.  The only thing i can of would be to have
> some sort of auto-expire mode that removes packages after they haven't been
> used for some amount of time.  This would require some changes in pulp and
> katello.
> 
> Its sounding more and more like a squid proxy :)

Pulp's yum importer does have a setting for how many versions to keep of any particular package. That addresses this use case of not wanting to keep older versions.

But it does not make older versions available on-demand, which is an interesting idea. It removes older versions from the repo.

Comment 15 Justin Sherrill 2017-01-11 18:05:05 UTC
Thats a good point Michael and that might be good enough for this use case.  

carlospec if we exposed that option would that help?  It would give you the ability to say only keep X number of versions of each package.  So you could only keep the latest 3 version for example.  

We could make that configurable globally for a capsule or possibly at the individual repo level on the main Satellite

Comment 16 Carlos Peón 2017-01-12 11:05:11 UTC
I think it would be preferable to make older versions available (downgrades, missed dependencies ¿?).

Anyway this parameter could open the door for a repository clean up (sync policy "on demand"):

"retain_old_count": "1" -> sync -> remove "retain_old_count" -> sync

I've tried it without success:

# hammer repository info --id 12 | grep Packages
    Packages:       18428
# grep retain /etc/pulp/server/plugins.conf.d/yum_importer.json
    "retain_old_count": "1"
# hammer repository synchronize --id 12
[...................................................................................................................................................] [100%]
No new packages.
# hammer repository info --id 12 | grep Packages
    Packages:       18428
# hammer package list --repository-id 12 --search sssd-client | wc -l
58

Maybe I'm missing something (I've restarted satellite services).

Comment 17 Justin Sherrill 2017-01-12 13:22:57 UTC
Carlos, 

That feature isn't there, but it is present in a subsystem of Satellite (pulp), so it would be trivial to expose and more likely to come sooner than a true 'expire old packages' feature.

I'll go ahead and close this BZ to an RFE to implement true "old package expiring" and use this one to implement 'retain_old_Count'

-Justin

Comment 18 Edu Alcaniz 2017-01-12 14:32:40 UTC
(In reply to Justin Sherrill from comment #17)
> Carlos, 
> 
> That feature isn't there, but it is present in a subsystem of Satellite
> (pulp), so it would be trivial to expose and more likely to come sooner than
> a true 'expire old packages' feature.
> 
> I'll go ahead and close this BZ to an RFE to implement true "old package
> expiring" and use this one to implement 'retain_old_Count'
> 
> -Justin

Thanks Justin, please notify to us the new RFE to attach into the support case.

Comment 27 Justin Sherrill 2017-10-19 13:56:31 UTC
Created redmine issue http://projects.theforeman.org/issues/21395 from this bug

Comment 28 Bryan Kearney 2017-11-10 20:11:31 UTC
Thank you for your interest in Satellite 6. We have evaluated this request, and we do not expect this to be implemented in product in the forseeable future. We are therefore closing this out as WONTFIX. If you have any concerns about this, please feel free to contact Rich Jerrido or Bryan Kearney. Thank you.


Note You need to log in before you can comment on or make changes to this bug.