Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2043089 - Orphan cleanup runs fine but does not clears anything from /var/lib/pulp/media/artifact of Red Hat Capsule 6.10
Summary: Orphan cleanup runs fine but does not clears anything from /var/lib/pulp/medi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Capsule - Content
Version: 6.10.1
Hardware: All
OS: All
high
high
Target Milestone: 6.14.0
Assignee: Samir Jha
QA Contact: Vladimír Sedmík
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-20 15:40 UTC by Sayan Das
Modified: 2023-11-08 14:18 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2211962 (view as bug list)
Environment:
Last Closed: 2023-11-08 14:17:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 36390 0 High Closed Orphan cleanup runs fine but does not clears anything from /var/lib/pulp/media/artifact of Red Hat Capsule 6.10 2023-05-31 14:09:20 UTC
Red Hat Issue Tracker SAT-13506 0 None None None 2023-06-01 14:32:57 UTC
Red Hat Knowledge Base (Solution) 6685201 0 None None None 2022-01-31 08:00:23 UTC
Red Hat Product Errata RHSA-2023:6818 0 None None None 2023-11-08 14:18:04 UTC

Description Sayan Das 2022-01-20 15:40:06 UTC
Description of problem:

I haven't tested directly on Satellite but the Orphan cleanup seems to be taking no effect on reducing the size of the /var/lib/pulp/media/artifact of Red Hat Capsule 6.10 at all.

Version-Release number of selected component (if applicable):

Satellite / Ca[sule 6.10.1

How reproducible:

100%

Steps to Reproduce:
1. Install a Satellite 6.10.1 and Capsule 6.10.1

2. Import manifest in satellite

3. Enable "Red Hat Satellite Tools 6.10 for RHEL 8 x86_64 RPMs" repo and it has 10 RPMs 

4. Sync it in satellite with Immediate download policy and confirm that /var/lib/pulp/media/artifact/*/* of the satellite has all 10 RPMs

    # ls /var/lib/pulp/media/artifact/*/*  | xargs file | grep RPM

5. Now, Associate the "Library" environment to the Capsule server and ensure that the download policy of the capsule is Immediate.

6. Sync the capsule and once done, Verify that /var/lib/pulp/media/artifact/*/* of the capsule has all 10 RPMs as Satellite.

    # ls /var/lib/pulp/media/artifact/*/*  | xargs file | grep RPM


7. Now, Disassociate the "Library" environment from the capsule and do an Optimized Sync on the same.

8. Run Orphan cleanup for that capsule where X is the ID of the capsule.

	# SMART_PROXY_ID=X foreman-rake katello:delete_orphaned_content RAILS_ENV=production

   And wait for the "Orphan Cleanup" task to get completed in Satellite UI.


8. Verify the content of /var/lib/pulp on Capsule afterward.



Actual results:


Steps 4, 6, and 8 results in the same content inside /var/lib/pulp without much of a change even after waiting for 30 mins to an hour. 


Expected results:

At Step 8, practically nothing should be left inside /var/lib/pulp of capsule. 

This is a big concern as without having immediate effect lots of troubleshooting efforts would be getting wasted and the file system will keep staying filled up.


Additional info:


I found out https://docs.pulpproject.org/pulpcore/configuration/settings.html#orphan-protection-time and Katello team confirmed that the ORPHAN_PROTECTION_TIME option is in play here on Satellite\Capsule 6.10 which is by default set to 24 hours or 1440 mins.

This means, After Step 7, one has to wait for 24 hours before that content gets marked as Orphaned, and then Step 8 can be executed.


Alternatively:

Assuming Capsule is the one where the orphan cleanup is needed:

** Edit /etc/pulp/settings.py of the capsule and add this line, to reduce that 24 hours to mere 3 mins.

ORPHAN_PROTECTION_TIME = 3

** Restart either just pulpcore specific services or all services of the capsule. 

** Wait for 3 - 4 mins which will allow the contents to become orphan and then run Orphan cleanup for that capsule where X is the ID of the capsule.

	# SMART_PROXY_ID=X foreman-rake katello:delete_orphaned_content RAILS_ENV=production

   And wait for the "Orphan Cleanup" task to get completed in Satellite UI.


That will now clean up the Orphan data.


** Either revert the changes made in /etc/pulp/settings.py followed by a restart of services or else keep the value of ORPHAN_PROTECTION_TIME set to 180 or so, followed by a restart of services.

Reason: This should ideally be longer than your longest-running task otherwise any content created during that task could be cleaned up before the task finishes. 


So, I guess:

A) We need to have something automated e.g. via Sat settings or installer using which value of ORPHAN_PROTECTION_TIME can be controlled.

B) If needed a child Doc BZ should be opened to explain how the Orphan Cleanup works in Pulp3\Sat 6.10 and what are the Do's and Don'ts to keep in mind about the same.

Comment 2 Daniel Alley 2022-07-06 16:56:36 UTC
The main purpose of ORPHAN_PROTECTION_TIME is that if you upload artifacts / content prior to associating it with any repository and run orphan cleanup (or e.g. it gets run on a schedule), it won't remove the content which you just recently uploaded.

For capsules, this isn't a relevant concern, so they can use a tight limit.

Comment 4 Daniel Alley 2022-10-19 17:47:50 UTC
Capsules should probably have "orphan protection time" configured as 0 (zero).

Comment 5 Daniel Alley 2022-10-19 17:50:14 UTC
I would also update the description of the parameter from " Time in minutes to consider orphan content as orphaned." to "Time in minutes before content not contained within a repo nor recently touched will be considered as orphaned"

The existing description doesn't really capture what is going on.

Comment 11 Samir Jha 2023-05-11 14:48:12 UTC
Created redmine issue https://projects.theforeman.org/issues/36390 from this bug

Comment 12 Bryan Kearney 2023-05-31 12:02:05 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/36390 has been resolved.

Comment 16 errata-xmlrpc 2023-11-08 14:17:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.14 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6818


Note You need to log in before you can comment on or make changes to this bug.