Bug 2043089

Description of problem:

I haven't tested directly on Satellite but the Orphan cleanup seems to be taking no effect on reducing the size of the /var/lib/pulp/media/artifact of Red Hat Capsule 6.10 at all.

Version-Release number of selected component (if applicable):

Satellite / Ca[sule 6.10.1

How reproducible:

100%

Steps to Reproduce:
1. Install a Satellite 6.10.1 and Capsule 6.10.1

2. Import manifest in satellite

3. Enable "Red Hat Satellite Tools 6.10 for RHEL 8 x86_64 RPMs" repo and it has 10 RPMs 

4. Sync it in satellite with Immediate download policy and confirm that /var/lib/pulp/media/artifact/*/* of the satellite has all 10 RPMs

    # ls /var/lib/pulp/media/artifact/*/*  | xargs file | grep RPM

5. Now, Associate the "Library" environment to the Capsule server and ensure that the download policy of the capsule is Immediate.

6. Sync the capsule and once done, Verify that /var/lib/pulp/media/artifact/*/* of the capsule has all 10 RPMs as Satellite.

    # ls /var/lib/pulp/media/artifact/*/*  | xargs file | grep RPM


7. Now, Disassociate the "Library" environment from the capsule and do an Optimized Sync on the same.

8. Run Orphan cleanup for that capsule where X is the ID of the capsule.

	# SMART_PROXY_ID=X foreman-rake katello:delete_orphaned_content RAILS_ENV=production

   And wait for the "Orphan Cleanup" task to get completed in Satellite UI.


8. Verify the content of /var/lib/pulp on Capsule afterward.



Actual results:


Steps 4, 6, and 8 results in the same content inside /var/lib/pulp without much of a change even after waiting for 30 mins to an hour. 


Expected results:

At Step 8, practically nothing should be left inside /var/lib/pulp of capsule. 

This is a big concern as without having immediate effect lots of troubleshooting efforts would be getting wasted and the file system will keep staying filled up.


Additional info:


I found out https://docs.pulpproject.org/pulpcore/configuration/settings.html#orphan-protection-time and Katello team confirmed that the ORPHAN_PROTECTION_TIME option is in play here on Satellite\Capsule 6.10 which is by default set to 24 hours or 1440 mins.

This means, After Step 7, one has to wait for 24 hours before that content gets marked as Orphaned, and then Step 8 can be executed.


Alternatively:

Assuming Capsule is the one where the orphan cleanup is needed:

** Edit /etc/pulp/settings.py of the capsule and add this line, to reduce that 24 hours to mere 3 mins.

ORPHAN_PROTECTION_TIME = 3

** Restart either just pulpcore specific services or all services of the capsule. 

** Wait for 3 - 4 mins which will allow the contents to become orphan and then run Orphan cleanup for that capsule where X is the ID of the capsule.

	# SMART_PROXY_ID=X foreman-rake katello:delete_orphaned_content RAILS_ENV=production

   And wait for the "Orphan Cleanup" task to get completed in Satellite UI.


That will now clean up the Orphan data.


** Either revert the changes made in /etc/pulp/settings.py followed by a restart of services or else keep the value of ORPHAN_PROTECTION_TIME set to 180 or so, followed by a restart of services.

Reason: This should ideally be longer than your longest-running task otherwise any content created during that task could be cleaned up before the task finishes. 


So, I guess:

A) We need to have something automated e.g. via Sat settings or installer using which value of ORPHAN_PROTECTION_TIME can be controlled.

B) If needed a child Doc BZ should be opened to explain how the Orphan Cleanup works in Pulp3\Sat 6.10 and what are the Do's and Don'ts to keep in mind about the same.