Solution for the time being proposed are as below:


DON'T TOUCH /var/lib/pulp manually at any cost

----

# rm -rf /var/lib/mongodb/*

# umount /var/lib/mongodb

# vi /etc/fstab 

And comment out the entry for /var/lib/mongodb/ mount


Repeat the exact same steps for /var/cache/pulp if it is also a separate mount point. 


Once done:

# satellite-maintain content remove-pulp2

# satellite-maintain service restart


NOTE: This seems like an expected behavior but foreman-maintain should be more intelligent in handling the deletion if /var/lib/mongodb/ and /var/cache/pulp are dedicated mounts.

The dirs that we try to remove are defined in https://github.com/theforeman/foreman_maintain/blob/dfd7f911363f51b112163dfeab0e74d1cc5abfaf/definitions/procedures/pulp/remove.rb#L18-L27

I think we can use that list and filter it using 'Pathname(dir).mountpoint?' (see https://ruby-doc.org/stdlib-2.0.0/libdoc/pathname/rdoc/Pathname.html#method-i-mountpoint-3F) to split it in two:
one part that we can clean up ourself, and the other we report to the user as "please do this yourself"

(In reply to Evgeni Golov from comment #4)
> The dirs that we try to remove are defined in
> https://github.com/theforeman/foreman_maintain/blob/
> dfd7f911363f51b112163dfeab0e74d1cc5abfaf/definitions/procedures/pulp/remove.
> rb#L18-L27
> 
> I think we can use that list and filter it using 'Pathname(dir).mountpoint?'
> (see
> https://ruby-doc.org/stdlib-2.0.0/libdoc/pathname/rdoc/Pathname.html#method-
> i-mountpoint-3F) to split it in two:
> one part that we can clean up ourself, and the other we report to the user
> as "please do this yourself"

So far, I have seen following dirs mostly used as dedicated mounts (either LV or NFS):

        '/var/lib/pulp/published'
        '/var/lib/pulp/content'
        '/var/lib/mongodb/'
        '/var/cache/pulp'


Considering that we don't only have the data inside active mount points to deal with but also need to ensure that, the filesystem is being unmounted, LV is being properly deactivated, and  /etc/fstab is modified to either comment out or delete the relevant entries, I would propose the following from my end:

* Check if the directory is an active mount point or not.

* If no, delete the contents as usual.

* If Yes, Detect the same and 

   --> Don't try to delete the data from that dir and don't fail the "remove-pulp2" action 

   --> Report the user back with details of what they need to do with that dir (delete,unmount,/etc/fstab modification)

   --> Also inform the user that once that cleanup has been done, the pulpcore* services are being restarted manually.

*** Bug 2052561 has been marked as a duplicate of this bug. ***

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34699 has been resolved.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite Maintenance 6.10.5 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1706