Bug 2027567

Summary:	Removing pulp2 content fails when one of those directories being removed are dedicated mount points
Product:	Red Hat Satellite	Reporter:	Akshay Gadhave <agadhave>
Component:	Satellite Maintain	Assignee:	Amit Upadhye <aupadhye>
Status:	CLOSED ERRATA	QA Contact:	Gaurav Talreja <gtalreja>
Severity:	medium	Docs Contact:
Priority:	high
Version:	6.10.1	CC:	ahumbe, apatel, aupadhye, avnkumar, dhjoshi, egolov, gscarbor, jjansky, kgaikwad, msunil, pcfe, pdwyer, peter.vreman, pmendezh, rcavalca, saydas, zhunting
Target Milestone:	6.10.5	Keywords:	PrioBumpGSS, Triaged, Upgrades
Target Release:	Unused
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	rubygem-foreman_maintain-0.8.26	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2105898 (view as bug list)		Environment:
Last Closed:	2022-05-04 12:23:36 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2105898

Description Akshay Gadhave 2021-11-30 05:06:19 UTC

Description of problem:

After upgrading to 6.10, the remove pulp2 content step fails if one of the directories being removed are dedicated mount points.


Version-Release number of selected component (if applicable):
Satellite 6.10


How reproducible:
Everytime

Steps to Reproduce:
1. Create a 6.9 Satellite with /var/lib/mongodb as a dedicated mount point.
2. Upgrade to 6.10
3. Try removing the pulp2 content:
~~~
# satellite-maintain content remove-pulp2
~~~

Actual results:
The below output is seen:
~~~
Failed executing rm -rf /var/lib/mongodb/, exit status 1:
 rm: cannot remove ‘/var/lib/mongodb/’: Device or resource busy
--------------------------------------------------------------------------------
Scenario [Remove Pulp2 and mongodb packages and data] failed.

The following steps ended up in failing state:

  [pulp-remove]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="pulp-remove"
~~~


Expected results:
Pulp2 content should be removed graciously, check if any of the directories being deleted are dedicated mounts or not. If they are, then delete the contents inside them but not delete the directory itself as that will still be an active mount.

Additional info:

Comment 1 Sayan Das 2021-12-15 17:03:33 UTC

Solution for the time being proposed are as below:


DON'T TOUCH /var/lib/pulp manually at any cost

----

# rm -rf /var/lib/mongodb/*

# umount /var/lib/mongodb

# vi /etc/fstab 

And comment out the entry for /var/lib/mongodb/ mount


Repeat the exact same steps for /var/cache/pulp if it is also a separate mount point. 


Once done:

# satellite-maintain content remove-pulp2

# satellite-maintain service restart


NOTE: This seems like an expected behavior but foreman-maintain should be more intelligent in handling the deletion if /var/lib/mongodb/ and /var/cache/pulp are dedicated mounts.

Comment 4 Evgeni Golov 2022-03-10 13:39:21 UTC

The dirs that we try to remove are defined in https://github.com/theforeman/foreman_maintain/blob/dfd7f911363f51b112163dfeab0e74d1cc5abfaf/definitions/procedures/pulp/remove.rb#L18-L27

I think we can use that list and filter it using 'Pathname(dir).mountpoint?' (see https://ruby-doc.org/stdlib-2.0.0/libdoc/pathname/rdoc/Pathname.html#method-i-mountpoint-3F) to split it in two:
one part that we can clean up ourself, and the other we report to the user as "please do this yourself"

Comment 5 Sayan Das 2022-03-20 14:09:43 UTC

(In reply to Evgeni Golov from comment #4)
> The dirs that we try to remove are defined in
> https://github.com/theforeman/foreman_maintain/blob/
> dfd7f911363f51b112163dfeab0e74d1cc5abfaf/definitions/procedures/pulp/remove.
> rb#L18-L27
> 
> I think we can use that list and filter it using 'Pathname(dir).mountpoint?'
> (see
> https://ruby-doc.org/stdlib-2.0.0/libdoc/pathname/rdoc/Pathname.html#method-
> i-mountpoint-3F) to split it in two:
> one part that we can clean up ourself, and the other we report to the user
> as "please do this yourself"

So far, I have seen following dirs mostly used as dedicated mounts (either LV or NFS):

        '/var/lib/pulp/published'
        '/var/lib/pulp/content'
        '/var/lib/mongodb/'
        '/var/cache/pulp'


Considering that we don't only have the data inside active mount points to deal with but also need to ensure that, the filesystem is being unmounted, LV is being properly deactivated, and  /etc/fstab is modified to either comment out or delete the relevant entries, I would propose the following from my end:

* Check if the directory is an active mount point or not.

* If no, delete the contents as usual.

* If Yes, Detect the same and 

   --> Don't try to delete the data from that dir and don't fail the "remove-pulp2" action 

   --> Report the user back with details of what they need to do with that dir (delete,unmount,/etc/fstab modification)

   --> Also inform the user that once that cleanup has been done, the pulpcore* services are being restarted manually.

Comment 6 Jan Jansky 2022-03-21 08:24:18 UTC

*** Bug 2052561 has been marked as a duplicate of this bug. ***

Comment 8 Bryan Kearney 2022-03-31 20:05:04 UTC

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34699 has been resolved.

Comment 14 errata-xmlrpc 2022-05-04 12:23:36 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite Maintenance 6.10.5 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1706