Bug 1574257

Summary: katello-remove does not completely remove data on mounted filesystems
Product: Red Hat Satellite Reporter: Sam Yangsao <syangsao>
Component: InfrastructureAssignee: Chris Roberts <chrobert>
Status: CLOSED ERRATA QA Contact: Lukas Pramuk <lpramuk>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.3.1CC: chrobert, ehelms, pcreech
Target Milestone: 6.5.0Keywords: Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
URL: https://projects.theforeman.org/issues/25195
Whiteboard:
Fixed In Version: katello-3.9.0-rc2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-14 12:37:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
two default/local capsules none

Description Sam Yangsao 2018-05-02 22:51:10 UTC 
Description of problem:

katello-remove does not completely remove the data on mounted directory structures

Version-Release number of selected component (if applicable):

Satellite 6.3.1
RHEL 7.5

How reproducible:

Always

Steps to Reproduce:

1.  Install a new RHEL host with mounted filesystems following guidelines [1]:

/var/lib/pgsql
/var/lib/mongodb

2.  Install Satellite server bits

# yum install satellite
# satellite-installer

3.  Remove [2] Satellite 6.3 server bits:

# katello-remove

4.  Change hostname to something else

# hostnamectl set-hostname satname2

5.  Install Satellite server bits again

# yum install satellite
# satellite-installer

Actual results:

Since the data does not remove completely, remnants of the old install still stick around, for example, in this scenario, the hostname was changed during the second installation and the capsule id is now "2" instead of "1" for a fresh installation.

/var/lib/pgsql was not cleaned up properly so the new installation took the data from this directory and used it for Satellite 6

Expected results:

katello-remove should remove *all* the data in these directories

Maybe add some logic to check if the filesystem is mounted, and ask the user if it's OK to remove the data in that path.

We see that /var/lib/pgsql is a mounted filesystem, is it OK to remove /var/lib/pgsql/* ?

Additional info:

[1] https://access.redhat.com/documentation/en-us/red_hat_satellite/6.3/html-single/installation_guide/#hardware_storage_prerequisites
[2] https://access.redhat.com/documentation/en-us/red_hat_satellite/6.3/html-single/installation_guide/#uninstalling_satellite_and_capsule_servers
Comment 1 Sam Yangsao 2018-05-02 22:54:04 UTC 
After running katello-remove ...

[root@sat6test pgsql]# df -k .
Filesystem              1K-blocks   Used Available Use% Mounted on
/dev/mapper/pgsql-pgsql  10471424 120560  10350864   2% /var/lib/pgsql
[root@sat6test pgsql]# pwd
/var/lib/pgsql
[root@sat6test pgsql]# ls
data
[root@sat6test pgsql]# cd data
[root@sat6test data]# ls
base     pg_hba.conf    pg_multixact  pg_snapshots  pg_tblspc    pg_xlog
global   pg_ident.conf  pg_notify     pg_stat_tmp   pg_twophase  postgresql.conf
pg_clog  pg_log         pg_serial     pg_subtrans   PG_VERSION   postmaster.opts
[root@sat6test data]# cd /var/lib/mongodb/
[root@sat6test mongodb]# ls
journal  local.0  local.ns  mongod.lock  pulp_database.0  pulp_database.ns
[root@sat6test mongodb]# df -k .
Filesystem              1K-blocks    Used Available Use% Mounted on
/dev/mapper/mongo-mongo  52399108 3342552  49056556   7% /var/lib/mongodb
[root@sat6test mongodb]#
Comment 2 Sam Yangsao 2018-05-03 01:00:44 UTC 
While attempting to refresh the manifest with the Satellite server in this state,  we encounter this error as both are claiming to be the "default/local" capsule.

"There was an issue with the backend service pulp:  Couldn't find SmartProxy with 'id'=1"
Comment 3 Sam Yangsao 2018-05-03 01:03:01 UTC 
Created attachment 1430370 [details]
two default/local capsules
Comment 5 Brad Buckingham 2018-05-15 17:48:39 UTC 
Note: when addressing this bugzilla, the solution should not destroy all databases on the server.  Instead, it should only be those that are used by Satellite (e.g. foreman, candlepin...etc).
Comment 7 Lukas Pramuk 2018-12-11 14:17:58 UTC 
VERIFIED.

@satellite-6.5.0-5.beta.el7sat.noarch
katello-common-3.10.0-0.6.rc1.el7sat.noarch

by the manual reproducer:

1) Set up a machine to have these mountpoints:

# cat /etc/fstab
...
/dev/mapper/machine01-pgsql /var/lib/pgsql            xfs     defaults        0 0
/dev/mapper/machine01-mongo /var/lib/mongodb          xfs     defaults        0 0

2) Install Satellite

3) Run katello-remove

# katello-remove

WARNING: This script will erase many packages and config files.
Important packages such as the following will be removed:
  * httpd (apache)
  * mongodb
  * tomcat
  * puppet
  * ruby
  * rubygems
  * All Katello and Foreman Packages
Once these packages and configuration files are removed there is no going back.
If you use this system for anything other than Katello and Foreman you probably
do not want to execute this script.

Read the source at /usr/sbin/katello-remove for a list of what is removed.  Are you sure(Y/N) 
y

ARE YOU SURE?: This script permanently deletes data and configurations.

Type [remove] to continue 
remove

Starting removal

Stopping services

Removing RPMs

Cleaning up configuration files

Cleaning up log files

Cleaning up certs

Cleaning up content

Removing Databases and Pulp storage on Filesystem

 /var/lib/mongodb is part of a mounted filesystem, skipping. You will need to remove /var/lib/mongodb manually.

 /var/lib/pgsql is part of a mounted filesystem, skipping. You will need to remove /var/lib/pgsql manually.

Finished

>>> katello-remove issues appropriate warnings about mountpoints to be deleted manually
Comment 10 errata-xmlrpc 2019-05-14 12:37:19 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222