Bug 1574257 - katello-remove does not completely remove data on mounted filesystems
Summary: katello-remove does not completely remove data on mounted filesystems
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Infrastructure
Version: 6.3.1
Hardware: All
OS: Linux
unspecified
high vote
Target Milestone: Released
Assignee: Chris Roberts
QA Contact: Lukas Pramuk
URL: https://projects.theforeman.org/issue...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-05-02 22:51 UTC by Sam Yangsao
Modified: 2019-10-07 17:18 UTC (History)
3 users (show)

Fixed In Version: katello-3.9.0-rc2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-14 12:37:19 UTC


Attachments (Terms of Use)
two default/local capsules (54.62 KB, image/png)
2018-05-03 01:03 UTC, Sam Yangsao
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:1222 None None None 2019-05-14 12:37:27 UTC
Github theforeman foreman-packaging pull 3092 None None None 2018-10-12 20:05:08 UTC
Foreman Issue Tracker 25195 None None None 2018-10-12 19:53:13 UTC

Description Sam Yangsao 2018-05-02 22:51:10 UTC
Description of problem:

katello-remove does not completely remove the data on mounted directory structures

Version-Release number of selected component (if applicable):

Satellite 6.3.1
RHEL 7.5

How reproducible:

Always

Steps to Reproduce:

1.  Install a new RHEL host with mounted filesystems following guidelines [1]:

/var/lib/pgsql
/var/lib/mongodb

2.  Install Satellite server bits

# yum install satellite
# satellite-installer

3.  Remove [2] Satellite 6.3 server bits:

# katello-remove

4.  Change hostname to something else

# hostnamectl set-hostname satname2

5.  Install Satellite server bits again

# yum install satellite
# satellite-installer

Actual results:

Since the data does not remove completely, remnants of the old install still stick around, for example, in this scenario, the hostname was changed during the second installation and the capsule id is now "2" instead of "1" for a fresh installation.

/var/lib/pgsql was not cleaned up properly so the new installation took the data from this directory and used it for Satellite 6

Expected results:

katello-remove should remove *all* the data in these directories

Maybe add some logic to check if the filesystem is mounted, and ask the user if it's OK to remove the data in that path.

We see that /var/lib/pgsql is a mounted filesystem, is it OK to remove /var/lib/pgsql/* ?

Additional info:

[1] https://access.redhat.com/documentation/en-us/red_hat_satellite/6.3/html-single/installation_guide/#hardware_storage_prerequisites
[2] https://access.redhat.com/documentation/en-us/red_hat_satellite/6.3/html-single/installation_guide/#uninstalling_satellite_and_capsule_servers

Comment 1 Sam Yangsao 2018-05-02 22:54:04 UTC
After running katello-remove ...

[root@sat6test pgsql]# df -k .
Filesystem              1K-blocks   Used Available Use% Mounted on
/dev/mapper/pgsql-pgsql  10471424 120560  10350864   2% /var/lib/pgsql
[root@sat6test pgsql]# pwd
/var/lib/pgsql
[root@sat6test pgsql]# ls
data
[root@sat6test pgsql]# cd data
[root@sat6test data]# ls
base     pg_hba.conf    pg_multixact  pg_snapshots  pg_tblspc    pg_xlog
global   pg_ident.conf  pg_notify     pg_stat_tmp   pg_twophase  postgresql.conf
pg_clog  pg_log         pg_serial     pg_subtrans   PG_VERSION   postmaster.opts
[root@sat6test data]# cd /var/lib/mongodb/
[root@sat6test mongodb]# ls
journal  local.0  local.ns  mongod.lock  pulp_database.0  pulp_database.ns
[root@sat6test mongodb]# df -k .
Filesystem              1K-blocks    Used Available Use% Mounted on
/dev/mapper/mongo-mongo  52399108 3342552  49056556   7% /var/lib/mongodb
[root@sat6test mongodb]#

Comment 2 Sam Yangsao 2018-05-03 01:00:44 UTC
While attempting to refresh the manifest with the Satellite server in this state,  we encounter this error as both are claiming to be the "default/local" capsule.

"There was an issue with the backend service pulp:  Couldn't find SmartProxy with 'id'=1"

Comment 3 Sam Yangsao 2018-05-03 01:03:01 UTC
Created attachment 1430370 [details]
two default/local capsules

Comment 5 Brad Buckingham 2018-05-15 17:48:39 UTC
Note: when addressing this bugzilla, the solution should not destroy all databases on the server.  Instead, it should only be those that are used by Satellite (e.g. foreman, candlepin...etc).

Comment 7 Lukas Pramuk 2018-12-11 14:17:58 UTC
VERIFIED.

@satellite-6.5.0-5.beta.el7sat.noarch
katello-common-3.10.0-0.6.rc1.el7sat.noarch

by the manual reproducer:

1) Set up a machine to have these mountpoints:

# cat /etc/fstab
...
/dev/mapper/machine01-pgsql /var/lib/pgsql            xfs     defaults        0 0
/dev/mapper/machine01-mongo /var/lib/mongodb          xfs     defaults        0 0

2) Install Satellite

3) Run katello-remove

# katello-remove

WARNING: This script will erase many packages and config files.
Important packages such as the following will be removed:
  * httpd (apache)
  * mongodb
  * tomcat
  * puppet
  * ruby
  * rubygems
  * All Katello and Foreman Packages
Once these packages and configuration files are removed there is no going back.
If you use this system for anything other than Katello and Foreman you probably
do not want to execute this script.

Read the source at /usr/sbin/katello-remove for a list of what is removed.  Are you sure(Y/N) 
y

ARE YOU SURE?: This script permanently deletes data and configurations.

Type [remove] to continue 
remove

Starting removal

Stopping services

Removing RPMs

Cleaning up configuration files

Cleaning up log files

Cleaning up certs

Cleaning up content

Removing Databases and Pulp storage on Filesystem

 /var/lib/mongodb is part of a mounted filesystem, skipping. You will need to remove /var/lib/mongodb manually.

 /var/lib/pgsql is part of a mounted filesystem, skipping. You will need to remove /var/lib/pgsql manually.

Finished

>>> katello-remove issues appropriate warnings about mountpoints to be deleted manually

Comment 10 errata-xmlrpc 2019-05-14 12:37:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222


Note You need to log in before you can comment on or make changes to this bug.