Bug 1497858

Summary: katello-restore is not properly restoring the incremental backup
Product: Red Hat Satellite Reporter: Christian Marineau <cmarinea>
Component: Backup & RestoreAssignee: Christine Fouant <cfouant>
Status: CLOSED ERRATA QA Contact: Kedar Bidarkar <kbidarka>
Severity: high Docs Contact:
Priority: high    
Version: 6.2.12CC: adujicek, ajambhul, bbuckingham, jomitsch, kbidarka, pcreech, sauchter, smane
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: katello-3.4.5-19 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1596354 (view as bug list) Environment:
Last Closed: 2018-08-22 20:07:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1316897    
Attachments:
Description Flags
Satellite6 full and incremental restore logs none

Description Christian Marineau 2017-10-02 21:51:18 UTC 
Description of problem:
When restoring (using katello-restore) a full backup taken via katello-backup, the full backup is correctly restored. And according to the documentation, we should then re-run katello-restore for each incremental directory. 

Looking at the katello-restore script logic, we can see that reset_katello is run every time. This would drop the database and make them unusable after restoring an incremental.

This issue is happening with the offline backup, which is the default behavior of katello-backup.

Version-Release number of selected component (if applicable):
6.2

How reproducible:
100%

Steps to Reproduce:
1. Take a full backup
   # katello-backup /backup
  
2. Once the full backup is finished, run an incremental
   # katello-backup /backup --incremental /backup/katello-backup-2017-10-02T14\:13\:51-04\:00/

3. Restore the full backup
   # katello-restore /backup/katello-backup-2017-10-02T14\:13\:51-04\:00/

4. Restore the incremental backup
   # katello-restore /backup/katello-backup-2017-10-02T14\:40\:29-04\:00/

Actual results:
Broken postgresql database.

FATAL:  database "foreman" does not exist
DETAIL:  The database subdirectory "base/41146" is missing.
Run `$ bin/rake db:create db:migrate` to create your database (ActiveRecord::NoDatabaseError)

Expected results:
Have the full backup and incremental to get restored correctly in a consistent state.

Additional info:
When decompressing the database archive manually (pgsql_data.tar.gz) one after the other, the postgresql database is restored to a working state.
Comment 3 Christian Marineau 2017-11-03 11:59:15 UTC 
Some additional information:
1. When the backup where created with the "--online-backup", as it takes database dumps, the restoration of incremental will be, in face, restoring a full database and thus the databases will be in a working state. However, the pulp filesystem will be inconsistent and broken.

2. In addition the the database, the pulp file system /var/lib/pulp is also suffering from being reset. Some directories are delete when the Satellite is reset, so the incremental will only restore some pieces and bits.
Comment 5 Christine Fouant 2018-04-03 16:15:51 UTC 
Created redmine issue http://projects.theforeman.org/issues/23107 from this bug
Comment 17 Kedar Bidarkar 2018-08-02 13:54:08 UTC 
Followed the below procedure to test and it works fine now.

1) satellite-backup -y /var/tmp/sat-content >> /var/log/satellite/backups.log 2>&1

2) satellite-backup -y /var/tmp/sat-content --incremental /var/tmp/sat-content/satellite-backup-20180801092000 >> /var/log/satellite/backups.log 2>&1

3) satellite-restore /var/tmp/sat-content/satellite-backup-20180801092000/

4) satellite-restore -i /var/tmp/sat-content/satellite-backup-20180801093900

Restored successfully and "hammer ping" working.

VERIFIED with Sat6.3.3 snap2.

Will attach some logs shortly.
Comment 18 Kedar Bidarkar 2018-08-02 13:56:09 UTC 
Created attachment 1472709 [details]
Satellite6 full and incremental restore logs
Comment 20 errata-xmlrpc 2018-08-22 20:07:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2550