Bug 1497858 - katello-restore is not properly restoring the incremental backup
Summary: katello-restore is not properly restoring the incremental backup
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Backup & Restore
Version: 6.2.12
Hardware: All
OS: Linux
Target Milestone: Unspecified
Assignee: Christine Fouant
QA Contact: Kedar Bidarkar
Depends On:
Blocks: 1316897
TreeView+ depends on / blocked
Reported: 2017-10-02 21:51 UTC by Christian Marineau
Modified: 2021-03-11 15:54 UTC (History)
8 users (show)

Fixed In Version: katello-3.4.5-19
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1596354 (view as bug list)
Last Closed: 2018-08-22 20:07:08 UTC
Target Upstream Version:

Attachments (Terms of Use)
Satellite6 full and incremental restore logs (36.99 KB, text/plain)
2018-08-02 13:56 UTC, Kedar Bidarkar
no flags Details

System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 23107 0 High Closed katello-restore is not properly restoring the incremental backup 2021-02-02 02:47:53 UTC
Red Hat Knowledge Base (Solution) 3231031 0 None None None 2017-11-06 15:29:45 UTC

Description Christian Marineau 2017-10-02 21:51:18 UTC
Description of problem:
When restoring (using katello-restore) a full backup taken via katello-backup, the full backup is correctly restored. And according to the documentation, we should then re-run katello-restore for each incremental directory. 

Looking at the katello-restore script logic, we can see that reset_katello is run every time. This would drop the database and make them unusable after restoring an incremental.

This issue is happening with the offline backup, which is the default behavior of katello-backup.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Take a full backup
   # katello-backup /backup
2. Once the full backup is finished, run an incremental
   # katello-backup /backup --incremental /backup/katello-backup-2017-10-02T14\:13\:51-04\:00/

3. Restore the full backup
   # katello-restore /backup/katello-backup-2017-10-02T14\:13\:51-04\:00/

4. Restore the incremental backup
   # katello-restore /backup/katello-backup-2017-10-02T14\:40\:29-04\:00/

Actual results:
Broken postgresql database.

FATAL:  database "foreman" does not exist
DETAIL:  The database subdirectory "base/41146" is missing.
Run `$ bin/rake db:create db:migrate` to create your database (ActiveRecord::NoDatabaseError)

Expected results:
Have the full backup and incremental to get restored correctly in a consistent state.

Additional info:
When decompressing the database archive manually (pgsql_data.tar.gz) one after the other, the postgresql database is restored to a working state.

Comment 3 Christian Marineau 2017-11-03 11:59:15 UTC
Some additional information:
1. When the backup where created with the "--online-backup", as it takes database dumps, the restoration of incremental will be, in face, restoring a full database and thus the databases will be in a working state. However, the pulp filesystem will be inconsistent and broken.

2. In addition the the database, the pulp file system /var/lib/pulp is also suffering from being reset. Some directories are delete when the Satellite is reset, so the incremental will only restore some pieces and bits.

Comment 5 Christine Fouant 2018-04-03 16:15:51 UTC
Created redmine issue http://projects.theforeman.org/issues/23107 from this bug

Comment 17 Kedar Bidarkar 2018-08-02 13:54:08 UTC
Followed the below procedure to test and it works fine now.

1) satellite-backup -y /var/tmp/sat-content >> /var/log/satellite/backups.log 2>&1

2) satellite-backup -y /var/tmp/sat-content --incremental /var/tmp/sat-content/satellite-backup-20180801092000 >> /var/log/satellite/backups.log 2>&1

3) satellite-restore /var/tmp/sat-content/satellite-backup-20180801092000/

4) satellite-restore -i /var/tmp/sat-content/satellite-backup-20180801093900

Restored successfully and "hammer ping" working.

VERIFIED with Sat6.3.3 snap2.

Will attach some logs shortly.

Comment 18 Kedar Bidarkar 2018-08-02 13:56:09 UTC
Created attachment 1472709 [details]
Satellite6 full and incremental restore logs

Comment 20 errata-xmlrpc 2018-08-22 20:07:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.