Red Hat Bugzilla – Bug 1497858
katello-restore is not properly restoring the incremental backup
Last modified: 2018-08-22 16:07:08 EDT
Description of problem: When restoring (using katello-restore) a full backup taken via katello-backup, the full backup is correctly restored. And according to the documentation, we should then re-run katello-restore for each incremental directory. Looking at the katello-restore script logic, we can see that reset_katello is run every time. This would drop the database and make them unusable after restoring an incremental. This issue is happening with the offline backup, which is the default behavior of katello-backup. Version-Release number of selected component (if applicable): 6.2 How reproducible: 100% Steps to Reproduce: 1. Take a full backup # katello-backup /backup 2. Once the full backup is finished, run an incremental # katello-backup /backup --incremental /backup/katello-backup-2017-10-02T14\:13\:51-04\:00/ 3. Restore the full backup # katello-restore /backup/katello-backup-2017-10-02T14\:13\:51-04\:00/ 4. Restore the incremental backup # katello-restore /backup/katello-backup-2017-10-02T14\:40\:29-04\:00/ Actual results: Broken postgresql database. FATAL: database "foreman" does not exist DETAIL: The database subdirectory "base/41146" is missing. Run `$ bin/rake db:create db:migrate` to create your database (ActiveRecord::NoDatabaseError) Expected results: Have the full backup and incremental to get restored correctly in a consistent state. Additional info: When decompressing the database archive manually (pgsql_data.tar.gz) one after the other, the postgresql database is restored to a working state.
Some additional information: 1. When the backup where created with the "--online-backup", as it takes database dumps, the restoration of incremental will be, in face, restoring a full database and thus the databases will be in a working state. However, the pulp filesystem will be inconsistent and broken. 2. In addition the the database, the pulp file system /var/lib/pulp is also suffering from being reset. Some directories are delete when the Satellite is reset, so the incremental will only restore some pieces and bits.
Created redmine issue http://projects.theforeman.org/issues/23107 from this bug
Followed the below procedure to test and it works fine now. 1) satellite-backup -y /var/tmp/sat-content >> /var/log/satellite/backups.log 2>&1 2) satellite-backup -y /var/tmp/sat-content --incremental /var/tmp/sat-content/satellite-backup-20180801092000 >> /var/log/satellite/backups.log 2>&1 3) satellite-restore /var/tmp/sat-content/satellite-backup-20180801092000/ 4) satellite-restore -i /var/tmp/sat-content/satellite-backup-20180801093900 Restored successfully and "hammer ping" working. VERIFIED with Sat6.3.3 snap2. Will attach some logs shortly.
Created attachment 1472709 [details] Satellite6 full and incremental restore logs
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2550