Description of problem:
When restoring (using katello-restore) a full backup taken via katello-backup, the full backup is correctly restored. And according to the documentation, we should then re-run katello-restore for each incremental directory.
Looking at the katello-restore script logic, we can see that reset_katello is run every time. This would drop the database and make them unusable after restoring an incremental.
This issue is happening with the offline backup, which is the default behavior of katello-backup.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Take a full backup
# katello-backup /backup
2. Once the full backup is finished, run an incremental
# katello-backup /backup --incremental /backup/katello-backup-2017-10-02T14\:13\:51-04\:00/
3. Restore the full backup
# katello-restore /backup/katello-backup-2017-10-02T14\:13\:51-04\:00/
4. Restore the incremental backup
# katello-restore /backup/katello-backup-2017-10-02T14\:40\:29-04\:00/
Broken postgresql database.
FATAL: database "foreman" does not exist
DETAIL: The database subdirectory "base/41146" is missing.
Run `$ bin/rake db:create db:migrate` to create your database (ActiveRecord::NoDatabaseError)
Have the full backup and incremental to get restored correctly in a consistent state.
When decompressing the database archive manually (pgsql_data.tar.gz) one after the other, the postgresql database is restored to a working state.
Some additional information:
1. When the backup where created with the "--online-backup", as it takes database dumps, the restoration of incremental will be, in face, restoring a full database and thus the databases will be in a working state. However, the pulp filesystem will be inconsistent and broken.
2. In addition the the database, the pulp file system /var/lib/pulp is also suffering from being reset. Some directories are delete when the Satellite is reset, so the incremental will only restore some pieces and bits.
Created redmine issue http://projects.theforeman.org/issues/23107 from this bug
Followed the below procedure to test and it works fine now.
1) satellite-backup -y /var/tmp/sat-content >> /var/log/satellite/backups.log 2>&1
2) satellite-backup -y /var/tmp/sat-content --incremental /var/tmp/sat-content/satellite-backup-20180801092000 >> /var/log/satellite/backups.log 2>&1
3) satellite-restore /var/tmp/sat-content/satellite-backup-20180801092000/
4) satellite-restore -i /var/tmp/sat-content/satellite-backup-20180801093900
Restored successfully and "hammer ping" working.
VERIFIED with Sat6.3.3 snap2.
Will attach some logs shortly.
Created attachment 1472709 [details]
Satellite6 full and incremental restore logs
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.