Red Hat Bugzilla – Bug 1465182
Add warning when using --online-backup
Last modified: 2018-02-21 12:32:56 EST
We offer the --online-backup flag in 6.2 but there needs to be a warning statement when running with this flag as there is a caveat with its use. Satellite 6 uses two database systems, Postgres and Mongo. There are records that exist in Postgres and Mongo that need to remain consistent and in sync between each system. When you shut down all the services you can ensure that there are no modifications occurring to either database. The --online-backup flag keeps all services running so there is a possibility that exists that data being modified while the backups are occurring. There is a basic check in place in the backup routine that checks to see if the database was modified during the backup. If this occurs, it will re-backup the data and start over. This check is rudimentary and can't ensure with 100% certainty that there were no modifications to the Pulp or Postgres database while the backup was running. This check can also result in repeated loops if there is continuous modification occurring to the database. If the user still wishes to use the --online-backup flag for production use they will need to ensure that there are no modifications that occur during the backup runs. We need to add a warning that explains this, for example (the wording is open for discussion): # katello-backup /var/tmp --online-backup *** WARNING: The online backup flag is intended for making a copy of the data *** for debugging purposes only. The backup routine can not ensure 100% consistency while the *** backup is taking place as there is a chance there may be data mismatch between *** Mongo and Postgres databases while the services are live. If you wish to utilize the --online-backup *** flag for production use you need to ensure that there are no modifications occurring during *** your backup run. Satellite 6.3 will offer LVM based snapshots for 'hot backup' function and we can then remove this warning.
We also need to document possible mitigation steps that users of Satellite 6.2 can do to minimize the chances of data mismatch between Postgres and Mongo. This would involve things like: * Bulk disable all sync plans while --online-backup is running * Disable pulp workers while --online-backup is running * Possible firewall rules to block API access to pulp to ensure nothing is modifying * Disable any cron tasks that operate on pulp
implementing comment #1 is very similar to https://bugzilla.redhat.com/show_bug.cgi?id=1420648, it may be possible to roll them both into one feature
Created redmine issue http://projects.theforeman.org/issues/20268 from this bug
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/20268 has been resolved.
clearing the needinfo.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:0336