Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1465182 - Add warning when using --online-backup
Summary: Add warning when using --online-backup
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Backup & Restore
Version: 6.2.10
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Christine Fouant
QA Contact: Ales Dujicek
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-06-26 23:09 UTC by Mike McCune
Modified: 2021-06-10 12:30 UTC (History)
8 users (show)

Fixed In Version: tfm-rubygem-katello-3.4.4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-21 17:32:20 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 20268 0 None None None 2017-07-10 19:47:24 UTC

Description Mike McCune 2017-06-26 23:09:54 UTC 
We offer the --online-backup flag in 6.2 but there needs to be a warning statement when running with this flag as there is a caveat with its use.

Satellite 6 uses two database systems, Postgres and Mongo. There are records that exist in Postgres and Mongo that need to remain consistent and in sync between each system. 

When you shut down all the services you can ensure that there are no modifications occurring to either database. The --online-backup flag keeps all services running so there is a possibility that exists that data being modified while the backups are occurring. There is a basic check in place in the backup routine that checks to see if the database was modified during the backup. If this occurs, it will re-backup the data and start over. 

This check is rudimentary and can't ensure with 100% certainty that there were no modifications to the Pulp or Postgres database while the backup was running. This check can also result in repeated loops if there is continuous modification occurring to the database. If the user still wishes to use the --online-backup flag for production use they will need to ensure that there are no modifications that occur during the backup runs. 

We need to add a warning that explains this, for example (the wording is open for discussion):

# katello-backup /var/tmp --online-backup

*** WARNING: The online backup flag is intended for making a copy of the data
*** for debugging purposes only. The backup routine can not ensure 100% consistency while the 
*** backup is taking place as there is a chance there may be data mismatch between 
*** Mongo and Postgres databases while the services are live. If you wish to utilize the --online-backup
*** flag for production use you need to ensure that there are no modifications occurring during 
*** your backup run. 

Satellite 6.3 will offer LVM based snapshots for 'hot backup' function and we can then remove this warning.
Comment 1 Mike McCune 2017-06-26 23:11:51 UTC 
We also need to document possible mitigation steps that users of Satellite 6.2 can do to minimize the chances of data mismatch between Postgres and Mongo.

This would involve things like:

* Bulk disable all sync plans while --online-backup is running
* Disable pulp workers while --online-backup is running
* Possible firewall rules to block API access to pulp to ensure nothing is modifying 
* Disable any cron tasks that operate on pulp
Comment 2 Chris Duryee 2017-06-27 00:58:46 UTC 
implementing comment #1 is very similar to https://bugzilla.redhat.com/show_bug.cgi?id=1420648, it may be possible to roll them both into one feature
Comment 5 Christine Fouant 2017-07-10 19:47:22 UTC 
Created redmine issue http://projects.theforeman.org/issues/20268 from this bug
Comment 6 Satellite Program 2017-07-10 20:06:30 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/20268 has been resolved.
Comment 8 Satellite Program 2017-08-03 22:08:21 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/20268 has been resolved.
Comment 14 Bryan Kearney 2017-10-18 19:48:15 UTC 
clearing the needinfo.
Comment 15 Bryan Kearney 2018-02-21 17:32:20 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336
Comment 16 Bryan Kearney 2018-02-21 17:32:56 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336
Note You need to log in before you can comment on or make changes to this bug.