Bug 2219008

Summary: Satellite restore of an online backup is resulting in duplicate entries of the settings table
Product: Red Hat Satellite Reporter: jalviso <jalviso>
Component: Satellite MaintainAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED MIGRATED QA Contact: Satellite QE Team <sat-qe-bz-list>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.13.2CC: ahumbe, ehelms, fhirtz, jalviso, jatan, rlavi
Target Milestone: UnspecifiedKeywords: MigratedToJIRA, Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-06-06 16:23:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description jalviso 2023-06-30 23:13:57 UTC
1. Proposed title of this feature request

Need a reliable Satellite Backup while the satellite services are online
 
3. What is the nature and description of the request?

The customer requires a reliable satellite backup mechanism to capture the full backup (preferable a snapshot on the DB) without satellite downtime.
At the moment, they uses the satellite online backup as they can't afford to stop the satellite services while the backup is running for hours.
They uses the backup to restore a clone production satellite to divert the satellite traffic to the clone while the production satellite is doing some maintenance works.

Currently, duplicate entries on the DB occurs during the DB dumped by the online backup. Specifically, there are duplicate entries on the foreman DB settings table which causes the backup restoration to fail. This issue happens multiple times during the customer's backup restore activities and has to be addressed before retrying the restore which takes hours. 

This is the customer's say:

~~~~~~~
So, I am looking for suggestions on an alternative method that we can get a clean DB backup without shutting down the primary satellite.  We do not want to have to keep manually sanitizing the DB backup so that it will restore to the backup without errors.  Our current automated solution does a full backup of the primary and restores to the backup so that we have a "warm" backup to failover to if needed.

One idea I have for getting a more consistent DB backup while the primary is online is:
 
- Creating an online synced read only copy of the primary's Postgres DB on a different server.  Taking an offline database backup from the different server to restore to the backup satellite.

My question would be is this possible with the RH postgresql version we are using with Satellite?
~~~~~~~~~

4. Why does the customer need this? (List the business requirements here)

The customer requires a backup method that can get a clean DB backup without shutting down the primary satellite.
The customer prefer to use the backup facility provided in Satellite than VMware vm snapshot of the primary Sat as their data is very huge (20+ external Capsules, 50K+ registered Hosts, TB of content synced)

5. How would the customer like to achieve this? (List the functional requirements here)

- Be able to run a full backup without shutting down the production satellite using a feature available/provided by Satellite that would copy the existing DB of the primary without duplicate entries
     on DB tables.
- Run satellite-restore in 1 try as each run takes ~4 hours
- Secondary satellite services running without failure, route the traffic seamlessly to the secondary Satellite.
   
6. For each functional requirement listed, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.

- Customer is willing to provide their DB backup for robust testing. 

7. Is there already an existing RFE upstream or in Red Hat Bugzilla?

No.

8. Does the customer have any specific timeline dependencies and which release would they like to target (i.e. RHEL5, RHEL6)?

If possible, the feature will be available on the next releases of Sat6.

9. Is the sales team involved in this request and do they have any additional input?

TAMs on this account (North America, Asia Pacific) are heavily involved.

10. List any affected packages or components.

rubygem-foreman_maintain

11. Would the customer be able to assist in testing this functionality if implemented?

Yes.

Comment 2 Eric Helms 2023-07-06 20:01:09 UTC
At first glance, this sounds like a bug rather than an RFE. Taking a backup and restoring it should not end up with duplicate entries and we need to better understand why this is happening.


Can you expand on this statement?

"""
Currently, duplicate entries on the DB occurs during the DB dumped by the online backup. Specifically, there are duplicate entries on the foreman DB settings table which causes the backup restoration to fail. This issue happens multiple times during the customer's backup restore activities and has to be addressed before retrying the restore which takes hours. 
"""


Where are you seeing the duplicate entries? Does it throw an error? What is the affect you are seeing?

Comment 4 Eric Helms 2024-06-06 16:23:38 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "SAT-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.