Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2230934 - restore of offline incremental backups is broken
Summary: restore of offline incremental backups is broken
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Satellite Maintain
Version: 6.13.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: 6.14.0
Assignee: Evgeni Golov
QA Contact: Lukas Pramuk
URL:
Whiteboard:
: 2250206 2269551 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-08-10 09:37 UTC by Evgeni Golov
Modified: 2024-04-05 11:47 UTC (History)
7 users (show)

Fixed In Version: rubygem-foreman_maintain-1.3.4
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-11-08 14:20:16 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 36668 0 Normal Closed restore of offline incremental backups is broken 2023-09-08 10:49:08 UTC
Red Hat Issue Tracker SAT-19478 0 None None None 2023-08-10 09:38:31 UTC
Red Hat Knowledge Base (Solution) 7045174 0 None None None 2023-11-20 13:28:02 UTC
Red Hat Product Errata RHSA-2023:6818 0 None None None 2023-11-08 14:21:23 UTC

Description Evgeni Golov 2023-08-10 09:37:09 UTC 
Originally reported in https://community.theforeman.org/t/how-is-foreman-maintain-restore-i-supposed-to-work/34519

Since "9d5f5c9315e71d08451d51fb2d41dc0f735bdfc9":https://github.com/theforeman/foreman_maintain/commit/9d5f5c9315e71d08451d51fb2d41dc0f735bdfc9 foreman-maintain performs a <code>reindexdb</code> after restoring a backup,
which seems to alter the DB in a way that incremental backups don't apply cleanly anymore
and PostgreSQL isn't able to operate after an incremental restore has been attempted:

<pre>
[postgres@centos8-stream-katello-4-9 ~]$ psql
psql: error: FATAL:  could not open file "base/13449/36756": No such file or directory
[postgres@centos8-stream-katello-4-9 ~]$ psql foreman
psql: error: FATAL:  could not open file "base/18254/33445": No such file or directory
[postgres@centos8-stream-katello-4-9 ~]$ psql candlepin
psql: error: FATAL:  could not open file "base/16385/35536": No such file or directory
</pre>

This is not an issue 
* if you use online backups (those always contain a full DB dump and don't require a REINDEX anyway)
* for non-incremental restores of offline backups

The REINDEX step was added to avoid issues with different versions of libc locales when changing operating system versions.
A possible short term fix would be to only execute the REINDEX when foreman-maintain detects that the backup was taken on a different OS than the restore is happening on.
A better solution would be to re-architecture the restore process to first extract *all* incremental steps and only afterwards perform the necessary DB steps.

The issue is present in the following foreman-maintain versions:
* 1.3.x
* 1.2.4+
* 1.1.10+
* 1.0.19+
Comment 1 Evgeni Golov 2023-08-10 09:37:13 UTC 
Created from redmine issue https://projects.theforeman.org/issues/36668
Comment 2 Evgeni Golov 2023-08-10 09:37:14 UTC 
Upstream bug assigned to egolov
Comment 3 Bryan Kearney 2023-08-11 12:03:19 UTC 
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/36668 has been resolved.
Comment 4 Lukas Pramuk 2023-08-28 16:19:31 UTC 
VERIFIED.

@Satellite 6.14.0 Snap13
rubygem-foreman_maintain-1.3.5-1.el8sat.noarch

by the following manual reproducer:

1) Populate some content (RHEL8 repo sync)

2) Create a full offline backup
# satellite-maintain backup offline -y /var/backup

3) Populate some more content (RHEL9 repo sync)

4) Create an incremental backup based on the full backup 
# satellite-maintain backup offline -y -i /var/backup/satellite-backup-2023-08-28-10-56-26 /var/backup

5) Restore the full backup
# satellite-maintain restore -y /var/backup/satellite-backup-2023-08-28-10-56-26

6) Try to restore the incremental backup on top of the full backup
# satellite-maintain restore -y /var/backup/satellite-backup-2023-08-28-11-09-43

REPRO:

Extract any existing tar files in backup: 
| Extracting pgsql data                                               [OK]      
--------------------------------------------------------------------------------
REINDEX databases: 
- Reindexing the databases                                            [FAIL]    
Failed executing runuser - postgres -c "reindexdb -a", exit status 1:
 reindexdb: error: could not connect to database template1: FATAL:  could not open file "base/1/46542": No such file or directory
--------------------------------------------------------------------------------
Scenario [Restore backup] failed.

vs.

FIX:

Extract any existing tar files in backup: 
- Extracting pgsql data                                               [OK]      
--------------------------------------------------------------------------------
Migrate pulpcore db: 
| Migrating pulpcore database                                         [OK]      
--------------------------------------------------------------------------------
Ensure Candlepin runs all migrations after restoring the database:    [OK]
--------------------------------------------------------------------------------

>>> incremental restore has finished successfully as the db reindex step is being run only when OS version changes
Comment 7 errata-xmlrpc 2023-11-08 14:20:16 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.14 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6818
Comment 8 Ron Lavi 2023-11-20 13:28:03 UTC 
*** Bug 2250206 has been marked as a duplicate of this bug. ***
Comment 9 momran 2024-03-14 16:16:15 UTC 
*** Bug 2269551 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.