Bug 2230934

Summary: restore of offline incremental backups is broken
Product: Red Hat Satellite Reporter: Evgeni Golov <egolov>
Component: Foreman MaintainAssignee: Evgeni Golov <egolov>
Status: ON_QA --- QA Contact: Satellite QE Team <sat-qe-bz-list>
Severity: high Docs Contact:
Priority: high    
Version: 6.13.3CC: ahumbe, ehelms, pcreech
Target Milestone: 6.14.0Keywords: PrioBumpGSS, Triaged, UserExperience
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-foreman_maintain-1.3.4 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Evgeni Golov 2023-08-10 09:37:09 UTC
Originally reported in https://community.theforeman.org/t/how-is-foreman-maintain-restore-i-supposed-to-work/34519

Since "9d5f5c9315e71d08451d51fb2d41dc0f735bdfc9":https://github.com/theforeman/foreman_maintain/commit/9d5f5c9315e71d08451d51fb2d41dc0f735bdfc9 foreman-maintain performs a <code>reindexdb</code> after restoring a backup,
which seems to alter the DB in a way that incremental backups don't apply cleanly anymore
and PostgreSQL isn't able to operate after an incremental restore has been attempted:

<pre>
[postgres@centos8-stream-katello-4-9 ~]$ psql
psql: error: FATAL:  could not open file "base/13449/36756": No such file or directory
[postgres@centos8-stream-katello-4-9 ~]$ psql foreman
psql: error: FATAL:  could not open file "base/18254/33445": No such file or directory
[postgres@centos8-stream-katello-4-9 ~]$ psql candlepin
psql: error: FATAL:  could not open file "base/16385/35536": No such file or directory
</pre>

This is not an issue 
* if you use online backups (those always contain a full DB dump and don't require a REINDEX anyway)
* for non-incremental restores of offline backups

The REINDEX step was added to avoid issues with different versions of libc locales when changing operating system versions.
A possible short term fix would be to only execute the REINDEX when foreman-maintain detects that the backup was taken on a different OS than the restore is happening on.
A better solution would be to re-architecture the restore process to first extract *all* incremental steps and only afterwards perform the necessary DB steps.

The issue is present in the following foreman-maintain versions:
* 1.3.x
* 1.2.4+
* 1.1.10+
* 1.0.19+

Comment 1 Evgeni Golov 2023-08-10 09:37:13 UTC
Created from redmine issue https://projects.theforeman.org/issues/36668

Comment 2 Evgeni Golov 2023-08-10 09:37:14 UTC
Upstream bug assigned to egolov

Comment 3 Bryan Kearney 2023-08-11 12:03:19 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/36668 has been resolved.