Bug 2164413

Summary: backup restore unable to cope with backups created via "-t" option
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: Satellite MaintainAssignee: Pavel Moravec <pmoravec>
Status: CLOSED ERRATA QA Contact: Jameer Pathan <jpathan>
Severity: high Docs Contact:
Priority: high    
Version: 6.12.0CC: ehelms, gsulliva, jeanbaptiste.dancre, momran, pcreech
Target Milestone: 6.13.0Keywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: rubygem-foreman_maintain-1.2.5 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-05-03 13:24:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2023-01-25 13:51:29 UTC
Description of problem:
When a backup is created with option "-t" (to split pulp_data.tar into smaller chunks), satellite-maintain restore is *completely* unable to restore that backup.

The cause is, -t option implies tar uses --tape-length option which implies -M option. To skip prompts, --new-volume-script=/usr/share/gems/gems/foreman_maintain-1.1.8/bin/foreman-maintain-rotate-tar mimics exchanging tapes.

That works great on backup, where pulp_data.part* files are created.

But restore routine does not use that at all, causing the tar --extract gets stuck forever.

It "just" needs to add:

https://github.com/theforeman/foreman_maintain/blob/9d5f5c9315e71d08451d51fb2d41dc0f735bdfc9/definitions/features/tar.rb#L56

to the tar --extract routine (every time? or just when backup contains some pulp_data.part* file?)


I can verify manually that just adding that argument to a "split backup" makes the tar to successfully complete..


Version-Release number of selected component (if applicable):
Sat 6.11 +


How reproducible:
100%


Steps to Reproduce:
1. create a backup split to smaller chunks:

mkdir /backup
satellite-maintain backup offline -t 50M --whitelist="sync-plans-enable" /backup

2. try to restore it:

satellite-maintain restore --whitelist="sync-plans-enable" -y /backup/satellite-backup-2023-01-25-14-19-21

3. Wait until "Extracting pulp data" occurs for a while.


Actual results:
3. When "Extracting pulp data" occurs, see that "ps aux | grep tar" shows some tar command being idle. B'cos it waits on user prompt to swap the tapes :(

And the "Extracting pulp data" never completes.


Expected results:
3. "Extracting pulp data" to complete in reasonable time.


Additional info:

Comment 1 Pavel Moravec 2023-01-25 15:43:35 UTC
Some additional notes:

satellite-maintain restore procedure does not have "-t" option (which adds the --new-volume-script option when creating the archive, and which misses during extract).


Potential workaround: 
1) run the command:

tar --selinux --extract --file=/path/to/satellite-backup-2023-01-23-14-17-07/pulp_data.tar --absolute-names --overwrite --listed-incremental=/dev/null -M --directory=/ --new-volume-script=/usr/share/gems/gems/foreman_maintain-1.1.8/bin/foreman-maintain-rotate-tar

(add the latest argument there)

2) rename the pulp_data.tar
3) create a dummy pulp_data.tar, like:

touch /tmp/empty
tar cf /PATH/TO/BACKUP/pulp_data.tar /tmp/empty

4) Let the restore run again - it will not stuck on the same user prompt, but extract the dummy file instead; pulp data extracted manually in the right way

(procedure is pending a review, use with caution until KCS is created)

Comment 2 Pavel Moravec 2023-01-25 21:44:51 UTC
Perspective patch:

--- /usr/share/gems/gems/foreman_maintain-1.1.8/definitions/features/tar.rb.orig	2023-01-25 20:40:20.781292213 +0100
+++ /usr/share/gems/gems/foreman_maintain-1.1.8/definitions/features/tar.rb.new	2023-01-25 20:39:40.311747413 +0100
@@ -51,13 +51,16 @@ class Features::Tar < ForemanMaintain::F
     end
 
     if volume_size
-      split_tar_script = default_split_tar_script
       tar_command << "--tape-length=#{volume_size}"
-      tar_command << "--new-volume-script=#{split_tar_script}"
     end
 
     tar_command << '--overwrite' if options[:overwrite]
-    tar_command << '--gzip' if options[:gzip]
+
+    if options[:gzip]
+      tar_command << '--gzip'
+    else
+      tar_command << '-M --new-volume-script=#{default_split_tar_script}'
+    end
 
     exclude = options.fetch(:exclude, [])
     exclude.each do |ex|

Comment 3 Pavel Moravec 2023-01-25 22:25:23 UTC
(In reply to Pavel Moravec from comment #2)
> +      tar_command << '-M --new-volume-script=#{default_split_tar_script}'

It must be:

+      tar_command << "-M --new-volume-script=#{default_split_tar_script}"

to eval the variable properly.

Comment 4 Jean-Baptiste Dancre 2023-01-26 08:33:22 UTC
One would need to confirm, but i'm not 100% sure the -M is needed when you pass the --new-volume-script flag.

In theory, extract from manpages of tar:
~~~
-F, --info-script=NAME, --new-volume-script=NAME
run script at end of each tape (implies -M)
~~~

Comment 5 Pavel Moravec 2023-01-26 14:04:31 UTC
I can confirm that the patch from #c2 + #c3 does work well on any backup, either with -t or without the option used. Also, it properly applies the -M --new-volume-script options only to pulp_data.tar - not to gzipped tarbals that cant be split that way.

And indeed, we can even omit the -M option there, since it is implicit from the other one. So the working patch is:

--- /usr/share/gems/gems/foreman_maintain-1.1.8/definitions/features/tar.rb.orig	2023-01-25 20:40:20.781292213 +0100
+++ /usr/share/gems/gems/foreman_maintain-1.1.8/definitions/features/tar.rb.new	2023-01-25 20:39:40.311747413 +0100
@@ -51,13 +51,16 @@ class Features::Tar < ForemanMaintain::F
     end
 
     if volume_size
-      split_tar_script = default_split_tar_script
       tar_command << "--tape-length=#{volume_size}"
-      tar_command << "--new-volume-script=#{split_tar_script}"
     end
 
     tar_command << '--overwrite' if options[:overwrite]
-    tar_command << '--gzip' if options[:gzip]
+
+    if options[:gzip]
+      tar_command << '--gzip'
+    else
+      tar_command << "--new-volume-script=#{default_split_tar_script}"
+    end
 
     exclude = options.fetch(:exclude, [])
     exclude.each do |ex|


It is not very clean patch as we should detect the tape size in a more clever way, but.. at least it works. As I successfully tested all combinations "backup|restore satellite with|without -t option".

Comment 6 Bryan Kearney 2023-01-27 00:03:34 UTC
Upstream bug assigned to pmoravec

Comment 7 Bryan Kearney 2023-01-27 00:03:36 UTC
Upstream bug assigned to pmoravec

Comment 8 Eric Helms 2023-02-09 13:55:05 UTC
*** Bug 2121082 has been marked as a duplicate of this bug. ***

Comment 9 Bryan Kearney 2023-02-16 20:03:28 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/36012 has been resolved.

Comment 10 Jameer Pathan 2023-03-20 16:49:04 UTC
Verified

Verified with:
- Satellite 6.13.0 snap 15
- rubygem-foreman_maintain-1.2.7-1

Test steps:
- mkdir /backup
- satellite-maintain backup offline -t 50M /backup
- satellite-maintain restore -y /backup/satellite-backup-2023-03-20-09-59-46/

Observation:
- No error during restore.

Comment 13 errata-xmlrpc 2023-05-03 13:24:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.13 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2097