Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1488525 - katello-backup does incremental backup of pulp data when checksum check is failing
Summary: katello-backup does incremental backup of pulp data when checksum check is fa...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Backup & Restore
Version: 6.2.11
Hardware: x86_64
OS: Linux
high
high
Target Milestone: 6.4.0
Assignee: Christine Fouant
QA Contact: Jan Hutař
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-09-05 15:31 UTC by Simon Reber
Modified: 2019-11-05 23:08 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-16 19:14:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 23029 0 High Closed katello-backup does incremental backup of pulp data when checksum check is failing 2020-07-02 11:58:58 UTC

Description Simon Reber 2017-09-05 15:31:36 UTC
Description of problem:

Referring to upstream code, but the problem is the same in Red Hat Satellite 6.2.

If we are doing an `--online-backup` with `katello-backup` we are running this part of the code:

    FileUtils.cd '/var/lib/pulp' do
      puts "Backing up Pulp data... "
      matching = false
      until matching
        checksum1 = run_cmd("find . -printf '%T@\n' | md5sum")
        create_pulp_data_tar
        checksum2 = run_cmd("find . -printf '%T@\n' | md5sum")
        matching = (checksum1 == checksum2)
      end

See https://github.com/Katello/katello-packaging/blob/master/katello/katello-backup#L269

The important part is the following:

        checksum1 = run_cmd("find . -printf '%T@\n' | md5sum")
        create_pulp_data_tar
        checksum2 = run_cmd("find . -printf '%T@\n' | md5sum")

We are making sure that nothing has changed during the time the `tar` command is run to make sure we have a consistent backup. This is OK. But in case we do hit an issue, we re-run the activity which is failing as the following option is set in the `tar` command:

run_cmd("tar --selinux --create --file=#{File.join(@dir, 'pulp_data.tar')} --exclude=var/lib/pulp/katello-export --listed-incremental=#{File.join(@dir, '.pulp.snar')} --transform 's,^,var/lib/pulp/,S' -S *")

See in https://github.com/Katello/katello-packaging/blob/master/katello/katello-backup#L349

According `tar` man page does `--listed-incremental` create the file if it does not exist or else use it when exists to allow incremental option.

"Handle new GNU-format incremental backups.  FILE is the name of a snapshot file, where tar stores additional information which is used to decide which files changed since the previous incremental dump and, consequently, must be dumped again.  If FILE does not exist when creating an archive, it will be created and all files will be added to the resulting archive"

Since we don't use an option archive name, that is changing between the re-run we end up with only the changes in `pulp` backup that happen between the first and second run.

This should not happen and we should make sure a full backup is taken if we re-run the `tar` command. Another option would be to provide dynamic file names to avoid that the first `pulp` tar is overwritten by the second run and thus is missing all the data.

Version-Release number of selected component (if applicable):

 - satellite-6.2.11-2.0.el7sat.noarch


How reproducible:

 - Always 

Steps to Reproduce:
1. Check out https://bugzilla.redhat.com/show_bug.cgi?id=1478047 where we found a possible cause that would trigger a re-run of the `tar` command to backup `/var/lib/pulp`

Actual results:

Empty `pulp` backup created due to slight modification happening during the first run of the `tar` command that does backup `/var/lib/pulp`. Once the job is rerun, we have a incremental file available and thus only changes are considered which might be very small and thus leaving incomplete backup.

Expected results:

Proper backup to be available after `katello-backup` run. No matter if it ran once or twice or even more.

Additional info:

See https://bugzilla.redhat.com/show_bug.cgi?id=1478047 which is related and which is the reason why we found the problem.

Comment 2 Christine Fouant 2018-03-27 14:37:34 UTC
Created redmine issue http://projects.theforeman.org/issues/23029 from this bug

Comment 4 Bryan Kearney 2018-10-16 19:14:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927


Note You need to log in before you can comment on or make changes to this bug.