Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1369864 - gzip backup files *after* restarting services to minimize downtime
Summary: gzip backup files *after* restarting services to minimize downtime
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Backup & Restore
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: Christine Fouant
QA Contact: jcallaha
URL: http://projects.theforeman.org/issues...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-24 14:47 UTC by Mike McCune
Modified: 2019-09-25 21:20 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-21 16:54:17 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 16732 0 None None None 2016-09-28 17:49:25 UTC

Description Mike McCune 2016-08-24 14:47:33 UTC 
Currently we gzip the backup files while all the services are stopped during katello-backup which can cause an extended amount of downtime for the Satellite during a backup run.

Lets move the gzipping routine to be after we bring services back online.

Some rough estimates are that we can reduce the downtime from ~45 minutes to ~5 minutes if we make this change.
Comment 2 Christine Fouant 2016-09-28 17:49:24 UTC 
Created redmine issue http://projects.theforeman.org/issues/16732 from this bug
Comment 3 Bryan Kearney 2016-10-03 16:19:09 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16732 has been resolved.
Comment 6 jcallaha 2017-08-10 19:50:27 UTC 
Failed QA in Satellite 6.3 Snap 10.

It looks like there is still a gzip in place before, services are restarted, in the 'Backing up config files' step. See below

-bash-4.2# katello-backup --skip-pulp-content .
Starting backup: 2017-08-10 15:38:11 -0400
Creating backup folder ./katello-backup-20170810153812
Generating metadata ... 
/opt/theforeman/tfm/root/usr/share/gems/gems/redhat_access-2.0.3/app/services/redhat_access/telemetry/look_ups.rb:171: warning: key :hosts is duplicated and overwritten on line 172
Done.
Backing up config files... 
Failed 'tar --selinux --create --gzip --file=./katello-backup-20170810153812/config_files.tar.gz --listed-incremental=./katello-backup-20170810153812/.config.snar /etc/foreman-proxy /etc/httpd /etc/foreman-installer /etc/pki/katello /etc/pki/katello-certs-tools /etc/pki/pulp /etc/pulp /etc/puppet /etc/qpid /etc/qpid-dispatch /root/ssl-build /var/www/html/pub /etc/squid /etc/puppetlabs /opt/puppetlabs/puppet/cache/foreman_cache_data /opt/puppetlabs/puppet/ssl/ /var/lib/puppet/foreman_cache_data /var/lib/puppet/ssl /etc/candlepin /etc/foreman /etc/hammer /etc/sysconfig/tomcat* /etc/tomcat* /var/lib/candlepin 2>/dev/null' with exit code 141
Cleaning up backup folder and starting any stopped services... 
/usr/share/ruby/fileutils.rb:125: warning: conflicting chdir during another chdir block
Redirecting to /bin/systemctl start mongod.service
Redirecting to /bin/systemctl start postgresql.service
Redirecting to /bin/systemctl start tomcat.service
Redirecting to /bin/systemctl start pulp_workers.service
Redirecting to /bin/systemctl start pulp_resource_manager.service
Redirecting to /bin/systemctl start pulp_streamer.service
Redirecting to /bin/systemctl start pulp_celerybeat.service
Redirecting to /bin/systemctl start httpd.service
Redirecting to /bin/systemctl start foreman-tasks.service
Done.
-bash-4.2# 
-bash-4.2# 
-bash-4.2# which katello-backup
/usr/sbin/katello-backup
Comment 7 Christine Fouant 2017-08-14 19:05:24 UTC 
(In reply to jcallaha from comment #6)
> Failed QA in Satellite 6.3 Snap 10.
> 
> It looks like there is still a gzip in place before, services are restarted,
> in the 'Backing up config files' step. See below
> 
> -bash-4.2# katello-backup --skip-pulp-content .
> Starting backup: 2017-08-10 15:38:11 -0400
> Creating backup folder ./katello-backup-20170810153812
> Generating metadata ... 
> /opt/theforeman/tfm/root/usr/share/gems/gems/redhat_access-2.0.3/app/
> services/redhat_access/telemetry/look_ups.rb:171: warning: key :hosts is
> duplicated and overwritten on line 172
> Done.
> Backing up config files... 
> Failed 'tar --selinux --create --gzip
> --file=./katello-backup-20170810153812/config_files.tar.gz
> --listed-incremental=./katello-backup-20170810153812/.config.snar
> /etc/foreman-proxy /etc/httpd /etc/foreman-installer /etc/pki/katello
> /etc/pki/katello-certs-tools /etc/pki/pulp /etc/pulp /etc/puppet /etc/qpid
> /etc/qpid-dispatch /root/ssl-build /var/www/html/pub /etc/squid
> /etc/puppetlabs /opt/puppetlabs/puppet/cache/foreman_cache_data
> /opt/puppetlabs/puppet/ssl/ /var/lib/puppet/foreman_cache_data
> /var/lib/puppet/ssl /etc/candlepin /etc/foreman /etc/hammer
> /etc/sysconfig/tomcat* /etc/tomcat* /var/lib/candlepin 2>/dev/null' with
> exit code 141
> Cleaning up backup folder and starting any stopped services... 
> /usr/share/ruby/fileutils.rb:125: warning: conflicting chdir during another
> chdir block
> Redirecting to /bin/systemctl start mongod.service
> Redirecting to /bin/systemctl start postgresql.service
> Redirecting to /bin/systemctl start tomcat.service
> Redirecting to /bin/systemctl start pulp_workers.service
> Redirecting to /bin/systemctl start pulp_resource_manager.service
> Redirecting to /bin/systemctl start pulp_streamer.service
> Redirecting to /bin/systemctl start pulp_celerybeat.service
> Redirecting to /bin/systemctl start httpd.service
> Redirecting to /bin/systemctl start foreman-tasks.service
> Done.
> -bash-4.2# 
> -bash-4.2# 
> -bash-4.2# which katello-backup
> /usr/sbin/katello-backup

The gzip'ing of config_files.tar.gz happens before services are stopped, so will not affect amount of time that services are stopped.
Comment 8 jcallaha 2017-08-14 19:12:04 UTC 
Moving back to on_qa based on #7.
Comment 10 jcallaha 2018-01-30 16:27:38 UTC 
Verified in Satellite 6.3 Snap 33. I added some text to the script to clarify exactly when services are stopped and started back up. The only parts that were backup up while services were stopped are related to pulp.


-bash-4.2# satellite-backup /tmp
Starting backup: 2018-01-30 17:21:08 +0100
Creating backup folder /tmp/satellite-backup-20180130172108
Generating metadata ... 
Done.
Backing up config files... 
Done.
WARNING: This script will stop your services. Do you want to proceed(y/n)? y
Stopping services...
Redirecting to /bin/systemctl stop foreman-tasks.service
Redirecting to /bin/systemctl stop httpd.service
Redirecting to /bin/systemctl stop pulp_celerybeat.service
Redirecting to /bin/systemctl stop pulp_streamer.service
Redirecting to /bin/systemctl stop pulp_resource_manager.service
Redirecting to /bin/systemctl stop pulp_workers.service
Redirecting to /bin/systemctl stop tomcat.service
Redirecting to /bin/systemctl stop postgresql.service
Redirecting to /bin/systemctl stop mongod.service
Backing up Pulp data... 
Done.
Backing up postgres db...
Done.
Backing up mongo db... 
Done.
Starting services...
Redirecting to /bin/systemctl start mongod.service
Redirecting to /bin/systemctl start postgresql.service
Redirecting to /bin/systemctl start tomcat.service
Redirecting to /bin/systemctl start pulp_workers.service
Redirecting to /bin/systemctl start pulp_resource_manager.service
Redirecting to /bin/systemctl start pulp_streamer.service
Redirecting to /bin/systemctl start pulp_celerybeat.service
Redirecting to /bin/systemctl start httpd.service
Redirecting to /bin/systemctl start foreman-tasks.service
Done with backup: 2018-01-30 17:25:19 +0100
**** BACKUP Complete, contents can be found in: /tmp/satellite-backup-20180130172108 ****
Comment 11 Satellite Program 2018-02-21 16:54:17 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336
Note You need to log in before you can comment on or make changes to this bug.