Bug 1369864

Summary:	gzip backup files after restarting services to minimize downtime
Product:	Red Hat Satellite	Reporter:	Mike McCune <mmccune>
Component:	Backup & Restore	Assignee:	Christine Fouant <cfouant>
Status:	CLOSED ERRATA	QA Contact:	jcallaha
Severity:	high	Docs Contact:
Priority:	high
Version:	6.2.0	CC:	bbuckingham, bkearney, cwelton, jcallaha
Target Milestone:	Unspecified	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
URL:	http://projects.theforeman.org/issues/16732
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-02-21 16:54:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Mike McCune 2016-08-24 14:47:33 UTC

Currently we gzip the backup files while all the services are stopped during katello-backup which can cause an extended amount of downtime for the Satellite during a backup run.

Lets move the gzipping routine to be after we bring services back online.

Some rough estimates are that we can reduce the downtime from ~45 minutes to ~5 minutes if we make this change.

Comment 2 Christine Fouant 2016-09-28 17:49:24 UTC

Created redmine issue http://projects.theforeman.org/issues/16732 from this bug

Comment 3 Bryan Kearney 2016-10-03 16:19:09 UTC

Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16732 has been resolved.

Comment 6 jcallaha 2017-08-10 19:50:27 UTC

Failed QA in Satellite 6.3 Snap 10.

It looks like there is still a gzip in place before, services are restarted, in the 'Backing up config files' step. See below

-bash-4.2# katello-backup --skip-pulp-content .
Starting backup: 2017-08-10 15:38:11 -0400
Creating backup folder ./katello-backup-20170810153812
Generating metadata ... 
/opt/theforeman/tfm/root/usr/share/gems/gems/redhat_access-2.0.3/app/services/redhat_access/telemetry/look_ups.rb:171: warning: key :hosts is duplicated and overwritten on line 172
Done.
Backing up config files... 
Failed 'tar --selinux --create --gzip --file=./katello-backup-20170810153812/config_files.tar.gz --listed-incremental=./katello-backup-20170810153812/.config.snar /etc/foreman-proxy /etc/httpd /etc/foreman-installer /etc/pki/katello /etc/pki/katello-certs-tools /etc/pki/pulp /etc/pulp /etc/puppet /etc/qpid /etc/qpid-dispatch /root/ssl-build /var/www/html/pub /etc/squid /etc/puppetlabs /opt/puppetlabs/puppet/cache/foreman_cache_data /opt/puppetlabs/puppet/ssl/ /var/lib/puppet/foreman_cache_data /var/lib/puppet/ssl /etc/candlepin /etc/foreman /etc/hammer /etc/sysconfig/tomcat* /etc/tomcat* /var/lib/candlepin 2>/dev/null' with exit code 141
Cleaning up backup folder and starting any stopped services... 
/usr/share/ruby/fileutils.rb:125: warning: conflicting chdir during another chdir block
Redirecting to /bin/systemctl start mongod.service
Redirecting to /bin/systemctl start postgresql.service
Redirecting to /bin/systemctl start tomcat.service
Redirecting to /bin/systemctl start pulp_workers.service
Redirecting to /bin/systemctl start pulp_resource_manager.service
Redirecting to /bin/systemctl start pulp_streamer.service
Redirecting to /bin/systemctl start pulp_celerybeat.service
Redirecting to /bin/systemctl start httpd.service
Redirecting to /bin/systemctl start foreman-tasks.service
Done.
-bash-4.2# 
-bash-4.2# 
-bash-4.2# which katello-backup
/usr/sbin/katello-backup

Comment 7 Christine Fouant 2017-08-14 19:05:24 UTC

(In reply to jcallaha from comment #6)
> Failed QA in Satellite 6.3 Snap 10.
> 
> It looks like there is still a gzip in place before, services are restarted,
> in the 'Backing up config files' step. See below
> 
> -bash-4.2# katello-backup --skip-pulp-content .
> Starting backup: 2017-08-10 15:38:11 -0400
> Creating backup folder ./katello-backup-20170810153812
> Generating metadata ... 
> /opt/theforeman/tfm/root/usr/share/gems/gems/redhat_access-2.0.3/app/
> services/redhat_access/telemetry/look_ups.rb:171: warning: key :hosts is
> duplicated and overwritten on line 172
> Done.
> Backing up config files... 
> Failed 'tar --selinux --create --gzip
> --file=./katello-backup-20170810153812/config_files.tar.gz
> --listed-incremental=./katello-backup-20170810153812/.config.snar
> /etc/foreman-proxy /etc/httpd /etc/foreman-installer /etc/pki/katello
> /etc/pki/katello-certs-tools /etc/pki/pulp /etc/pulp /etc/puppet /etc/qpid
> /etc/qpid-dispatch /root/ssl-build /var/www/html/pub /etc/squid
> /etc/puppetlabs /opt/puppetlabs/puppet/cache/foreman_cache_data
> /opt/puppetlabs/puppet/ssl/ /var/lib/puppet/foreman_cache_data
> /var/lib/puppet/ssl /etc/candlepin /etc/foreman /etc/hammer
> /etc/sysconfig/tomcat* /etc/tomcat* /var/lib/candlepin 2>/dev/null' with
> exit code 141
> Cleaning up backup folder and starting any stopped services... 
> /usr/share/ruby/fileutils.rb:125: warning: conflicting chdir during another
> chdir block
> Redirecting to /bin/systemctl start mongod.service
> Redirecting to /bin/systemctl start postgresql.service
> Redirecting to /bin/systemctl start tomcat.service
> Redirecting to /bin/systemctl start pulp_workers.service
> Redirecting to /bin/systemctl start pulp_resource_manager.service
> Redirecting to /bin/systemctl start pulp_streamer.service
> Redirecting to /bin/systemctl start pulp_celerybeat.service
> Redirecting to /bin/systemctl start httpd.service
> Redirecting to /bin/systemctl start foreman-tasks.service
> Done.
> -bash-4.2# 
> -bash-4.2# 
> -bash-4.2# which katello-backup
> /usr/sbin/katello-backup

The gzip'ing of config_files.tar.gz happens before services are stopped, so will not affect amount of time that services are stopped.

Comment 8 jcallaha 2017-08-14 19:12:04 UTC

Moving back to on_qa based on #7.

Comment 10 jcallaha 2018-01-30 16:27:38 UTC

Verified in Satellite 6.3 Snap 33. I added some text to the script to clarify exactly when services are stopped and started back up. The only parts that were backup up while services were stopped are related to pulp.


-bash-4.2# satellite-backup /tmp
Starting backup: 2018-01-30 17:21:08 +0100
Creating backup folder /tmp/satellite-backup-20180130172108
Generating metadata ... 
Done.
Backing up config files... 
Done.
WARNING: This script will stop your services. Do you want to proceed(y/n)? y
Stopping services...
Redirecting to /bin/systemctl stop foreman-tasks.service
Redirecting to /bin/systemctl stop httpd.service
Redirecting to /bin/systemctl stop pulp_celerybeat.service
Redirecting to /bin/systemctl stop pulp_streamer.service
Redirecting to /bin/systemctl stop pulp_resource_manager.service
Redirecting to /bin/systemctl stop pulp_workers.service
Redirecting to /bin/systemctl stop tomcat.service
Redirecting to /bin/systemctl stop postgresql.service
Redirecting to /bin/systemctl stop mongod.service
Backing up Pulp data... 
Done.
Backing up postgres db...
Done.
Backing up mongo db... 
Done.
Starting services...
Redirecting to /bin/systemctl start mongod.service
Redirecting to /bin/systemctl start postgresql.service
Redirecting to /bin/systemctl start tomcat.service
Redirecting to /bin/systemctl start pulp_workers.service
Redirecting to /bin/systemctl start pulp_resource_manager.service
Redirecting to /bin/systemctl start pulp_streamer.service
Redirecting to /bin/systemctl start pulp_celerybeat.service
Redirecting to /bin/systemctl start httpd.service
Redirecting to /bin/systemctl start foreman-tasks.service
Done with backup: 2018-01-30 17:25:19 +0100
**** BACKUP Complete, contents can be found in: /tmp/satellite-backup-20180130172108 ****

Comment 11 Satellite Program 2018-02-21 16:54:17 UTC

Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336