Bug 1431971

Summary: [regression] Unable to locate pgsql_data.tar & mongo_data.tar while compressing the katello-backup content
Product: Red Hat Satellite Reporter: Rajan Gupta <rajgupta>
Component: Backup & RestoreAssignee: Christine Fouant <cfouant>
Status: CLOSED ERRATA QA Contact: jcallaha
Severity: high Docs Contact:
Priority: high    
Version: 6.2.8CC: andrew, asahni, bbuckingham, bkearney, egolov, jcallaha, ktordeur, mjahangi, mmccune, nitthoma, pdwyer, pierre-yves.goubet, rajgupta, sellis, xdmoon, zhunting
Target Milestone: UnspecifiedKeywords: PrioBumpGSS, Regression, Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: katello-3.0.0-20 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1446720 (view as bug list) Environment:
Last Closed: 2017-05-16 14:38:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
1431971.patch none

Description Rajan Gupta 2017-03-14 08:07:56 UTC
Description of problem:
Unable to locate pgsql_data.tar & mongo_data.tar while compressing the katello-backup content


Version-Release number of selected component (if applicable):
6.2.8

How reproducible:
Execute "katello-backup" on satellite 6.2.8.

Steps to Reproduce:
1. Install/upgrade to satellite 6.2.8.
2. run "katello-backup <Path>


Actual results:
[root@sat6 /]# katello-backup --skip-pulp-content /katello-backup/
Starting backup: 2017-03-14 13:01:57 +0530
Creating backup folder /katello-backup/katello-backup-2017-03-14T13:01:57+05:30
Redirecting to /bin/systemctl stop  foreman-tasks.service
Redirecting to /bin/systemctl stop  httpd.service
Redirecting to /bin/systemctl stop  pulp_workers.service
Redirecting to /bin/systemctl stop  foreman-proxy.service
Redirecting to /bin/systemctl stop  pulp_streamer.service
Redirecting to /bin/systemctl stop  pulp_resource_manager.service
Redirecting to /bin/systemctl stop  pulp_celerybeat.service
Redirecting to /bin/systemctl stop  smart_proxy_dynflow_core.service
Redirecting to /bin/systemctl stop  tomcat.service
Redirecting to /bin/systemctl stop  squid.service
Redirecting to /bin/systemctl stop  qdrouterd.service
Redirecting to /bin/systemctl stop  qpidd.service
Redirecting to /bin/systemctl stop  postgresql.service
Redirecting to /bin/systemctl stop  mongod.service
Backing up config files... 
tar: Removing leading `/' from member names
Done.
Redirecting to /bin/systemctl stop  foreman-tasks.service
Redirecting to /bin/systemctl stop  httpd.service
Redirecting to /bin/systemctl stop  pulp_workers.service
Redirecting to /bin/systemctl stop  foreman-proxy.service
Redirecting to /bin/systemctl stop  pulp_streamer.service
Redirecting to /bin/systemctl stop  pulp_resource_manager.service
Redirecting to /bin/systemctl stop  pulp_celerybeat.service
Redirecting to /bin/systemctl stop  smart_proxy_dynflow_core.service
Redirecting to /bin/systemctl stop  tomcat.service
Redirecting to /bin/systemctl stop  squid.service
Redirecting to /bin/systemctl stop  qdrouterd.service
Redirecting to /bin/systemctl stop  qpidd.service
Redirecting to /bin/systemctl stop  postgresql.service
Redirecting to /bin/systemctl stop  mongod.service
Backing up postgres db... 
tar: Removing leading `/' from member names
Done.
Backing up mongo db... 
tar: Removing leading `/' from member names
Done.
Redirecting to /bin/systemctl start  mongod.service
Redirecting to /bin/systemctl start  postgresql.service
Redirecting to /bin/systemctl start  qpidd.service
Redirecting to /bin/systemctl start  qdrouterd.service
Redirecting to /bin/systemctl start  squid.service
Redirecting to /bin/systemctl start  tomcat.service
Redirecting to /bin/systemctl start  smart_proxy_dynflow_core.service
Redirecting to /bin/systemctl start  pulp_celerybeat.service
Redirecting to /bin/systemctl start  pulp_resource_manager.service
Redirecting to /bin/systemctl start  pulp_streamer.service
Redirecting to /bin/systemctl start  foreman-proxy.service
Redirecting to /bin/systemctl start  pulp_workers.service
Redirecting to /bin/systemctl start  httpd.service
Redirecting to /bin/systemctl start  foreman-tasks.service
gzip: pgsql_data.tar: No such file or directory
gzip: mongo_data.tar: No such file or directory
**** BACKUP Complete, contents can be found in: /katello-backup/katello-backup-2017-03-14T13:01:57+05:30 ****
[root@sat6 /]# 


Expected results:

[root@sat6 katello-backup]# katello-backup --skip-pulp-content /katello-backup/
Starting backup: 2017-03-14 13:15:45 +0530
Creating backup folder /katello-backup/katello-backup-2017-03-14T13:15:45+05:30
Redirecting to /bin/systemctl stop  foreman-tasks.service
Redirecting to /bin/systemctl stop  httpd.service
Redirecting to /bin/systemctl stop  pulp_workers.service
Redirecting to /bin/systemctl stop  foreman-proxy.service
Redirecting to /bin/systemctl stop  pulp_streamer.service
Redirecting to /bin/systemctl stop  pulp_resource_manager.service
Redirecting to /bin/systemctl stop  pulp_celerybeat.service
Redirecting to /bin/systemctl stop  smart_proxy_dynflow_core.service
Redirecting to /bin/systemctl stop  tomcat.service
Redirecting to /bin/systemctl stop  squid.service
Redirecting to /bin/systemctl stop  qdrouterd.service
Redirecting to /bin/systemctl stop  qpidd.service
Redirecting to /bin/systemctl stop  postgresql.service
Redirecting to /bin/systemctl stop  mongod.service
Backing up config files... 
tar: Removing leading `/' from member names
Done.
Redirecting to /bin/systemctl stop  foreman-tasks.service
Redirecting to /bin/systemctl stop  httpd.service
Redirecting to /bin/systemctl stop  pulp_workers.service
Redirecting to /bin/systemctl stop  foreman-proxy.service
Redirecting to /bin/systemctl stop  pulp_streamer.service
Redirecting to /bin/systemctl stop  pulp_resource_manager.service
Redirecting to /bin/systemctl stop  pulp_celerybeat.service
Redirecting to /bin/systemctl stop  smart_proxy_dynflow_core.service
Redirecting to /bin/systemctl stop  tomcat.service
Redirecting to /bin/systemctl stop  squid.service
Redirecting to /bin/systemctl stop  qdrouterd.service
Redirecting to /bin/systemctl stop  qpidd.service
Redirecting to /bin/systemctl stop  postgresql.service
Redirecting to /bin/systemctl stop  mongod.service
Backing up postgres db... 
tar: Removing leading `/' from member names
Done.
Backing up mongo db... 
tar: Removing leading `/' from member names
Done.
Redirecting to /bin/systemctl start  mongod.service
Redirecting to /bin/systemctl start  postgresql.service
Redirecting to /bin/systemctl start  qpidd.service
Redirecting to /bin/systemctl start  qdrouterd.service
Redirecting to /bin/systemctl start  squid.service
Redirecting to /bin/systemctl start  tomcat.service
Redirecting to /bin/systemctl start  smart_proxy_dynflow_core.service
Redirecting to /bin/systemctl start  pulp_celerybeat.service
Redirecting to /bin/systemctl start  pulp_resource_manager.service
Redirecting to /bin/systemctl start  pulp_streamer.service
Redirecting to /bin/systemctl start  foreman-proxy.service
Redirecting to /bin/systemctl start  pulp_workers.service
Redirecting to /bin/systemctl start  httpd.service
Redirecting to /bin/systemctl start  foreman-tasks.service
**** BACKUP Complete, contents can be found in: /katello-backup/katello-backup-2017-03-14T13:15:45+05:30 ****
[root@sat6 katello-backup]# 

Additional info:

While Analysing the "/usr/bin/katello-backup" script, I found below corrections:

1. Line number 68:
`tar --selinux --create --file=pgsql_data.tar.gz  --listed-incremental=.postgres.snar /var/lib/pgsql/data/`

change to >>>

`tar --selinux --create --file=pgsql_data.tar  --listed-incremental=.postgres.snar /var/lib/pgsql/data/`

AND

2. Line number 72: 
`tar --selinux --create --file=mongo_data.tar.gz --listed-incremental=.mongo.snar --exclude=mongod.lock /var/lib/mongodb/`

change to >>>

`tar --selinux --create --file=mongo_data.tar --listed-incremental=.mongo.snar --exclude=mongod.lock /var/lib/mongodb/`

This runs successfully.

Comment 6 Steven Ellis 2017-03-29 00:58:19 UTC
There are two sides to the problem.

Backup script shouldn't complain about the missing tar files.

Recovery script should work correctly.

Comment 7 Muhammad Selim Jahangir 2017-04-12 23:13:21 UTC
case#01718495,
Customer requested hotfix for satellite 6.2.8. Any update on this will be appreciated. 

Regards
Selim

Comment 9 Mike McCune 2017-04-25 05:07:18 UTC
** HOTFIX For Satellite 6.2.8 ** 

1) download attached 1431971.patch to your Satellite

2) Copy to /usr/bin

3) patch -p2 < 1431971.patch

4) run katello-backup as normal, files should have the .tar.gz file extension and restore properly via katello-restore

If you upgrade to 6.2.9 please re-run the above hotfix patch to continue to carry this fix.

Comment 10 Mike McCune 2017-04-25 05:08:40 UTC
Created attachment 1273813 [details]
1431971.patch

Comment 11 Xixi 2017-04-25 22:17:04 UTC
(In reply to Mike McCune from comment #9)
> ** HOTFIX For Satellite 6.2.8 ** 
> 
Thank you - releasing hofix for Case 01718495 (Air New Zealand Limited, strategic)

Comment 12 Mike McCune 2017-04-27 14:38:17 UTC
*** Bug 1433888 has been marked as a duplicate of this bug. ***

Comment 14 Rajan Gupta 2017-05-05 09:16:01 UTC
This is partially fixed in 6.2.9

The online backup and restore now works
The offline backup is still creating tar files with a .gz extension and then trying to compress a non existant .tar file

gzip: pgsql_data.tar: No such file or directory
gzip: mongo_data.tar: No such file or directory

[root katello-backup-2017-05-05T14:49:08+08:00]# ls -la
total 34759604
drwxrwx--- 2 root postgres        4096 May  5 14:50 .
drwxr-xr-x 3 root root            4096 May  5 14:49 ..
-rw-r--r-- 1 root root        38785326 May  5 14:49 config_files.tar.gz
-rw-r--r-- 1 root root           95792 May  5 14:49 .config.snar
-rw-r--r-- 1 root root             346 May  5 14:53 metadata
-rw-r--r-- 1 root root     27720161280 May  5 14:53 mongo_data.tar.gz
-rw-r--r-- 1 root root             516 May  5 14:53 .mongo.snar
-rw-r--r-- 1 root root      7695124480 May  5 14:50 pgsql_data.tar.gz
-rw-r--r-- 1 root root           26159 May  5 14:50 .postgres.snar

[root katello-backup-2017-05-05T14:49:08+08:00]# file mongo_data.tar.gz
mongo_data.tar.gz: POSIX tar archive

[root katello-backup-2017-05-05T14:49:08+08:00]# file pgsql_data.tar.gz
pgsql_data.tar.gz: POSIX tar archive

See line 68 and 72 in /bin/katello-backup where it sets --file=pgsql_data.tar.gz, should be --file=pgsql_data.tar
and Line 83 and 84 where is tries to compress pgsql_data.tar"

Comment 15 jcallaha 2017-05-15 20:11:28 UTC
Verified in Satellite 6.2.9 Async.

RHEL 7
------
-bash-4.2# katello-backup . --incremental katello-backup-2017-05-15T18\:05\:40+02\:00/
Starting backup: 2017-05-15 21:55:26 +0200
Creating backup folder ./katello-backup-2017-05-15T21:55:26+02:00
Generating metadata ... 
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.42/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.125/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
Backing up config files... 
Done.
Redirecting to /bin/systemctl stop  foreman-tasks.service
Redirecting to /bin/systemctl stop  httpd.service
Redirecting to /bin/systemctl stop  pulp_workers.service
Redirecting to /bin/systemctl stop  foreman-proxy.service
Redirecting to /bin/systemctl stop  pulp_streamer.service
Redirecting to /bin/systemctl stop  pulp_resource_manager.service
Redirecting to /bin/systemctl stop  pulp_celerybeat.service
Redirecting to /bin/systemctl stop  smart_proxy_dynflow_core.service
Redirecting to /bin/systemctl stop  tomcat.service
Redirecting to /bin/systemctl stop  squid.service
Redirecting to /bin/systemctl stop  qdrouterd.service
Redirecting to /bin/systemctl stop  qpidd.service
Redirecting to /bin/systemctl stop  postgresql.service
Redirecting to /bin/systemctl stop  mongod.service
Backing up postgres db... 
tar: Removing leading `/' from member names
Done.
Backing up mongo db... 
tar: Removing leading `/' from member names
Done.
Backing up Pulp data... 
tar: Removing leading `/' from member names
Done.
Redirecting to /bin/systemctl start  mongod.service
Redirecting to /bin/systemctl start  postgresql.service
Redirecting to /bin/systemctl start  qpidd.service
Redirecting to /bin/systemctl start  qdrouterd.service
Redirecting to /bin/systemctl start  squid.service
Redirecting to /bin/systemctl start  tomcat.service
Redirecting to /bin/systemctl start  smart_proxy_dynflow_core.service
Redirecting to /bin/systemctl start  pulp_celerybeat.service
Redirecting to /bin/systemctl start  pulp_resource_manager.service
Redirecting to /bin/systemctl start  pulp_streamer.service
Redirecting to /bin/systemctl start  foreman-proxy.service
Redirecting to /bin/systemctl start  pulp_workers.service
Redirecting to /bin/systemctl start  httpd.service
Redirecting to /bin/systemctl start  foreman-tasks.service
Done with backup: 2017-05-15 21:57:45 +0200
**** BACKUP Complete, contents can be found in: ./katello-backup-2017-05-15T21:55:26+02:00 ****


RHEL 6
------

-bash-4.1# katello-backup .
Starting backup: Mon May 15 21:58:12 +0200 2017
Creating backup folder ./katello-backup-2017-05-15T21:58:12+02:00
Generating metadata ... 
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-0.1.42/app/models/concerns/satellite_packages.rb:4: warning: already initialized constant Katello::Ping::PACKAGES
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.0.0.125/app/models/katello/ping.rb:7: warning: previous definition of PACKAGES was here
Backing up config files... 
Done.
celery multi v3.1.11 (Cipater)
> Stopping nodes...
	> reserved_resource_worker-6.eng.brq.redhat.com: QUIT -> 2335
	> reserved_resource_worker-1.eng.brq.redhat.com: QUIT -> 2217
	> reserved_resource_worker-5.eng.brq.redhat.com: QUIT -> 2312
	> reserved_resource_worker-7.eng.brq.redhat.com: QUIT -> 2358
	> reserved_resource_worker-0.eng.brq.redhat.com: QUIT -> 2201
	> reserved_resource_worker-2.eng.brq.redhat.com: QUIT -> 2242
	> reserved_resource_worker-4.eng.brq.redhat.com: QUIT -> 2289
	> reserved_resource_worker-3.eng.brq.redhat.com: QUIT -> 2266
> Waiting for 8 nodes -> 2335, 2217, 2312, 2358, 2201, 2242, 2289, 2266....................
	> reserved_resource_worker-6.eng.brq.redhat.com: OK
> Waiting for 7 nodes -> 2217, 2312, 2358, 2201, 2242, 2289, 2266....
	> reserved_resource_worker-1.eng.brq.redhat.com: OK
> Waiting for 6 nodes -> 2312, 2358, 2201, 2242, 2289, 2266....
	> reserved_resource_worker-5.eng.brq.redhat.com: OK
> Waiting for 5 nodes -> 2358, 2201, 2242, 2289, 2266....
	> reserved_resource_worker-7.eng.brq.redhat.com: OK
> Waiting for 4 nodes -> 2201, 2242, 2289, 2266....
	> reserved_resource_worker-0.eng.brq.redhat.com: OK
> Waiting for 3 nodes -> 2242, 2289, 2266....
	> reserved_resource_worker-2.eng.brq.redhat.com: OK
> Waiting for 2 nodes -> 2289, 2266....
	> reserved_resource_worker-4.eng.brq.redhat.com: OK
> Waiting for 1 node -> 2266....
	> reserved_resource_worker-3.eng.brq.redhat.com: OK

celery multi v3.1.11 (Cipater)
> Stopping nodes...
	> resource_manager.eng.brq.redhat.com: QUIT -> 1626
> Waiting for 1 node -> 1626......
	> resource_manager.eng.brq.redhat.com: OK

Backing up postgres db... 
tar: Removing leading `/' from member names
Done.
Backing up mongo db... 
tar: Removing leading `/' from member names
Done.
Backing up Pulp data... 
tar: Removing leading `/' from member names
Done.
celery multi v3.1.11 (Cipater)
> Starting nodes...
	> resource_manager.eng.brq.redhat.com: OK
celery multi v3.1.11 (Cipater)
> Starting nodes...
	> reserved_resource_worker-0.eng.brq.redhat.com: OK
	> reserved_resource_worker-1.eng.brq.redhat.com: OK
	> reserved_resource_worker-2.eng.brq.redhat.com: OK
	> reserved_resource_worker-3.eng.brq.redhat.com: OK
	> reserved_resource_worker-4.eng.brq.redhat.com: OK
	> reserved_resource_worker-5.eng.brq.redhat.com: OK
	> reserved_resource_worker-6.eng.brq.redhat.com: OK
	> reserved_resource_worker-7.eng.brq.redhat.com: OK
/usr/lib/python2.6/site-packages/pulp/server/db/connection.py:159: DeprecationWarning: add_son_manipulator is deprecated
  _DATABASE.add_son_manipulator(NamespaceInjector())
/usr/lib/python2.6/site-packages/pulp/server/db/model/base.py:96: DeprecationWarning: ensure_index is deprecated. Use create_index instead.
  unique=unique, background=True)
[Mon May 15 22:01:50 2017] [warn] module passenger_module is already loaded, skipping
Done with backup: Mon May 15 22:03:24 +0200 2017
**** BACKUP Complete, contents can be found in: ./katello-backup-2017-05-15T21:58:12+02:00 ****
-bash-4.1# ll
total 4.0K
drwxrwx---. 2 root postgres 4.0K May 15 22:03 katello-backup-2017-05-15T21:58:12+02:00
-bash-4.1# ll katello-backup-2017-05-15T21\:58\:12+02\:00/
total 507M
-rw-r--r--. 1 root root 610K May 15 21:58 config_files.tar.gz
-rw-r--r--. 1 root root  349 May 15 22:03 metadata
-rw-r--r--. 1 root root 401M May 15 22:00 mongo_data.tar.gz
-rw-r--r--. 1 root root  28M May 15 21:59 pgsql_data.tar.gz
-rw-r--r--. 1 root root  79M May 15 22:00 pulp_data.tar

Comment 17 errata-xmlrpc 2017-05-16 14:38:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1234