Bug 1659888

Summary:	Not a good idea to store undercloud backups in Swift
Product:	Red Hat OpenStack	Reporter:	Dan Macpherson <dmacpher>
Component:	python-tripleoclient	Assignee:	Juan Badia Payno <jbadiapa>
Status:	CLOSED WONTFIX	QA Contact:	Gurenko Alex <agurenko>
Severity:	low	Docs Contact:
Priority:	low
Version:	13.0 (Queens)	CC:	aschultz, beth.white, hbrock, jbadiapa, jbuchta, jslagle, mburns, rmccabe, tdunnon
Target Milestone:	---	Keywords:	FutureFeature, Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-09 14:56:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Dan Macpherson 2018-12-17 06:33:20 UTC

When running the "openstack undercloud backup" command, tripleo creates an archive containing the full undercloud backup and the undercloud filesystem and saves it as an object in swift.

The problem is the filesystem backup also includes the swift filesystem, which will include the previous backups. This means that any sequential backup you take will exponentially increase in size.

For example, here are two backups I took the same day:

(undercloud) [stack@ccsosp-undercloud ~]$ ls -lh UC*
-rw-rw-r--. 1 stack stack 6.6G Dec 17 15:53 UC-backup-20181217154339.tar
-rw-rw-r--. 1 stack stack  14G Dec 17 16:13 UC-backup-20181217155754.tar

The second backup jumps in size because it also contains the first backup as a swift object.

The next backup I take will be 6.6GB (inital filesystem) + 6.6.GB (backup 1) + 14GB (backup 2) ~= 28GB

The next backup after that will be 6.6GB + 6.6.GB + 14GB + 28GB ~= 56GB

etc, etc, etc

There are probably a few other reasons not to store the archive in Swift:

* If the undercloud disk gets corrupted and causes the undercloud to become inaccessible, you won't be able to recreate the undercloud since the backup is stored in the undercloud.
* The undercloud has a recommended disk size of 100GB, which can fill up quickly if performing a daily backup.

Might be an idea to simply export the archive as a file so the user can move it to an external location.

Comment 1 Adriano Petrich 2018-12-18 13:49:42 UTC

I feel that this is more related to user action than to the swift system. Probably changing the default would be enough. What would be the default action?

The exponential backup problem seems bad but it seems like the correct behaviour if you want a backup of your backup these things happen.

Comment 2 Dan Macpherson 2018-12-18 14:44:48 UTC

(In reply to Adriano Petrich from comment #1)
> I feel that this is more related to user action than to the swift system.
> Probably changing the default would be enough. What would be the default
> action?

When you say "default action", are we talking about mistral actions? If so, the relevant actions are:

* tripleo.undercloud.create_file_system_backup, which maps to class CreateFileSystemBackup in tripleo-common -- this creates the filesystem backup, including the swift objects
* tripleo.undercloud.upload_backup_to_swift which maps to class UploadUndercloudBackupToSwift in tripleo-common -- this saves the resulting backup to swift

So might need to change the component of this BZ to tripleo-common

> The exponential backup problem seems bad but it seems like the correct
> behaviour if you want a backup of your backup these things happen.

It seems a little redundant and impractical to backup a backup and then upload it to the same location as the original backup. I'm not sure why anyone would want this to be standard behaviour.

Comment 3 Adriano Petrich 2019-07-23 14:30:29 UTC

Agreed in all counts.

When I said the default action it was bad semantics. I meant should the default be store to disk? because that is already an intermediary step. What could be done is that uploading to swift and deleting the temporary directory be an optional step. The new backups would not contain the previous ones if they are not in swift and the backup dir is outside the regular folders to be backupped.

Comment 4 Dan Macpherson 2019-07-23 14:38:21 UTC

Pretty much. Ideally you'd want store the backup in a location not included in the filesystem backup to avoid the exponential backup size.

Comment 6 Juan Badia Payno 2019-11-27 16:10:51 UTC

We are moving all the backup and restore feature to use ReaR.

Comment 7 Toure Dunnon 2020-03-09 14:56:30 UTC

With the use of ReaR as our main method of backup and recovery this bug will be closed.