Bug 1188132 - [RFE] - Engine-backup should expose options related to backup\restore of database that affect restore time.
Summary: [RFE] - Engine-backup should expose options related to backup\restore of data...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: oVirt
Classification: Retired
Component: ovirt-engine-installer
Version: 3.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.6.0
Assignee: Yedidyah Bar David
QA Contact: Gonza
URL: http://www.ovirt.org/Features/BackupA...
Whiteboard: integration
Depends On: 1188121
Blocks: 1188119 1213152 1213153
TreeView+ depends on / blocked
 
Reported: 2015-02-02 06:53 UTC by Yaniv Lavi
Modified: 2015-11-04 11:25 UTC (History)
13 users (show)

Fixed In Version: ovirt-3.6.0-alpha1.2
Doc Type: Enhancement
Doc Text:
See output of 'engine-backup --help' and commit message in [1]. The following options were added to engine-backup: --archive-compressor --files-compressor --db-compressor --db-dump-format --db-restore-jobs --dwh-db-compressor --dwh-db-dump-format --dwh-db-restore-jobs --reports-db-compressor --reports-db-dump-format --reports-db-restore-jobs These should allow fine-tuning backup/restore size/speed. [1] https://gerrit.ovirt.org/39896
Clone Of:
Environment:
Last Closed: 2015-11-04 11:25:17 UTC
oVirt Team: ---


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 39896 0 master MERGED packaging: engine-backup: allow custom dump/compression Never
oVirt gerrit 40715 0 master MERGED packaging: engine-backup: Updated --help output Never
oVirt gerrit 40723 0 master MERGED packaging: engine-backup: Correctly apply --archive-compressor Never

Description Yaniv Lavi 2015-02-02 06:53:34 UTC
Description of problem:
Flags like number of concurrent restore threads and backup format should be exposed to user.
The considerations of use of these flags should be added to wiki based on BZ #1188121.

Comment 1 Doron Fediuck 2015-03-22 12:20:29 UTC
(In reply to Yaniv Dary from comment #0)
> Description of problem:
> Flags like number of concurrent restore threads and backup format should be
> exposed to user.

This request is too generic. We cannot and will not support any possible
flag pg_dump supports, as we need to be consistent with previous and possibly
future versions.

Current options are available in [1]. If there's a specific pg_dump
flag which is needed, please specify which one and why is it needed.


[1] https://gerrit.ovirt.org/gitweb?p=ovirt-engine.git;a=blob;f=packaging/bin/engine-backup.sh;h=ba1f6771e97b2200462e696703c76afa8aa047fb;hb=HEAD#l136

Comment 2 Yedidyah Bar David 2015-03-22 13:52:53 UTC
(In reply to Doron Fediuck from comment #1)
> (In reply to Yaniv Dary from comment #0)
> > Description of problem:
> > Flags like number of concurrent restore threads and backup format should be
> > exposed to user.
> 
> This request is too generic. We cannot and will not support any possible
> flag pg_dump supports, as we need to be consistent with previous and possibly
> future versions.

We can simply allow the user to pass arbitrary options.

We'll then need to think how to restore. Perhaps just add to the backup file the added options, and if restore fails, hint to the user asking to pass relevant restore options. See also bug 1073421.

> 
> Current options are available in [1]. If there's a specific pg_dump
> flag which is needed, please specify which one and why is it needed.
> 
> 
> [1]
> https://gerrit.ovirt.org/gitweb?p=ovirt-engine.git;a=blob;f=packaging/bin/
> engine-backup.sh;h=ba1f6771e97b2200462e696703c76afa8aa047fb;hb=HEAD#l136

I already started thinking about a patch for engine-setup that will allow more control over backup/restore there (used for rollback).

In the past we used to use plain format there, then, for bug 1059286, wanted to compress (see e.g. [1]), then decided to have general support for pipes in otopi [2], but eventually didn't actually use that, and just use the custom format, which is compressed (see the final version of [1], [3]).

[1] https://gerrit.ovirt.org/#/c/26073/2/packaging/setup/ovirt_engine_setup/database.py
[2] https://gerrit.ovirt.org/26213
[3] https://gerrit.ovirt.org/#/c/26073/7/packaging/setup/ovirt_engine_setup/database.py

During that time, I did some tests and reported internally. Bottom line - with a 4 core, 12 GB VM, fastest restore was using custom format and 6 jobs (-j 6), best compression was with plain dump + bzip2 (or xz, don't remember). I searched and understood that pg_dump does not seem like it will support anything other than zlib (gzip) compression.

So to offer enough flexibility, I think we want to support (a subset(?) of):
1. either of the dump formats (plain, custom, or tar(?))
2. gzip, bzip2, xz or no extra compression
3. -Z n, the compression level
4. -j n, number of jobs

When we rewrite engine-backup using the engine-setup framework (see bug 1064503), we'll enjoy this there too.

I can envision users regularly running engine-backup with different options then - custom format to allow fast restores, plain + xz for longer archiving.

Comment 3 Doron Fediuck 2015-03-30 09:25:28 UTC
(In reply to Yedidyah Bar David from comment #2)
> (In reply to Doron Fediuck from comment #1)
> > (In reply to Yaniv Dary from comment #0)

> 
> When we rewrite engine-backup using the engine-setup framework (see bug
> 1064503), we'll enjoy this there too.
> 
> I can envision users regularly running engine-backup with different options
> then - custom format to allow fast restores, plain + xz for longer archiving.

We cannot support any potential flag, which may create partial backups
or backups we're not sure how to restore from. The backups we generate
should have a predictable output which we should be able to restore from
without loosing substantial data.

Hence, I'm fine with considering 2 additional 'backup profiles'-
1. Lean and Mean- may be used for fast restoration
2. Full backup- will allow complete DB backup
These will be added to the current default.

So a user may be able to choose 1 of 3 reasonable profiles based on his
needs, or change the tool which is fully opensourced and handle the results.

Yaniv, if this is what you have in mind it's fine. If not we should discuss it.

Comment 4 Yaniv Lavi 2015-03-30 09:45:07 UTC
(In reply to Doron Fediuck from comment #3)
> (In reply to Yedidyah Bar David from comment #2)
> > (In reply to Doron Fediuck from comment #1)
> > > (In reply to Yaniv Dary from comment #0)
> 
> > 
> > When we rewrite engine-backup using the engine-setup framework (see bug
> > 1064503), we'll enjoy this there too.
> > 
> > I can envision users regularly running engine-backup with different options
> > then - custom format to allow fast restores, plain + xz for longer archiving.
> 
> We cannot support any potential flag, which may create partial backups
> or backups we're not sure how to restore from. The backups we generate
> should have a predictable output which we should be able to restore from
> without loosing substantial data.

Didi didn't mention any flag that might cause partial backup. No sure where this concern is arising from since we choose what to expose.

> 
> Hence, I'm fine with considering 2 additional 'backup profiles'-
> 1. Lean and Mean- may be used for fast restoration
> 2. Full backup- will allow complete DB backup
> These will be added to the current default.
> 
> So a user may be able to choose 1 of 3 reasonable profiles based on his
> needs, or change the tool which is fully opensourced and handle the results.
> 
> Yaniv, if this is what you have in mind it's fine. If not we should discuss
> it.

I'm not against profiles as well, if Didi thinks that we can sum up the backup options into such profiles.

Comment 5 Yedidyah Bar David 2015-03-31 07:44:44 UTC
My current opinion is easily summarized in [1], the current pending patch which will be the basis for these options. It's currently for engine-setup, which is a bit different, but not by much.

In particular:

1. It does not (easily, currently) allow passing arbitrary options to pg_dump/pg_restore/gzip/bzip2/etc.

2. It does allow a bit more than just 2 or 3 profiles, but not much more

3. It reasonably controls the irrelevant/unwanted options (such as passing -j N to psql, or passing it to pg_restore when reading from a pipe), and matching needed ones (such as restoring a custom format dump with pg_restore and not psql).

Extending this to engine-backup will require adding some configuration to the backup file, but I do not see a problem with that, as we already require there restoring with the same version used to backup (so a backup done by 3.5 which does not include this configuration will not be restorable by 3.6, but so it is today).

4. It reasonably allows future changes/additions/etc, which will require changing the code (not merely editing the answer file). In particular, we can make the allowed options (functions _backup_restore_filters_info and _backup_restore_dumpers_info in [2] there) be saved early in the environment, thus allowing external plugins extend them without changing code.

Doron - is that ok with you?

[1] https://gerrit.ovirt.org/#/c/39190/6//COMMIT_MSG
[2] https://gerrit.ovirt.org/#/c/39190/6/packaging/setup/ovirt_engine_setup/engine_common/database.py

Comment 6 Yaniv Lavi 2015-04-07 13:47:39 UTC
*** Bug 1073421 has been marked as a duplicate of this bug. ***

Comment 7 Doron Fediuck 2015-04-12 10:20:11 UTC
(In reply to Yedidyah Bar David from comment #5)
> My current opinion is easily summarized in [1], the current pending patch
> which will be the basis for these options. It's currently for engine-setup,
> which is a bit different, but not by much.
> 
> In particular:
> 
> 1. It does not (easily, currently) allow passing arbitrary options to
> pg_dump/pg_restore/gzip/bzip2/etc.
> 
> 2. It does allow a bit more than just 2 or 3 profiles, but not much more
> 
> 3. It reasonably controls the irrelevant/unwanted options (such as passing
> -j N to psql, or passing it to pg_restore when reading from a pipe), and
> matching needed ones (such as restoring a custom format dump with pg_restore
> and not psql).
> 
> Extending this to engine-backup will require adding some configuration to
> the backup file, but I do not see a problem with that, as we already require
> there restoring with the same version used to backup (so a backup done by
> 3.5 which does not include this configuration will not be restorable by 3.6,
> but so it is today).
> 
> 4. It reasonably allows future changes/additions/etc, which will require
> changing the code (not merely editing the answer file). In particular, we
> can make the allowed options (functions _backup_restore_filters_info and
> _backup_restore_dumpers_info in [2] there) be saved early in the
> environment, thus allowing external plugins extend them without changing
> code.
> 
> Doron - is that ok with you?
> 
> [1] https://gerrit.ovirt.org/#/c/39190/6//COMMIT_MSG
> [2]
> https://gerrit.ovirt.org/#/c/39190/6/packaging/setup/ovirt_engine_setup/
> engine_common/database.py


According to the above if I backup my 3.5 setup, and then upgrade
I won't be able to use my ~2 days old backup to restore the system in case
of a crash?

Comment 8 Yedidyah Bar David 2015-04-12 10:32:11 UTC
(In reply to Doron Fediuck from comment #7)
> According to the above if I backup my 3.5 setup, and then upgrade

Upgrade to what?

> I won't be able to use my ~2 days old backup to restore the system in case
> of a crash?

engine-backup always required same major.minor. So engine-backup 3.5 refuses to restore a backup done with 3.4. Same will be true for 3.5 and 3.6.

Comment 9 Yedidyah Bar David 2015-04-12 10:37:29 UTC
(In reply to Yedidyah Bar David from comment #8)
> (In reply to Doron Fediuck from comment #7)
> > According to the above if I backup my 3.5 setup, and then upgrade
> 
> Upgrade to what?
> 
> > I won't be able to use my ~2 days old backup to restore the system in case
> > of a crash?

BTW, if that's a real question, the answer is:

1. You should backup often :-)

2. Specifically you should backup after verifying that an upgrade was successful.

IMO this is true for every product/system/whatever, not ovirt-specific.

3. If you didn't, you'll need to downgrade (or wipe and reinstall for that matter) to the version used to do the backup. After a successful restore you can of course upgrade again.

4. engine-setup also keeps its own db dump, to allow rollback, which was already used sometimes to restore systems that ran no other backup at all. That's obviously not recommended. engine-backup does not support this, pg_restore (or psql in previous version) has to be ran manually.

Comment 10 Doron Fediuck 2015-04-14 12:24:11 UTC
(In reply to Yedidyah Bar David from comment #9)
> (In reply to Yedidyah Bar David from comment #8)
> > (In reply to Doron Fediuck from comment #7)
> > > According to the above if I backup my 3.5 setup, and then upgrade
> > 
> > Upgrade to what?
> > 
> > > I won't be able to use my ~2 days old backup to restore the system in case
> > > of a crash?
> 

Even if I backup every hour.
What you're suggesting is that once I upgrade I need to immediately run
another back so I'll have something to restore from?
ie- if I upgrade 3.5->3.6 and post upgrade my system will crash I won't be able to
use my last good backup (3.5), unless I immediately create a 3.6 backup to
restore from?

Comment 11 Yedidyah Bar David 2015-04-15 18:05:34 UTC
(In reply to Doron Fediuck from comment #10)
> (In reply to Yedidyah Bar David from comment #9)
> > (In reply to Yedidyah Bar David from comment #8)
> > > (In reply to Doron Fediuck from comment #7)
> > > > According to the above if I backup my 3.5 setup, and then upgrade
> > > 
> > > Upgrade to what?
> > > 
> > > > I won't be able to use my ~2 days old backup to restore the system in case
> > > > of a crash?
> > 
> 
> Even if I backup every hour.
> What you're suggesting is that once I upgrade I need to immediately run
> another back so I'll have something to restore from?

Yes, or reinstall the version you had with your last backup

> ie- if I upgrade 3.5->3.6 and post upgrade my system will crash I won't be
> able to
> use my last good backup (3.5), unless I immediately create a 3.6 backup to
> restore from?

or downgrade to 3.5

Comment 12 Gonza 2015-07-03 08:09:17 UTC
Verified with:
ovirt-engine-3.6.0-0.0.master.20150627185750.git6f063c1.el6.noarch

Comment 13 Sandro Bonazzola 2015-11-04 11:25:17 UTC
oVirt 3.6.0 has been released on November 4th, 2015 and should fix this issue.
If problems still persist, please open a new BZ and reference this one.


Note You need to log in before you can comment on or make changes to this bug.