Bug 1059286 - [DWH-OTOPI-SETUP] - Backup handling was removed and restore is done automatically
Summary: [DWH-OTOPI-SETUP] - Backup handling was removed and restore is done automatic...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-dwh
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.4.0
Assignee: Yedidyah Bar David
QA Contact: Barak Dagan
URL:
Whiteboard: integration
Depends On:
Blocks: 1054226 rhev3.4rc
TreeView+ depends on / blocked
 
Reported: 2014-01-29 15:11 UTC by Yaniv Lavi
Modified: 2014-06-09 15:18 UTC (History)
12 users (show)

Fixed In Version: av8 - rhevm-dwh-3.4.0-8.el6ev.noarch.rpm
Doc Type: Bug Fix
Doc Text:
Users are now prompted whether to take a backup of the ovirt_engine_history database when upgrading the Data Warehouse or Reports features using the engine-setup command.
Clone Of:
Environment:
Last Closed: 2014-06-09 15:18:09 UTC
oVirt Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2014:0601 0 normal SHIPPED_LIVE rhevm-dwh 3.4 bug fix and enhancement update 2014-06-09 19:15:53 UTC
oVirt gerrit 26073 0 None None None Never
oVirt gerrit 26082 0 master MERGED packaging: setup: Ask whether to backup the database Never
oVirt gerrit 27046 0 ovirt-engine-3.4 MERGED packaging: setup: Ask whether to backup the database Never
oVirt gerrit 27126 0 ovirt-engine-3.4 MERGED packaging: setup: Use custom pg_dump format Never

Description Yaniv Lavi 2014-01-29 15:11:56 UTC
Description of problem:
Backup handling was removed and restore is done automatically. This can cause issues in cases that DB is very large, since backup can take up a lot of space and restore can take very long. Like in previous installer user should be prompt if to backup db with data on available disk space and predicted backup size. User should be prompt about if he wants to restore db after failure as well.  

Version-Release number of selected component (if applicable):
3.4.0

How reproducible:
Always

Comment 1 Barak 2014-03-02 15:28:36 UTC
Arthur,

You need to decide what is the desired behaviour.

Either have the db backed up automatically - which might take a long time - but will enable restore.

Or leaving the behaviour like in the previous installer - asking whether to backup the db.

Comment 2 Arthur Berezin 2014-03-03 12:36:48 UTC
Agree with Yaniv, otopi installer should ask the user if he is interested in backing up DWH.

Comment 3 Barak 2014-03-20 10:08:04 UTC
Please note that in case the db is nor backed up anyway, there will be no rollback.

Are you sure about this answer ?

Comment 5 Arthur Berezin 2014-03-23 12:52:46 UTC
1. User should be prompted if he wants to backup data-warehouse(dwh) database.
2. It should be clearly stated what data will be lost(reports, etc') if the user chooses not to backup data-warehouse db.
3. default answer should be to backup DWH - [YES] in case the user hits enter without reading the questions.

Comment 6 Barak 2014-03-23 15:28:08 UTC
didi,

when asking the questions we need to calculate whether the diskspace available for backup is sufficient.

Comment 7 Barak 2014-03-23 15:36:57 UTC
On the restore phase (if upgrade fails),

I would not restore the history db by default (automatically), because this might take too long and might leaves the customer to hang (not knowing how much time it takes).

But this allows the engine to get back to operational state faster.

And on the end of the restore have appropriate message about how to restore the history db.

Comment 8 Arthur Berezin 2014-03-23 15:52:13 UTC
(In reply to Barak from comment #7)
> On the restore phase (if upgrade fails),
> 
> I would not restore the history db by default (automatically), because this
> might take too long and might leaves the customer to hang (not knowing how
> much time it takes).
> 
> But this allows the engine to get back to operational state faster.
> 
> And on the end of the restore have appropriate message about how to restore
> the history db.

Great! ACK

Comment 9 Yedidyah Bar David 2014-03-27 14:00:20 UTC
I already had here a reply waiting to be sent which I wrote and forgot before today's meeting. Instead I'll summarize the meeting and a discussion with Alon I had later.

Concerns:
1. Some users might want the upgrade process to be "atomic" - have a working 3.3 system (engine+dwh+reports), stop everything, upgrade everthing, start. If it fails in the middle - restore everything to the initial state.
2. Some users might want minimal downtime for the engine. These users might prefer to upgrade each separately, and might prefer, if we do everything in a single engine-setup run, a possibly complex setup process that upgrades the engine as quickly as possible, and specifically rolls back the engine quickly and then either leaves dwh for the user to restore manually or restore it after the engine is already up.

Current code:
1. Allows upgrading either each separately or all at once. I still need to verify the details but that's the intention.
2. Engine is always "upgraded". If there is no update to the rpm, it basically does nothing, but the user does see a complete upgrade process - backup, refresh db schema, on failure restore from backup. Barak was concerned that users will have issues with that, but Alon said changing this currently for 3.4 is too complex.

For this bug I currently only intend to make one change - to ask the user if they want to backup. If they said yes, on rollback I think we'll always restore. If you want to never, or make this optional, please state exactly what you want. I agree with Alon that if you want something else, it should be another bug, but we'll probably not make significant structural changes before 3.5.

Barak/Yaniv - please ack and/or detail exactly what you want. Thanks.

Comment 10 Barak 2014-03-30 18:03:06 UTC
(In reply to Yedidyah Bar David from comment #9)
> I already had here a reply waiting to be sent which I wrote and forgot
> before today's meeting. Instead I'll summarize the meeting and a discussion
> with Alon I had later.
> 
> Concerns:
> 1. Some users might want the upgrade process to be "atomic" - have a working
> 3.3 system (engine+dwh+reports), stop everything, upgrade everthing, start.
> If it fails in the middle - restore everything to the initial state.
> 2. Some users might want minimal downtime for the engine. These users might
> prefer to upgrade each separately, and might prefer, if we do everything in
> a single engine-setup run, a possibly complex setup process that upgrades
> the engine as quickly as possible, and specifically rolls back the engine
> quickly and then either leaves dwh for the user to restore manually or
> restore it after the engine is already up.

(#1 & #2) I don't see any point in doing rollback on the engine if reports failed to upgrade so I would keep the process the same as today (= first upgrade engine ... than dwh and than reports ...). This even grows stronger when you think on the potential long time it takes to restore the history db.
From past experience users tend to CTRL-C the shell once they see upgrade/rollback takes too long ... and it might take hours. 


> 
> Current code:
> 1. Allows upgrading either each separately or all at once. I still need to
> verify the details but that's the intention.

see above 

> 2. Engine is always "upgraded". If there is no update to the rpm, it
> basically does nothing, but the user does see a complete upgrade process -
> backup, refresh db schema, on failure restore from backup. Barak was
> concerned that users will have issues with that, but Alon said changing this
> currently for 3.4 is too complex.

I assume you refer to the dwh-upgrade using otopi will also do the upgrade sequence of the engine (including prints to the screen) and will do nothing eventually. 
Although this is ugly This is o.k. for now. Arthur ? 


> 
> For this bug I currently only intend to make one change - to ask the user if
> they want to backup. If they said yes, on rollback I think we'll always
> restore. If you want to never, or make this optional, please state exactly
> what you want. I agree with Alon that if you want something else, it should
> be another bug, but we'll probably not make significant structural changes
> before 3.5.

Bottom line I don't think we should do automatic restore of the history db (see above), it will only cause a massive amount of support tickets.

> 
> Barak/Yaniv - please ack and/or detail exactly what you want. Thanks.

Comment 12 Arthur Berezin 2014-05-01 07:38:37 UTC
(In reply to Barak from comment #10)
> 
> I assume you refer to the dwh-upgrade using otopi will also do the upgrade
> sequence of the engine (including prints to the screen) and will do nothing
> eventually. 
> Although this is ugly This is o.k. for now. Arthur ? 
> 
> 
it is ugly but for 3.4 it's fine, we should raise a BZ to check for package updates before printing the complete upgrade sequence to the screen.

Comment 13 Yedidyah Bar David 2014-05-01 08:32:32 UTC
(In reply to Arthur Berezin from comment #12)
> (In reply to Barak from comment #10)
> > 
> > I assume you refer to the dwh-upgrade using otopi will also do the upgrade
> > sequence of the engine (including prints to the screen) and will do nothing
> > eventually. 
> > Although this is ugly This is o.k. for now. Arthur ? 
> > 
> > 
> it is ugly but for 3.4 it's fine, we should raise a BZ to check for package
> updates before printing the complete upgrade sequence to the screen.

Note that it's not completely a no-op - e.g., it also runs a "task cleaner" - and users might want to run engine-setup with no updated packages to get these side-effects.

Comment 14 Barak Dagan 2014-05-04 12:24:31 UTC
Verified on av8.1:

          --== DATABASE CONFIGURATION ==--
         
          The detected DWH database size is 148 MB.
          Setup can backup the existing database. The time and space required for the database backup depend on its size. This process takes time, and in some cases (for instance, when the size is few GBs) may take several hours to complete.
          If you choose to not back up the database, and Setup later fails for some reason, it will not be able to restore the database and all DWH data will be lost.
          Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]: 

Restoring:
1) Befiore dwh was backed up - no rollback
2) After dwh was backed up - rollback performed:

[ INFO  ] Backing up database localhost:ovirt_engine_history to '/var/lib/ovirt-engine-dwh/backups/dwh-20140504151626.3cNWnV.dump'.
[ INFO  ] Creating/refreshing DWH database schema
^C[ ERROR ] Failed to execute stage 'Misc configuration': Command '/usr/share/ovirt-engine-dwh/dbscripts/schema.sh' failed to execute
[ INFO  ] Rolling back database schema
[ INFO  ] Clearing Engine database engine
[ INFO  ] Restoring Engine database engine
[WARNING] Rollback of DWH database postponed to Stage "Clean up"
[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20140504151420.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20140504151700-setup.conf'
[WARNING] Rollback of DWH database started
          This might be a long process, but it should be safe to start the engine service before it finishes, if needed.
[ INFO  ] Clearing DWH database ovirt_engine_history
[ INFO  ] Restoring DWH database ovirt_engine_history
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Execution of setup failed

Comment 15 errata-xmlrpc 2014-06-09 15:18:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0601.html


Note You need to log in before you can comment on or make changes to this bug.