Bug 1111749 - DWH/Reports upgrade from 3.2 to 3.5 fails
Summary: DWH/Reports upgrade from 3.2 to 3.5 fails
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-dwh
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.5.0
Assignee: Yedidyah Bar David
QA Contact: movciari
URL:
Whiteboard: integration
: 1121792 (view as bug list)
Depends On:
Blocks: 1071020 1077775 rhev3.4rc 1123751 rhev3.5beta 1156165
TreeView+ depends on / blocked
 
Reported: 2014-06-20 22:49 UTC by Anitha Udgiri
Modified: 2019-05-20 11:16 UTC (History)
17 users (show)

Fixed In Version: vt1.5 - rhevm-dwh-3.5.0-1.el6_5
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1123751 (view as bug list)
Environment:
Last Closed: 2015-02-11 18:15:22 UTC
oVirt Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 1144503 0 None None None Never
Red Hat Knowledge Base (Solution) 1144563 0 None None None Never
Red Hat Product Errata RHEA-2015:0177 0 normal SHIPPED_LIVE rhevm-dwh 3.5 bug fix and enhancement update 2015-02-11 23:11:50 UTC

Description Anitha Udgiri 2014-06-20 22:49:52 UTC
Description of problem:
In customer's words :

"I upgrade RHEV to 3.4 and now engine-backup failed with command :
/usr/bin/engine-backup --scope=all --mode=backup --log=/disk1/rhev_backup/backup.$(/bin/date +%Y%m%d%H%M%S).out  --file=/disk1/rhev_backup/backup.$(/bin/date +%Y%m%d%H%M%S).tar.bz2

with error :

2014-06-12 09:27:37 23531: Start of engine-backup mode backup scope all file /disk1/rhev_backup/archives/engine_20140612_09:27.bz2
2014-06-12 09:27:37 23531: Backing up:
2014-06-12 09:27:37 23531: Generating pgpass
2014-06-12 09:27:37 23531: Creating temp folder /tmp/engine-backup.cDTix814MV/tar
2014-06-12 09:27:37 23531: - Files
2014-06-12 09:27:37 23531: Backing up files to /tmp/engine-backup.cDTix814MV/tar/files
2014-06-12 09:27:37 23531: - Engine database 'engine'
2014-06-12 09:27:37 23531: Backing up database to /tmp/engine-backup.cDTix814MV/tar/db/engine_backup.db
2014-06-12 09:27:39 23531: - DWH database 'ovirt_engine_history'
2014-06-12 09:27:39 23531: Backing up dwh database to /tmp/engine-backup.cDTix814MV/tar/db/dwh_backup.db
pg_dump: [programme d'archivage (db)] la connexion à la base de données « ovirt_engine_history » a échoué : FATAL:  authentification Ident échouée pour l'utilisateur « engine_history »
2014-06-12 09:27:39 23531: FATAL: Database ovirt_engine_history backup failed


so when i run engine-setup after upgrade i have this error :


[ INFO  ] Exporting data out of Jasper
[ ERROR ] Failed to execute stage 'Misc configuration': Command './js-export.sh' failed to execute
[ INFO  ] Yum Performing yum transaction rollback
[ INFO  ] Rolling back database schema
[ INFO  ] Clearing Engine database engine
[ INFO  ] Restoring Engine database engine
[WARNING] Rollback of DWH database postponed to Stage "Clean up"
[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20140612095519-8fzr5d.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20140612095631-setup.conf'
[WARNING] Rollback of DWH database started
          This might be a long process, but it should be safe to start the engine service before it finishes, if needed.
[ INFO  ] Clearing DWH database ovirt_engine_history
[ INFO  ] Restoring DWH database ovirt_engine_history
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Execution of setup failed

Comment 2 Yedidyah Bar David 2014-06-22 13:28:54 UTC
Hi,

Based on the sosreport attached to the case, it seems to me like reports was never upgraded to 3.3. So the flow was, more-or-less:

1. 3.2 engine/dwh/reports installed and setup (perhaps upgraded from prev)
2. engine was upgraded to various 3.3 versions
3. reports package was updated to 3.3 but never setup
4. engine package was updated to 3.4 and engine-setup ran, upgraded engine
5. reports package was updated to 3.4, engine-setup ran and failed during the upgrade of reports.

During all this, the 3.4 version of engine-backup was ran, and failed too.

It seems to me that these two independent failures were caused by not running rhevm-reports-setup after updating the package to 3.3.

So:

Please try to reproduce. If you succeed, try to solve, perhaps by:
- downgrade rhevm-setup to 3.3
- run rhevm-reports-setup
- upgrade reports package to 3.4
- run engine-setup
Please report if this works or not, and if fails, please provide access to a test machine showing the failure.

Thanks and good luck.

Comment 3 Yedidyah Bar David 2014-06-22 13:34:02 UTC
(In reply to Yedidyah Bar David from comment #2)
> Hi,
> 
> Based on the sosreport attached to the case, it seems to me like reports was
> never upgraded to 3.3. So the flow was, more-or-less:
> 
> 1. 3.2 engine/dwh/reports installed and setup (perhaps upgraded from prev)
> 2. engine was upgraded to various 3.3 versions
> 3. reports package was updated to 3.3 but never setup
> 4. engine package was updated to 3.4 and engine-setup ran, upgraded engine
> 5. reports package was updated to 3.4, engine-setup ran and failed during
> the upgrade of reports.
> 
> During all this, the 3.4 version of engine-backup was ran, and failed too.
> 
> It seems to me that these two independent failures were caused by not
> running rhevm-reports-setup after updating the package to 3.3.
> 
> So:
> 
> Please try to reproduce. If you succeed, try to solve, perhaps by:
> - downgrade rhevm-setup to 3.3

Sorry, I meant rhevm-reports.
You might need to remove rhevm-reports-setup, that's ok. If you need to remove other stuff, please consult us.

> - run rhevm-reports-setup
> - upgrade reports package to 3.4
> - run engine-setup
> Please report if this works or not, and if fails, please provide access to a
> test machine showing the failure.
> 
> Thanks and good luck.

Comment 4 Yedidyah Bar David 2014-06-22 13:53:45 UTC
(In reply to Yedidyah Bar David from comment #3)
> (In reply to Yedidyah Bar David from comment #2)
> > Hi,
> > 
> > Based on the sosreport attached to the case, it seems to me like reports was
> > never upgraded to 3.3. So the flow was, more-or-less:
> > 
> > 1. 3.2 engine/dwh/reports installed and setup (perhaps upgraded from prev)
> > 2. engine was upgraded to various 3.3 versions
> > 3. reports package was updated to 3.3 but never setup

Same for dwh.

rhevm-dwh-setup 3.3 was ran a few times and never finished successfully. Last time it was killed at the prompt asking whether to backup the db.

> > 4. engine package was updated to 3.4 and engine-setup ran, upgraded engine
> > 5. reports package was updated to 3.4, engine-setup ran and failed during
> > the upgrade of reports.

Indeed. Note that dwh might not be in perfect condition either, as it too was not upgraded to 3.3. I did not analyze potential issues this might cause in the future, but one definite difference between this and a properly upgraded system is that dwh and reports still use the 'postgres' db user and not their own (as happens on an upgrade to 3.3).

> > 
> > During all this, the 3.4 version of engine-backup was ran, and failed too.
> > 
> > It seems to me that these two independent failures were caused by not
> > running rhevm-reports-setup after updating the package to 3.3.
> > 
> > So:
> > 
> > Please try to reproduce. If you succeed, try to solve, perhaps by:
> > - downgrade rhevm-setup to 3.3
> 
> Sorry, I meant rhevm-reports.
> You might need to remove rhevm-reports-setup, that's ok. If you need to
> remove other stuff, please consult us.
> 
> > - run rhevm-reports-setup
> > - upgrade reports package to 3.4
> > - run engine-setup
> > Please report if this works or not, and if fails, please provide access to a
> > test machine showing the failure.
> > 
> > Thanks and good luck.

Comment 5 Yedidyah Bar David 2014-06-22 14:13:15 UTC
(In reply to Yedidyah Bar David from comment #4)
> (In reply to Yedidyah Bar David from comment #3)
> > (In reply to Yedidyah Bar David from comment #2)
> > > Hi,
> > > 
> > > Based on the sosreport attached to the case, it seems to me like reports was
> > > never upgraded to 3.3. So the flow was, more-or-less:
> > > 
> > > 1. 3.2 engine/dwh/reports installed and setup (perhaps upgraded from prev)
> > > 2. engine was upgraded to various 3.3 versions
> > > 3. reports package was updated to 3.3 but never setup
> 
> Same for dwh.
> 
> rhevm-dwh-setup 3.3 was ran a few times and never finished successfully.
> Last time it was killed at the prompt asking whether to backup the db.
> 
> > > 4. engine package was updated to 3.4 and engine-setup ran, upgraded engine
> > > 5. reports package was updated to 3.4, engine-setup ran and failed during
> > > the upgrade of reports.

BTW, the actual error is:

Caused by: org.postgresql.util.PSQLException: ERREUR: la relation « jivirtualdatasource » n'existe pas
  Position : 5082

Probably meaning the relation jivirtualdatasource does not exist.

This rings a bell - I stumbled upon this error at some point, not sure what
was the outcome. Adding Yaniv to add more details, although I am pretty certain
that this was supposed to be solved during the upgrade to 3.3 (which was not
done).

Comment 6 Yaniv Lavi 2014-06-23 06:13:27 UTC
(In reply to Yedidyah Bar David from comment #5)
> (In reply to Yedidyah Bar David from comment #4)
> > (In reply to Yedidyah Bar David from comment #3)
> > > (In reply to Yedidyah Bar David from comment #2)
> > > > Hi,
> > > > 
> > > > Based on the sosreport attached to the case, it seems to me like reports was
> > > > never upgraded to 3.3. So the flow was, more-or-less:
> > > > 
> > > > 1. 3.2 engine/dwh/reports installed and setup (perhaps upgraded from prev)
> > > > 2. engine was upgraded to various 3.3 versions
> > > > 3. reports package was updated to 3.3 but never setup
> > 
> > Same for dwh.
> > 
> > rhevm-dwh-setup 3.3 was ran a few times and never finished successfully.
> > Last time it was killed at the prompt asking whether to backup the db.
> > 
> > > > 4. engine package was updated to 3.4 and engine-setup ran, upgraded engine
> > > > 5. reports package was updated to 3.4, engine-setup ran and failed during
> > > > the upgrade of reports.
> 
> BTW, the actual error is:
> 
> Caused by: org.postgresql.util.PSQLException: ERREUR: la relation «
> jivirtualdatasource » n'existe pas
>   Position : 5082
> 
> Probably meaning the relation jivirtualdatasource does not exist.
> 
> This rings a bell - I stumbled upon this error at some point, not sure what
> was the outcome. Adding Yaniv to add more details, although I am pretty
> certain
> that this was supposed to be solved during the upgrade to 3.3 (which was not
> done).

I this supposed to be a upgrade? because this test is done to check if the db is empty and it seems it is.


Yaniv

Comment 7 Yedidyah Bar David 2014-06-23 06:38:54 UTC
(In reply to Yaniv Dary from comment #6)
> (In reply to Yedidyah Bar David from comment #5)
> > BTW, the actual error is:
> > 
> > Caused by: org.postgresql.util.PSQLException: ERREUR: la relation «
> > jivirtualdatasource » n'existe pas
> >   Position : 5082
> > 
> > Probably meaning the relation jivirtualdatasource does not exist.
> > 
> > This rings a bell - I stumbled upon this error at some point, not sure what
> > was the outcome. Adding Yaniv to add more details, although I am pretty
> > certain
> > that this was supposed to be solved during the upgrade to 3.3 (which was not
> > done).
> 
> I this supposed to be a upgrade? because this test is done to check if the
> db is empty and it seems it is.

No, db is not empty:

2014-06-12 09:20:01 DEBUG otopi.context context._executeMethod:138 Stage setup METHOD otopi.plugins.ovirt_engine_common.ovirt_engine_reports.db.connection.Plugin._setup
2014-06-12 09:20:01 DEBUG otopi.ovirt_engine_setup.database database.execute:167 Database: 'None', Statement: '
                    select 1
                ', args: {}
2014-06-12 09:20:01 DEBUG otopi.ovirt_engine_setup.database database.execute:172 Creating own connection
2014-06-12 09:20:01 DEBUG otopi.ovirt_engine_setup.database database.execute:217 Result: [{'?column?': 1}]
2014-06-12 09:20:01 DEBUG otopi.ovirt_engine_setup.database database.tryDatabaseConnect:439 Connection succeeded
2014-06-12 09:20:01 DEBUG otopi.ovirt_engine_setup.database database.execute:167 Database: 'None', Statement: '
                select count(*) as count
                from pg_catalog.pg_tables
                where schemaname = 'public';
            ', args: {}
2014-06-12 09:20:01 DEBUG otopi.ovirt_engine_setup.database database.execute:172 Creating own connection
2014-06-12 09:20:01 DEBUG otopi.ovirt_engine_setup.database database.execute:217 Result: [{'count': 86L}]
2014-06-12 09:20:01 DEBUG otopi.context context.dumpEnvironment:468 ENVIRONMENT DUMP - BEGIN
2014-06-12 09:20:01 DEBUG otopi.context context.dumpEnvironment:478 ENV OVESETUP_REPORTS_CORE/enable=bool:'True'
2014-06-12 09:20:01 DEBUG otopi.context context.dumpEnvironment:478 ENV OVESETUP_REPORTS_DB/database=unicode:'rhevmreports'
2014-06-12 09:20:01 DEBUG otopi.context context.dumpEnvironment:478 ENV OVESETUP_REPORTS_DB/host=unicode:'localhost'
2014-06-12 09:20:01 DEBUG otopi.context context.dumpEnvironment:478 ENV OVESETUP_REPORTS_DB/newDatabase=bool:'False'

Comment 8 Yedidyah Bar David 2014-06-23 07:29:03 UTC
(In reply to Yedidyah Bar David from comment #2)
> Hi,
> 
> Based on the sosreport attached to the case, it seems to me like reports was
> never upgraded to 3.3. So the flow was, more-or-less:
> 
> 1. 3.2 engine/dwh/reports installed and setup (perhaps upgraded from prev)
> 2. engine was upgraded to various 3.3 versions
> 3. reports package was updated to 3.3 but never setup
> 4. engine package was updated to 3.4 and engine-setup ran, upgraded engine
> 5. reports package was updated to 3.4, engine-setup ran and failed during
> the upgrade of reports.
> 
> During all this, the 3.4 version of engine-backup was ran, and failed too.

Just to clarify - this happened because engine-backup expects the reports db
creds to be in [1]. In <=3.3 they are kept in [2] and the upgrade from 3.3 to 3.4
copies them to the new location.

Generally speaking, engine-backup of version X should be used only to backup
and restore a system running version X.

I will keep this bug open until we know exactly what happened and decide what
needs to be fixed, but from what I know now, the only problem is that dwh and
reports were not upgraded. So one might rephrase the summary as "dwh/reports
should force upgrade from version X.Y to version X.Y+1 and not a later one".
This is currently done in the engine using versionlock, not sure we want to go
this way.

[1] /var/lib/ovirt-engine-reports/build-conf/master.properties
[2] /usr/share/jasperreports-server-pro/buildomatic/build_conf/default/master.properties

Comment 9 Anitha Udgiri 2014-06-23 16:32:30 UTC
(In reply to Yedidyah Bar David from comment #2)
> Hi,
> 
> Based on the sosreport attached to the case, it seems to me like reports was
> never upgraded to 3.3. So the flow was, more-or-less:
> 
> 1. 3.2 engine/dwh/reports installed and setup (perhaps upgraded from prev)
> 2. engine was upgraded to various 3.3 versions
> 3. reports package was updated to 3.3 but never setup
> 4. engine package was updated to 3.4 and engine-setup ran, upgraded engine
> 5. reports package was updated to 3.4, engine-setup ran and failed during
> the upgrade of reports.
> 
> During all this, the 3.4 version of engine-backup was ran, and failed too.
> 
> It seems to me that these two independent failures were caused by not
> running rhevm-reports-setup after updating the package to 3.3.
> 
> So:
> 
> Please try to reproduce. If you succeed, try to solve, perhaps by:
> - downgrade rhevm-setup to 3.3
> - run rhevm-reports-setup
> - upgrade reports package to 3.4
> - run engine-setup
> Please report if this works or not, and if fails, please provide access to a
> test machine showing the failure.
> 
> Thanks and good luck.

Will reach out to the customer to confirm on the actions performed.
I will in the interim, set up a system and try reproducing this problem.

Comment 15 Yedidyah Bar David 2014-07-22 08:32:05 UTC
*** Bug 1121792 has been marked as a duplicate of this bug. ***

Comment 16 Yedidyah Bar David 2014-07-22 08:37:05 UTC
Decided to re-open after getting yet another case like this.

Comment 19 Yedidyah Bar David 2014-07-22 13:29:45 UTC
Now reproduced:
1. Had 3.2 engine/dwh/reports installed and setup
2. yum update rhevm-setup to 3.3
3. engine-setup
4. yum update rhevm-dwh rhevm-reports to 3.3, did not setup
5. yum update rhevm-setup to 3.4
6. engine-setup
7. yum update rhevm-dwh rhevm-reports to 3.4
8. engine-setup fails

Seems to have managed to solved by:

1. As root:
# yum downgrade rhevm-dwh rhevm-reports jasperreports-server-pro

Make sure you got the expected versions of dwh/reports (3.3.x). Doing this without jasperreports-server-pro fails, because we require exact versions of it.

2. In the engine database (probably 'psql engine' as user postgres, otherwise need to use the correct credentials):

engine=# SELECT * from vdc_options where option_name = 'MinimalETLVersion';

Note the current version. It's currently '3.4.1'.

engine=# update vdc_options set option_value='3.3.0' where option_name = 'MinimalETLVersion';

3. Upgrade dwh to 3.3:

As root:

# rhevm-dwh-setup

4. Change back:

As root:

# service ovirt-engine-dwhd stop

In engine db:

engine=# update vdc_options set option_value='3.4.1' where option_name = 'MinimalETLVersion';

There is a delicate issue here - dwhd of 3.3 will try (at end of dwh setup) to connect to engine db and potentially read incorrect data (as it expects a 3.3 db). I now consulted Yaniv, and he said that dwhd will simply fail. To be extra careful, it's probably better to do this step (updating back MinimalETLVersion to 3.4.1) right after rhevm-dwh-setup started, e.g. after it outputs:
Backing up the DB...                                  [ DONE ]

5. Reports:

As root:

# rhevm-reports-setup

6. Now upgrade to 3.4:

As root:

# yum update rhevm-dwh rhevm-reports
# engine-setup

Comment 20 Yedidyah Bar David 2014-07-22 13:33:57 UTC
Anitha - please carefully try the above workaround (comments 19), and if it works you can publish it in a kb article and suggest that to customers that can't/won't use other means (such as restoring the entire machine from a backup done prior to updating the engine to 3.4).

This is obviously a workaround. I am still looking at preventing this altogether.

Comment 22 Yedidyah Bar David 2014-07-28 08:21:40 UTC
Following a discussion with Tomas, it seems we can't do much for 3.5.

GSS might want to prepare an article, or document this using other means.

Basically, users that will do:
* 3.2 engine/dwh/reports setup
* update engine to 3.3.x, then to 3.4.0 or 3.4.1 (3.4.2 will prevent that)

Will not be able to do anything simple to upgrade dwh to 3.3. The 3.3 version of rhevm-dwh-setup will query the engine db, see that minimal etl version there is 3.4.x, and will refuse to continue. It might be quite simple to use a workaround based on comment 19 above. We might decide we want that - that setup will notice and alert the user pointing to a relevant article.

Tomas - if you do want this, please create such an article and we'll prepare a patch for 3.5 to point to it. Otherwise we can just close wontfix. Thanks!

Comment 23 Tomas Dosek 2014-07-28 08:41:30 UTC
Knowledgebase article covering future needs of covering the topic for 3.5 was linked to this BZ.

Comment 24 Anitha Udgiri 2014-07-28 20:10:35 UTC
Thanks Toams and Didi.

Comment 25 Yedidyah Bar David 2014-07-31 10:30:42 UTC
Note to QE - please see https://bugzilla.redhat.com/show_bug.cgi?id=1123751#c1 .

This affects 3.5 if trying to upgrade to it from an unfixed version (3.4.0 or 3.4.1). Since you already upgraded the engine to 3.4, rhevm-dwh-setup of 3.3 will refuse to upgrade dwh (from 3.2 to 3.3). So your only choice will be to use a workaround which is basically comment 19 above. engine-setup will merely point you at an article.

Comment 27 movciari 2014-09-09 13:09:27 UTC
verified - installed 3.2 rhevm & dwh & reports, updated all rpms to 3.3, ran engine-setup, updated rhevm-setup rpm to 3.4 (av10.6), ran engine-setup, then with repos for 3.5 i tried to run "yum update rhevm-setup" which failed with: 

Error: Package: rhevm-setup-plugin-ovirt-engine-3.5.0-0.10.master.el6ev.noarch (rhevm35)
           Requires: rhevm >= 3.4.2
           Installed: rhevm-3.4.1-0.31.el6ev.noarch (@rhevm34)
               rhevm = 3.4.1-0.31.el6ev
Error: rhevm-setup-plugin-ovirt-engine conflicts with rhevm-3.4.1-0.31.el6ev.noarch

so to upgrade to 3.5, customer first has to upgrade to 3.4.2 which points him to a solution on customer portal and makes him fix problems with his setup
this should be in documentation

Comment 28 Yedidyah Bar David 2014-11-04 13:49:26 UTC
Not sure what should be in the doc text.

We definitely want to document, perhaps elsewhere, that we now recommend, and usually enforce, not skipping upgrades. This was already discussed elsewhere, e.g. bug 1123751 .

Comment 30 errata-xmlrpc 2015-02-11 18:15:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2015-0177.html


Note You need to log in before you can comment on or make changes to this bug.