Bug 1304658 - [Upgrade][DWH] Upgrade 3.5->3.6 failed - column "pm_ip_address" does not exist
[Upgrade][DWH] Upgrade 3.5->3.6 failed - column "pm_ip_address" does not exist
Status: CLOSED DUPLICATE of bug 1130764
Product: ovirt-engine-dwh
Classification: oVirt
Component: Setup (Show other bugs)
3.6.3
Unspecified Unspecified
unspecified Severity urgent (vote)
: ovirt-3.6.3
: ---
Assigned To: Shirly Radco
Pavel Stehlik
: Regression
Depends On:
Blocks: RHEV3.6Upgrade
  Show dependency treegraph
 
Reported: 2016-02-04 04:37 EST by Gil Klein
Modified: 2017-05-11 05:26 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-10 03:00:29 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Integration
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
gklein: ovirt‑3.6.z?
gklein: blocker?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)
ovirt engine setup log (2.88 MB, text/plain)
2016-02-04 04:37 EST, Gil Klein
no flags Details

  None (edit)
Description Gil Klein 2016-02-04 04:37:15 EST
Created attachment 1121052 [details]
ovirt engine setup log

Description of problem:
When upgrading a production system from 3.5.7 engine to 3.6.3, upgrade of DWH is failing
[ INFO  ] Rolling back database schema
[ INFO  ] Clearing Engine database engine
[ INFO  ] Restoring Engine database engine
[ INFO  ] Restoring file '/var/lib/ovirt-engine/backups/engine-20160204111121.qK55vP.dump' to database localhost:engine.
[ INFO  ] Rolling back DWH database schema
[ INFO  ] Clearing DWH database ovirt_engine_history
[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20160204105216-56hx8p.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20160204111532-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Execution of setup failed


Version-Release number of selected component (if applicable):
rhevm-dwh-3.5.5-1.el6ev.noarch -> rhevm-dwh-3.6.2-1.el6ev.noarch 


How reproducible:
100% on the specific system 


Steps to Reproduce:
1. Run engine-setup on the system


Actual results:
[ ERROR ] Execution of setup failed


Expected results:
Upgrade of DWH should succeeded


Additional info:

2016-02-04 11:12:33 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine_dwh.db.schema plugin.execute:941 execute-output: ['/usr/share/ovirt-engine-dwh/dbscripts/schema.sh', '-s', 'localhost', '-p', '5432', '-u', 'ovirt_engine_history', '-d', 'ovirt_engine_history', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20160204105216-56hx8p.log', '-c', 'apply'] stderr:
psql:/usr/share/ovirt-engine-dwh/dbscripts/common_sp.sql:566: NOTICE:  drop cascades to function fn_db_get_async_tasks()
psql:/usr/share/ovirt-engine-dwh/dbscripts/common_sp.sql:566: NOTICE:  drop cascades to function fn_db_get_async_tasks()
psql:/usr/share/ovirt-engine-dwh/dbscripts/create_views_3_0.sql:228: ERROR:  column "pm_ip_address" does not exist
LINE 16:       pm_ip_address as pm_ip_address,
               ^
FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine-dwh/dbscripts/create_views_3_0.sql

2016-02-04 11:12:33 DEBUG otopi.context context._executeMethod:156 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/otopi/context.py", line 146, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine-dwh/db/schema.py", line 364, in _misc
    odwhcons.DBEnv.PGPASS_FILE
  File "/usr/lib/python2.6/site-packages/otopi/plugin.py", line 946, in execute
    command=args[0],
RuntimeError: Command '/usr/share/ovirt-engine-dwh/dbscripts/schema.sh' failed to execute
2016-02-04 11:12:33 ERROR otopi.context context._executeMethod:165 Failed to execute stage 'Misc configuration': Command '/usr/share/ovirt-engine-dwh/dbscripts/schema.sh' failed to execute
Comment 2 Shirly Radco 2016-02-04 05:38:05 EST
This does not make sense and probably an environment issue.
Please supply step to reproduce.

This is the way "pm_ip_address" is represented in the  3_0 views:

cast(NULL as VARCHAR(255)) as pm_ip_address,

for legacy support.
Comment 3 Shirly Radco 2016-02-04 05:41:36 EST
Did you backup the dwh db during the first engine-setup?
Comment 4 Gil Klein 2016-02-04 06:21:17 EST
(In reply to Shirly Radco from comment #3)
> Did you backup the dwh db during the first engine-setup?
No. But I can rerun the flow with the reverted db, and take this backup if it may help somehow (I've attached the original db in comment 1).

I'm really not sure how to reproduce it. This system is a clone of out production and was fully restored with "engine-backup --mode=restore --scope=all"

I can provide you access to the server if it may help.
Comment 5 Yaniv Lavi (Dary) 2016-02-04 08:21:54 EST
(In reply to Gil Klein from comment #4)
> (In reply to Shirly Radco from comment #3)
> > Did you backup the dwh db during the first engine-setup?
> No. But I can rerun the flow with the reverted db, and take this backup if
> it may help somehow (I've attached the original db in comment 1).
> 
> I'm really not sure how to reproduce it. This system is a clone of out
> production and was fully restored with "engine-backup --mode=restore
> --scope=all"
> 
> I can provide you access to the server if it may help.

Are you sure the database is 3.5? I don't think you will be able to recreate this in real upgrade.
Comment 6 Gil Klein 2016-02-04 08:53:01 EST
(In reply to Yaniv Dary from comment #5)
> (In reply to Gil Klein from comment #4)
> > (In reply to Shirly Radco from comment #3)
> > > Did you backup the dwh db during the first engine-setup?
> > No. But I can rerun the flow with the reverted db, and take this backup if
> > it may help somehow (I've attached the original db in comment 1).
> > 
> > I'm really not sure how to reproduce it. This system is a clone of out
> > production and was fully restored with "engine-backup --mode=restore
> > --scope=all"
> > 
> > I can provide you access to the server if it may help.
> 
> Are you sure the database is 3.5? I don't think you will be able to recreate
> this in real upgrade.
Yes, it was restored from a 3.5.7 system.

I can try retry restoring it again, and retry.
Comment 8 Shirly Radco 2016-02-07 06:18:32 EST
Didi,

Can you please take a look?
Comment 9 Yaniv Lavi (Dary) 2016-02-08 08:51:56 EST
It doesn't make since the create views is not from 3.6 (the column is nulled in 3.6) and being run in upgrade. Since in the one in 3.6 this can not happen.
Comment 10 Yaniv Lavi (Dary) 2016-02-08 09:40:08 EST
Please check the integrity of the 3.6 DWH RPM.
Comment 11 Gil Klein 2016-02-08 11:11:39 EST
(In reply to Yaniv Dary from comment #10)
> Please check the integrity of the 3.6 DWH RPM.
Looks like the system got to a state where both old and new DHW pacakges are installed

# rpm -qa|grep dwh
rhevm-dwh-setup-3.6.2-1.el6ev.noarch
rhevm-dwh-3.6.2-1.el6ev.noarch
rhevm-dwh-3.5.5-1.el6ev.noarch
Comment 12 Yedidyah Bar David 2016-02-09 05:28:08 EST
Gil, isn't this a duplicate of bug 1289660?

For reference, the ^C bug is tracked in bug 1130764.
Comment 13 Yaniv Lavi (Dary) 2016-02-10 03:00:29 EST

*** This bug has been marked as a duplicate of bug 1130764 ***

Note You need to log in before you can comment on or make changes to this bug.