Bug 1304658

Summary: [Upgrade][DWH] Upgrade 3.5->3.6 failed - column "pm_ip_address" does not exist
Product: [oVirt] ovirt-engine-dwh Reporter: Gil Klein <gklein>
Component: SetupAssignee: Shirly Radco <sradco>
Status: CLOSED DUPLICATE QA Contact: Pavel Stehlik <pstehlik>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 3.6.3CC: bugs, didi, ylavi
Target Milestone: ovirt-3.6.3Keywords: Regression
Target Release: ---Flags: gklein: ovirt-3.6.z?
gklein: blocker?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-10 08:00:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1285700    
Attachments:
Description Flags
ovirt engine setup log none

Description Gil Klein 2016-02-04 09:37:15 UTC
Created attachment 1121052 [details]
ovirt engine setup log

Description of problem:
When upgrading a production system from 3.5.7 engine to 3.6.3, upgrade of DWH is failing
[ INFO  ] Rolling back database schema
[ INFO  ] Clearing Engine database engine
[ INFO  ] Restoring Engine database engine
[ INFO  ] Restoring file '/var/lib/ovirt-engine/backups/engine-20160204111121.qK55vP.dump' to database localhost:engine.
[ INFO  ] Rolling back DWH database schema
[ INFO  ] Clearing DWH database ovirt_engine_history
[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20160204105216-56hx8p.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20160204111532-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Execution of setup failed


Version-Release number of selected component (if applicable):
rhevm-dwh-3.5.5-1.el6ev.noarch -> rhevm-dwh-3.6.2-1.el6ev.noarch 


How reproducible:
100% on the specific system 


Steps to Reproduce:
1. Run engine-setup on the system


Actual results:
[ ERROR ] Execution of setup failed


Expected results:
Upgrade of DWH should succeeded


Additional info:

2016-02-04 11:12:33 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine_dwh.db.schema plugin.execute:941 execute-output: ['/usr/share/ovirt-engine-dwh/dbscripts/schema.sh', '-s', 'localhost', '-p', '5432', '-u', 'ovirt_engine_history', '-d', 'ovirt_engine_history', '-l', '/var/log/ovirt-engine/setup/ovirt-engine-setup-20160204105216-56hx8p.log', '-c', 'apply'] stderr:
psql:/usr/share/ovirt-engine-dwh/dbscripts/common_sp.sql:566: NOTICE:  drop cascades to function fn_db_get_async_tasks()
psql:/usr/share/ovirt-engine-dwh/dbscripts/common_sp.sql:566: NOTICE:  drop cascades to function fn_db_get_async_tasks()
psql:/usr/share/ovirt-engine-dwh/dbscripts/create_views_3_0.sql:228: ERROR:  column "pm_ip_address" does not exist
LINE 16:       pm_ip_address as pm_ip_address,
               ^
FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine-dwh/dbscripts/create_views_3_0.sql

2016-02-04 11:12:33 DEBUG otopi.context context._executeMethod:156 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/otopi/context.py", line 146, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine-dwh/db/schema.py", line 364, in _misc
    odwhcons.DBEnv.PGPASS_FILE
  File "/usr/lib/python2.6/site-packages/otopi/plugin.py", line 946, in execute
    command=args[0],
RuntimeError: Command '/usr/share/ovirt-engine-dwh/dbscripts/schema.sh' failed to execute
2016-02-04 11:12:33 ERROR otopi.context context._executeMethod:165 Failed to execute stage 'Misc configuration': Command '/usr/share/ovirt-engine-dwh/dbscripts/schema.sh' failed to execute

Comment 2 Shirly Radco 2016-02-04 10:38:05 UTC
This does not make sense and probably an environment issue.
Please supply step to reproduce.

This is the way "pm_ip_address" is represented in the  3_0 views:

cast(NULL as VARCHAR(255)) as pm_ip_address,

for legacy support.

Comment 3 Shirly Radco 2016-02-04 10:41:36 UTC
Did you backup the dwh db during the first engine-setup?

Comment 4 Gil Klein 2016-02-04 11:21:17 UTC
(In reply to Shirly Radco from comment #3)
> Did you backup the dwh db during the first engine-setup?
No. But I can rerun the flow with the reverted db, and take this backup if it may help somehow (I've attached the original db in comment 1).

I'm really not sure how to reproduce it. This system is a clone of out production and was fully restored with "engine-backup --mode=restore --scope=all"

I can provide you access to the server if it may help.

Comment 5 Yaniv Lavi 2016-02-04 13:21:54 UTC
(In reply to Gil Klein from comment #4)
> (In reply to Shirly Radco from comment #3)
> > Did you backup the dwh db during the first engine-setup?
> No. But I can rerun the flow with the reverted db, and take this backup if
> it may help somehow (I've attached the original db in comment 1).
> 
> I'm really not sure how to reproduce it. This system is a clone of out
> production and was fully restored with "engine-backup --mode=restore
> --scope=all"
> 
> I can provide you access to the server if it may help.

Are you sure the database is 3.5? I don't think you will be able to recreate this in real upgrade.

Comment 6 Gil Klein 2016-02-04 13:53:01 UTC
(In reply to Yaniv Dary from comment #5)
> (In reply to Gil Klein from comment #4)
> > (In reply to Shirly Radco from comment #3)
> > > Did you backup the dwh db during the first engine-setup?
> > No. But I can rerun the flow with the reverted db, and take this backup if
> > it may help somehow (I've attached the original db in comment 1).
> > 
> > I'm really not sure how to reproduce it. This system is a clone of out
> > production and was fully restored with "engine-backup --mode=restore
> > --scope=all"
> > 
> > I can provide you access to the server if it may help.
> 
> Are you sure the database is 3.5? I don't think you will be able to recreate
> this in real upgrade.
Yes, it was restored from a 3.5.7 system.

I can try retry restoring it again, and retry.

Comment 8 Shirly Radco 2016-02-07 11:18:32 UTC
Didi,

Can you please take a look?

Comment 9 Yaniv Lavi 2016-02-08 13:51:56 UTC
It doesn't make since the create views is not from 3.6 (the column is nulled in 3.6) and being run in upgrade. Since in the one in 3.6 this can not happen.

Comment 10 Yaniv Lavi 2016-02-08 14:40:08 UTC
Please check the integrity of the 3.6 DWH RPM.

Comment 11 Gil Klein 2016-02-08 16:11:39 UTC
(In reply to Yaniv Dary from comment #10)
> Please check the integrity of the 3.6 DWH RPM.
Looks like the system got to a state where both old and new DHW pacakges are installed

# rpm -qa|grep dwh
rhevm-dwh-setup-3.6.2-1.el6ev.noarch
rhevm-dwh-3.6.2-1.el6ev.noarch
rhevm-dwh-3.5.5-1.el6ev.noarch

Comment 12 Yedidyah Bar David 2016-02-09 10:28:08 UTC
Gil, isn't this a duplicate of bug 1289660?

For reference, the ^C bug is tracked in bug 1130764.

Comment 13 Yaniv Lavi 2016-02-10 08:00:29 UTC

*** This bug has been marked as a duplicate of bug 1130764 ***