Bug 892532

Summary: [ovirt-engine-backend] DB upgrade from 3.0 to 3.1 fails
Product: Red Hat Enterprise Virtualization Manager Reporter: Eli Mesika <emesika>
Component: ovirt-engineAssignee: Eli Mesika <emesika>
Status: CLOSED ERRATA QA Contact: Pavel Stehlik <pstehlik>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.2.0CC: bazulay, cboyle, dyasny, iheim, italkohe, lpeer, mpavlik, pep, Rhev-m-bugs, sgrinber, yeylon, ykaul, yzaslavs
Target Milestone: ---Keywords: TestBlocker, ZStream
Target Release: 3.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: infra
Fixed In Version: sf7 Doc Type: Bug Fix
Doc Text:
--no doc text required
Story Points: ---
Clone Of:
: 893012 909948 (view as bug list) Environment:
Last Closed: 2013-06-10 21:31:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 880665, 890324, 893012, 909948, 915537    
Attachments:
Description Flags
upgrade si26.1 -> SF5
none
ovirt-engine-upgrade_2013_02_07_14_26_15.log.tar
none
ovirt-engine-upgrade_2013_02_07_14_26_15.log.tar
none
ovirt-engine-upgrade_2013_02_07_14_26_15.log.tar
none
sosreport-LogCollector-admin-20130207143840-eec8.tar none

Description Eli Mesika 2013-01-07 10:19:21 UTC
Description of problem:
After running on about 250 customer databases we have got errors from the following types:

1) psql --set ON_ERROR_STOP=1 -U postgres -f ./restore.sql rhevm


2)psql:upgrade/03_01_0100_migrate_templates_to_vm_static.sql:120: ERROR:  constraint "vm_templates_vm_static" of relation "vm_static" does not exist
CONTEXT:  SQL statement "ALTER TABLE vm_static DROP CONSTRAINT vm_templates_vm_static"


3)psql:upgrade/03_01_0070_populate_disks_table.sql:107: ERROR:  duplicate key value violates unique constraint "pk_disks"
CONTEXT:  SQL statement "INSERT INTO disks (disk_id, status, internal_drive_mapping, active_image_id, disk_type, disk_interface, wipe_after_delete, propagate_errors) VALUES ( $1 , CASE WHEN  $2  = 1 THEN 'OK' WHEN  $3  = 2 THEN 'LOCKED' WHEN  $4  = 3 THEN 'INVALID' WHEN  $5  = 4 THEN 'ILLEGAL' ELSE 'Unassigned' END, CAST( $6  AS INTEGER),  $7 , CASE WHEN  $8  = 1 THEN 'System' WHEN  $9  = 2 THEN 'Data' WHEN  $10  = 3 THEN 'Shared' WHEN  $11  = 4 THEN 'Swap' WHEN  $12  = 5 THEN 'Temp' ELSE 'Unassigned' END, CASE WHEN  $13  = 1 THEN 'SCSI' WHEN  $14  = 2 THEN 'VirtIO' ELSE 'IDE' END,  $15 , CASE WHEN  $16  = 1 THEN 'On' ELSE 'Off' END)"
PL/pgSQL function "__temp_upgrade_populatediskstable" line 16 at SQL statement


4)psql:upgrade/03_01_0950_add_permissions_on_storage_to_users.sql:122: ERROR:  duplicate key value violates unique constraint "idx_combined_ad_role_object"
CONTEXT:  SQL statement "INSERT INTO permissions (id, role_id, ad_element_id, object_id, object_type_id) (SELECT uuid_generate_v1(),  $1 ,  $2 , storage_pool_iso_map.storage_id,  $3  FROM vds_groups INNER JOIN storage_pool_iso_map ON vds_groups.storage_pool_id = storage_pool_iso_map.storage_pool_id WHERE vds_groups.vds_group_id =  $4 )"
PL/pgSQL function "__temp_insert_storage_permissions" line 60 at SQL statement

Follow upstream commit : 0a736cabab1d9c442cab34ecb96ad175544f7fd3

5)psql:upgrade/03_01_0100_migrate_templates_to_vm_static.sql:120: ERROR:  column "entity_type" contains null values
CONTEXT:  SQL statement "ALTER TABLE vm_static ALTER COLUMN entity_type SET NOT NULL"
PL/pgSQL function "__temp_upgrade_mergetemplatestovmstatic" line 13 at SQL statement

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1.dropdb 
2.create db with 3.0 schema
3.restore db from tar file
4.run upgrade
  
Actual results:
upgrade fails

Expected results:
upgrade should succeed on all databases 

Additional info:

Comment 2 Eli Mesika 2013-01-07 10:24:12 UTC
fixed in commits : 36c65d8, 877b0a4, 2e57263

Comment 9 Eli Mesika 2013-01-07 17:04:05 UTC
http://gerrit.ovirt.org/#/c/10735/

Comment 11 Eli Mesika 2013-01-09 10:27:56 UTC
fix in commit: 1280938

Comment 12 Eli Mesika 2013-01-23 13:14:19 UTC
*** Bug 902686 has been marked as a duplicate of this bug. ***

Comment 15 Eli Mesika 2013-01-28 11:57:50 UTC
fixed in commit: dff85e8

Comment 16 Martin Pavlik 2013-01-31 15:29:27 UTC
Upgrade si26.1 -> SF5 failed

attached:
logcollector

ovirt-engine-upgrade_2013_01_31_16_19_39.log

Running  upgrade sql script upgrade/03_02_0330_add_vds_event_notifications.sql ...
Creating views...

2013-01-31 16:21:05::DEBUG::common_utils::473::root:: stderr = psql:create_views.sql:382: ERROR:  column vm_templates.is_smartcard_enabled does not exist
LINE 34:        vm_templates.is_smartcard_enabled as is_smartcard_ena...
                ^

2013-01-31 16:21:05::DEBUG::common_utils::474::root:: retcode = 3
2013-01-31 16:21:05::ERROR::rhevm-upgrade::1078::root:: Traceback (most recent call last):
  File "/usr/bin/rhevm-upgrade", line 1065, in main
    runFunc([db.update], MSG_INFO_DB_UPDATE)
  File "/usr/bin/rhevm-upgrade", line 594, in runFunc
    func()
  File "/usr/bin/rhevm-upgrade", line 429, in update
    output, rc = utils.execCmd(cmdList=cmd, failOnError=True, msg=MSG_ERROR_UPDATE_DB)
  File "/usr/share/ovirt-engine/scripts/common_utils.py", line 477, in execCmd
    raise Exception(msg)
Exception: Error: Database update failed

2013-01-31 16:21:05::ERROR::rhevm-upgrade::1079::root:: Rolling back update
2013-01-31 16:21:05::ERROR::rhevm-upgrade::563::root:: PKI: cannot remove '/etc/httpd/conf.d/ssl.conf.tmp'

Comment 17 Martin Pavlik 2013-01-31 15:30:12 UTC
Created attachment 691007 [details]
upgrade si26.1 -> SF5

Comment 18 Eli Mesika 2013-01-31 16:46:07 UTC
I am moving it to QA again since this commit was not included in SF5
Please reacheck in SF6

Comment 19 Martin Pavlik 2013-02-07 13:40:10 UTC
Upgrade si26.1 -> SF6 failed

attached:
upgradelog
ovirt-engine-upgrade_2013_02_07_14_26_15.log

logcollector
sosreport-LogCollector-admin-20130207143840-eec8.tar.xz

2013-02-07 14:27:12::DEBUG::common_utils::473::root:: stderr = psql:create_views.sql:654: ERROR:  column vds_static.sshkeyfingerprint does not exist
LINE 29:    vds_static.recoverable as recoverable, vds_static.sshKeyF...
                                                   ^

2013-02-07 14:27:12::DEBUG::common_utils::474::root:: retcode = 3
2013-02-07 14:27:12::ERROR::rhevm-upgrade::1102::root:: Traceback (most recent call last):
  File "/usr/bin/rhevm-upgrade", line 1089, in main
    runFunc([db.update], MSG_INFO_DB_UPDATE)
  File "/usr/bin/rhevm-upgrade", line 594, in runFunc
    func()
  File "/usr/bin/rhevm-upgrade", line 429, in update
    output, rc = utils.execCmd(cmdList=cmd, failOnError=True, msg=MSG_ERROR_UPDATE_DB)
  File "/usr/share/ovirt-engine/scripts/common_utils.py", line 477, in execCmd
    raise Exception(msg)
Exception: Error: Database update failed

Comment 20 Martin Pavlik 2013-02-07 13:41:15 UTC
Created attachment 694481 [details]
ovirt-engine-upgrade_2013_02_07_14_26_15.log.tar

Comment 21 Martin Pavlik 2013-02-07 13:43:06 UTC
Created attachment 694482 [details]
ovirt-engine-upgrade_2013_02_07_14_26_15.log.tar

Comment 22 Martin Pavlik 2013-02-07 13:43:52 UTC
Created attachment 694484 [details]
ovirt-engine-upgrade_2013_02_07_14_26_15.log.tar

Comment 23 Martin Pavlik 2013-02-07 13:45:52 UTC
Created attachment 694486 [details]
sosreport-LogCollector-admin-20130207143840-eec8.tar

Comment 26 Martin Pavlik 2013-03-19 13:16:24 UTC
si27.4 to sf10.1 works fine

Comment 27 Cheryn Tan 2013-04-03 06:53:32 UTC
This bug is currently attached to errata RHEA-2013:14491. If this change is not to be documented in the text for this errata please either remove it from the errata, set the requires_doc_text flag to minus (-), or leave a "Doc Text" value of "--no tech note required" if you do not have permission to alter the flag.

Otherwise to aid in the development of relevant and accurate release documentation, please fill out the "Doc Text" field above with these four (4) pieces of information:

* Cause: What actions or circumstances cause this bug to present.

* Consequence: What happens when the bug presents.

* Fix: What was done to fix the bug.

* Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')

Once filled out, please set the "Doc Type" field to the appropriate value for the type of change made and submit your edits to the bug.

For further details on the Cause, Consequence, Fix, Result format please refer to:

https://bugzilla.redhat.com/page.cgi?id=fields.html#cf_release_notes

Thanks in advance.

Comment 28 errata-xmlrpc 2013-06-10 21:31:33 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0888.html