Hide Forgot
Description of problem: Customer was trying to perform standard upgrade to his RHEV environment (minor version number, NOT 3.1 to 3.2). Upgrade failed due to datbase related issue. From the sosreport it appears that some RPMs are on 3.2.3, while others are at 3.2.0 or 3.2.1. How reproducible: Unknown Actual results: Database change fails, upgrade script rollsback and exits with error Expected results: Cleqan upgrade to 3.2.3 Additional info: From upgrade log: 2013-10-21 15:28:29::DEBUG::rhevm-upgrade::454::root:: DB Update started 2013-10-21 15:28:29::DEBUG::common_utils::453::root:: Executing command --> '/usr/share/ovirt-engine/dbscripts/ upgrade.sh -l /var/log/ovirt-engine/engine-db-upgrade-2013_10_21_15_28_29.log -s localhost -p 5432 -u engine -d engine_2013_10_21_15_25_27' in working directory '/usr/share/ovirt-engine/dbscripts' 2013-10-21 15:28:30::DEBUG::common_utils::491::root:: output = /usr/share/ovirt-engine/dbscripts /usr/share/ovi rt-engine/dbscripts upgrade script detected a change in Config, View or Stored Procedure... 2013-10-21 15:28:30::DEBUG::common_utils::492::root:: stderr = psql:common_sp.sql:20: ERROR: must be owner of function fn_db_add_column 2013-10-21 15:28:30::DEBUG::common_utils::493::root:: retcode = 3 2013-10-21 15:28:30::ERROR::rhevm-upgrade::1455::root:: Traceback (most recent call last): File "/usr/bin/rhevm-upgrade", line 1442, in main runFunc([db.update], MSG_INFO_DB_UPDATE) File "/usr/bin/rhevm-upgrade", line 757, in runFunc func() File "/usr/bin/rhevm-upgrade", line 473, in update output, rc = utils.execCmd(cmdList=cmd, failOnError=True, msg=MSG_ERROR_UPDATE_DB) File "/usr/share/ovirt-engine/scripts/common_utils.py", line 496, in execCmd raise Exception(msg) Exception: Error: Database update failed 2013-10-21 15:28:30::ERROR::rhevm-upgrade::1456::root:: Rolling back update 2013-10-21 15:28:30::DEBUG::rhevm-upgrade::404::root:: DB Restore started $ grep rhevm installed-rpms rhevm-3.2.3-0.42.el6ev.noarch Mon 21 Oct 2013 03:33:40 PM CDT rhevm-backend-3.2.3-0.42.el6ev.noarch Mon 21 Oct 2013 03:33:29 PM CDT rhevm-cli-3.2.0.9-1.el6ev.noarch Tue 10 Sep 2013 11:14:02 AM CDT rhevm-config-3.2.3-0.42.el6ev.noarch Mon 21 Oct 2013 03:33:28 PM CDT rhevm-dbscripts-3.2.3-0.42.el6ev.noarch Mon 21 Oct 2013 03:33:40 PM CDT rhevm-doc-3.2.1-2.el6eng.noarch Tue 10 Sep 2013 11:13:48 AM CDT rhevm-dwh-3.2.1-2.el6ev.noarch Wed 11 Sep 2013 09:59:36 AM CDT rhevm-genericapi-3.2.3-0.42.el6ev.noarch Mon 21 Oct 2013 03:33:40 PM CDT rhevm-image-uploader-3.2.2-2.el6ev.noarch Tue 10 Sep 2013 11:13:16 AM CDT rhevm-iso-uploader-3.2.2-3.el6ev.noarch Tue 10 Sep 2013 11:13:16 AM CDT rhevm-log-collector-3.2.2-4.el6ev.noarch Tue 10 Sep 2013 11:13:16 AM CDT rhevm-notification-service-3.2.3-0.42.el6ev.noarch Mon 21 Oct 2013 03:33:40 PM CDT rhevm-reports-3.2.1-6.el6ev.noarch Wed 11 Sep 2013 10:00:10 AM CDT rhevm-restapi-3.2.3-0.42.el6ev.noarch Mon 21 Oct 2013 03:33:29 PM CDT rhevm-sdk-3.2.1.1-1.el6ev.noarch Tue 10 Sep 2013 11:12:54 AM CDT rhevm-setup-3.2.3-0.43.el6ev.noarch Fri 18 Oct 2013 08:16:45 AM CDT rhevm-spice-client-x64-cab-3.2-13.el6ev.noarch Tue 10 Sep 2013 11:15:29 AM CDT rhevm-spice-client-x86-cab-3.2-13.el6ev.noarch Tue 10 Sep 2013 11:15:29 AM CDT rhevm-tools-common-3.2.3-0.42.el6ev.noarch Mon 21 Oct 2013 03:33:28 PM CDT rhevm-userportal-3.2.3-0.42.el6ev.noarch Mon 21 Oct 2013 03:33:40 PM CDT rhevm-webadmin-portal-3.2.3-0.42.el6ev.noarch Mon 21 Oct 2013 03:33:55 PM CDT
ERROR: must be owner of function fn_db_add_column for some reason you have objects in database that are not owned by the engine user. had this environment restored from backup or database manually updated?
flagging for tracking until analyzed. also setting needinfo for comment 1
(In reply to Alon Bar-Lev from comment #1) > ERROR: must be owner of function fn_db_add_column > > for some reason you have objects in database that are not owned by the > engine user. had this environment restored from backup or database manually > updated? From the customer: This box was restored from a another box that had a failed raid controller. I installed a fresh RH 6.4 on a newer server and then installed the latest rhevm 3.2 software then I followed the backup/restore onto a different machine article. It was never upgraded from a rhevm 3.1 server. Every time that I moved from 3.0 to 3.1 to 3.2 , I installed a fresh OS & rhevmX.X then "exported" the VM's then imported them into the new environment.
OK, this is known issue, the procedure mistakenly used the postgres user instead of the engine user. We are working on a script[1] that will enable to change the owner of all object to engine user. [1] http://gerrit.ovirt.org/#/c/18682/
i assume we don't need customer to wait for the z-stream of this, rather just run the script when its ready and continue with their upgrade.
(In reply to Itamar Heim from comment #5) > i assume we don't need customer to wait for the z-stream of this, rather > just run the script when its ready and continue with their upgrade. right. eli can pack up something workable. it is not z-stream it is part of 3.3 as toolbox for these cases.
(In reply to Itamar Heim from comment #5) > i assume we don't need customer to wait for the z-stream of this, rather > just run the script when its ready and continue with their upgrade. Itamar, Alon, I'm happy to pass this on to my customer as a resolution, thank you for pointing this out. Will this bug be updated when that script has been given the OK to be run by the customer, or is this something that they are currently able to use? Thanks, Wallace
Hi Eli, Can you please provide workable script for 3.2? Thanks.
Alon, Eli, Thank you for your help with this so far. The customer now has a second case open with us that also seems to be linked to this, and we're hoping to get a time frame on when we could get a copy of this script. Thanks, Wallace
Created attachment 817189 [details] dbutils.tar.gz Backup database. Extract attachment into temp directory. Execute: # PGPASSWORD=<postgres user password> ./changedbowner.sh -d engine -f postgres -t engine Provided the database name is engine and the user name is engine, you can check this at /etc/ovirt-engine/.pgpass Eli, Please ACK.
(In reply to Alon Bar-Lev from comment #10) > Created attachment 817189 [details] > dbutils.tar.gz > > Backup database. > > Extract attachment into temp directory. > > Execute: > > # PGPASSWORD=<postgres user password> ./changedbowner.sh -d engine -f > postgres -t engine > > Provided the database name is engine and the user name is engine, you can > check this at /etc/ovirt-engine/.pgpass > > Eli, > > Please ACK. ACK
As far as I am concerned , this issue was resolved by sending the changedbowner.sh to the customer (see comment 10)
(In reply to Eli Mesika from comment #12) > As far as I am concerned , this issue was resolved by sending the > changedbowner.sh to the customer (see comment 10) Is there no way to automate this during a 3.2.z upgrade?
since this seems to solved, can you close it as "CLOSED CURRENT RELEASE?" (according to comment 12)