Bug 975938

Summary: failed to force remove datacenter cause "ERROR: function uuid_generate_v1() does not exist"
Product: Red Hat Enterprise Virtualization Manager Reporter: Aharon Canan <acanan>
Component: ovirt-engineAssignee: Eli Mesika <emesika>
Status: CLOSED WORKSFORME QA Contact: Pavel Stehlik <pstehlik>
Severity: high Docs Contact:
Priority: urgent    
Version: 3.1.4CC: acanan, acathrow, amureini, bazulay, emesika, hateya, iheim, jkt, lpeer, maci, Rhev-m-bugs, yeylon
Target Milestone: ---Keywords: Regression
Target Release: 3.1.5   
Hardware: Unspecified   
OS: Linux   
Whiteboard: infra
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-08 14:10:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
engine logs
none
Installation script for pg uuid functions for rhevm 3.1 on PG 8.4.x none

Description Aharon Canan 2013-06-19 15:56:36 UTC
Created attachment 763029 [details]
engine logs

Description of problem:
=======================
while trying to "force remove" the datacenter, I got "failed to remove datacenter" message, 

amureini: "seems to me as though force-remove leaves crap in the vds_spm_id_map, so when you move a vds to another cluster, it has two entries, hence the error you're seeing"


in engine logs I see:
=====================
2013-06-19 17:58:36,486 INFO  [org.ovirt.engine.core.bll.UpdateVdsGroupCommand] (ajp-/127.0.0.1:8702-10) [42c1cc23] Running command: UpdateVdsGroupCommand internal: false. Entities affected :  ID: 99408929-82cf-4dc7-a532-9d998063fa95 Typ
e: VdsGroups
2013-06-19 17:59:08,846 WARN  [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-3) [20c62106] Adding new value to configuration cache.
2013-06-19 17:59:08,847 WARN  [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ajp-/127.0.0.1:8702-3) [20c62106] Didnt find the value of BootstrapNodeIDCommand in DB for version general - using default: IDFILE=/etc/vdsm/vdsm.i
d; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUID="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}"
 > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UUID}" ] && echo "${UUID}"; fi
2013-06-19 17:59:08,847 INFO  [org.ovirt.engine.core.utils.hostinstall.VdsInstallerSSH] (ajp-/127.0.0.1:8702-3) SSH execute camel-vdsb.qa.lab.tlv.redhat.com 'IDFILE=/etc/vdsm/vdsm.id; if [ -r "${IDFILE}" ]; then cat "${IDFILE}"; else UUI
D="$(dmidecode -s system-uuid 2> /dev/null | sed -e 's/.*Not.*//' )"; if [ -z "${UUID}" ]; then UUID="$(uuidgen 2> /dev/null)" && mkdir -p "$(dirname "${IDFILE}")" && echo "${UUID}" > "${IDFILE}" && chmod 0644 "${IDFILE}"; fi; [ -n "${UU
ID}" ] && echo "${UUID}"; fi'
2013-06-19 17:59:09,112 INFO  [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-3) [20c62106] Running command: AddVdsCommand internal: false. Entities affected :  ID: 99408929-82cf-4dc7-a532-9d998063fa95 Type: VdsGroups
2013-06-19 17:59:09,124 INFO  [org.ovirt.engine.core.utils.transaction.TransactionSupport] (ajp-/127.0.0.1:8702-3) [20c62106] transaction rolled back
2013-06-19 17:59:09,124 ERROR [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-3) [20c62106] Command org.ovirt.engine.core.bll.AddVdsCommand throw exception: org.springframework.jdbc.BadSqlGrammarException: CallableStatemen
tCallback; bad SQL grammar [{call insertvdsstatic(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}]; nested exception is org.postgresql.util.PSQLException: ERROR: function uuid_generate_v1() does not exist
  Hint: No function matches the given name and argument types. You might need to add explicit type casts.



Version-Release number of selected component (if applicable):
rhevm-genericapi-3.1.0-55.el6ev

How reproducible:
not sure

Steps to Reproduce:
I only tried to remove the datacenter, nothing else before

Actual results:
remove datacenter failed

Expected results:
should succeed 

Additional info:

Comment 3 Allon Mureinik 2013-06-26 14:13:13 UTC
There are two issues here:
1. The missing generate_uuid_v1 - IIRC, there is already a bug on this - Mooli please check this.

2. The unique constraint violation in vds_spm_id_map - this was solved for 3.2 in bug 878344. If this is required in z-stream, please flag accordingly.

I'm leaving this bug open so issue #1 can be checked. IMHO, once it's resolved, this bug can be marked as MODIFIED.

Comment 5 Eli Mesika 2013-06-27 08:38:55 UTC
Created attachment 766006 [details]
Installation script for pg uuid functions for rhevm 3.1 on PG 8.4.x

Comment 7 Aharon Canan 2013-06-27 13:02:29 UTC
I do not have the same testing environment so I am not sure if the result of "select version from schema_version order by id desc limit 1;" really can help

anyway, I got below ouput while running it - 
 version  
----------
 03020440
(1 row)

Comment 8 Barak 2013-06-30 15:25:08 UTC
Aharon,

Can you please supply answer for "psql -U postgres -f uuid-ossp.sql engine"

Comment 9 Aharon Canan 2013-07-01 08:09:46 UTC
Barak, 

the answer already appears in comment #7

Comment 10 Marcel Wysocki 2013-07-01 12:11:22 UTC
running into the same bug on rhev-m 3.1.0-55.el6ev

installing the stored procedures from the attachement works

Comment 11 Eli Mesika 2013-07-01 21:03:48 UTC
Please check if this is a missing ossp_uuid package installation and if so close the bug

Comment 12 Barak 2013-07-04 14:22:05 UTC
(In reply to Aharon Canan from comment #7)
> I do not have the same testing environment so I am not sure if the result of
> "select version from schema_version order by id desc limit 1;" really can
> help
> 
> anyway, I got below ouput while running it - 
>  version  
> ----------
>  03020440
> (1 row)

No it's not ,
The above is the output of 

select version from schema_version order by id desc limit 1

please supply the out put of "psql -U postgres -f uuid-ossp.sql engine"

on the setup you failed on.

Comment 13 Aharon Canan 2013-07-04 14:38:56 UTC
I do not have the same testing environment.
Maybe Marcel who run into it as well can do it as well...

Comment 14 Marcel Wysocki 2013-07-05 10:21:01 UTC
output of "psql -U postgres -f uuid-ossp.sql engine"

was just that its adding the stuff.


my setup was the following:

rhevm 3.0 upgraded to 3.1 ,
later exported the DB, backed up the pki folder
removed everything rhevm, also /var/lib/psql
reinstalled rhevm 3.1  imported the db and restored the pki files

ran into some problem about DB not being owned by engine role, fixed it

later upgraded to 3.1 (rev 55, iirc)

then ran into the uuid generate SP problems when trying to install a new host

so after adding the SPs it worked again

had to do this in 3 environments, including our production so it was quite painful

Comment 15 Eli Mesika 2013-07-07 06:55:37 UTC
(In reply to Marcel Wysocki from comment #14)
> output of "psql -U postgres -f uuid-ossp.sql engine"
> 
> was just that its adding the stuff.
> 
> 
> my setup was the following:
> 
> rhevm 3.0 upgraded to 3.1 ,
> later exported the DB, backed up the pki folder
> removed everything rhevm, also /var/lib/psql

The uuis_ossp functions are not part of the PG installation in 8.x , so if you had removed PG, you will have to reinstall those functions manually.
So, I don't think we have any bug here....

> reinstalled rhevm 3.1  imported the db and restored the pki files
> 
> ran into some problem about DB not being owned by engine role, fixed it
> 
> later upgraded to 3.1 (rev 55, iirc)
> 
> then ran into the uuid generate SP problems when trying to install a new host
> 
> so after adding the SPs it worked again
> 
> had to do this in 3 environments, including our production so it was quite
> painful

Comment 16 Andrew Cathrow 2013-07-08 14:10:23 UTC
Based on comment 15 and based on lack of way to reproduce - we're closing. 
Let's review if we can reproduce and triage further