Created attachment 1052033 [details] engine logs Description of problem: Upgrade of oVirt 3.5.3 to oVirt 3.6 master fails with error during taskcleaner execution: Failed to execute stage 'Setup validation': Failed to clear zombie commands. Please access support in attempt to resolve the problem Following error appears in corresponding setup log: ERROR: function deleteallcommandswithzombietasks() does not exist LINE 1: SELECT DeleteAllCommandsWithZombieTasks(); ^ HINT: No function matches the given name and argument types. You might need to add explicit type casts. FATAL: Cannot execute sql command: --command=SELECT DeleteAllCommandsWithZombieTasks(); 2015-07-14 17:25:55 DEBUG otopi.context context._executeMethod:155 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 145, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 458, in _validateZombies self._clearZombies() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 146, in _clearZombies 'Failed to clear zombie commands. ' Version-Release number of selected component (if applicable): 3.5.3.1 How reproducible: 100% Steps to Reproduce: 1. Install oVirt 3.5.3 2. Run engine-setup 3. Install oVirt 3.6 master 4. Run engine-setup Actual results: engine-setup fails after upgrade to oVirt 3.6 master Expected results: engine-setup is completed successfully after upgrade to oVirt 3.6 master Additional info:
Created attachment 1057004 [details] log for upgrade 3.6-3 -> 3.6-4
It fails on upgrade between minor releases too. ovirt 3.6.0-3 -> 3.6.0-4 I was told that may be related to unfinished tasks in database. I was able to find two stucked tasks there (Removing XYZ from system), both in 'Finalizing' stage. See attached logs for this case.
The database was in a clean state (just after successful engine-setup) in original scenario, so definitely without any unfinished tasks. Looking at the error it seems that stored procedure DeleteAllCommandsWithZombieTasks was removed/renamed during 3.6 development, but it's still called during upgrade.
(In reply to Lukas Bednar from comment #2) > It fails on upgrade between minor releases too. > ovirt 3.6.0-3 -> 3.6.0-4 > > I was told that may be related to unfinished tasks in database. > I was able to find two stucked tasks there (Removing XYZ from system), both > in 'Finalizing' stage. That is a separate issue which deserves investigation on its own. There should be no orphaned/unfinished tasks left. Oved, would you please look into that too and perhaps track it separately?
(In reply to Michal Skrivanek from comment #4) > (In reply to Lukas Bednar from comment #2) > > It fails on upgrade between minor releases too. > > ovirt 3.6.0-3 -> 3.6.0-4 > > > > I was told that may be related to unfinished tasks in database. > > I was able to find two stucked tasks there (Removing XYZ from system), both > > in 'Finalizing' stage. > > That is a separate issue which deserves investigation on its own. There > should be no orphaned/unfinished tasks left. Oved, would you please look > into that too and perhaps track it separately? As far as I know this was fixed already by Liran. Perhaps they indeed weren't finished? CC-ing Liran.
No need for needinfo here on Liran, as it is not the issue reported in the original. Is upgrade from ovirt 3.5 to latest 3.6 master work? If so, this should be moved to verified. Now, we did some fixed with regards to stuck jobs on 3.6. That has nothing to do with 3.5 to 3.6 upgrade. An issue upgrading 3.6.0-3 to 3.6 master can't be an urgent issue.... So, please verify the original use-case.
Lukas, if there is an issue between 3.6.0-3 to 3.6.0-master,please open a separate bug on that.
I can confirm that I also had this issue when upgrading from 3.5.3 to 3.6.0-master. Only way around was to follow instructions here: https://bugzilla.redhat.com/show_bug.cgi?id=1235707#c9
The missing sql functions issue? When did you take master? Today?
Examining the issue, it was introduced by commit: https://gerrit.ovirt.org/#/c/42943 Comment by Eli specifically warns from that, but Didi replies that it won't break the upgrade Question in: https://gerrit.ovirt.org/#/c/42943/3/packaging/setup/dbutils/taskcleaner_sp_3_5.sql Explanation in: https://gerrit.ovirt.org/#/c/42943/3/packaging/setup/dbutils/taskcleaner.sh Didi, seems like it does break. Currently, no one creates this stored procedure. Can you take a look?
Sorry :-(
This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
verified with rhevm-setup-plugins-3.6.0-1.el6ev.noarch
Since oVirt 3.6.0 has been released, moving from verified to closed current release.