I've got a problem where since upgrading to 4.0.5 I am still unable to create a vm from a template. The issue now is that it seems to be stuck in a loop trying to and failing to create the vm, which is making my logs explode with the following (which is still going, I've tried taskcleaner.sh and deleting from the db and it keeps filling up) 2016-11-24 14:35:55,466 ERROR [org.ovirt.engine.core.bll.storage.disk.cinder.RemoveCinderDiskCommand] (DefaultQuartzScheduler1) [216b9d17] Ending command 'org.ovirt.engine.core.bll.storage.disk.cinder.RemoveCinderDiskCommand' with failure. 2016-11-24 14:35:55,466 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (DefaultQuartzScheduler1) [216b9d17] transaction rolled back 2016-11-24 14:35:55,468 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler1) [216b9d17] Failed invoking callback end method 'onFailed' for command '9b6f4dc2-3331-4838-bad6-d607b4123b0d' with exception 'Index: 0, Size: 0', the callback is marked for end method retries 2016-11-24 14:35:55,473 ERROR [org.ovirt.engine.core.bll.storage.disk.cinder.RemoveCinderDiskCommand] (DefaultQuartzScheduler1) [1c709434] Ending command 'org.ovirt.engine.core.bll.storage.disk.cinder.RemoveCinderDiskCommand' with failure. 2016-11-24 14:35:55,473 INFO [org.ovirt.engine.core.utils.transaction.TransactionSupport] (DefaultQuartzScheduler1) [1c709434] transaction rolled back 2016-11-24 14:35:55,475 ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] (DefaultQuartzScheduler1) [1c709434] Failed invoking callback end method 'onFailed' for command '145b5c75-a9e1-47a6-94c4-1f6fa81f4bb3' with exception 'Index: 0, Size: 0', the callback is marked for end method retries And in the engine UI "Failed to complete VM fromtemplate-1 creation." thousands of times. The issue started I believe with this sql error ERROR: duplicate key value violates unique constraint "uq_command_id_entity_id" DETAIL: Key (command_id, entity_id, entity_type, snapshot_type)=(43466eb8-cb63-4b3d-94be-a674a9946771, 6e8bc30f-7742-46c3-9495-7def0c47ad05, org.ovirt.engine.core.common.businessentities.VmDynamic, 2) already exists. CONTEXT: SQL statement "INSERT INTO business_entity_snapshot ( id, command_id, command_type, entity_id, entity_type, entity_snapshot, snapshot_class, snapshot_type, insertion_order ) VALUES ( v_id, v_command_id, v_command_type, v_entity_id, v_entity_type, v_entity_snapshot, v_snapshot_class, v_snapshot_type, v_insertion_order )" PL/pgSQL function insert_entity_snapshot(uuid,uuid,character varying,character varying,character varying,text,character varying,integer,integer) line 4 at SQL statement STATEMENT: select * from insert_entity_snapshot($1, $2, $3, $4, $5, $6, $7, $8, $9) as result Any insight on how I can stop the persistent error would be appreciated, as it is also preventing me from running engine-setup due to running jobs.
Hi Jordan, can you please attach the engine.log file? we need a bit more of logs to determine the issue. thanks.
Created attachment 1225395 [details] enginelog Here's the last 500 lines of my engine log, I've got several gigs just like this. I am unable to get meaningful information out of it due to this.
Thanks, I'd appreciate if you could add more logs from the time the issue has started (the sql exception you quoted). In order to stop the persistent error you can clear the command_entities table. I'd suggest to do it only after you've made a db backup when there are no running operations and the engine is down. thanks, Liron
Created attachment 1225398 [details] original error engine log Found the original bit where the error was along with the next 5 minutes or so of logs. I hope this is more helpful.
Thanks Jordan. I'd appreciate if you could upload the logs starting at 2016-11-23 18:30 - the beginning of the execution is missing. thanks, Liron
Created attachment 1225414 [details] lines previous to provided logs in ovirt-engine.log.origin Hi Liron, This is what precedes those lines. Thank you for the tip, I had previously tried clearing command_entities but with ovirt-engine running it kept repopulating, this did the trick! Thanks, Jordan
Now that I've got quiet logs again, I can restart the create template, create vm from template process to get a fresh look at it to see how reproducible it is if you would like.
Thanks Jordan, I really appreciate it, but right now it's not needed - will update you if it will be. The log in comment #6 is between 18:30 and 18:32, While the execution started in around 18:37 - so we have a gap. I will be helpful if you could paste the entire log between 2016-11-23 18:30 and 2016-11-23 18:45. thanks, Liron.
Created attachment 1225451 [details] engine There is nothing before the 18:31 entry until unrelated things at 18:12, attached is everything between 18:30 and 18:45
Thanks Jordan, as it seems we have few issues here: 1. flow related issue - the context passed to the CloneSingleCinderDisk commands - the compensation context has the parent commmand id leading to the FK violation. The command should be executed with the compensation context of its own. 2. infrastructural issue - the AddVmFromTemplate command isn't marked as one requires compensation context, yet one is still created (that's handled by the infrastructure). 3. AddVmFromTemplate should be using a compensation context (so it should be marked as command using it). I'm cloning this bug to a new bug to tackle to infrastructural issues, this flow should tackle the flow related issues.
Thanks Liron, Daniel please take it from here
4.0.6 has been the last oVirt 4.0 release, please re-target this bug.
*** Bug 1421293 has been marked as a duplicate of this bug. ***
VM creation from template with Cinder disks works as expected on latest build (due to recent infrastructural CoCo changes). Moving to MODIFIED for verification.
verified on 4.2.1.6-0.1.el7
This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017. Since the problem described in this bug report should be resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.