Bug 1266250 - Make template from cinder-ceph based VM errors out.
Make template from cinder-ceph based VM errors out.
Status: CLOSED CURRENTRELEASE
Product: ovirt-engine
Classification: oVirt
Component: General (Show other bugs)
3.6.0
x86_64 Linux
unspecified Severity high (vote)
: ovirt-3.6.0-ga
: 3.6.0.3
Assigned To: Maor
Ori Gofen
storage
:
Depends On: 1264691
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-24 17:19 EDT by joseph.blasi
Modified: 2016-05-25 21:49 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-27 02:54:45 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
amureini: ovirt‑3.6.0?
rule-engine: planning_ack?
rule-engine: devel_ack+
rule-engine: testing_ack+


Attachments (Terms of Use)
engine log file (48.03 KB, text/plain)
2015-09-24 17:19 EDT, joseph.blasi
no flags Details
rc2 log (187.09 KB, text/plain)
2015-10-14 11:36 EDT, joseph.blasi
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 47408 master MERGED engine: Update command associated entities instantly Never

  None (edit)
Description joseph.blasi 2015-09-24 17:19:02 EDT
Created attachment 1076724 [details]
engine log file

Description of problem:
When I try to make an template from an cinder-ceph based VM it processes for some time and then errors out with "Error while executing action New VM Template: Internal Engine Error"

Version-Release number of selected component (if applicable):


How reproducible:
all the time

Steps to Reproduce:
1. Make a new vm with cinder-ceph
2. Install os and shutdown the VM
3. Make a template from the VM

Actual results:

Get a Error while executing action New VM Template: Internal Engine Error and see the copied volume in the cinder store.

The new volume is able to be booted up and ran if attached to a new VM.

Expected results:
To have the VM show in a Templates list.

Additional info:
Log clipping attached
Comment 1 Allon Mureinik 2015-10-14 03:50:30 EDT
Maor - please take a look?
Comment 2 Allon Mureinik 2015-10-14 03:54:34 EDT
Joseph, thanks for reporting this bug!
Can you specify which RPM version you're using?
Comment 3 Maor 2015-10-14 04:09:02 EDT
looks like it is the same issue described in https://bugzilla.redhat.com/1264677

The transaction got timed out  after 10 minutes
2015-09-24 15:54:46,930 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-113) [51809cd7] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{runAsync='true', vmId='48b3a875-94d2-4239-abf7-e42e0e240843', status='ImageLocked', exitStatus='Normal'}), log id: 13d06319
2015-09-24 15:54:46,933 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-113) [51809cd7] FINISH, SetVmStatusVDSCommand, log id: 13d06319
2015-09-24 16:04:46,994 INFO  [org.ovirt.engine.core.bll.storage.CloneCinderDisksCommand] (pool-7-thread-1) [332a1ca] Running command: CloneCinderDisksCommand internal: true.
Comment 4 joseph.blasi 2015-10-14 11:16:23 EDT
After updating to RC2 templates are showing up in the Templates list.
Comment 5 joseph.blasi 2015-10-14 11:36 EDT
Created attachment 1082881 [details]
rc2 log
Comment 6 joseph.blasi 2015-10-14 11:38:09 EDT
Trying to start a new VM from an Template leads to a long wait and errors out.
Comment 7 Maor 2015-10-15 03:04:53 EDT
It looks like this is the same issue mentioned at https://bugzilla.redhat.com/1264691
Looking at your logs it looks that you got a transaction timeout after 10 minutes:

2015-10-14 10:15:22,780 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-87) [] FINISH, SetVmStatusVDSCommand, log id: 169ec556
2015-10-14 10:25:22,589 ERROR [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (Transaction Reaper Worker 0) [] Transaction rolled-back for command 'org.ovirt.engine.core.bll.AddVmFromTemplateCommand'.
2015-10-14 10:25:22,639 INFO  [org.ovirt.engine.core.bll.storage.CloneCinderDisksCommand] (pool-7-thread-3) [355b7e7a] Running command: CloneCinderDisksCommand internal: true.
2015-10-14 10:25:22,689 INFO  [org.ovirt.engine.core.bll.storage.CloneSingleCinderDiskCommand] (pool-7-thread-4) [38241192] Running command: CloneSingleCinderDiskCommand internal: true.
Comment 8 Maor 2015-10-15 03:06:43 EDT
Joseph,

Can u please change the summary of the bug to indicate this is now an issue of creating a VM from Template, or open a separate bug on this specific issue.
Comment 9 Maor 2015-10-15 03:09:02 EDT
Moti,

That looks like the same issue at https://bugzilla.redhat.com/1264691, (see https://bugzilla.redhat.com/show_bug.cgi?id=1266250#c7).
Probably worth to check this once BZ1264691 will be solved.
Comment 10 Moti Asayag 2015-10-16 11:06:15 EDT
The root cause of this bug is a difference between the behavior of postgres 8.4 and 9.1.

In the described scenarios (and other similar to this issue as we faced on AddVmTemplate) we have the following sequence:

On The main thread:
Open new transaction (either explicit or as default command's scope):
CommandCoordinatorUtil.executeAsyncCommand(...)
  |
  +-- coco.persistCommandAssociatedEntities(...)
      which internally updates command_assoc_entities table

On the submitted thread we calls:
  CommandBase.setCommandStatus()
  which updates the command_entities table.

The first update of 'command_assoc_entities' table is done on transaction Tx1 a, which is open. While  Tx1 is open, the update for 'command_entities' is executed in other transaction Tx2.

In postgres 9.1, that scenario is permitted without any locking on the command_entities table.

However in postgres 8.4, an update for 'command_entities' isn't permitted, unless the referred table 'command_assoc_entities' is closed.

In order to make the code compliant for both postgres versions, the first update to 'command_assoc_entities' should be immediately committed.
Comment 11 Red Hat Bugzilla Rules Engine 2015-10-19 06:57:11 EDT
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Comment 12 Maor 2015-10-19 11:26:41 EDT
Verified with patch 47408.
Moving to modify
Comment 13 Allon Mureinik 2015-10-29 05:03:08 EDT
(In reply to Maor from comment #12)
> Verified with patch 47408.
> Moving to modify
Moving to ON_QA by that logic.
Comment 14 Red Hat Bugzilla Rules Engine 2015-11-02 07:27:04 EST
Bug tickets that are moved to testing must have target release set to make sure tester knows what to test. Please set the correct target release before moving to ON_QA.
Comment 15 Ori Gofen 2015-11-11 10:02:22 EST
Verified on rhevm-3.6-0.3
Comment 16 Sandro Bonazzola 2015-11-27 02:54:45 EST
Since oVirt 3.6.0 has been released, moving from verified to closed current release.

Note You need to log in before you can comment on or make changes to this bug.