Bug 1266250

Summary: Make template from cinder-ceph based VM errors out.
Product: [oVirt] ovirt-engine Reporter: joseph.blasi
Component: GeneralAssignee: Maor <mlipchuk>
Status: CLOSED CURRENTRELEASE QA Contact: Ori Gofen <ogofen>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.6.0CC: acanan, amureini, bugs, joseph.blasi, masayag, mlipchuk, tnisan
Target Milestone: ovirt-3.6.0-gaFlags: amureini: ovirt-3.6.0?
rule-engine: planning_ack?
rule-engine: devel_ack+
rule-engine: testing_ack+
Target Release: 3.6.0.3   
Hardware: x86_64   
OS: Linux   
Whiteboard: storage
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-27 07:54:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1264691    
Bug Blocks:    
Attachments:
Description Flags
engine log file
none
rc2 log none

Description joseph.blasi 2015-09-24 21:19:02 UTC
Created attachment 1076724 [details]
engine log file

Description of problem:
When I try to make an template from an cinder-ceph based VM it processes for some time and then errors out with "Error while executing action New VM Template: Internal Engine Error"

Version-Release number of selected component (if applicable):


How reproducible:
all the time

Steps to Reproduce:
1. Make a new vm with cinder-ceph
2. Install os and shutdown the VM
3. Make a template from the VM

Actual results:

Get a Error while executing action New VM Template: Internal Engine Error and see the copied volume in the cinder store.

The new volume is able to be booted up and ran if attached to a new VM.

Expected results:
To have the VM show in a Templates list.

Additional info:
Log clipping attached

Comment 1 Allon Mureinik 2015-10-14 07:50:30 UTC
Maor - please take a look?

Comment 2 Allon Mureinik 2015-10-14 07:54:34 UTC
Joseph, thanks for reporting this bug!
Can you specify which RPM version you're using?

Comment 3 Maor 2015-10-14 08:09:02 UTC
looks like it is the same issue described in https://bugzilla.redhat.com/1264677

The transaction got timed out  after 10 minutes
2015-09-24 15:54:46,930 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-113) [51809cd7] START, SetVmStatusVDSCommand( SetVmStatusVDSCommandParameters:{runAsync='true', vmId='48b3a875-94d2-4239-abf7-e42e0e240843', status='ImageLocked', exitStatus='Normal'}), log id: 13d06319
2015-09-24 15:54:46,933 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-113) [51809cd7] FINISH, SetVmStatusVDSCommand, log id: 13d06319
2015-09-24 16:04:46,994 INFO  [org.ovirt.engine.core.bll.storage.CloneCinderDisksCommand] (pool-7-thread-1) [332a1ca] Running command: CloneCinderDisksCommand internal: true.

Comment 4 joseph.blasi 2015-10-14 15:16:23 UTC
After updating to RC2 templates are showing up in the Templates list.

Comment 5 joseph.blasi 2015-10-14 15:36:22 UTC
Created attachment 1082881 [details]
rc2 log

Comment 6 joseph.blasi 2015-10-14 15:38:09 UTC
Trying to start a new VM from an Template leads to a long wait and errors out.

Comment 7 Maor 2015-10-15 07:04:53 UTC
It looks like this is the same issue mentioned at https://bugzilla.redhat.com/1264691
Looking at your logs it looks that you got a transaction timeout after 10 minutes:

2015-10-14 10:15:22,780 INFO  [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (default task-87) [] FINISH, SetVmStatusVDSCommand, log id: 169ec556
2015-10-14 10:25:22,589 ERROR [org.ovirt.engine.core.bll.AddVmFromTemplateCommand] (Transaction Reaper Worker 0) [] Transaction rolled-back for command 'org.ovirt.engine.core.bll.AddVmFromTemplateCommand'.
2015-10-14 10:25:22,639 INFO  [org.ovirt.engine.core.bll.storage.CloneCinderDisksCommand] (pool-7-thread-3) [355b7e7a] Running command: CloneCinderDisksCommand internal: true.
2015-10-14 10:25:22,689 INFO  [org.ovirt.engine.core.bll.storage.CloneSingleCinderDiskCommand] (pool-7-thread-4) [38241192] Running command: CloneSingleCinderDiskCommand internal: true.

Comment 8 Maor 2015-10-15 07:06:43 UTC
Joseph,

Can u please change the summary of the bug to indicate this is now an issue of creating a VM from Template, or open a separate bug on this specific issue.

Comment 9 Maor 2015-10-15 07:09:02 UTC
Moti,

That looks like the same issue at https://bugzilla.redhat.com/1264691, (see https://bugzilla.redhat.com/show_bug.cgi?id=1266250#c7).
Probably worth to check this once BZ1264691 will be solved.

Comment 10 Moti Asayag 2015-10-16 15:06:15 UTC
The root cause of this bug is a difference between the behavior of postgres 8.4 and 9.1.

In the described scenarios (and other similar to this issue as we faced on AddVmTemplate) we have the following sequence:

On The main thread:
Open new transaction (either explicit or as default command's scope):
CommandCoordinatorUtil.executeAsyncCommand(...)
  |
  +-- coco.persistCommandAssociatedEntities(...)
      which internally updates command_assoc_entities table

On the submitted thread we calls:
  CommandBase.setCommandStatus()
  which updates the command_entities table.

The first update of 'command_assoc_entities' table is done on transaction Tx1 a, which is open. While  Tx1 is open, the update for 'command_entities' is executed in other transaction Tx2.

In postgres 9.1, that scenario is permitted without any locking on the command_entities table.

However in postgres 8.4, an update for 'command_entities' isn't permitted, unless the referred table 'command_assoc_entities' is closed.

In order to make the code compliant for both postgres versions, the first update to 'command_assoc_entities' should be immediately committed.

Comment 11 Red Hat Bugzilla Rules Engine 2015-10-19 10:57:11 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 12 Maor 2015-10-19 15:26:41 UTC
Verified with patch 47408.
Moving to modify

Comment 13 Allon Mureinik 2015-10-29 09:03:08 UTC
(In reply to Maor from comment #12)
> Verified with patch 47408.
> Moving to modify
Moving to ON_QA by that logic.

Comment 14 Red Hat Bugzilla Rules Engine 2015-11-02 12:27:04 UTC
Bug tickets that are moved to testing must have target release set to make sure tester knows what to test. Please set the correct target release before moving to ON_QA.

Comment 15 Ori Gofen 2015-11-11 15:02:22 UTC
Verified on rhevm-3.6-0.3

Comment 16 Sandro Bonazzola 2015-11-27 07:54:45 UTC
Since oVirt 3.6.0 has been released, moving from verified to closed current release.