Bug 1294511

Summary: Pool's template version doesn't update when set to latest and a new version is created
Product: [oVirt] ovirt-engine Reporter: sefi litmanovich <slitmano>
Component: BLL.VirtAssignee: Arik <ahadas>
Status: CLOSED CURRENTRELEASE QA Contact: sefi litmanovich <slitmano>
Severity: urgent Docs Contact:
Priority: high    
Version: 3.6.2CC: ahadas, alukiano, bugs, eedri, gklein, jmcdonal, mavital, mgoldboi, michal.skrivanek, ncredi, sbonazzo, slitmano, tjelinek
Target Milestone: ovirt-3.6.5Keywords: AutomationBlocker, Regression
Target Release: 3.6.5.1Flags: rule-engine: ovirt-3.6.z+
rule-engine: blocker+
mgoldboi: planning_ack+
tjelinek: devel_ack+
mavital: testing_ack+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-21 14:36:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
engine log none

Description sefi litmanovich 2015-12-28 16:21:22 UTC
Created attachment 1110038 [details]
engine log

Version-Release number of selected component (if applicable):

rhevm-3.6.2-0.1.el6.noarch

How reproducible:
always

Steps to Reproduce:

1. Create a template of some OS.
2. Create a pool with some vms (in my case 5) using the template you created - set version to latest.
3. Start 2 vms from the pool and create a new file in one of them.
4. stop the vm with the new file. 
5. make a template from that vm and set it as a new version of the previously used template.

Actual results:

1.The vm which is still up doesn't get the delta sign expected.
2.The other vms which are down are not deleted and re created with the new template.
3. When the started vm is stopped it then IS deleted and re created with the correct version.
4. If one of the vms which were down is started and the stopped again it also is deleted and re createed with the correct version.

Expected results:

Once the new template version is created the delta sign should appear on the started vm and the vms down should be deleted and re created with the right template version.


Additional info:

In the attached log, the vm pool name is 'template_version' and has 5 vms.
the name of the template is 'golden_mixed_virtio_template' and the scenario was creating the 5th sub version from vm 'template_version-1' while the pool is set to 'latest' which was the 4th sub - version. the vm which was up was 'template_version-2' and the other vms down were 'template_version-3/4/5'.

Creation of the template can be found in the log:

2015-12-28 17:41:32,193 INFO  [org.ovirt.engine.core.bll.AddVmTemplateCommand] (ajp-/127.0.0.1:8702-18) [b6c75f1] Lock Acquired to object 'EngineLock:{exclusiveLocks='null', sharedLocks='[2ab87f2c-0b59-4c98-8f27-6c555e898bef=<TEMPLATE, ACTION_TYPE_FAILED_OBJECT_LOCKED>]'}'

Comment 1 Michal Skrivanek 2016-01-29 13:47:49 UTC
what kind of pool is it?

when you stop the VM from pool the snapshot is reverted and you've lost that file...so then when you create a new version from the same VM it is exactly the same as the previous version

there's no immediate reaction when you add a version. On next VM startup the new version is used, that's all.

Comment 2 sefi litmanovich 2016-02-07 12:58:54 UTC
It's an automatic pool, and the VMs are started by admin and not as user, therefore not started as stateless vms as you suggest (there is no snapshot)

I also don't agree with the last part of you comment, as far as I know, when a new version is created which is different than the previous version then any vm that is up (thus works on top of previous version) should be marked for reboot to apply new version.
And the other VMs are supposed to be deleted and re-created on top of the new version.
That is at least the behaviour when replacing a version to a different version (not as latest pointer).
e.g. If I would do the same flow but instead of keeping the template version pointer on latest I'd change the flow to:

1. Create a template of some OS.
2. Create a pool with some vms (in my case 5) using the template you created - set version to (1).
3. Start 2 vms from the pool and create a new file in one of them.
4. stop the vm with the new file. 
5. make a template from that vm and set it as a new version of the previously used template.
6. edit pool -> set template version to (2).

This flow will result with the expected behaviour. Seeing as this flow and the original should be equivalent I insist this is a bug with usage of the 'latest' pointer.

Comment 5 Red Hat Bugzilla Rules Engine 2016-02-29 11:27:41 UTC
Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

Comment 6 Red Hat Bugzilla Rules Engine 2016-02-29 11:42:52 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 7 Arik 2016-03-21 17:10:48 UTC
Lets separate two things:

1. The indication for 'next-run' configuration (that will be applied on next run) - it was designed in parallel to the template version feature. There was no requirement to show this indication for a VM that is set with 'latest' template when new version is created (it was designed only for new configuration that is set for the specific VM). If we want to present it in that case as well - then it should be tracked as RFE and not as a regression.

2. The problem that some of the VMs were not upgraded to the latest version - I can't reproduce that. Looking at the attached log, I see that the VM template_version-3 for example was deleted and recreated.
Sefi, can try to reproduce it on latest 3.6 and see if it is reproduced?

Comment 8 sefi litmanovich 2016-03-22 08:19:19 UTC
2. Re produced once more on rhevm-3.6.4-0.1.el6.noarch.
Re iterating on the steps:

1. Created a vm pool with 5 VMs from some template with rhel 7.2 on it, set to latest template version.
2. Started 2/5 of the VMs as admin (vms start normally - not stateless).
3. Create a file in 1 of the running vms then stop it.
4. Create a new template version from that VM.

Result:
The 3 Vms are not re created.
The running vm has no indicator - I think there should be an indicator, I can open the RFE so discussion can continue there.

5. start 1 of the 3 VMs which were untouched.

Result:
Vm doesn't contain the new file.

6. stop VM-2 (the one that was started and untouched) and VM-3 from last step.

Result:
Now these VMs are re created - file appear when started.
the last 2 untouched VMs (4,5) are still not re created.

Comment 9 Arik 2016-03-22 09:07:05 UTC
(In reply to sefi litmanovich from comment #8)
Alright, that's good that you managed to reproduce it.
So please attach the engine log and specify the IDs of the VMs in the pool and specifically these that have not been upgraded to the latest template version

Comment 11 Arik 2016-03-24 09:13:15 UTC
We didn't manage to reproduce it with a debugger on Sefi's environment.

We identified a possible race and I was able to reproduce it with some breakpoints. Not sure that this the cause to this bug, but chances are relatively high (in my opinion) and with no further information that's probably the best we can do at the moment.

Therefore, proposing a simple fix to prevent this race and hopefully it will eliminate this issue.

Comment 12 Mike McCune 2016-03-28 22:13:31 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 13 Michal Skrivanek 2016-03-31 13:47:51 UTC
not really in 3.6.5 yet

Comment 14 Red Hat Bugzilla Rules Engine 2016-03-31 13:47:55 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 15 Michal Skrivanek 2016-03-31 13:49:54 UTC
*** Bug 1322865 has been marked as a duplicate of this bug. ***

Comment 16 Michal Skrivanek 2016-04-04 06:31:03 UTC
*** Bug 1323493 has been marked as a duplicate of this bug. ***

Comment 17 sefi litmanovich 2016-04-04 16:39:03 UTC
Verified on same env where the bug appeared after upgrade to: rhevm-3.6.5.1-0.1.el6.noarch according to the steps in description.

Comment 18 Roy Golan 2016-04-05 08:06:50 UTC
*** Bug 1323595 has been marked as a duplicate of this bug. ***