Bug 1009100 - VM live snapshot creation failed - when qemu does not support this operation
VM live snapshot creation failed - when qemu does not support this operation
Status: CLOSED CURRENTRELEASE
Product: oVirt
Classification: Community
Component: ovirt-engine-core (Show other bugs)
3.3
All Linux
high Severity medium
: ---
: 3.5.0
Assigned To: Francesco Romani
Lukas Svaty
virt
: Reopened, Triaged
: 1050835 1057587 1063979 1064194 (view as bug list)
Depends On:
Blocks: 1105846
  Show dependency treegraph
 
Reported: 2013-09-17 13:02 EDT by Jakub Bittner
Modified: 2016-02-10 14:49 EST (History)
23 users (show)

See Also:
Fixed In Version: ovirt-engine-3.5.0_beta
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1018947 (view as bug list)
Environment:
Last Closed: 2014-10-17 08:32:56 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Virt
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
engine.log (26.08 KB, text/x-log)
2013-09-18 09:14 EDT, Jakub Bittner
no flags Details
vdsm.log (168.32 KB, text/x-log)
2013-09-18 09:15 EDT, Jakub Bittner
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 26149 None MERGED caps: report if QEMU supports live snapshots Never
oVirt gerrit 27677 master MERGED core, webadmin: report QEMU live snapshot support Never
oVirt gerrit 28171 None NEW restapi: expose the host live snapshot support Never
oVirt gerrit 28182 master MERGED core: check QEMU compatibility on livesnapshot Never

  None (edit)
Description Jakub Bittner 2013-09-17 13:02:38 EDT
Description of problem:
I upgraded oVirt from 3.2 to 3.3.beta, after importing all of VMs they disappeared from oVirt WUI after DB restart. This was fixed in 3.3 stable (more info http://lists.ovirt.org/pipermail/users/2013-August/015628.html). I was not able to create snapshots before 3.3.0-4. Today I upgraded to 3.3.0-4 and I can create snapshots, but live snapshot creating still failling.

Version-Release number of selected component (if applicable):
 oVirt 3.3.0-4 on Centos 6.4, all nodes Centos 6.4

How reproducible:
In oVirt WUI.
Maybe some prerequisite steps are needed:
Install oVirt 3.2. create some VMs, export some of them, delete them from oVirt, upgrade to 3.3 beta (or RC), import VMs, restart DB(or whole oVirt server), check VMs (You should see that some of them are missing) 

Steps to Reproduce:
1. Select VM
2. Click snapshot
3. Click Create
4. Assign name to snapshot (for ex. test)
5. Wait until snapshot is done (or fails)

Actual results:
Snapshot 'test' creation for VM 'server1.testlab.org' was initiated by admin.
Failed to create live snapshot 'test' for VM 'server1.testlab.org'. VM restart is recommended.
Snapshot 'test' creation for VM 'server1.testlab.org' has been completed.
I can see newly created snapshot in oVirt GUI.

(no Livesnapshot is created, but normal snapshot is done and works)

Expected results:
Snapshot 'test' creation for VM 'server1.testlab.org' was initiated by admin.
Live snapshot 'test' for VM 'server1.testlab.org' has been completed.
Snapshot 'test' creation for VM 'server1.testlab.org' has been completed.
I can see newly created snapshot in oVirt GUI.

Live snapshot should be created.


Additional info:
None
Comment 1 Allon Mureinik 2013-09-18 05:12:16 EDT
Jakub, thanks for posting this bug.
Can you attach oVirt Engine and VDSM logs from your environment please?
Comment 2 Jakub Bittner 2013-09-18 09:14:38 EDT
Created attachment 799357 [details]
engine.log
Comment 3 Jakub Bittner 2013-09-18 09:15:19 EDT
Created attachment 799358 [details]
vdsm.log
Comment 4 Ayal Baron 2013-09-24 07:06:31 EDT
looks like your libvirt version is not up to date:
libvirtError: unsupported configuration: reuse is not supported with this QEMU binary

just update your libvirt version and it will work
Comment 5 Jakub Bittner 2013-09-30 07:45:34 EDT
I upgraded libvirt and it still does not work. My versions are:

RHEL - 6 - 4.el6.centos.10
libvirt-0.10.2-18.el6_4.14
vdsm-4.12.1-2.el6
Comment 6 Ayal Baron 2013-09-30 17:53:17 EDT
Please attach updated logs.
Thanks.
Comment 7 Jakub Bittner 2013-10-08 11:21:33 EDT
Hello,

I searched thru updated logs and I have this in it:

Thread-4647::DEBUG::2013-10-08 17:16:42,087::libvirtconnection::101::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 84 edom: 10 level: 2 message: Operation not supported: live disk snapshot not supported with this QEMU binary
Thread-4647::ERROR::2013-10-08 17:16:42,090::vm::3644::vm.Vm::(snapshot) vmId=`b5a53cac-7a3a-4004-b4d8-f98490c04b90`::Unable to take snapshot
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 3642, in snapshot
    self._dom.snapshotCreateXML(snapxml, snapFlags)
  File "/usr/share/vdsm/vm.py", line 826, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1623, in snapshotCreateXML
    if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self)
libvirtError: Operation not supported: live disk snapshot not supported with this QEMU binary

Packages in node system (Centos 6.4):

ovirt-release-el6-8-1.noarch
vdsm-4.12.1-2.el6.x86_64
libvirt-0.10.2-18.el6_4.14.x86_64
qemu-kvm-0.12.1.2-2.355.0.1.el6_4.9.x86_64
qemu-kvm-tools-0.12.1.2-2.355.0.1.el6_4.9.x86_64
qemu-img-0.12.1.2-2.355.0.1.el6_4.9.x86_64
Comment 8 Itamar Heim 2013-10-08 17:30:36 EDT
you need qemu-kvm-rhev for live snapshots. i'm pretty sure its available for centos
Comment 9 Jakub Bittner 2013-10-09 04:40:19 EDT
I did not found qemu-kvm-rhev package in centos mirrors. I found only http://www.dreyou.org/ovirt/vdsm/Packages/ which contains this package, but it is third party and from January.
Comment 10 R P Herrold 2013-10-09 10:14:06 EDT
I maintain a full mirror of CentOS sources -- CentOS does not build from the RHEV sub-tree

[herrold@centos-6 ~]$ srcfind rhev | grep qemu | grep -v centos | wc
     17      17    2050
[herrold@centos-6 ~]$ srcfind rhev | grep qemu | wc
     17      17    2050
[herrold@centos-6 ~]$ srcfind rhev | grep qemu | tail -n 1
         /mnt/nfs/var/ftp/pub/mirror2/redhat/rhel/SRPMSonly/6Server/en/RHEV/SRPMS/qemu-kvm-rhev-0.12.1.2-2.295.el6_3.5.src.rpm
[herrold@centos-6 ~]$
Comment 11 Dan Kenigsberg 2013-10-09 17:45:31 EDT
Vdsm should check (hopefully via libvirt) if the underlying qemu supports live snapshot, and report this feature to Engine. If it does not, block this feature in UI.
Comment 12 Ayal Baron 2013-11-11 16:10:32 EST
Fede, any update on this?
Comment 13 Federico Simoncelli 2013-12-10 10:52:00 EST
(In reply to Ayal Baron from comment #12)
> Fede, any update on this?

No, we still have two options here, either we provide a qemu-kvm-ovirt build (with live snapshot support) on the ovirt repositories [1], or we start working on comment 11 (probably not 3.3 material).

[1] I suppose it will conflict with qemu-kvm.el6
Comment 14 Ayal Baron 2013-12-17 10:52:34 EST
(In reply to Federico Simoncelli from comment #13)
> (In reply to Ayal Baron from comment #12)
> > Fede, any update on this?
> 
> No, we still have two options here, either we provide a qemu-kvm-ovirt build
> (with live snapshot support) on the ovirt repositories [1], or we start
> working on comment 11 (probably not 3.3 material).
> 
> [1] I suppose it will conflict with qemu-kvm.el6

either way this is not a storage issue.
Comment 15 Michal Skrivanek 2014-01-07 03:05:00 EST
regardless if we eventually manage to build qemu-kvm-rhev equivalent we should implement comment #11 with a clear message in UI (grey out and display feature support status in Host subtab)
Comment 16 Itamar Heim 2014-01-12 03:43:21 EST
setting target release to current version for consideration and review. please do not push non-RFE bugs to an undefined target release to make sure bugs are reviewed for relevancy, fix, closure, etc.
Comment 17 Allon Mureinik 2014-01-12 10:18:12 EST
*** Bug 1050835 has been marked as a duplicate of this bug. ***
Comment 18 Francesco Romani 2014-01-15 12:38:52 EST
libvirt doesn't yet export the information if the underlying qemu supports live snapshot.

https://www.redhat.com/archives/libvirt-users/2014-January/msg00033.html

A patch exporting this information on the capabilities XML is being crafted and will be submitted soon;
VDSM will be updated accordingly once the XML format is finalized
Comment 19 Francesco Romani 2014-01-21 06:52:54 EST
Patch submitted upstream. Revision still pending

https://www.redhat.com/archives/libvir-list/2014-January/msg00824.html
Comment 20 Michal Skrivanek 2014-01-21 08:49:53 EST
just note this is going to take some time to be available for us. Till then the part of the functionality will simply fail to work.
decreasing prio/severity as no other functionality is affected
Comment 21 Federico Simoncelli 2014-02-11 16:32:06 EST
*** Bug 1063979 has been marked as a duplicate of this bug. ***
Comment 22 Ayal Baron 2014-02-17 16:13:35 EST
*** Bug 1064194 has been marked as a duplicate of this bug. ***
Comment 23 Francesco Romani 2014-02-25 13:15:23 EST
Patch still pending review on libvirt list. Being pinged regulary on list.
Comment 24 Michal Skrivanek 2014-03-05 03:59:34 EST
*** Bug 1057587 has been marked as a duplicate of this bug. ***
Comment 25 Francesco Romani 2014-03-17 12:57:41 EDT
review in progress. XML format seems to have reached consensus. Will post VDSM patch once XML format is finalized.
Comment 26 amedeo 2014-04-24 02:43:13 EDT
(In reply to Francesco Romani from comment #25)
> review in progress. XML format seems to have reached consensus. Will post
> VDSM patch once XML format is finalized.

I don't understood if ovirt 3.4.1 will release qemu-kvm-(rhev|ovirt) rpms?
or will disable live snapshot on wui if qemu-kvm packages installed on hypervisor|node doesn't support that?
Comment 27 Francesco Romani 2014-04-24 02:59:23 EDT
The implemented fix is to disable the libe snapshot support from the WUI if the hypervisor does not have the support enabled.
Comment 28 amedeo 2014-04-24 05:52:14 EDT
understood... 

last question: there are an rfe for providing on ovirt repositories qemu-kvm-rhev rpms? if not, can I open a new rfe?

best regards
a
Comment 29 Francesco Romani 2014-04-24 05:59:47 EDT
Just one thing, before to open a RFE for a qemu-kvm-{rhev,ovirt} (I'm not aware of any existing request), please make sure the vanilla QEMU (not the rhev flavour) still lacks the live snapshot support, they made huge improvements on many fields since this bug was filed.
Comment 30 Sandro Bonazzola 2014-05-08 09:50:44 EDT
This is an automated message.

oVirt 3.4.1 has been released.
This issue has been retargeted to 3.4.2 as it has priority high, please retarget if needed.
If this is a blocker please add it to the tracker Bug #1095370
Comment 31 Francesco Romani 2014-05-09 09:22:34 EDT
Patch merged on VDSM master.
Not changed to MODIFIED because the engine part is still missing.
Comment 32 Francesco Romani 2014-05-15 10:00:27 EDT
retargeted for 3.5.0
Comment 33 Francesco Romani 2014-05-23 10:59:07 EDT
engine patch under review.
Comment 34 Nicolas Ecarnot 2014-06-23 08:03:15 EDT
Hi,

I don't consider this bug as a blocker, so I will be patient, no issue.
I understood reading you that the next answer will be a GUI disabling in case the adequate Qemu support is anavailable/not reported.

Right.

But I also read comment #29 about huge efforts on the qemu side. Were those efforts witnessed regarding this bug, or at least the reporting part?

Thank you.
Comment 35 Lukas Svaty 2014-07-29 11:57:00 EDT
Thanks to Cluster emulation modes it is unable to add older version of qemu to 3.5 cluster 

"Host blue does not comply with the cluster Default emulated machines. The Hosts emulated machines are rhel6.4.0,pc,rhel6.3.0,rhel6.2.0,rhel6.1.0,rhel6.0.0,rhel5.5.0,rhel5.4.4,rhel5.4.0 and the cluster is rhel6.5.0"

As the new qemu has live snapshot support, you should not come in contact with such a scenario that qemu-kvm won't have live snapshot support and you will be able to add host to 3.5 cluster.

Either way Francesco suggested a scenario how to test this, which requires a bit of more work but will be able to verify all components of this fix.

1. Edit source of new qemu-kvm and disable live snapshot support in code
2. Build qemu-kvm and add host to 3.5 engine
3. See if Live Snapshot option is disabled in engine

These are our suggested verification steps. If anybody got better solution I will be more than pleased to try it, rather than go into qemu code. :)
Comment 36 Allon Mureinik 2014-07-29 13:51:55 EDT
Another possible way could be to install qemu-kvm and update oVirt's database to allow this host to join the cluster:

UPDATE vds_groups
SET    emulated_mahcine = 'rhel6.4.0,pc,rhel6.3.0,rhel6.2.0,rhel6.1.0,rhel6.0.0,rhel5.5.0,rhel5.4.4,rhel5.4.0'
WHERE  vds_group_id = '<id of your cluster'

Note that you need to do this while the engine is down.
Comment 37 Lukas Svaty 2014-07-30 07:14:29 EDT
Thank you Allon for this suggestion, I've been trying to change vds_dynamic table but the configuration was realoaded each time.
This works perfectly

Current behaviour:
1. Create remove qemu-kvm-rhev package from host
2. Install 
qemu-img-0.12.1.2-2.355.el6_4.9.x86_64
gpxe-roms-qemu-0.9.7-6.10.el6.noarch
qemu-kvm-0.12.1.2-2.355.el6_4.9.x86_64
3. Install vdsm 
4. service vdsm-tool configure --force && service vdsmd start)
5. Add host to engine, start VM on host, wait for UP state
6. VMs->Snapshot subtab - Create

Finalization stage of create snapshot task failed.
attaching engine.log

I would prefer If create snapshot button was disabled while the VM is in up state rather than Fail as suggester in comment #11.

Plus the snapshot is created in engine. When trying to preview this VM stays down as the memory of VM was not saved.

I can provide testing environment if necessary.
Moving to ASSIGNED as snapshot creation fails, snapshot is created eitherway, and user is able to create live snapshot with qemu-kvm that does not support it.
Comment 38 Francesco Romani 2014-07-30 07:20:58 EDT
Hi Lukas,

please provide the output of

'vdsClient hypervisorHost getVdsCaps' of the host running VDSM
Comment 39 Lukas Svaty 2014-07-30 07:42:26 EDT
posting just necessary stuff from host

[~]# vdsClient -s 0 getVdsCaps 
	clusterLevels = ['3.0', '3.1', '3.2', '3.3', '3.4', '3.5']
	cpuCores = '4'
	cpuFlags = 'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,f16c,rdrand,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,fsgsbase,smep,erms,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge'
	cpuModel = 'Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz'
	cpuSockets = '1'
	cpuSpeed = '3301.000'
	cpuThreads = '8'
	emulatedMachines = ['rhel6.4.0',
	                    'pc',
	                    'rhel6.3.0',
	                    'rhel6.2.0',
	                    'rhel6.1.0',
	                    'rhel6.0.0',
	                    'rhel5.5.0',
	                    'rhel5.4.4',
	                    'rhel5.4.0']
	guestOverhead = '65'
	hooks = {}
	kdumpStatus = 1
	kvmEnabled = 'true'
	operatingSystem = {'name': 'RHEL', 'release': '6.5.0.1.el6', 'version': '6Server'}
	packages2 = {'kernel': {'buildtime': 1384118394.0,
	                        'release': '431.el6.x86_64',
	                        'version': '2.6.32'},
	             'libvirt': {'buildtime': 1401355377,
	                         'release': '29.el6_5.9',
	                         'version': '0.10.2'},
	             'mom': {'buildtime': 1403794344, 'release': '2.el6', 'version': '0.4.1'},
	             'qemu-img': {'buildtime': 1379320642,
	                          'release': '2.355.el6_4.9',
	                          'version': '0.12.1.2'},
	             'qemu-kvm': {'buildtime': 1379320642,
	                          'release': '2.355.el6_4.9',
	                          'version': '0.12.1.2'},
	             'spice-server': {'buildtime': 1401730350,
	                              'release': '6.el6_5.2',
	                              'version': '0.12.4'},
	             'vdsm': {'buildtime': 1406372956,
	                      'release': '0.gita4d9abf.el6',
	                      'version': '4.16.1'}}
	reservedMem = '321'
	selinux = {'mode': '1'}
	software_revision = '0'
	software_version = '4.16'
	supportedENGINEs = ['3.0', '3.1', '3.2', '3.3', '3.4', '3.5']
	supportedProtocols = ['2.2', '2.3']
	uuid = '4C4C4544-0034-5310-8052-B1C04F4A354A'
	version_name = 'Snow Man'
	vlans = {}
	vmTypes = ['kvm']

[~]# vdsClient -s 0 getVdsCaps | grep -i live
Comment 40 Francesco Romani 2014-07-30 07:48:35 EDT
I see the 'liveSnapshot' is missing in the caps output.

That explains the behaviour.
If Engine doesn't know about live snapshot support, it falls back to the pre-bug behaviour, and always allow live snapsnotting for backwards compatibility.

If liveSnapshot is reported to be unsupported, then the new code kicks in and Engine should block the feature as reported.

This should also be visible in the host tab in Engine, among the other host capabilties.

AFAIK/IIRC libvirt 35_el6 should provide this reporting. U/S libvirt >= 1.2.3 is fine as well.

To summarize, what is needed is:
- qemu without live snapshotting support (either old or configured this way)
- reporting libvirt
- oVirt >= 3.5.0
Comment 41 Francesco Romani 2014-07-31 09:28:34 EDT
As stated in https://bugzilla.redhat.com/show_bug.cgi?id=1009100#c40 , I believe that no code changes are needed, thus I'm putting back the state to MODIFIED.
Comment 42 Sandro Bonazzola 2014-10-17 08:32:56 EDT
oVirt 3.5 has been released and should include the fix for this issue.

Note You need to log in before you can comment on or make changes to this bug.