Description of problem: I upgraded oVirt from 3.2 to 3.3.beta, after importing all of VMs they disappeared from oVirt WUI after DB restart. This was fixed in 3.3 stable (more info http://lists.ovirt.org/pipermail/users/2013-August/015628.html). I was not able to create snapshots before 3.3.0-4. Today I upgraded to 3.3.0-4 and I can create snapshots, but live snapshot creating still failling. Version-Release number of selected component (if applicable): oVirt 3.3.0-4 on Centos 6.4, all nodes Centos 6.4 How reproducible: In oVirt WUI. Maybe some prerequisite steps are needed: Install oVirt 3.2. create some VMs, export some of them, delete them from oVirt, upgrade to 3.3 beta (or RC), import VMs, restart DB(or whole oVirt server), check VMs (You should see that some of them are missing) Steps to Reproduce: 1. Select VM 2. Click snapshot 3. Click Create 4. Assign name to snapshot (for ex. test) 5. Wait until snapshot is done (or fails) Actual results: Snapshot 'test' creation for VM 'server1.testlab.org' was initiated by admin. Failed to create live snapshot 'test' for VM 'server1.testlab.org'. VM restart is recommended. Snapshot 'test' creation for VM 'server1.testlab.org' has been completed. I can see newly created snapshot in oVirt GUI. (no Livesnapshot is created, but normal snapshot is done and works) Expected results: Snapshot 'test' creation for VM 'server1.testlab.org' was initiated by admin. Live snapshot 'test' for VM 'server1.testlab.org' has been completed. Snapshot 'test' creation for VM 'server1.testlab.org' has been completed. I can see newly created snapshot in oVirt GUI. Live snapshot should be created. Additional info: None
Jakub, thanks for posting this bug. Can you attach oVirt Engine and VDSM logs from your environment please?
Created attachment 799357 [details] engine.log
Created attachment 799358 [details] vdsm.log
looks like your libvirt version is not up to date: libvirtError: unsupported configuration: reuse is not supported with this QEMU binary just update your libvirt version and it will work
I upgraded libvirt and it still does not work. My versions are: RHEL - 6 - 4.el6.centos.10 libvirt-0.10.2-18.el6_4.14 vdsm-4.12.1-2.el6
Please attach updated logs. Thanks.
Hello, I searched thru updated logs and I have this in it: Thread-4647::DEBUG::2013-10-08 17:16:42,087::libvirtconnection::101::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 84 edom: 10 level: 2 message: Operation not supported: live disk snapshot not supported with this QEMU binary Thread-4647::ERROR::2013-10-08 17:16:42,090::vm::3644::vm.Vm::(snapshot) vmId=`b5a53cac-7a3a-4004-b4d8-f98490c04b90`::Unable to take snapshot Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 3642, in snapshot self._dom.snapshotCreateXML(snapxml, snapFlags) File "/usr/share/vdsm/vm.py", line 826, in f ret = attr(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1623, in snapshotCreateXML if ret is None:raise libvirtError('virDomainSnapshotCreateXML() failed', dom=self) libvirtError: Operation not supported: live disk snapshot not supported with this QEMU binary Packages in node system (Centos 6.4): ovirt-release-el6-8-1.noarch vdsm-4.12.1-2.el6.x86_64 libvirt-0.10.2-18.el6_4.14.x86_64 qemu-kvm-0.12.1.2-2.355.0.1.el6_4.9.x86_64 qemu-kvm-tools-0.12.1.2-2.355.0.1.el6_4.9.x86_64 qemu-img-0.12.1.2-2.355.0.1.el6_4.9.x86_64
you need qemu-kvm-rhev for live snapshots. i'm pretty sure its available for centos
I did not found qemu-kvm-rhev package in centos mirrors. I found only http://www.dreyou.org/ovirt/vdsm/Packages/ which contains this package, but it is third party and from January.
I maintain a full mirror of CentOS sources -- CentOS does not build from the RHEV sub-tree [herrold@centos-6 ~]$ srcfind rhev | grep qemu | grep -v centos | wc 17 17 2050 [herrold@centos-6 ~]$ srcfind rhev | grep qemu | wc 17 17 2050 [herrold@centos-6 ~]$ srcfind rhev | grep qemu | tail -n 1 /mnt/nfs/var/ftp/pub/mirror2/redhat/rhel/SRPMSonly/6Server/en/RHEV/SRPMS/qemu-kvm-rhev-0.12.1.2-2.295.el6_3.5.src.rpm [herrold@centos-6 ~]$
Vdsm should check (hopefully via libvirt) if the underlying qemu supports live snapshot, and report this feature to Engine. If it does not, block this feature in UI.
Fede, any update on this?
(In reply to Ayal Baron from comment #12) > Fede, any update on this? No, we still have two options here, either we provide a qemu-kvm-ovirt build (with live snapshot support) on the ovirt repositories [1], or we start working on comment 11 (probably not 3.3 material). [1] I suppose it will conflict with qemu-kvm.el6
(In reply to Federico Simoncelli from comment #13) > (In reply to Ayal Baron from comment #12) > > Fede, any update on this? > > No, we still have two options here, either we provide a qemu-kvm-ovirt build > (with live snapshot support) on the ovirt repositories [1], or we start > working on comment 11 (probably not 3.3 material). > > [1] I suppose it will conflict with qemu-kvm.el6 either way this is not a storage issue.
regardless if we eventually manage to build qemu-kvm-rhev equivalent we should implement comment #11 with a clear message in UI (grey out and display feature support status in Host subtab)
setting target release to current version for consideration and review. please do not push non-RFE bugs to an undefined target release to make sure bugs are reviewed for relevancy, fix, closure, etc.
*** Bug 1050835 has been marked as a duplicate of this bug. ***
libvirt doesn't yet export the information if the underlying qemu supports live snapshot. https://www.redhat.com/archives/libvirt-users/2014-January/msg00033.html A patch exporting this information on the capabilities XML is being crafted and will be submitted soon; VDSM will be updated accordingly once the XML format is finalized
Patch submitted upstream. Revision still pending https://www.redhat.com/archives/libvir-list/2014-January/msg00824.html
just note this is going to take some time to be available for us. Till then the part of the functionality will simply fail to work. decreasing prio/severity as no other functionality is affected
*** Bug 1063979 has been marked as a duplicate of this bug. ***
*** Bug 1064194 has been marked as a duplicate of this bug. ***
Patch still pending review on libvirt list. Being pinged regulary on list.
*** Bug 1057587 has been marked as a duplicate of this bug. ***
review in progress. XML format seems to have reached consensus. Will post VDSM patch once XML format is finalized.
(In reply to Francesco Romani from comment #25) > review in progress. XML format seems to have reached consensus. Will post > VDSM patch once XML format is finalized. I don't understood if ovirt 3.4.1 will release qemu-kvm-(rhev|ovirt) rpms? or will disable live snapshot on wui if qemu-kvm packages installed on hypervisor|node doesn't support that?
The implemented fix is to disable the libe snapshot support from the WUI if the hypervisor does not have the support enabled.
understood... last question: there are an rfe for providing on ovirt repositories qemu-kvm-rhev rpms? if not, can I open a new rfe? best regards a
Just one thing, before to open a RFE for a qemu-kvm-{rhev,ovirt} (I'm not aware of any existing request), please make sure the vanilla QEMU (not the rhev flavour) still lacks the live snapshot support, they made huge improvements on many fields since this bug was filed.
This is an automated message. oVirt 3.4.1 has been released. This issue has been retargeted to 3.4.2 as it has priority high, please retarget if needed. If this is a blocker please add it to the tracker Bug #1095370
Patch merged on VDSM master. Not changed to MODIFIED because the engine part is still missing.
retargeted for 3.5.0
engine patch under review.
Hi, I don't consider this bug as a blocker, so I will be patient, no issue. I understood reading you that the next answer will be a GUI disabling in case the adequate Qemu support is anavailable/not reported. Right. But I also read comment #29 about huge efforts on the qemu side. Were those efforts witnessed regarding this bug, or at least the reporting part? Thank you.
Thanks to Cluster emulation modes it is unable to add older version of qemu to 3.5 cluster "Host blue does not comply with the cluster Default emulated machines. The Hosts emulated machines are rhel6.4.0,pc,rhel6.3.0,rhel6.2.0,rhel6.1.0,rhel6.0.0,rhel5.5.0,rhel5.4.4,rhel5.4.0 and the cluster is rhel6.5.0" As the new qemu has live snapshot support, you should not come in contact with such a scenario that qemu-kvm won't have live snapshot support and you will be able to add host to 3.5 cluster. Either way Francesco suggested a scenario how to test this, which requires a bit of more work but will be able to verify all components of this fix. 1. Edit source of new qemu-kvm and disable live snapshot support in code 2. Build qemu-kvm and add host to 3.5 engine 3. See if Live Snapshot option is disabled in engine These are our suggested verification steps. If anybody got better solution I will be more than pleased to try it, rather than go into qemu code. :)
Another possible way could be to install qemu-kvm and update oVirt's database to allow this host to join the cluster: UPDATE vds_groups SET emulated_mahcine = 'rhel6.4.0,pc,rhel6.3.0,rhel6.2.0,rhel6.1.0,rhel6.0.0,rhel5.5.0,rhel5.4.4,rhel5.4.0' WHERE vds_group_id = '<id of your cluster' Note that you need to do this while the engine is down.
Thank you Allon for this suggestion, I've been trying to change vds_dynamic table but the configuration was realoaded each time. This works perfectly Current behaviour: 1. Create remove qemu-kvm-rhev package from host 2. Install qemu-img-0.12.1.2-2.355.el6_4.9.x86_64 gpxe-roms-qemu-0.9.7-6.10.el6.noarch qemu-kvm-0.12.1.2-2.355.el6_4.9.x86_64 3. Install vdsm 4. service vdsm-tool configure --force && service vdsmd start) 5. Add host to engine, start VM on host, wait for UP state 6. VMs->Snapshot subtab - Create Finalization stage of create snapshot task failed. attaching engine.log I would prefer If create snapshot button was disabled while the VM is in up state rather than Fail as suggester in comment #11. Plus the snapshot is created in engine. When trying to preview this VM stays down as the memory of VM was not saved. I can provide testing environment if necessary. Moving to ASSIGNED as snapshot creation fails, snapshot is created eitherway, and user is able to create live snapshot with qemu-kvm that does not support it.
Hi Lukas, please provide the output of 'vdsClient hypervisorHost getVdsCaps' of the host running VDSM
posting just necessary stuff from host [~]# vdsClient -s 0 getVdsCaps clusterLevels = ['3.0', '3.1', '3.2', '3.3', '3.4', '3.5'] cpuCores = '4' cpuFlags = 'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,sse4_1,sse4_2,x2apic,popcnt,tsc_deadline_timer,aes,xsave,avx,f16c,rdrand,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,fsgsbase,smep,erms,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270,model_SandyBridge' cpuModel = 'Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz' cpuSockets = '1' cpuSpeed = '3301.000' cpuThreads = '8' emulatedMachines = ['rhel6.4.0', 'pc', 'rhel6.3.0', 'rhel6.2.0', 'rhel6.1.0', 'rhel6.0.0', 'rhel5.5.0', 'rhel5.4.4', 'rhel5.4.0'] guestOverhead = '65' hooks = {} kdumpStatus = 1 kvmEnabled = 'true' operatingSystem = {'name': 'RHEL', 'release': '6.5.0.1.el6', 'version': '6Server'} packages2 = {'kernel': {'buildtime': 1384118394.0, 'release': '431.el6.x86_64', 'version': '2.6.32'}, 'libvirt': {'buildtime': 1401355377, 'release': '29.el6_5.9', 'version': '0.10.2'}, 'mom': {'buildtime': 1403794344, 'release': '2.el6', 'version': '0.4.1'}, 'qemu-img': {'buildtime': 1379320642, 'release': '2.355.el6_4.9', 'version': '0.12.1.2'}, 'qemu-kvm': {'buildtime': 1379320642, 'release': '2.355.el6_4.9', 'version': '0.12.1.2'}, 'spice-server': {'buildtime': 1401730350, 'release': '6.el6_5.2', 'version': '0.12.4'}, 'vdsm': {'buildtime': 1406372956, 'release': '0.gita4d9abf.el6', 'version': '4.16.1'}} reservedMem = '321' selinux = {'mode': '1'} software_revision = '0' software_version = '4.16' supportedENGINEs = ['3.0', '3.1', '3.2', '3.3', '3.4', '3.5'] supportedProtocols = ['2.2', '2.3'] uuid = '4C4C4544-0034-5310-8052-B1C04F4A354A' version_name = 'Snow Man' vlans = {} vmTypes = ['kvm'] [~]# vdsClient -s 0 getVdsCaps | grep -i live
I see the 'liveSnapshot' is missing in the caps output. That explains the behaviour. If Engine doesn't know about live snapshot support, it falls back to the pre-bug behaviour, and always allow live snapsnotting for backwards compatibility. If liveSnapshot is reported to be unsupported, then the new code kicks in and Engine should block the feature as reported. This should also be visible in the host tab in Engine, among the other host capabilties. AFAIK/IIRC libvirt 35_el6 should provide this reporting. U/S libvirt >= 1.2.3 is fine as well. To summarize, what is needed is: - qemu without live snapshotting support (either old or configured this way) - reporting libvirt - oVirt >= 3.5.0
As stated in https://bugzilla.redhat.com/show_bug.cgi?id=1009100#c40 , I believe that no code changes are needed, thus I'm putting back the state to MODIFIED.
oVirt 3.5 has been released and should include the fix for this issue.