Created attachment 922156 [details] vdsm+engine logs Description of problem: vdsm reports: "Thread-567::ERROR::2014-07-29 16:37:45,012::vm::5542::vm.Vm::(merge) vmId=`f4aa40f2-8b43-427a-bdfd-bf790ebe6ff8`::merge: base volume '9c9f9c84-278e-4711-a9c6-7ebcf50a0dda' not found" when trying to live merge a snapshot,engine throws several exceptions seemingly concerning vdsm's failure "Failed to live merge, still in volume chain[]" 2014-07-29 16:37:43,179 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [154700c3] Failed in MergeVDS method 2014-07-29 16:37:43,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [154700c3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand retu rn value· StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=52, mMessage=Merge failed]] 2014-07-29 16:37:43,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [154700c3] HostName = vdsb 2014-07-29 16:37:43,203 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [154700c3] Command MergeVDSCommand(HostName = vdsb, MergeVDSCommandParameters{Hos tId = b20b9d2d-d5de-4bef-ab1a-d044df43d90b, vmId=f4aa40f2-8b43-427a-bdfd-bf790ebe6ff8, storagePoolId=00000002-0002-0002-0002-000000000200, storageDomainId=c0259524-af0c-4860-b6fc-58cbd8e97f 68, imageGroupId=8400c2f4-e0c7-4eff-aca2-65448d2cf81c, imageId=2758889b-2b68-4c64-9e04-a92d84281e7a, baseImageId=9c9f9c84-278e-4711-a9c6-7ebcf50a0dda, topImageId=73c14704-db08-403f-9b32-79f 83b1243ab, bandwidth=0}) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 2014-07-29 16:37:43,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [154700c3] FINISH, MergeVDSCommand, log id: 6549f990 2014-07-29 16:37:43,250 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-2) [154700c3] Command org.ovirt.engine.core.bll.MergeCommand throw Vdc Bll exception. With error messag e VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with erro r mergeErr and code 52) 2014-07-29 16:37:43,280 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-2) [154700c3] Transaction rolled-back for command: org.ovirt.engine.core.bll.MergeCommand. 2014-07-29 16:37:52,927 INFO [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-15) [4a7d215a] Waiting on Live Merge command step MERGE to compl ete 2014-07-29 16:37:52,956 INFO [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler_Worker-15) [154700c3] Merge command has completed for images 9c9f9c84-278e-4711-a9c6-7 ebcf50a0dda..73c14704-db08-403f-9b32-79f83b1243ab 2014-07-29 16:37:52,966 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-15) [154700c3] Waiting on Live Merge child commands to complete 2014-07-29 16:38:02,977 INFO [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-44) [4a7d215a] Executing Live Merge command step MERGE_STATUS 2014-07-29 16:38:03,027 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-44) Waiting on Live Merge child commands to complete 2014-07-29 16:38:03,029 INFO [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-7-thread-3) [5339fe0b] Running command: MergeStatusCommand internal: true. Entities affected : ID: c02595 24-af0c-4860-b6fc-58cbd8e97f68 Type: Storage 2014-07-29 16:38:03,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (pool-7-thread-3) [5339fe0b] START, FullListVdsCommand(HostName = vdsb, HostId = b20b9d2d-d5de-4 bef-ab1a-d044df43d90b, vds=Host[vdsb,b20b9d2d-d5de-4bef-ab1a-d044df43d90b], vmIds=[f4aa40f2-8b43-427a-bdfd-bf790ebe6ff8]), log id: 2022a280 2014-07-29 16:38:03,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (pool-7-thread-3) [5339fe0b] FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEn able=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.116.2, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true , vmId=f4aa40f2-8b43-427a-bdfd-bf790ebe6ff8, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up , memGuaranteedSize=1024, display=qxl, pid=1628, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numa Tune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@41da91fb, vmName=vm_1, fileTransferEnable=true, cpuType=Nehalem}], log id: 2022a280 2014-07-29 16:38:03,100 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-7-thread-3) [5339fe0b] Failed to live merge, still in volume chain: [] 2014-07-29 16:38:03,140 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-7-thread-3) [5339fe0b] Transaction rolled-back for command: org.ovirt.engine.core.bll.MergeStatusCommand. 2014-07-29 16:38:13,033 ERROR [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-29) [4a7d215a] Failed to merge: failed child command status for step MERGE_STATUS 2014-07-29 16:38:13,048 INFO [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-29) [18925ea3] All Live Merge child commands have completed, status FA ILED 2014-07-29 16:38:23,100 ERROR [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (DefaultQuartzScheduler_Worker-72) [18925ea3] Ending command with failure: org.ovirt.engine.core.bll.RemoveSn apshotCommand 2014-07-29 16:38:23,263 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [18925ea3] Correlation ID: 18925ea3, Call Stack: null , Custom Event ID: -1, Message: Failed to delete snapshot '11' for VM 'vm_1'. Version-Release number of selected component (if applicable): beta.2 How reproducible: 100% Steps to Reproduce: 1.delete a middle snapshot in a running vm(have two at least) Actual results: vdsm fails to locate base volume which leads to engine's exceptions and failure Expected results: operation should succeed Additional info:
This is probably due to a known SELinux issue with the libvirt/qemu packages that are installed from: http://files.pipo.sk/ Please run 'setenforce 0' on your vdsm host and see if the problem goes away. If so, then this issue will be resolved in the upstream versions of qemu and libvirt once officially released.
setting SELinux to permissive did seem to solve this issue.
(In reply to Adam Litke from comment #1) > This is probably due to a known SELinux issue with the libvirt/qemu packages > that are installed from: http://files.pipo.sk/ Do we/libvirt have a BZ to track this issue?
Since you are using an unreleased and privately patched version of libvirt there is no formal bug. Peter Krempa is aware of the problem in the implementation and he has provided me with a new build to test. I'll see if it resolves the issue and place a link to the new libvirt rpms here.
Actually, I reported the issue to Peter here: https://bugzilla.redhat.com/show_bug.cgi?id=1102881
*** This bug has been marked as a duplicate of bug 1102881 ***