Bug 1124445

Summary: [vdsm] Live deletion of a snapshot (live merge) fails with exceptions since vdsm can't locate base volume
Product: [Retired] oVirt Reporter: Ori Gofen <ogofen>
Component: vdsmAssignee: Adam Litke <alitke>
Status: CLOSED DUPLICATE QA Contact: Gil Klein <gklein>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.5CC: acanan, acathrow, alitke, amureini, bazulay, bugs, ecohen, gklein, iheim, mgoldboi, ogofen, yeylon
Target Milestone: ---Keywords: Triaged
Target Release: 3.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: storage
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-08-06 14:39:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
vdsm+engine logs none

Description Ori Gofen 2014-07-29 13:55:02 UTC
Created attachment 922156 [details]
vdsm+engine logs

Description of problem:

vdsm reports:
 "Thread-567::ERROR::2014-07-29 16:37:45,012::vm::5542::vm.Vm::(merge) vmId=`f4aa40f2-8b43-427a-bdfd-bf790ebe6ff8`::merge: base volume '9c9f9c84-278e-4711-a9c6-7ebcf50a0dda' not found"

when trying to live merge a snapshot,engine throws several exceptions seemingly concerning vdsm's failure 

"Failed to live merge, still in volume chain[]"

2014-07-29 16:37:43,179 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [154700c3] Failed in MergeVDS method
  2014-07-29 16:37:43,180 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [154700c3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand retu  rn value·
   StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=52, mMessage=Merge failed]]
  2014-07-29 16:37:43,189 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [154700c3] HostName = vdsb
  2014-07-29 16:37:43,203 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [154700c3] Command MergeVDSCommand(HostName = vdsb, MergeVDSCommandParameters{Hos  tId = b20b9d2d-d5de-4bef-ab1a-d044df43d90b, vmId=f4aa40f2-8b43-427a-bdfd-bf790ebe6ff8, storagePoolId=00000002-0002-0002-0002-000000000200, storageDomainId=c0259524-af0c-4860-b6fc-58cbd8e97f  68, imageGroupId=8400c2f4-e0c7-4eff-aca2-65448d2cf81c, imageId=2758889b-2b68-4c64-9e04-a92d84281e7a, baseImageId=9c9f9c84-278e-4711-a9c6-7ebcf50a0dda, topImageId=73c14704-db08-403f-9b32-79f  83b1243ab, bandwidth=0}) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52
  2014-07-29 16:37:43,244 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (pool-7-thread-2) [154700c3] FINISH, MergeVDSCommand, log id: 6549f990
  2014-07-29 16:37:43,250 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-2) [154700c3] Command org.ovirt.engine.core.bll.MergeCommand throw Vdc Bll exception. With error messag  e VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with erro  r mergeErr and code 52)
  2014-07-29 16:37:43,280 ERROR [org.ovirt.engine.core.bll.MergeCommand] (pool-7-thread-2) [154700c3] Transaction rolled-back for command: org.ovirt.engine.core.bll.MergeCommand.
  2014-07-29 16:37:52,927 INFO  [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-15) [4a7d215a] Waiting on Live Merge command step MERGE to compl  ete
  2014-07-29 16:37:52,956 INFO  [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler_Worker-15) [154700c3] Merge command has completed for images 9c9f9c84-278e-4711-a9c6-7  ebcf50a0dda..73c14704-db08-403f-9b32-79f83b1243ab
  2014-07-29 16:37:52,966 INFO  [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-15) [154700c3] Waiting on Live Merge child commands to complete
  2014-07-29 16:38:02,977 INFO  [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-44) [4a7d215a] Executing Live Merge command step MERGE_STATUS
  2014-07-29 16:38:03,027 INFO  [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-44) Waiting on Live Merge child commands to complete
  2014-07-29 16:38:03,029 INFO  [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-7-thread-3) [5339fe0b] Running command: MergeStatusCommand internal: true. Entities affected :  ID: c02595  24-af0c-4860-b6fc-58cbd8e97f68 Type: Storage
  2014-07-29 16:38:03,050 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (pool-7-thread-3) [5339fe0b] START, FullListVdsCommand(HostName = vdsb, HostId = b20b9d2d-d5de-4  bef-ab1a-d044df43d90b, vds=Host[vdsb,b20b9d2d-d5de-4bef-ab1a-d044df43d90b], vmIds=[f4aa40f2-8b43-427a-bdfd-bf790ebe6ff8]), log id: 2022a280
  2014-07-29 16:38:03,074 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (pool-7-thread-3) [5339fe0b] FINISH, FullListVdsCommand, return: [{displaySecurePort=5901, kvmEn  able=true, nicModel=rtl8139,pv, keyboardLayout=en-us, displayIp=10.35.116.2, pauseCode=NOERR, nice=0, displayNetwork=ovirtmgmt, copyPasteEnable=true, timeOffset=0, transparentHugePages=true  , vmId=f4aa40f2-8b43-427a-bdfd-bf790ebe6ff8, acpiEnable=true, custom={}, spiceSslCipherSuite=DEFAULT, memSize=1024, smp=1, displayPort=5900, emulatedMachine=rhel6.5.0, vmType=kvm, status=Up  , memGuaranteedSize=1024, display=qxl, pid=1628, smartcardEnable=false, bootMenuEnable=false, spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard, numa  Tune={mode=preferred}, smpCoresPerSocket=1, maxVCpus=160, clientIp=, devices=[Ljava.lang.Object;@41da91fb, vmName=vm_1, fileTransferEnable=true, cpuType=Nehalem}], log id: 2022a280
  2014-07-29 16:38:03,100 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-7-thread-3) [5339fe0b] Failed to live merge, still in volume chain: []
  2014-07-29 16:38:03,140 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-7-thread-3) [5339fe0b] Transaction rolled-back for command: org.ovirt.engine.core.bll.MergeStatusCommand.
  2014-07-29 16:38:13,033 ERROR [org.ovirt.engine.core.bll.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler_Worker-29) [4a7d215a] Failed to merge: failed child command status for   step MERGE_STATUS
  2014-07-29 16:38:13,048 INFO  [org.ovirt.engine.core.bll.RemoveSnapshotCommandCallback] (DefaultQuartzScheduler_Worker-29) [18925ea3] All Live Merge child commands have completed, status FA  ILED
  2014-07-29 16:38:23,100 ERROR [org.ovirt.engine.core.bll.RemoveSnapshotCommand] (DefaultQuartzScheduler_Worker-72) [18925ea3] Ending command with failure: org.ovirt.engine.core.bll.RemoveSn  apshotCommand
  2014-07-29 16:38:23,263 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [18925ea3] Correlation ID: 18925ea3, Call Stack: null  , Custom Event ID: -1, Message: Failed to delete snapshot '11' for VM 'vm_1'.

Version-Release number of selected component (if applicable):
beta.2

How reproducible:
100%

Steps to Reproduce:
1.delete a middle snapshot in a running vm(have two at least)

Actual results:
vdsm fails to locate base volume which leads to engine's exceptions and failure

Expected results:
operation should succeed 

Additional info:

Comment 1 Adam Litke 2014-07-29 20:00:45 UTC
This is probably due to a known SELinux issue with the libvirt/qemu packages that are installed from: http://files.pipo.sk/

Please run 'setenforce 0' on your vdsm host and see if the problem goes away.  If so, then this issue will be resolved in the upstream versions of qemu and libvirt once officially released.

Comment 2 Ori Gofen 2014-07-30 11:05:42 UTC
setting SELinux to permissive did seem to solve this issue.

Comment 3 Allon Mureinik 2014-07-30 11:19:42 UTC
(In reply to Adam Litke from comment #1)
> This is probably due to a known SELinux issue with the libvirt/qemu packages
> that are installed from: http://files.pipo.sk/
Do we/libvirt have a BZ to track this issue?

Comment 4 Adam Litke 2014-07-30 13:59:56 UTC
Since you are using an unreleased and privately patched version of libvirt there is no formal bug.  Peter Krempa is aware of the problem in the implementation and he has provided me with a new build to test.  I'll see if it resolves the issue and place a link to the new libvirt rpms here.

Comment 5 Adam Litke 2014-07-30 14:01:43 UTC
Actually, I reported the issue to Peter here: https://bugzilla.redhat.com/show_bug.cgi?id=1102881

Comment 6 Adam Litke 2014-08-06 14:39:17 UTC

*** This bug has been marked as a duplicate of bug 1102881 ***