Bug 1147085

Summary: Memory volumes not deleted when removing a vm with snapshots
Product: [Retired] oVirt Reporter: Nir Soffer <nsoffer>
Component: ovirt-engine-coreAssignee: Idan Shaby <ishaby>
Status: CLOSED CURRENTRELEASE QA Contact: Ori Gofen <ogofen>
Severity: high Docs Contact:
Priority: high    
Version: 3.5CC: acanan, amureini, ecohen, gklein, iheim, lsurette, rbalakri, yeylon
Target Milestone: ---Keywords: Regression
Target Release: 3.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: storage
Fixed In Version: ovirt-3.5.0_rc4 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-17 12:38:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1073943    
Attachments:
Description Flags
engine log
none
vdsm log none

Description Nir Soffer 2014-09-26 21:01:27 UTC
Created attachment 941720 [details]
engine log

Description of problem:

When creating snapshot including memory snapshot, a new volume is
created for the memory contents. When removing a vm, the memory 
snapshot is not deleted. 

Version-Release number of selected component (if applicable):
oVirt Engine Version: 3.5.0-0.0.master.20140911091402.gite1c5ffd.fc20
vdsm master 29defc3faab3d7

How reproducible:
Always

Steps to Reproduce:
1. Create vm with 1G memory and one 1G thin provisioning disk
2. Run vm
3. Create snapshot (ensure that "save memory" is checked)
4. Stop vm
5. Remove vm

Actual results:
Memory volumes not removed, and the domain holding them free memory is smaller
then expected (10G instead of 15G).

The only way to remove the volume is to login to one of the hosts and remove
the unwanted lvs, but there is no easy way to detect that these volumes are 
indeed the unused.

Expected results:
Memory volumes should be removed when vm is removed

Additional info:

For testing this issue, I created two new iscsi storage domains, and repeated
the steps above 2 times.

This is the master domain contents at the end of the test.

# lvs ff559f46-c495-4f6b-901c-2a624042a050
  LV                                   VG                                   Attr       LSize
  3149a0f8-82d3-43a4-b7e6-d6033485afb0 ff559f46-c495-4f6b-901c-2a624042a050 -wi------- 128.00m
  75b7d7b4-5c1a-48fe-a079-0f6529bd8968 ff559f46-c495-4f6b-901c-2a624042a050 -wi------- 128.00m
  ids                                  ff559f46-c495-4f6b-901c-2a624042a050 -wi-ao---- 128.00m
  inbox                                ff559f46-c495-4f6b-901c-2a624042a050 -wi-a----- 128.00m
  leases                               ff559f46-c495-4f6b-901c-2a624042a050 -wi-a-----   2.00g
  master                               ff559f46-c495-4f6b-901c-2a624042a050 -wi-ao----   1.00g
  metadata                             ff559f46-c495-4f6b-901c-2a624042a050 -wi-a----- 512.00m
  outbox                               ff559f46-c495-4f6b-901c-2a624042a050 -wi-a----- 128.00m

The two 128M volumes are ovf store volumes.

This is the other domain contents at the end of the test:

# lvs 3e419414-ee73-47e8-809d-60de4a88403c
  LV                                   VG                                   Attr       LSize
  0ffd2268-ef48-4fbf-aaf6-73f4249d5f40 3e419414-ee73-47e8-809d-60de4a88403c -wi-------   1.38g
  58b9b9f8-b2c7-441f-acdb-2c0616dff271 3e419414-ee73-47e8-809d-60de4a88403c -wi-------   1.38g
  70aafeeb-6307-4451-a558-d707310667ab 3e419414-ee73-47e8-809d-60de4a88403c -wi-------   1.00g
  e89a3dc7-947c-4692-b5d1-eb8f5a4958cb 3e419414-ee73-47e8-809d-60de4a88403c -wi-------   1.00g
  ids                                  3e419414-ee73-47e8-809d-60de4a88403c -wi-ao---- 128.00m
  inbox                                3e419414-ee73-47e8-809d-60de4a88403c -wi-a----- 128.00m
  leases                               3e419414-ee73-47e8-809d-60de4a88403c -wi-a-----   2.00g
  master                               3e419414-ee73-47e8-809d-60de4a88403c -wi-a-----   1.00g
  metadata                             3e419414-ee73-47e8-809d-60de4a88403c -wi-a----- 512.00m
  outbox                               3e419414-ee73-47e8-809d-60de4a88403c -wi-a----- 128.00m

Note that the vm was created on the master domain 
(ff559f46-c495-4f6b-901c-2a624042a050), but the memory volumes are created
on the other domain (3e419414-ee73-47e8-809d-60de4a88403c) - this does not 
make sense and probably the root cause of this.

Looking in vdsm log, we can see that only images in the master domain
(ff559f46-c495-4f6b-901c-2a624042a050) were deleted. There is no other
deleteImage or deleteVolume request in vdsm log.

# grep 'Run and protect: deleteImage' vdsm.log 
Thread-34::INFO::2014-09-26 22:57:25,005::logUtils::48::dispatcher::(wrapper) Run and protect: deleteImage(sdUUID='ff559f46-c495-4f6b-901c-2a624042a050', spUUID='b86b687a-d073-497a-ac8a-249025419a3e', imgUUID='a256fc11-f9ab-4391-af78-2a96e27fe39b', postZero='false', force='false')
Thread-34::INFO::2014-09-26 22:57:25,462::logUtils::51::dispatcher::(wrapper) Run and protect: deleteImage, Return response: None
Thread-34::INFO::2014-09-26 23:00:45,883::logUtils::48::dispatcher::(wrapper) Run and protect: deleteImage(sdUUID='ff559f46-c495-4f6b-901c-2a624042a050', spUUID='b86b687a-d073-497a-ac8a-249025419a3e', imgUUID='9288c7e8-43fd-4174-9889-634fb2e0437a', postZero='false', force='false')
Thread-34::INFO::2014-09-26 23:00:46,338::logUtils::51::dispatcher::(wrapper) Run and protect: deleteImage, Return response: None
Thread-34::INFO::2014-09-26 23:07:24,205::logUtils::48::dispatcher::(wrapper) Run and protect: deleteImage(sdUUID='ff559f46-c495-4f6b-901c-2a624042a050', spUUID='b86b687a-d073-497a-ac8a-249025419a3e', imgUUID='60e28edc-4bb3-4f87-b6a2-5dc45f1a0971', postZero='false', force='false')
Thread-34::INFO::2014-09-26 23:07:24,659::logUtils::51::dispatcher::(wrapper) Run and protect: deleteImage, Return response: None

Since vdsm was not asked to delete the images, this is clearly an engine bug.

I did not test it with older engine version, but I'm sure this is a regression.
I'm creating and deleting vms regularly while verifying, and I would notice
if my storage domain loose free space.

Comment 1 Nir Soffer 2014-09-26 21:02:40 UTC
Created attachment 941721 [details]
vdsm log

Comment 2 Allon Mureinik 2014-09-28 08:22:27 UTC
Idan, I think this is another instance of a bug you're already working on. Please verify (and solve :-))

Comment 3 Sandro Bonazzola 2014-10-17 12:38:25 UTC
oVirt 3.5 has been released and should include the fix for this issue.