Bug 1099894
| Summary: | [engine-backend] live storage migration isn't blocked in case there is not enough free space in the source domain | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Elad <ebenahar> | ||||
| Component: | ovirt-engine | Assignee: | Vered Volansky <vered> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Elad <ebenahar> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 3.4.0 | CC: | amureini, derez, gklein, lpeer, rbalakri, Rhev-m-bugs, scohen, tnisan, vered, yeylon | ||||
| Target Milestone: | ovirt-3.6.0-rc | Keywords: | ZStream | ||||
| Target Release: | 3.6.0 | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | ovirt-engine-3.6.0_qa1 | Doc Type: | Bug Fix | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1185570 (view as bug list) | Environment: | |||||
| Last Closed: | Type: | Bug | |||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1185570 | ||||||
| Attachments: |
|
||||||
Severity was set to high since this is a common scenario because one of the main reasons for administrator to live migrate a disk is that the source storage domain is lack of free space. Looks like this should have been solved as part of bug 970932. Vered, please take a look? In any event, the CDA validation is a nicety, not a blocker (as its racy by nature). Moving to 3.4.1. (In reply to Allon Mureinik from comment #2) > Looks like this should have been solved as part of bug 970932. > Vered, please take a look? > > In any event, the CDA validation is a nicety, not a blocker (as its racy by > nature). > Moving to 3.4.1. I'm not sure since the fix of 970932 is for blocking LSM if the DST domain has not enough free space, it was verified according to it. (In reply to Elad from comment #3) > (In reply to Allon Mureinik from comment #2) > > Looks like this should have been solved as part of bug 970932. > > Vered, please take a look? > > > > In any event, the CDA validation is a nicety, not a blocker (as its racy by > > nature). > > Moving to 3.4.1. > > I'm not sure since the fix of 970932 is for blocking LSM if the DST domain > has not enough free space, it was verified according to it. Good point, thanks. In any event, I'm devel-acking this one - this is obviously a bug, and should be solved. bz 970932 only handles MoveOrCopyDiskCommand, which is a cold move. It validates the destination storage domain sapce reserves. LiveMigrateVmDisksCommand handles the snapshot per running VM, which is where the source SD space validation should be handled. Currently there are no space validations in this command. Note that the fix for bug 1056922 makes LSM only snapshot the relevant disk, not the entire VM, so there is some mitigation for this issue in 3.4.0. In case there is not enough space in the source storage domain for the live snapshots of the LSM to be created on, the LSM operation is blocked with CDA: 2014-08-27 10:17:02,335 WARN [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (ajp--127.0.0.1-8702-5) [38148f72] CanDoAction of action LiveMigrateVmDisks failed. Reasons:VAR__ACTION__MOVE,VAR__TYPE__VM_DISK,ACTION_TYPE_FAILED_DISK_SPACE_LOW_ON_STORAGE_DOMAIN,$storageName block1 Verified using upstream ovirt-3.5 RC1.1 Missed the cutoff, will be included when 3.5.0-1 is built. An attempt to live migrate disks while the source domain doesn't have enough free space for the snapshots is blocked on CDA. Verified using ovirt-engine-3.6.0_qa1 ovirt-engine-3.6.0-0.0.master.20150412172306.git55ba764.el6.noarch vdsm-4.17.0-632.git19a83a2.el7.x86_64 RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE |
Created attachment 897954 [details] logs from engine and vdsm Description of problem: I tried to live migrate several disks (5) from one domain to another, while the source domain had only 5 GB free space. Live storage migration started and wasn't blocked. It failed due to a lack of space in the source domain with 'CannotCreateLogicalVolume' error message in vdsm.log Version-Release number of selected component (if applicable): av9.1 rhevm-3.4.0-0.20.el6ev.noarch vdsm-4.14.7-2.el6ev.x86_64 How reproducible: Always Steps to Reproduce: On a shared DC: 1. Create 2 iSCSI storage domains, one of them with capacity of 20G - SD1. After its creation, the 20G domain (SD1) is supposed to have 15G free space. 2. Create a VM with 1 5G preacllocated disk attached, located on SD1. Start the VM and attach another 4 disks located on SD1 to it: 2 disks of 2G and 2 disks of 1G 3. Mark all the 5 disks and live migrate them to the second domain Actual results: Domain 1-2-iscsi has only 4G free space before the LSM: id | storage_name | available_disk_size --------------------------------------+--------------+--------------------- 393d9b97-476c-42ab-a6df-fcd383af98f2 | ISO_DOMAIN | 0 ce1e07a9-9264-48ae-91b2-06d1463b0ec1 | 1-2-iscsi | 4 f2718e18-cf10-4afe-9b55-9967d147709b | 1-1-iscsi | 32 Live storage migration begins although the source domain has only 4G free space. 2014-05-21 15:03:59,988 INFO [org.ovirt.engine.core.bll.lsm.LiveMigrateVmDisksCommand] (org.ovirt.thread.pool-4-thread-7) [2bb9a4a4] Running command: LiveMigrateVmDisksCommand Task handler: LiveSnapshotTaskHandler internal: false. Entities affected : ID: 6902bcdf-75c4-4d6e-a9ae-072366c569cb Type: Disk, ID: f2718e18-cf10-4afe-9b55-9967d147709b Type: Storage, ID: 434caca6-dece-4e5b-a01b-5a9324cd6a5f Type: Disk, ID: f2718e18-cf10-4afe-9b55-9967d147709b Type: Storage, ID: bf10b4cd-a1a5-440f-8673-3d0ec9ee492a Type: Disk, ID: f2718e18-cf10-4afe-9b55-9967d147709b Type: Storage, ID: 7a2b70b8-d886-480c-af87-443328962084 Type: Disk, ID: f2718e18-cf10-4afe-9b55-9967d147709b Type: Storage, ID: 9a41bb74-db54-42f6-8961-2efb5af24c25 Type: Disk, ID: f2718e18-cf10-4afe-9b55-9967d147709b Type: Storage The source domain is supposed to allocate 5 new 1G logical volumes for the live snapshot new volumes that are being created on it as part of the LSM. The operation fails on vdsm: 426ce720-8e3b-40e4-8422-927a30c31426::ERROR::2014-05-21 15:04:11,249::task::866::TaskManager.Task::(_setError) Task=`426ce720-8e3b-40e4-8422-927a30c31426`::Unexpected error Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 873, in _run return fn(*args, **kargs) File "/usr/share/vdsm/storage/task.py", line 334, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/share/vdsm/storage/securable.py", line 75, in wrapper return method(self, *args, **kwargs) File "/usr/share/vdsm/storage/sp.py", line 1770, in createVolume desc=desc, srcImgUUID=srcImgUUID, srcVolUUID=srcVolUUID) File "/usr/share/vdsm/storage/sd.py", line 421, in createVolume preallocate, diskType, volUUID, desc, srcImgUUID, srcVolUUID) File "/usr/share/vdsm/storage/volume.py", line 468, in create raise e CannotCreateLogicalVolume: Cannot create Logical Volume: ('ce1e07a9-9264-48ae-91b2-06d1463b0ec1', 'c2b13de4-ffc0-4765-ab35-b5333b580243') Expected results: If there is not enough free space on the source storage domain, the live storage migration shouldn't be allowed Additional info: logs from engine and vdsm