Bug 1178012

Summary: RHEV: Faulty storage allocation checks when importing a disk from glance
Product: Red Hat Enterprise Virtualization Manager Reporter: Allon Mureinik <amureini>
Component: ovirt-engineAssignee: Vered Volansky <vered>
Status: CLOSED CURRENTRELEASE QA Contact: Elad <ebenahar>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.4.0CC: acanan, amureini, bugs, dfediuck, ebenahar, eedri, gklein, lpeer, lsurette, nlevinki, rbalakri, Rhev-m-bugs, scohen, srevivo, tnisan, vered, yeylon, ykaul, ylavi
Target Milestone: ovirt-3.6.0-rc   
Target Release: 3.6.0   
Hardware: Unspecified   
OS: All   
Whiteboard:
Fixed In Version: ovirt-engine-3.5.0_vt6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1053750 Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1053750    
Bug Blocks: 960934    
Attachments:
Description Flags
engine log amd db dump
none
vdsm log none

Description Allon Mureinik 2015-01-01 12:29:32 UTC
+++ This bug was initially created as a clone of Bug #1053750 +++

+++ This bug was initially created as a clone of Bug #1053746 +++

+++ This bug was initially created as a clone of Bug #1053733 +++

+++ This bug was initially created as a clone of Bug #960934 +++

The storage requirements for the imported disk:

      | File Domain                             | Block Domain
 -----|-----------------------------------------|-------------
 qcow | preallocated : 1.1 * disk capacity      |1.1 * min(used ,capacity) 
      | sparse: 1.1 * min(used ,capacity)       |
 -----|-----------------------------------------|-------------
 raw  | preallocated: disk capacity             |disk capacity
      | sparse: min(used,capacity)

-------------------------------------------------------------------------------
This bug is a RHEV tracker for the QE team to verify against RHEVM 3.5.0

Comment 1 Allon Mureinik 2015-01-01 12:30:43 UTC
bug was put on ovirt-engine-config by mistake, please ignore the original assignment.

Comment 2 Elad 2015-01-12 10:03:58 UTC
The same as https://bugzilla.redhat.com/show_bug.cgi?id=1178010
Import an image from glance, while its size equales to the free space of the domain, is not blocked and the import operation is performed. After that, the domain is reported to have 0G free space.

Image size is 5368709120:


2015-01-12 11:53:40,402 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (org.ovirt.thread.pool-7-thread-31) [6cba6dbe] START, CreateImageVDSCommand( stor
agePoolId = 00000002-0002-0002-0002-0000000003e1, ignoreFailoverLimit = false, storageDomainId = 51ab53aa-9b94-4dec-b180-01a0c9de8915, imageGroupId = 93c411bb-699c-4987-8820-71a
5611f73ac, imageSizeInBytes = 5368709120, volumeFormat = RAW, newImageId = 4c0bd0f9-96af-4c6f-9f90-a6f62a32ae71, newImageDescription = vm-1_Disk1 (ae529be)), log id: 7f281ee9


After the import, the domain is reported with 0 GB free space:

2015-01-12 11:54:05,425 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [285829e8] Correlation ID: null, Call Sta
ck: null, Custom Event ID: -1, Message: Critical, Low disk space. fc1 domain has 0 GB of free space


Vered, the same as https://bugzilla.redhat.com/show_bug.cgi?id=1178010, I don't think it is the right behavior.

Comment 3 Vered Volansky 2015-01-12 15:57:34 UTC
(In reply to Elad from comment #2)
> The same as https://bugzilla.redhat.com/show_bug.cgi?id=1178010
> Import an image from glance, while its size equales to the free space of the
> domain, is not blocked and the import operation is performed. After that,
> the domain is reported to have 0G free space.

Equality is enough for us to do the operation, we're blocking at less than.
OGB is not a desirable status, but also not an illegal one, and is certainly not in the scope of this bz.

> Vered, the same as https://bugzilla.redhat.com/show_bug.cgi?id=1178010, I
> don't think it is the right behavior.
In that bug you described a less than situation where free_space < needed_space, so I don't think it's the same issue, please correct me if I'm wrong.

Comment 4 Elad 2015-01-13 10:56:07 UTC
I managed to import an image from Glance, which its size was bigger than the free space on the domain. Occurred on NFS domain. 
On iSCSI, the same action was blocked.


Free space on the domain was 6G and the image size is 7G  (imageSizeInBytes = 7516192768):

2015-01-13 12:06:47,806 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (org.ovirt.thread.pool-7-thread-12) [2384aa32] START, CreateImageVDSCommand( storagePoolId = 00000002-0002-0002-0002-0000000003e1, ignoreFailoverLimit = false, storageDomainId = 4b9b6b0a-31c7-4ed4-928b-771ef29e8c60, imageGroupId = 2bf52586-a3f1-4e33-9037-b121574aaab3, imageSizeInBytes = 7516192768, volumeFormat = RAW, newImageId = 122c26a9-7daf-4897-9105-0a8714b95c3c, newImageDescription = 1 (0aef12f)), log id: 1213fc86

During the import, the domain free space is decreasing until it runs out of space:

2015-01-13 12:07:06,620 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [befb354] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Messag
e: Critical, Low disk space. lion1 domain has 5 GB of free space

The import operation failed:

2015-01-13 12:08:47,084 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-7-thread-18) [541c8b14] Correlation ID: 573c8183, Call Stack: null, Custom Event ID: -1, Message: User admin failed to import image GlanceDisk-0aef12f to domain lion1.


The image gets removed after the failure:

2015-01-13 12:08:43,560 INFO  [org.ovirt.engine.core.bll.RemoveDiskCommand] (org.ovirt.thread.pool-7-thread-18) [19fff775] Running command: RemoveDiskCommand internal: true. Entities affected :  ID: 2bf52586-a3f1-4e33-9037-b121574aaab3 Type: DiskAction group DELETE_DISK with role type USER


I checked using vt13.6

Comment 5 Elad 2015-01-13 10:59:25 UTC
Created attachment 979562 [details]
engine log amd db dump

Comment 6 Elad 2015-01-13 11:11:56 UTC
Created attachment 979568 [details]
vdsm log

The failure in vdsm:


769a74e3-b33a-4c67-a3b5-ed2803ee795d::ERROR::2015-01-13 12:08:35,005::task::866::Storage.TaskManager.Task::(_setError) Task=`769a74e3-b33a-4c67-a3b5-ed2803ee795d`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/task.py", line 334, in run
    return self.cmd(*self.argslist, **self.argsdict)
  File "/usr/share/vdsm/storage/securable.py", line 77, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 1634, in downloadImage
    .download(methodArgs, sdUUID, imgUUID, volUUID)
  File "/usr/share/vdsm/storage/image.py", line 1254, in download
    imageSharing.download(vol.getVolumePath(), methodArgs)
  File "/usr/share/vdsm/storage/imageSharing.py", line 168, in download
    downloadImageImpl(dstImgPath, methodArgs)
  File "/usr/share/vdsm/storage/imageSharing.py", line 69, in httpDownloadImage
    methodArgs.get("headers", {}))
  File "/usr/share/vdsm/storage/curlImgWrap.py", line 71, in download
    raise CurlError(rc, out, err)
CurlError: ecode=1, stdout=[], stderr=["fork_exec('/usr/bin/curl', 'curl', '-q', '--silent', '--fail', '--show-error', '-H', 'X-Auth-Token: 8930cf80cf334eb88133088ba6e1ed92', 'http://10.35.160.139:9292/v1/images/0aef12f3-ef2f-4b1e-930e-caea9b388da6')", "fork_exec('/bin/dd', 'dd', 'bs=2M', 'of=/rhev/data-center/00000002-0002-0002-0002-0000000003e1/4b9b6b0a-31c7-4ed4-928b-771ef29e8c60/images/2bf52586-a3f1-4e33-9037-b121574aaab3/122c26a9-7daf-4897-9105-0a8714b95c3c')", "dd: writing `/rhev/data-center/00000002-0002-0002-0002-0000000003e1/4b9b6b0a-31c7-4ed4-928b-771ef29e8c60/images/2bf52586-a3f1-4e33-9037-b121574aaab3/122c26a9-7daf-4897-9105-0a8714b95c3c': No space left on device", '28+605162 records in', '28+605161 records out', '7335698432 bytes (7.3 GB) copied, 104.057 s, 70.5 MB/s', 'curl: (23) Failed writing body (2560 != 15928)'], message=None

Comment 7 Yaniv Lavi 2015-01-19 13:03:37 UTC
Doesn't seem urgent, will be fixed in 3.6 only, unless a customer will want this for 3.5.

Comment 8 Vered Volansky 2015-01-28 10:56:24 UTC
Elad, which glance image did you use?
The heaviest is 1GB.
Please state all steps to reproduce, since we're out of synch on this.

Comment 9 Elad 2015-01-28 11:40:27 UTC
I didn't use the ovirt-glance public repository. I used one of RHOS QE group's setups.
The image size I used was 7G  (imageSizeInBytes = 7516192768).

Comment 10 Elad 2015-04-26 10:12:31 UTC
An attempt to import an image from Glance repository to a storage domain that does not have enough free space is block on CanDoAction

Verified using ovirt 3.6.0-1
ovirt-engine-3.6.0-0.0.master.20150412172306.git55ba764.el6.noarch

Comment 11 Allon Mureinik 2016-03-10 10:46:26 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE

Comment 12 Allon Mureinik 2016-03-10 10:49:30 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE

Comment 13 Allon Mureinik 2016-03-10 12:06:01 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE