Bug 967513 - [Image-uploader] image-uploader requisite space on the local system calculation time is very long
[Image-uploader] image-uploader requisite space on the local system calculati...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-image-uploader (Show other bugs)
3.2.0
Unspecified Unspecified
high Severity medium
: ---
: 3.5.0
Assigned To: Simone Tiraboschi
Petr Kubica
integration
: Triaged
Depends On:
Blocks: rhev3.5beta 1156165
  Show dependency treegraph
 
Reported: 2013-05-27 06:29 EDT by Ilanit Stein
Modified: 2015-02-12 03:06 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
This update provides performance improvements via better handling of sparse files.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-02-11 12:46:38 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Ilanit Stein 2013-05-27 06:29:44 EDT
Description of problem:


image-uploader calculation time of the requisite disk space takes a lot of time.

After ~45 min, for image with 20GB disk, on a local machine, with low disk space (200M), error was printed: "Not enough space in /tmp: 20480Mb are needed." 

Version-Release number of selected component (if applicable):
sf17

Additional info on the long calculation:

"It is really slow calculating the required space because it has to read the archive in order to get the size of the files it contains.
image-uploader use tarfile for traversing the tar.gz content and get the size of the files it contains. this require to read the whole archive for having the file sizes
and the archive is quite big because it includes a whole VM that easily can grow up over 20Gb
I've seen gzip -l but it works only for files smaller than 4Gb" 
by sbonazzo

Comment was found during verify of Bug 786887
Comment 1 Sandro Bonazzola 2013-05-27 07:56:18 EDT
We're thinking of leaving the space test as optional doing the following:

- Warn the user to ensure there is enough space in $TMPDIR with an
additional alert if there is less than 10*size of the tar free space.
- Ask the user if he want to check if there is enough space, warning him
that it could take a lot of time.
- Avoid the question in unattended mode by supporting command line
option and configuration option.

We'll add also a suggestion for using tar -S at archive creation to handle correctly the sparse files in man page.

It has been suggested also to use qcow2 format and compression with qemu-img:

  qemu-img convert -p -c -O qcow2 disk${i}.img sparse/disk${i}.img

for having smaller tar archives, but this last suggestion has to be tested because I'm not sure it will work wit image-uploader.
Comment 2 Keith Robertson 2013-05-28 10:05:43 EDT
(In reply to Sandro Bonazzola from comment #1)
> We're thinking of leaving the space test as optional doing the following:
> 
I think the better way would be to allow them to 'opt-out' of the check as opposed to 'opt-in'.  This is an important safety check to ensure that they don't fill up their temp space and OSes generally don't handle it well when they have no room to move.  

Ex: Checking for requisite space in /var/tmp/blah/. Skip? (default: No): 

> - Warn the user to ensure there is enough space in $TMPDIR with an
> additional alert if there is less than 10*size of the tar free space.
> - Ask the user if he want to check if there is enough space, warning him
> that it could take a lot of time.
Ack.
> - Avoid the question in unattended mode by supporting command line
> option and configuration option.

Again. Default shd be to check for requisite space.

> 
> We'll add also a suggestion for using tar -S at archive creation to handle
> correctly the sparse files in man page.
> 
Ack

> It has been suggested also to use qcow2 format and compression with qemu-img:
> 
>   qemu-img convert -p -c -O qcow2 disk${i}.img sparse/disk${i}.img
> 
> for having smaller tar archives, but this last suggestion has to be tested
> because I'm not sure it will work wit image-uploader.

I don't think the uploader will care. As long as OVF format supports it, then you're good.
Comment 3 Sandro Bonazzola 2013-08-12 10:57:35 EDT
Another way could be opening the .ovf xml file and require as much space as the image size declared in the xml file. If the xml file is the first added to the archive this should be quite fast.
In this case however we're requiring more space than the minimum needed and we don't gain speed if the xml file is the last added to the archive.
Comment 4 Keith Robertson 2013-08-12 11:13:14 EDT
(In reply to Sandro Bonazzola from comment #3)
> Another way could be opening the .ovf xml file and require as much space as
> the image size declared in the xml file. If the xml file is the first added
> to the archive this should be quite fast.
> In this case however we're requiring more space than the minimum needed and
> we don't gain speed if the xml file is the last added to the archive.

Yes extracting only the OVF xml from the tar.gz and doing an approximate calculation would work.
Comment 5 Sandro Bonazzola 2013-08-23 04:43:48 EDT
So, changes needed:

UX: Warn the user to ensure there is enough space in $TMPDIR with an
additional alert if there is less than 10*size of the tar free space.
UX: Checking for requisite space in $TMPDIR may take a lot of time. Skip? (default: No):
UX: unattended mode should allow to skip the space check but perform it as default.
UX: allow to choose to use the .ovf space requirements for the check (may or may not be faster depending on how the tar.gz was created)

DOC / help: add a suggestion for:
 - using tar -S at archive creation to handle correctly the sparse files
 - convert the image for reducing the image size
 - first add .ovf file to the tar archive

I think we can add here:
- Allow to import from a directory, leaving to the user the option of extracting himself the archive allowing us to skip the tar extraction and $TMPDIR space check (but this may be done in a separate bug).

Am I missing something?
Comment 6 Simone Tiraboschi 2014-09-10 04:24:29 EDT
(In reply to Sandro Bonazzola from comment #5)

See also: https://bugzilla.redhat.com/1119798

> UX: Warn the user to ensure there is enough space in $TMPDIR with an
> additional alert if there is less than 10*size of the tar free space.
> UX: Checking for requisite space in $TMPDIR may take a lot of time. Skip?
> (default: No):

done

> UX: unattended mode should allow to skip the space check but perform it as
> default.

done

> UX: allow to choose to use the .ovf space requirements for the check (may or
> may not be faster depending on how the tar.gz was created)

now we have the capability to skip the whole test if the user knows that it has sufficient space: on my  opinion no really need to add also this.

> DOC / help: add a suggestion for:
>  - using tar -S at archive creation to handle correctly the sparse files

done

>  - convert the image for reducing the image size

now we are handling correctly sparse file so no need to explicit convert

>  - first add .ovf file to the tar archive
> 
> I think we can add here:
> - Allow to import from a directory, leaving to the user the option of
> extracting himself the archive allowing us to skip the tar extraction and
> $TMPDIR space check (but this may be done in a separate bug).

already done in the past.
Comment 7 Petr Beňas 2014-10-21 11:47:57 EDT
Might be related to similar issue pkubica was facing with the ovirt appliance.
Comment 8 Petr Kubica 2014-10-23 14:42:18 EDT
Verified in:
rhevm 3.5.0-0.17.beta.el6ev
rhevm-image-uploader-3.5.0-3.el6ev.noarch

with .ovf image the VM with 50GB disk was uploaded in ~5 minutes
and when I didn't have a space in /tmp, I got an error within 1 minute (ERROR: Not enough space in /tmp: up to 51200Mb are needed.)
Comment 10 errata-xmlrpc 2015-02-11 12:46:38 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0192.html

Note You need to log in before you can comment on or make changes to this bug.