Description of problem: image-uploader calculation time of the requisite disk space takes a lot of time. After ~45 min, for image with 20GB disk, on a local machine, with low disk space (200M), error was printed: "Not enough space in /tmp: 20480Mb are needed." Version-Release number of selected component (if applicable): sf17 Additional info on the long calculation: "It is really slow calculating the required space because it has to read the archive in order to get the size of the files it contains. image-uploader use tarfile for traversing the tar.gz content and get the size of the files it contains. this require to read the whole archive for having the file sizes and the archive is quite big because it includes a whole VM that easily can grow up over 20Gb I've seen gzip -l but it works only for files smaller than 4Gb" by sbonazzo Comment was found during verify of Bug 786887
We're thinking of leaving the space test as optional doing the following: - Warn the user to ensure there is enough space in $TMPDIR with an additional alert if there is less than 10*size of the tar free space. - Ask the user if he want to check if there is enough space, warning him that it could take a lot of time. - Avoid the question in unattended mode by supporting command line option and configuration option. We'll add also a suggestion for using tar -S at archive creation to handle correctly the sparse files in man page. It has been suggested also to use qcow2 format and compression with qemu-img: qemu-img convert -p -c -O qcow2 disk${i}.img sparse/disk${i}.img for having smaller tar archives, but this last suggestion has to be tested because I'm not sure it will work wit image-uploader.
(In reply to Sandro Bonazzola from comment #1) > We're thinking of leaving the space test as optional doing the following: > I think the better way would be to allow them to 'opt-out' of the check as opposed to 'opt-in'. This is an important safety check to ensure that they don't fill up their temp space and OSes generally don't handle it well when they have no room to move. Ex: Checking for requisite space in /var/tmp/blah/. Skip? (default: No): > - Warn the user to ensure there is enough space in $TMPDIR with an > additional alert if there is less than 10*size of the tar free space. > - Ask the user if he want to check if there is enough space, warning him > that it could take a lot of time. Ack. > - Avoid the question in unattended mode by supporting command line > option and configuration option. Again. Default shd be to check for requisite space. > > We'll add also a suggestion for using tar -S at archive creation to handle > correctly the sparse files in man page. > Ack > It has been suggested also to use qcow2 format and compression with qemu-img: > > qemu-img convert -p -c -O qcow2 disk${i}.img sparse/disk${i}.img > > for having smaller tar archives, but this last suggestion has to be tested > because I'm not sure it will work wit image-uploader. I don't think the uploader will care. As long as OVF format supports it, then you're good.
Another way could be opening the .ovf xml file and require as much space as the image size declared in the xml file. If the xml file is the first added to the archive this should be quite fast. In this case however we're requiring more space than the minimum needed and we don't gain speed if the xml file is the last added to the archive.
(In reply to Sandro Bonazzola from comment #3) > Another way could be opening the .ovf xml file and require as much space as > the image size declared in the xml file. If the xml file is the first added > to the archive this should be quite fast. > In this case however we're requiring more space than the minimum needed and > we don't gain speed if the xml file is the last added to the archive. Yes extracting only the OVF xml from the tar.gz and doing an approximate calculation would work.
So, changes needed: UX: Warn the user to ensure there is enough space in $TMPDIR with an additional alert if there is less than 10*size of the tar free space. UX: Checking for requisite space in $TMPDIR may take a lot of time. Skip? (default: No): UX: unattended mode should allow to skip the space check but perform it as default. UX: allow to choose to use the .ovf space requirements for the check (may or may not be faster depending on how the tar.gz was created) DOC / help: add a suggestion for: - using tar -S at archive creation to handle correctly the sparse files - convert the image for reducing the image size - first add .ovf file to the tar archive I think we can add here: - Allow to import from a directory, leaving to the user the option of extracting himself the archive allowing us to skip the tar extraction and $TMPDIR space check (but this may be done in a separate bug). Am I missing something?
(In reply to Sandro Bonazzola from comment #5) See also: https://bugzilla.redhat.com/1119798 > UX: Warn the user to ensure there is enough space in $TMPDIR with an > additional alert if there is less than 10*size of the tar free space. > UX: Checking for requisite space in $TMPDIR may take a lot of time. Skip? > (default: No): done > UX: unattended mode should allow to skip the space check but perform it as > default. done > UX: allow to choose to use the .ovf space requirements for the check (may or > may not be faster depending on how the tar.gz was created) now we have the capability to skip the whole test if the user knows that it has sufficient space: on my opinion no really need to add also this. > DOC / help: add a suggestion for: > - using tar -S at archive creation to handle correctly the sparse files done > - convert the image for reducing the image size now we are handling correctly sparse file so no need to explicit convert > - first add .ovf file to the tar archive > > I think we can add here: > - Allow to import from a directory, leaving to the user the option of > extracting himself the archive allowing us to skip the tar extraction and > $TMPDIR space check (but this may be done in a separate bug). already done in the past.
Might be related to similar issue pkubica was facing with the ovirt appliance.
Verified in: rhevm 3.5.0-0.17.beta.el6ev rhevm-image-uploader-3.5.0-3.el6ev.noarch with .ovf image the VM with 50GB disk was uploaded in ~5 minutes and when I didn't have a space in /tmp, I got an error within 1 minute (ERROR: Not enough space in /tmp: up to 51200Mb are needed.)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-0192.html