Bug 1316479

Created attachment 1134811 [details]
v2v log 1.5M space left

Created attachment 1134812 [details]
v2v log 68M space left

I think for both this bug and bug 1316487 we should just check
if the host $TMPDIR etc has "sufficient" space and warn or error
out before we start if not.

*** Bug 1316487 has been marked as a duplicate of this bug. ***

Fixed in upstream commits:

https://github.com/libguestfs/libguestfs/commit/24130d787256dc3d9e37e85b1ba35ac7dbeb86a1
https://github.com/libguestfs/libguestfs/commit/a7fb3d601ab2ae3cdf2074d4c47afe59f00eea6d
https://github.com/libguestfs/libguestfs/commit/d8a465a14c3596a9fea53560bc7ad2ee6c3111bc
https://github.com/libguestfs/libguestfs/commit/9221ef6f7959ae3b979fd05007363810bc66062b

Note that this just displays an error message if there is insufficient
free space (under 1GB) before starting the conversion.  The error
message would be something like:

  virt-v2v: error: insufficient free space in the conversion server temporary 
  directory /var/tmp (853.8M).

  Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR 
  environment variable to point to another directory with more than 1GB of 
  free space.

  See also the virt-v2v(1) manual, section "Minimum free space check in the 
  host".

I can reproduce this bug with package:
libguestfs-1.28.1-1.55.el7_2.1.x86_64
virt-v2v-1.28.1-1.55.el7_2.1.x86_64

Then try to verify this bug with new build:

# rpm -q libguestfs virt-v2v 
libguestfs-1.32.5-9.el7.x86_64
virt-v2v-1.32.5-9.el7.x86_64

Steps:
1. Check virt-v2v manual page:
# man virt-v2v
...
   Minimum free space check in the host
       You must have sufficient free space in the host directory used to store temporary overlays (except in [--in-place] mode).  To
       find out which directory this is, use:

        $ df -h "`guestfish get-cachedir`"
        Filesystem        Size  Used Avail Use% Mounted on
        /dev/mapper/root   50G   40G  6.8G  86% /

       and look under the "Avail" column.  Virt-v2v will refuse to do the conversion at all unless at least 1GB is available there.

       See also "RESOURCE REQUIREMENTS" above.

Result: We find that 'Virt-v2v will refuse to do the conversion at all unless at least 1GB is available there.', so while i verify this bug i just cover 2 scenario around '3M' and '248M'.

Scenario-1 
1.1 Left about 3M free space on host
# df -h /var/tmp/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       150G  150G  3.0M 100% /

1.2. Convert a xen guest to rhevm
# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
virt-v2v: error: insufficient free space in the conversion server temporary 
directory /var/tmp (2.9M).

Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR 
environment variable to point to another directory with more than 1GB of 
free space.

See also the virt-v2v(1) manual, section "Minimum free space check in the 
host".

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Result: We can get a clear error while host free space less than 1GB.

1.3. Repeat step1.2 after set LIBGUESTFS_CACHEDIR value to a directory with more than 1GB of free space.

# df -h /home/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda7       2.0G  6.0M  1.8G   1% /home

# export LIBGUESTFS_CACHEDIR=/home
# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on juzhou1
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
[   0.5] Creating an overlay to protect the source from being modified
[   0.9] Initializing the target -o rhev -os 10.73.72.63:/home/nfs_export
[   1.1] Opening the overlay
virt-v2v: error: libguestfs error: /usr/bin/supermin5 exited with error 
status 1.
To see full error messages you may need to enable debugging.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

I will attach virt-v2v debug log for scenario-1, called virt-v2v-log1.

Scenario-2 
2.1 Left about 258M free space on host
# df -h /var/tmp/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       150G  150G  248M 100% /

2.2. Convert a xen guest to rhevm

# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on juzhou1 
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
virt-v2v: error: insufficient free space in the conversion server temporary 
directory /var/tmp (247.9M).

Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR 
environment variable to point to another directory with more than 1GB of 
free space.

See also the virt-v2v(1) manual, section "Minimum free space check in the 
host".

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Result: We can get a clear error while host free space less than 1GB.

2.3. Repeat step2.2 after set LIBGUESTFS_CACHEDIR value to a directory with more than 1GB of free space.
# df -h /home/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda7       2.0G  6.0M  1.8G   1% /home

# export LIBGUESTFS_CACHEDIR=/home

# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on juzhou1

Result: conversion finished without error.


so rjones, please help me have a look of scenario-1 issue, thanks.

Created attachment 1176828 [details]
virt-v2v-log1 for scenario-1(verify-bug)

# export LIBGUESTFS_CACHEDIR=/home
# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on juzhou1
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
[   0.5] Creating an overlay to protect the source from being modified
[   0.9] Initializing the target -o rhev -os 10.73.72.63:/home/nfs_export
[   1.1] Opening the overlay
virt-v2v: error: libguestfs error: /usr/bin/supermin5 exited with error 
status 1.

...

supermin: ext2: creating empty ext2 filesystem '/home/.guestfs-0/appliance.d.hib4dgn3/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
supermin: ext2: copying kernel modules
supermin: ext2: creating minimal initrd '/home/.guestfs-0/appliance.d.hib4dgn3/initrd'
supermin: ext2: wrote 27 modules to minimal initrd
supermin: exception: Sys_error("No space left on device")
libguestfs: trace: v2v: launch = -1 (error)
virt-v2v: error: libguestfs error: /usr/bin/supermin5 exited with error 


I think it's just a really bad idea to set LIBGUESTFS_CACHEDIR
to /home.  However since you had 1.8GB free I don't really know why
it would have failed.

I suppose you could try this command (as root):

/usr/bin/supermin5 --build -v -v -v --copy-kernel -f ext2 \
  --host-cpu x86_64 /usr/lib64/guestfs/supermin.d -o /home/.guestfs-0/appliance.d

That will produce a much larger amount of debugging and should show why
it is failing.

Hi rjones,

After reproduce that error, then i tried as you said
1.
# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on juzhou1
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
virt-v2v: error: insufficient free space in the conversion server temporary 
directory /var/tmp (3.6M).

2.
# df -h /home/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda7       2.0G  6.1M  1.8G   1% /home

3.
# /usr/bin/supermin5 --build -v -v -v --copy-kernel -f ext2 \
>  --host-cpu x86_64 /usr/lib64/guestfs/supermin.d -o /home/.guestfs-0/appliance.d
supermin: version: 5.1.16
supermin: rpm: detected RPM version 4.11
supermin: package handler: fedora/rpm
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
tar: usr/sbin/guestfsd: Wrote only 8192 of 10240 bytes
tar: usr/share/guestfs: Cannot mkdir: No space left on device
tar: Exiting with failure status due to previous errors
supermin: zcat '/usr/lib64/guestfs/supermin.d/daemon.tar.gz' | tar -C '/tmp/supermin5a0221.tmpdir/base.d' -xf -: command failed, see earlier errors


Does it mean although there is free space on /home, but it cannot be used by usr/share/guestfs command, thanks.
If so, can you give some tips about set directory to 'LIBGUESTFS_CACHEDIR', thanks.

OK I see, what actually happens here is:

(1) LIBGUESTFS_CACHEDIR asks libguestfs to put the cache in that directory

(2) supermin doesn't know or care about any LIBGUESTFS_* variables

(3) supermin uses /tmp (or TMPDIR) for temporary files

So the fix is incomplete.  It should also check for free space
in /tmp (or TMPDIR).

I added this supermin patch which makes it use /var/tmp instead of /tmp
if TMPDIR is not set.

https://github.com/libguestfs/supermin/commit/12fc339efb1d287593b53da1dfddc88f876f8e7d

Try to verify this bug again with new build:
libguestfs-1.32.5-10.el7.x86_64
virt-v2v-1.32.5-10.el7.x86_64
libvirt-2.0.0-1.el7.x86_64

Steps:
1. Left about 3M free space on host

# df -h  /var/tmp/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       150G  150G  3.8M 100% /


2. Convert a xen guest to rhevm
# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on jtest1
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
virt-v2v: error: insufficient free space in the conversion server temporary 
directory /var/tmp (3.6M).

Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR 
environment variable to point to another directory with more than 1GB of 
free space.

See also the virt-v2v(1) manual, section "Minimum free space check in the 
host".

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Result: We can get a clear error while host free space less than 1GB, then try to set the LIBGUESTFS_CACHEDIR and do coversion again.

3.
Scenario-1 # export LIBGUESTFS_CACHEDIR=/opt

# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on jtest1
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
[   0.6] Creating an overlay to protect the source from being modified
[   1.0] Initializing the target -o rhev -os 10.73.72.63:/home/nfs_export
[   1.2] Opening the overlay
virt-v2v: error: libguestfs error: /usr/bin/supermin5 exited with error 
status 1.
To see full error messages you may need to enable debugging.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Result: Conversion failed and can also reproduce Comment 11 issue.

Scenario-2

# unset LIBGUESTFS_CACHEDIR
# export TMPDIR=/opt
# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on jtest1

Result: Conversion finished without error.

So rjones, please help me have a look, i think bug issue not fixed yet, thanks.

Yes, the new build (of supermin) only changes the location of
temporary files to /var/tmp.

It doesn't ensure there is sufficient space on /var/tmp, but these errors
are easy to diagnose.  If the failure happened the user would run
virt-v2v with -v -x, or libguestfs-test-tool, and the problem is obvious.
This same thing could happen with any program that uses temporary files.

(In reply to Richard W.M. Jones from comment #15)
> Yes, the new build (of supermin) only changes the location of
> temporary files to /var/tmp.
> 
> It doesn't ensure there is sufficient space on /var/tmp, but these errors
> are easy to diagnose.  If the failure happened the user would run
> virt-v2v with -v -x, or libguestfs-test-tool, and the problem is obvious.
> This same thing could happen with any program that uses temporary files.

Hi rjones,
do you mean we can move this bug to verified since the error prompt is obvious to users, thanks.

I think so.  The original error ("block I/O error in device") shouldn't
happen now.

(In reply to Richard W.M. Jones from comment #17)
> I think so.  The original error ("block I/O error in device") shouldn't
> happen now.

OK, thanks for your reply, move this bug to VERIFIED.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2576.html