Bug 1316479 - v2v cmd cannot exit and "block I/O error in device 'appliance': No space left on device (28)" is printed when specified "-v -x"
v2v cmd cannot exit and "block I/O error in device 'appliance': No space left...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs (Show other bugs)
7.3
Unspecified Unspecified
medium Severity medium
: rc
: ---
Assigned To: Richard W.M. Jones
Virtualization Bugs
V2V
:
: 1316487 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-03-10 06:00 EST by xiaodwan
Modified: 2016-11-03 13:59 EDT (History)
6 users (show)

See Also:
Fixed In Version: libguestfs-1.32.5-10.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-03 13:59:37 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
v2v log 1.5M space left (51.69 KB, text/plain)
2016-03-10 06:01 EST, xiaodwan
no flags Details
v2v log 68M space left (361.93 KB, text/plain)
2016-03-10 06:01 EST, xiaodwan
no flags Details
virt-v2v-log1 for scenario-1(verify-bug) (10.51 KB, text/plain)
2016-07-06 05:37 EDT, zhoujunqin
no flags Details

  None (edit)
Description xiaodwan 2016-03-10 06:00:09 EST
Description of problem:
v2v cmd cannot exit and "block I/O error in device 'appliance': No space left on device (28)" is printed when specified "-v -x".

Version-Release number of selected component (if applicable):
# rpm -q virt-v2v libguestfs
virt-v2v-1.28.1-1.55.el7_2.1.x86_64
libguestfs-1.28.1-1.55.el7_2.1.x86_64

How reproducible:
100%

Steps to Reproduce:
1)Seneriao 1:
1. About 68M left.
# df -h /var/tmp/
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root  130G  130G   68M 100% /
2. Run "virt-v2v  -ic xen+ssh://10.66.106.64 -o rhev -os 10.73.2.1:/home/nfs_export --network rhevm xen-hvm-win7-x86_64 -of raw".

Actual results:
# virt-v2v  -ic xen+ssh://10.66.106.64 -o rhev -os 10.73.2.1:/home/nfs_export --network rhevm xen-hvm-win7-x86_64 -of raw
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-hvm-win7-x86_64
[   1.0] Creating an overlay to protect the source from being modified
[   1.0] Opening the overlay
[   6.0] Initializing the target -o rhev -os 10.73.2.1:/home/nfs_export
virt-v2v: warning: cannot write files to the NFS server as 36:36, even 
though we appear to be running as root. This probably means the NFS client 
or idmapd is not configured properly.

You will have to chown the files that virt-v2v creates after the run, 
otherwise RHEV-M will not be able to import the VM.
[   6.0] Inspecting the overlay
[  11.0] Checking for sufficient free disk space in the guest
[  11.0] Estimating space required on target for each disk
[  11.0] Converting Windows 7 Ultimate to run on KVM


The command always stay here and cannot exit and there is "block I/O error indevice 'hd0': No space left on device (28)" when specifying '-v -x'.

Please see file "win7_v2v_log_68M_left".

Expected results:
The command should report error and exit.

2) Seneriao 2:
1. Make sure the "/var/tmp" has not enough disks.
# df -h /var/tmp/
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root  130G  130G  1.2M 100% /


2. Run ""virt-v2v  -ic xen+ssh://10.66.106.64 -o rhev -os 10.73.2.1:/home/nfs_export --network rhevm xen-hvm-win7-x86_64 -of raw"

Actual results:

virt-v2v: error: mkdtemp: No space left on device: /tmp/v2v.XXXXXX

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

About error reports but the cmd cannot exit and there is "block I/O error indevice 'hd0': No space left on device (28)" when specifying '-v -x'.

Please see file "v2v_log_1.5M_left".

Expected results:
virt-v2v command should exit after reporting a error.

Additional info:
Comment 1 xiaodwan 2016-03-10 06:01 EST
Created attachment 1134811 [details]
v2v log 1.5M space left
Comment 2 xiaodwan 2016-03-10 06:01 EST
Created attachment 1134812 [details]
v2v log 68M space left
Comment 4 Richard W.M. Jones 2016-03-31 08:33:13 EDT
I think for both this bug and bug 1316487 we should just check
if the host $TMPDIR etc has "sufficient" space and warn or error
out before we start if not.
Comment 5 Richard W.M. Jones 2016-06-23 08:00:34 EDT
*** Bug 1316487 has been marked as a duplicate of this bug. ***
Comment 6 Richard W.M. Jones 2016-06-23 08:49:07 EDT
Fixed in upstream commits:

https://github.com/libguestfs/libguestfs/commit/24130d787256dc3d9e37e85b1ba35ac7dbeb86a1
https://github.com/libguestfs/libguestfs/commit/a7fb3d601ab2ae3cdf2074d4c47afe59f00eea6d
https://github.com/libguestfs/libguestfs/commit/d8a465a14c3596a9fea53560bc7ad2ee6c3111bc
https://github.com/libguestfs/libguestfs/commit/9221ef6f7959ae3b979fd05007363810bc66062b

Note that this just displays an error message if there is insufficient
free space (under 1GB) before starting the conversion.  The error
message would be something like:

  virt-v2v: error: insufficient free space in the conversion server temporary 
  directory /var/tmp (853.8M).

  Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR 
  environment variable to point to another directory with more than 1GB of 
  free space.

  See also the virt-v2v(1) manual, section "Minimum free space check in the 
  host".
Comment 8 zhoujunqin 2016-07-06 05:36:32 EDT
I can reproduce this bug with package:
libguestfs-1.28.1-1.55.el7_2.1.x86_64
virt-v2v-1.28.1-1.55.el7_2.1.x86_64

Then try to verify this bug with new build:

# rpm -q libguestfs virt-v2v 
libguestfs-1.32.5-9.el7.x86_64
virt-v2v-1.32.5-9.el7.x86_64

Steps:
1. Check virt-v2v manual page:
# man virt-v2v
...
   Minimum free space check in the host
       You must have sufficient free space in the host directory used to store temporary overlays (except in [--in-place] mode).  To
       find out which directory this is, use:

        $ df -h "`guestfish get-cachedir`"
        Filesystem        Size  Used Avail Use% Mounted on
        /dev/mapper/root   50G   40G  6.8G  86% /

       and look under the "Avail" column.  Virt-v2v will refuse to do the conversion at all unless at least 1GB is available there.

       See also "RESOURCE REQUIREMENTS" above.

Result: We find that 'Virt-v2v will refuse to do the conversion at all unless at least 1GB is available there.', so while i verify this bug i just cover 2 scenario around '3M' and '248M'.

Scenario-1 
1.1 Left about 3M free space on host
# df -h /var/tmp/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       150G  150G  3.0M 100% /

1.2. Convert a xen guest to rhevm
# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
virt-v2v: error: insufficient free space in the conversion server temporary 
directory /var/tmp (2.9M).

Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR 
environment variable to point to another directory with more than 1GB of 
free space.

See also the virt-v2v(1) manual, section "Minimum free space check in the 
host".

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Result: We can get a clear error while host free space less than 1GB.

1.3. Repeat step1.2 after set LIBGUESTFS_CACHEDIR value to a directory with more than 1GB of free space.

# df -h /home/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda7       2.0G  6.0M  1.8G   1% /home

# export LIBGUESTFS_CACHEDIR=/home
# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on juzhou1
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
[   0.5] Creating an overlay to protect the source from being modified
[   0.9] Initializing the target -o rhev -os 10.73.72.63:/home/nfs_export
[   1.1] Opening the overlay
virt-v2v: error: libguestfs error: /usr/bin/supermin5 exited with error 
status 1.
To see full error messages you may need to enable debugging.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

I will attach virt-v2v debug log for scenario-1, called virt-v2v-log1.

Scenario-2 
2.1 Left about 258M free space on host
# df -h /var/tmp/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       150G  150G  248M 100% /

2.2. Convert a xen guest to rhevm

# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on juzhou1 
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
virt-v2v: error: insufficient free space in the conversion server temporary 
directory /var/tmp (247.9M).

Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR 
environment variable to point to another directory with more than 1GB of 
free space.

See also the virt-v2v(1) manual, section "Minimum free space check in the 
host".

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Result: We can get a clear error while host free space less than 1GB.

2.3. Repeat step2.2 after set LIBGUESTFS_CACHEDIR value to a directory with more than 1GB of free space.
# df -h /home/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda7       2.0G  6.0M  1.8G   1% /home

# export LIBGUESTFS_CACHEDIR=/home

# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on juzhou1

Result: conversion finished without error.


so rjones, please help me have a look of scenario-1 issue, thanks.
Comment 9 zhoujunqin 2016-07-06 05:37 EDT
Created attachment 1176828 [details]
virt-v2v-log1 for scenario-1(verify-bug)
Comment 10 Richard W.M. Jones 2016-07-06 05:52:56 EDT
# export LIBGUESTFS_CACHEDIR=/home
# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on juzhou1
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
[   0.5] Creating an overlay to protect the source from being modified
[   0.9] Initializing the target -o rhev -os 10.73.72.63:/home/nfs_export
[   1.1] Opening the overlay
virt-v2v: error: libguestfs error: /usr/bin/supermin5 exited with error 
status 1.

...

supermin: ext2: creating empty ext2 filesystem '/home/.guestfs-0/appliance.d.hib4dgn3/root'
supermin: ext2: populating from base image
supermin: ext2: copying files from host filesystem
supermin: ext2: copying kernel modules
supermin: ext2: creating minimal initrd '/home/.guestfs-0/appliance.d.hib4dgn3/initrd'
supermin: ext2: wrote 27 modules to minimal initrd
supermin: exception: Sys_error("No space left on device")
libguestfs: trace: v2v: launch = -1 (error)
virt-v2v: error: libguestfs error: /usr/bin/supermin5 exited with error 


I think it's just a really bad idea to set LIBGUESTFS_CACHEDIR
to /home.  However since you had 1.8GB free I don't really know why
it would have failed.

I suppose you could try this command (as root):

/usr/bin/supermin5 --build -v -v -v --copy-kernel -f ext2 \
  --host-cpu x86_64 /usr/lib64/guestfs/supermin.d -o /home/.guestfs-0/appliance.d

That will produce a much larger amount of debugging and should show why
it is failing.
Comment 11 zhoujunqin 2016-07-06 06:33:34 EDT
Hi rjones,

After reproduce that error, then i tried as you said
1.
# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on juzhou1
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
virt-v2v: error: insufficient free space in the conversion server temporary 
directory /var/tmp (3.6M).

2.
# df -h /home/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda7       2.0G  6.1M  1.8G   1% /home

3.
# /usr/bin/supermin5 --build -v -v -v --copy-kernel -f ext2 \
>  --host-cpu x86_64 /usr/lib64/guestfs/supermin.d -o /home/.guestfs-0/appliance.d
supermin: version: 5.1.16
supermin: rpm: detected RPM version 4.11
supermin: package handler: fedora/rpm
supermin: build: /usr/lib64/guestfs/supermin.d
supermin: reading the supermin appliance
supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar)
supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar)
tar: usr/sbin/guestfsd: Wrote only 8192 of 10240 bytes
tar: usr/share/guestfs: Cannot mkdir: No space left on device
tar: Exiting with failure status due to previous errors
supermin: zcat '/usr/lib64/guestfs/supermin.d/daemon.tar.gz' | tar -C '/tmp/supermin5a0221.tmpdir/base.d' -xf -: command failed, see earlier errors


Does it mean although there is free space on /home, but it cannot be used by usr/share/guestfs command, thanks.
If so, can you give some tips about set directory to 'LIBGUESTFS_CACHEDIR', thanks.
Comment 12 Richard W.M. Jones 2016-07-06 07:29:11 EDT
OK I see, what actually happens here is:

(1) LIBGUESTFS_CACHEDIR asks libguestfs to put the cache in that directory

(2) supermin doesn't know or care about any LIBGUESTFS_* variables

(3) supermin uses /tmp (or TMPDIR) for temporary files

So the fix is incomplete.  It should also check for free space
in /tmp (or TMPDIR).
Comment 13 Richard W.M. Jones 2016-07-06 09:21:34 EDT
I added this supermin patch which makes it use /var/tmp instead of /tmp
if TMPDIR is not set.

https://github.com/libguestfs/supermin/commit/12fc339efb1d287593b53da1dfddc88f876f8e7d
Comment 14 zhoujunqin 2016-07-08 03:19:27 EDT
Try to verify this bug again with new build:
libguestfs-1.32.5-10.el7.x86_64
virt-v2v-1.32.5-10.el7.x86_64
libvirt-2.0.0-1.el7.x86_64

Steps:
1. Left about 3M free space on host

# df -h  /var/tmp/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       150G  150G  3.8M 100% /


2. Convert a xen guest to rhevm
# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on jtest1
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
virt-v2v: error: insufficient free space in the conversion server temporary 
directory /var/tmp (3.6M).

Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR 
environment variable to point to another directory with more than 1GB of 
free space.

See also the virt-v2v(1) manual, section "Minimum free space check in the 
host".

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Result: We can get a clear error while host free space less than 1GB, then try to set the LIBGUESTFS_CACHEDIR and do coversion again.

3.
Scenario-1 # export LIBGUESTFS_CACHEDIR=/opt

# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on jtest1
[   0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64
[   0.6] Creating an overlay to protect the source from being modified
[   1.0] Initializing the target -o rhev -os 10.73.72.63:/home/nfs_export
[   1.2] Opening the overlay
virt-v2v: error: libguestfs error: /usr/bin/supermin5 exited with error 
status 1.
To see full error messages you may need to enable debugging.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Result: Conversion failed and can also reproduce Comment 11 issue.

Scenario-2

# unset LIBGUESTFS_CACHEDIR
# export TMPDIR=/opt
# virt-v2v  -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64  --network rhevm  -of raw -on jtest1

Result: Conversion finished without error.

So rjones, please help me have a look, i think bug issue not fixed yet, thanks.
Comment 15 Richard W.M. Jones 2016-07-08 03:42:22 EDT
Yes, the new build (of supermin) only changes the location of
temporary files to /var/tmp.

It doesn't ensure there is sufficient space on /var/tmp, but these errors
are easy to diagnose.  If the failure happened the user would run
virt-v2v with -v -x, or libguestfs-test-tool, and the problem is obvious.
This same thing could happen with any program that uses temporary files.
Comment 16 zhoujunqin 2016-07-08 04:49:10 EDT
(In reply to Richard W.M. Jones from comment #15)
> Yes, the new build (of supermin) only changes the location of
> temporary files to /var/tmp.
> 
> It doesn't ensure there is sufficient space on /var/tmp, but these errors
> are easy to diagnose.  If the failure happened the user would run
> virt-v2v with -v -x, or libguestfs-test-tool, and the problem is obvious.
> This same thing could happen with any program that uses temporary files.

Hi rjones,
do you mean we can move this bug to verified since the error prompt is obvious to users, thanks.
Comment 17 Richard W.M. Jones 2016-07-08 04:50:42 EDT
I think so.  The original error ("block I/O error in device") shouldn't
happen now.
Comment 18 zhoujunqin 2016-07-08 06:13:49 EDT
(In reply to Richard W.M. Jones from comment #17)
> I think so.  The original error ("block I/O error in device") shouldn't
> happen now.

OK, thanks for your reply, move this bug to VERIFIED.
Comment 20 errata-xmlrpc 2016-11-03 13:59:37 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2576.html

Note You need to log in before you can comment on or make changes to this bug.