Red Hat Bugzilla – Bug 1316479
v2v cmd cannot exit and "block I/O error in device 'appliance': No space left on device (28)" is printed when specified "-v -x"
Last modified: 2016-11-03 13:59:37 EDT
Description of problem: v2v cmd cannot exit and "block I/O error in device 'appliance': No space left on device (28)" is printed when specified "-v -x". Version-Release number of selected component (if applicable): # rpm -q virt-v2v libguestfs virt-v2v-1.28.1-1.55.el7_2.1.x86_64 libguestfs-1.28.1-1.55.el7_2.1.x86_64 How reproducible: 100% Steps to Reproduce: 1)Seneriao 1: 1. About 68M left. # df -h /var/tmp/ Filesystem Size Used Avail Use% Mounted on /dev/mapper/rhel-root 130G 130G 68M 100% / 2. Run "virt-v2v -ic xen+ssh://10.66.106.64 -o rhev -os 10.73.2.1:/home/nfs_export --network rhevm xen-hvm-win7-x86_64 -of raw". Actual results: # virt-v2v -ic xen+ssh://10.66.106.64 -o rhev -os 10.73.2.1:/home/nfs_export --network rhevm xen-hvm-win7-x86_64 -of raw [ 0.0] Opening the source -i libvirt -ic xen+ssh://10.66.106.64 xen-hvm-win7-x86_64 [ 1.0] Creating an overlay to protect the source from being modified [ 1.0] Opening the overlay [ 6.0] Initializing the target -o rhev -os 10.73.2.1:/home/nfs_export virt-v2v: warning: cannot write files to the NFS server as 36:36, even though we appear to be running as root. This probably means the NFS client or idmapd is not configured properly. You will have to chown the files that virt-v2v creates after the run, otherwise RHEV-M will not be able to import the VM. [ 6.0] Inspecting the overlay [ 11.0] Checking for sufficient free disk space in the guest [ 11.0] Estimating space required on target for each disk [ 11.0] Converting Windows 7 Ultimate to run on KVM The command always stay here and cannot exit and there is "block I/O error indevice 'hd0': No space left on device (28)" when specifying '-v -x'. Please see file "win7_v2v_log_68M_left". Expected results: The command should report error and exit. 2) Seneriao 2: 1. Make sure the "/var/tmp" has not enough disks. # df -h /var/tmp/ Filesystem Size Used Avail Use% Mounted on /dev/mapper/rhel-root 130G 130G 1.2M 100% / 2. Run ""virt-v2v -ic xen+ssh://10.66.106.64 -o rhev -os 10.73.2.1:/home/nfs_export --network rhevm xen-hvm-win7-x86_64 -of raw" Actual results: virt-v2v: error: mkdtemp: No space left on device: /tmp/v2v.XXXXXX If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] About error reports but the cmd cannot exit and there is "block I/O error indevice 'hd0': No space left on device (28)" when specifying '-v -x'. Please see file "v2v_log_1.5M_left". Expected results: virt-v2v command should exit after reporting a error. Additional info:
Created attachment 1134811 [details] v2v log 1.5M space left
Created attachment 1134812 [details] v2v log 68M space left
I think for both this bug and bug 1316487 we should just check if the host $TMPDIR etc has "sufficient" space and warn or error out before we start if not.
*** Bug 1316487 has been marked as a duplicate of this bug. ***
Fixed in upstream commits: https://github.com/libguestfs/libguestfs/commit/24130d787256dc3d9e37e85b1ba35ac7dbeb86a1 https://github.com/libguestfs/libguestfs/commit/a7fb3d601ab2ae3cdf2074d4c47afe59f00eea6d https://github.com/libguestfs/libguestfs/commit/d8a465a14c3596a9fea53560bc7ad2ee6c3111bc https://github.com/libguestfs/libguestfs/commit/9221ef6f7959ae3b979fd05007363810bc66062b Note that this just displays an error message if there is insufficient free space (under 1GB) before starting the conversion. The error message would be something like: virt-v2v: error: insufficient free space in the conversion server temporary directory /var/tmp (853.8M). Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR environment variable to point to another directory with more than 1GB of free space. See also the virt-v2v(1) manual, section "Minimum free space check in the host".
I can reproduce this bug with package: libguestfs-1.28.1-1.55.el7_2.1.x86_64 virt-v2v-1.28.1-1.55.el7_2.1.x86_64 Then try to verify this bug with new build: # rpm -q libguestfs virt-v2v libguestfs-1.32.5-9.el7.x86_64 virt-v2v-1.32.5-9.el7.x86_64 Steps: 1. Check virt-v2v manual page: # man virt-v2v ... Minimum free space check in the host You must have sufficient free space in the host directory used to store temporary overlays (except in [--in-place] mode). To find out which directory this is, use: $ df -h "`guestfish get-cachedir`" Filesystem Size Used Avail Use% Mounted on /dev/mapper/root 50G 40G 6.8G 86% / and look under the "Avail" column. Virt-v2v will refuse to do the conversion at all unless at least 1GB is available there. See also "RESOURCE REQUIREMENTS" above. Result: We find that 'Virt-v2v will refuse to do the conversion at all unless at least 1GB is available there.', so while i verify this bug i just cover 2 scenario around '3M' and '248M'. Scenario-1 1.1 Left about 3M free space on host # df -h /var/tmp/ Filesystem Size Used Avail Use% Mounted on /dev/sda1 150G 150G 3.0M 100% / 1.2. Convert a xen guest to rhevm # virt-v2v -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64 --network rhevm -of raw [ 0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64 virt-v2v: error: insufficient free space in the conversion server temporary directory /var/tmp (2.9M). Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR environment variable to point to another directory with more than 1GB of free space. See also the virt-v2v(1) manual, section "Minimum free space check in the host". If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] Result: We can get a clear error while host free space less than 1GB. 1.3. Repeat step1.2 after set LIBGUESTFS_CACHEDIR value to a directory with more than 1GB of free space. # df -h /home/ Filesystem Size Used Avail Use% Mounted on /dev/sda7 2.0G 6.0M 1.8G 1% /home # export LIBGUESTFS_CACHEDIR=/home # virt-v2v -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64 --network rhevm -of raw -on juzhou1 [ 0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64 [ 0.5] Creating an overlay to protect the source from being modified [ 0.9] Initializing the target -o rhev -os 10.73.72.63:/home/nfs_export [ 1.1] Opening the overlay virt-v2v: error: libguestfs error: /usr/bin/supermin5 exited with error status 1. To see full error messages you may need to enable debugging. Do: export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 and run the command again. For further information, read: http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs You can also run 'libguestfs-test-tool' and post the *complete* output into a bug report or message to the libguestfs mailing list. If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] I will attach virt-v2v debug log for scenario-1, called virt-v2v-log1. Scenario-2 2.1 Left about 258M free space on host # df -h /var/tmp/ Filesystem Size Used Avail Use% Mounted on /dev/sda1 150G 150G 248M 100% / 2.2. Convert a xen guest to rhevm # virt-v2v -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64 --network rhevm -of raw -on juzhou1 [ 0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64 virt-v2v: error: insufficient free space in the conversion server temporary directory /var/tmp (247.9M). Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR environment variable to point to another directory with more than 1GB of free space. See also the virt-v2v(1) manual, section "Minimum free space check in the host". If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] Result: We can get a clear error while host free space less than 1GB. 2.3. Repeat step2.2 after set LIBGUESTFS_CACHEDIR value to a directory with more than 1GB of free space. # df -h /home/ Filesystem Size Used Avail Use% Mounted on /dev/sda7 2.0G 6.0M 1.8G 1% /home # export LIBGUESTFS_CACHEDIR=/home # virt-v2v -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64 --network rhevm -of raw -on juzhou1 Result: conversion finished without error. so rjones, please help me have a look of scenario-1 issue, thanks.
Created attachment 1176828 [details] virt-v2v-log1 for scenario-1(verify-bug)
# export LIBGUESTFS_CACHEDIR=/home # virt-v2v -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64 --network rhevm -of raw -on juzhou1 [ 0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64 [ 0.5] Creating an overlay to protect the source from being modified [ 0.9] Initializing the target -o rhev -os 10.73.72.63:/home/nfs_export [ 1.1] Opening the overlay virt-v2v: error: libguestfs error: /usr/bin/supermin5 exited with error status 1. ... supermin: ext2: creating empty ext2 filesystem '/home/.guestfs-0/appliance.d.hib4dgn3/root' supermin: ext2: populating from base image supermin: ext2: copying files from host filesystem supermin: ext2: copying kernel modules supermin: ext2: creating minimal initrd '/home/.guestfs-0/appliance.d.hib4dgn3/initrd' supermin: ext2: wrote 27 modules to minimal initrd supermin: exception: Sys_error("No space left on device") libguestfs: trace: v2v: launch = -1 (error) virt-v2v: error: libguestfs error: /usr/bin/supermin5 exited with error I think it's just a really bad idea to set LIBGUESTFS_CACHEDIR to /home. However since you had 1.8GB free I don't really know why it would have failed. I suppose you could try this command (as root): /usr/bin/supermin5 --build -v -v -v --copy-kernel -f ext2 \ --host-cpu x86_64 /usr/lib64/guestfs/supermin.d -o /home/.guestfs-0/appliance.d That will produce a much larger amount of debugging and should show why it is failing.
Hi rjones, After reproduce that error, then i tried as you said 1. # virt-v2v -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64 --network rhevm -of raw -on juzhou1 [ 0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64 virt-v2v: error: insufficient free space in the conversion server temporary directory /var/tmp (3.6M). 2. # df -h /home/ Filesystem Size Used Avail Use% Mounted on /dev/sda7 2.0G 6.1M 1.8G 1% /home 3. # /usr/bin/supermin5 --build -v -v -v --copy-kernel -f ext2 \ > --host-cpu x86_64 /usr/lib64/guestfs/supermin.d -o /home/.guestfs-0/appliance.d supermin: version: 5.1.16 supermin: rpm: detected RPM version 4.11 supermin: package handler: fedora/rpm supermin: build: /usr/lib64/guestfs/supermin.d supermin: reading the supermin appliance supermin: build: visiting /usr/lib64/guestfs/supermin.d/base.tar.gz type gzip base image (tar) supermin: build: visiting /usr/lib64/guestfs/supermin.d/daemon.tar.gz type gzip base image (tar) tar: usr/sbin/guestfsd: Wrote only 8192 of 10240 bytes tar: usr/share/guestfs: Cannot mkdir: No space left on device tar: Exiting with failure status due to previous errors supermin: zcat '/usr/lib64/guestfs/supermin.d/daemon.tar.gz' | tar -C '/tmp/supermin5a0221.tmpdir/base.d' -xf -: command failed, see earlier errors Does it mean although there is free space on /home, but it cannot be used by usr/share/guestfs command, thanks. If so, can you give some tips about set directory to 'LIBGUESTFS_CACHEDIR', thanks.
OK I see, what actually happens here is: (1) LIBGUESTFS_CACHEDIR asks libguestfs to put the cache in that directory (2) supermin doesn't know or care about any LIBGUESTFS_* variables (3) supermin uses /tmp (or TMPDIR) for temporary files So the fix is incomplete. It should also check for free space in /tmp (or TMPDIR).
I added this supermin patch which makes it use /var/tmp instead of /tmp if TMPDIR is not set. https://github.com/libguestfs/supermin/commit/12fc339efb1d287593b53da1dfddc88f876f8e7d
Try to verify this bug again with new build: libguestfs-1.32.5-10.el7.x86_64 virt-v2v-1.32.5-10.el7.x86_64 libvirt-2.0.0-1.el7.x86_64 Steps: 1. Left about 3M free space on host # df -h /var/tmp/ Filesystem Size Used Avail Use% Mounted on /dev/sda1 150G 150G 3.8M 100% / 2. Convert a xen guest to rhevm # virt-v2v -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64 --network rhevm -of raw -on jtest1 [ 0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64 virt-v2v: error: insufficient free space in the conversion server temporary directory /var/tmp (3.6M). Either free up space in that directory, or set the LIBGUESTFS_CACHEDIR environment variable to point to another directory with more than 1GB of free space. See also the virt-v2v(1) manual, section "Minimum free space check in the host". If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] Result: We can get a clear error while host free space less than 1GB, then try to set the LIBGUESTFS_CACHEDIR and do coversion again. 3. Scenario-1 # export LIBGUESTFS_CACHEDIR=/opt # virt-v2v -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64 --network rhevm -of raw -on jtest1 [ 0.0] Opening the source -i libvirt -ic xen+ssh://10.73.3.21 xen-hvm-rhel6.7-x86_64 [ 0.6] Creating an overlay to protect the source from being modified [ 1.0] Initializing the target -o rhev -os 10.73.72.63:/home/nfs_export [ 1.2] Opening the overlay virt-v2v: error: libguestfs error: /usr/bin/supermin5 exited with error status 1. To see full error messages you may need to enable debugging. Do: export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 and run the command again. For further information, read: http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs You can also run 'libguestfs-test-tool' and post the *complete* output into a bug report or message to the libguestfs mailing list. If reporting bugs, run virt-v2v with debugging enabled and include the complete output: virt-v2v -v -x [...] Result: Conversion failed and can also reproduce Comment 11 issue. Scenario-2 # unset LIBGUESTFS_CACHEDIR # export TMPDIR=/opt # virt-v2v -ic xen+ssh://10.73.3.21 -o rhev -os 10.73.72.63:/home/nfs_export xen-hvm-rhel6.7-x86_64 --network rhevm -of raw -on jtest1 Result: Conversion finished without error. So rjones, please help me have a look, i think bug issue not fixed yet, thanks.
Yes, the new build (of supermin) only changes the location of temporary files to /var/tmp. It doesn't ensure there is sufficient space on /var/tmp, but these errors are easy to diagnose. If the failure happened the user would run virt-v2v with -v -x, or libguestfs-test-tool, and the problem is obvious. This same thing could happen with any program that uses temporary files.
(In reply to Richard W.M. Jones from comment #15) > Yes, the new build (of supermin) only changes the location of > temporary files to /var/tmp. > > It doesn't ensure there is sufficient space on /var/tmp, but these errors > are easy to diagnose. If the failure happened the user would run > virt-v2v with -v -x, or libguestfs-test-tool, and the problem is obvious. > This same thing could happen with any program that uses temporary files. Hi rjones, do you mean we can move this bug to verified since the error prompt is obvious to users, thanks.
I think so. The original error ("block I/O error in device") shouldn't happen now.
(In reply to Richard W.M. Jones from comment #17) > I think so. The original error ("block I/O error in device") shouldn't > happen now. OK, thanks for your reply, move this bug to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-2576.html