Bug 1246032

What I have found is that the hivex_close is slow.  It takes 120 seconds
or so.  This is just because the storage is remote (https to the VMware
server).  But nevertheless it could mean that there are lots of writes
still "in flight" when umount is called.

libguestfs: trace: hivex_value_key = "DhcpDomain"
libguestfs: trace: hivex_close
guestfsd: main_loop: new request, len 0x28
hivex: hivex_close: hivex_close
libguestfs: trace: hivex_close = 0
libguestfs: trace: umount_all
guestfsd: main_loop: proc 351 (hivex_close) took 0.00 seconds
guestfsd: main_loop: new request, len 0x28
umount-all: /proc/mounts: fsname=rootfs dir=/ type=rootfs opts=rw freq=0 passno=0
umount-all: /proc/mounts: fsname=proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev/root dir=/ type=ext2 opts=rw,noatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/sys dir=/sys type=sysfs opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=tmpfs dir=/run type=tmpfs opts=rw,nosuid,relatime,size=97728k,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev dir=/dev type=devtmpfs opts=rw,relatime,size=241872k,nr_inodes=60468,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev/sda2 dir=/sysroot type=fuseblk opts=ro,relatime,user_id=0,group_id=0,allow_other,blksize=4096 freq=0 passno=0
umount /sysroot
umount: /sysroot: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

Patch posted:
https://www.redhat.com/archives/libguestfs/2015-July/msg00137.html

The patch in comment 5 doesn't work.

This works - at least some of the time, but it's a complete hack:

diff --git a/daemon/mount.c b/daemon/mount.c
index c5b7d89..74a40de 100644
--- a/daemon/mount.c
+++ b/daemon/mount.c
@@ -26,6 +26,8 @@
 #include <sys/types.h>
 #include <mntent.h>
 
+#include "ignore-value.h"
+
 #include "daemon.h"
 #include "actions.h"
 
@@ -417,6 +419,9 @@ do_umount_all (void)
   if (mounts.size > 0)
     qsort (mounts.argv, mounts.size, sizeof (char *), compare_longest_first);
 
+  /* Hack to work around RHBZ#1246032. */
+  ignore_value (system ("lsof /sysroot"));
+
   /* Unmount them. */
   for (i = 0; i < mounts.size; ++i) {
     CLEANUP_FREE char *err = NULL;

I can reproduce this issue with package:
libguestfs-1.28.1-1.46.el7.x86_64
virt-v2v-1.28.1-1.46.el7.x86_64

Then try to verify this bug with new build:
libguestfs-1.28.1-1.47.el7.x86_64
virt-v2v-1.28.1-1.47.el7.x86_64

Steps:
1.A newly installed win7 guest on esx server,use virt-v2v to convert it:
# virt-v2v  -ic vpx://root.4.103/tzheng-demo/10.66.106.63/?no_verify=1 -o rhev -os 10.66.90.115:/vol/v2v_auto/fc_export -of raw -b rhevm -n rhevm esx5.5-win7-i386 --password-file /tmp/v2v_passwd 
[   0.0] Opening the source -i libvirt -ic vpx://root.4.103/tzheng-demo/10.66.106.63/?no_verify=1 esx5.5-win7-i386
[   2.0] Creating an overlay to protect the source from being modified
[   3.0] Opening the overlay
[ 104.0] Initializing the target -o rhev -os 10.66.90.115:/vol/v2v_auto/fc_export
virt-v2v: warning: cannot write files to the NFS server as 36:36, even 
though we appear to be running as root. This probably means the NFS client 
or idmapd is not configured properly.

You will have to chown the files that virt-v2v creates after the run, 
otherwise RHEV-M will not be able to import the VM.
[ 104.0] Inspecting the overlay
virt-v2v: error: libguestfs error: umount_all: umount: /sysroot: umount: 
/sysroot: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Result: I can also reproduce bug issue, so move this bug from ON_QA to ASSIGNED.

Created attachment 1056533 [details]
virt-v2v debug log

From the build log:

  sh: lsof: command not found

There was a mistake in my previous commit, which meant that the
workaround was not invoked.

The -1.48 package, building now, should fix it.

Try to verify this bug again with new build:
libguestfs-1.28.1-1.48.el7.x86_64
virt-v2v-1.28.1-1.48.el7.x86_64

steps:

# virt-v2v  -ic vpx://root.4.103/tzheng-demo/10.66.106.63/?no_verify=1 -o rhev -os 10.66.90.115:/vol/v2v_auto/fc_export -of raw -b rhevm -n rhevm esx5.5-win7-i386 -on esx5.5-win7-i386-juzhou  --password-file /tmp/v2v_passwd 
[   0.0] Opening the source -i libvirt -ic vpx://root.4.103/tzheng-demo/10.66.106.63/?no_verify=1 esx5.5-win7-i386
[   1.0] Creating an overlay to protect the source from being modified
[   2.0] Opening the overlay
[  76.0] Initializing the target -o rhev -os 10.66.90.115:/vol/v2v_auto/fc_export
virt-v2v: warning: cannot write files to the NFS server as 36:36, even 
though we appear to be running as root. This probably means the NFS client 
or idmapd is not configured properly.

You will have to chown the files that virt-v2v creates after the run, 
otherwise RHEV-M will not be able to import the VM.
[  76.0] Inspecting the overlay
[ 122.0] Checking for sufficient free disk space in the guest
[ 122.0] Estimating space required on target for each disk
[ 122.0] Converting Windows 7 Ultimate to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 146.0] Mapping filesystem data to avoid copying unused and blank areas
[ 147.0] Closing the overlay
[ 147.0] Checking if the guest needs BIOS or UEFI to boot
[ 147.0] Copying disk 1/1 to /tmp/v2v.J46wn1/f58033a8-ba4f-491b-b96e-28fe03792e1e/images/c37c1d03-02d7-4ee7-ba11-ec74817b1668/b55f3203-a473-4ad3-8035-7f58ec6e78e1 (raw)
    (100.00/100%)
[1093.0] Creating output metadata
[1093.0] Finishing off

Result: Conversion finished with no error, and can import and boot up on rhevm server, so move this bug from ON_QA to VERIFIED.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2183.html