RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1246032 - virt-v2v: error: libguestfs error: umount_all: umount: /sysroot: umount: /sysroot: target is busy
Summary: virt-v2v: error: libguestfs error: umount_all: umount: /sysroot: umount: /sy...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs
Version: 7.2
Hardware: x86_64
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard: V2V
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-07-23 11:00 UTC by tingting zheng
Modified: 2015-11-19 07:02 UTC (History)
5 users (show)

Fixed In Version: libguestfs-1.28.1-1.48.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-11-19 07:02:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Debug info (87.49 KB, text/plain)
2015-07-23 11:00 UTC, tingting zheng
no flags Details
virt-v2v debug log (88.16 KB, text/plain)
2015-07-27 09:47 UTC, zhoujunqin
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2183 0 normal SHIPPED_LIVE libguestfs bug fix and enhancement update 2015-11-19 08:18:42 UTC

Description tingting zheng 2015-07-23 11:00:22 UTC
Created attachment 1055282 [details]
Debug info

Description of problem:
Error shows when converting win7 from esx server:virt-v2v: error: libguestfs error: umount_all: umount: /sysroot: umount:  /sysroot: target is busy


Version-Release number of selected component (if applicable):
virt-v2v-1.28.1-1.46.el7.x86_64
libguestfs-1.28.1-1.46.el7.x86_64
libvirt-1.2.17-2.el7.x86_64
qemu-kvm-1.5.3-97.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.A newly installed win7 guest on esx server,use virt-v2v to convert it:
# /usr/bin/virt-v2v  -ic vpx://root.4.103/tzheng-demo/10.66.106.63/?no_verify=1 -o rhev -os 10.66.90.115:/vol/v2v_auto/fc_export -of raw -b rhevm -n rhevm esx5.5-win7-i386 --password-file /tmp/v2v_passwd 
[   0.0] Opening the source -i libvirt -ic vpx://root.4.103/tzheng-demo/10.66.106.63/?no_verify=1 esx5.5-win7-i386
[   1.0] Creating an overlay to protect the source from being modified
[   2.0] Opening the overlay
[  15.0] Initializing the target -o rhev -os 10.66.90.115:/vol/v2v_auto/fc_export
virt-v2v: warning: cannot write files to the NFS server as 36:36, even 
though we appear to be running as root. This probably means the NFS client 
or idmapd is not configured properly.

You will have to chown the files that virt-v2v creates after the run, 
otherwise RHEV-M will not be able to import the VM.
[  15.0] Inspecting the overlay
virt-v2v: error: libguestfs error: umount_all: umount: /sysroot: umount: 
/sysroot: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Actual results:
As description

Expected results:
Guest can be converted successfully.

Additional info:
Only this guest met this error,win7 on other hypervisors can be converted successfully.

Comment 4 Richard W.M. Jones 2015-07-23 16:21:10 UTC
What I have found is that the hivex_close is slow.  It takes 120 seconds
or so.  This is just because the storage is remote (https to the VMware
server).  But nevertheless it could mean that there are lots of writes
still "in flight" when umount is called.

libguestfs: trace: hivex_value_key = "DhcpDomain"
libguestfs: trace: hivex_close
guestfsd: main_loop: new request, len 0x28
hivex: hivex_close: hivex_close
libguestfs: trace: hivex_close = 0
libguestfs: trace: umount_all
guestfsd: main_loop: proc 351 (hivex_close) took 0.00 seconds
guestfsd: main_loop: new request, len 0x28
umount-all: /proc/mounts: fsname=rootfs dir=/ type=rootfs opts=rw freq=0 passno=0
umount-all: /proc/mounts: fsname=proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev/root dir=/ type=ext2 opts=rw,noatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/sys dir=/sys type=sysfs opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=tmpfs dir=/run type=tmpfs opts=rw,nosuid,relatime,size=97728k,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev dir=/dev type=devtmpfs opts=rw,relatime,size=241872k,nr_inodes=60468,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev/sda2 dir=/sysroot type=fuseblk opts=ro,relatime,user_id=0,group_id=0,allow_other,blksize=4096 freq=0 passno=0
umount /sysroot
umount: /sysroot: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

Comment 5 Richard W.M. Jones 2015-07-23 16:31:00 UTC
Patch posted:
https://www.redhat.com/archives/libguestfs/2015-July/msg00137.html

Comment 6 Richard W.M. Jones 2015-07-23 20:52:17 UTC
The patch in comment 5 doesn't work.

This works - at least some of the time, but it's a complete hack:

diff --git a/daemon/mount.c b/daemon/mount.c
index c5b7d89..74a40de 100644
--- a/daemon/mount.c
+++ b/daemon/mount.c
@@ -26,6 +26,8 @@
 #include <sys/types.h>
 #include <mntent.h>
 
+#include "ignore-value.h"
+
 #include "daemon.h"
 #include "actions.h"
 
@@ -417,6 +419,9 @@ do_umount_all (void)
   if (mounts.size > 0)
     qsort (mounts.argv, mounts.size, sizeof (char *), compare_longest_first);
 
+  /* Hack to work around RHBZ#1246032. */
+  ignore_value (system ("lsof /sysroot"));
+
   /* Unmount them. */
   for (i = 0; i < mounts.size; ++i) {
     CLEANUP_FREE char *err = NULL;

Comment 8 zhoujunqin 2015-07-27 09:46:57 UTC
I can reproduce this issue with package:
libguestfs-1.28.1-1.46.el7.x86_64
virt-v2v-1.28.1-1.46.el7.x86_64

Then try to verify this bug with new build:
libguestfs-1.28.1-1.47.el7.x86_64
virt-v2v-1.28.1-1.47.el7.x86_64

Steps:
1.A newly installed win7 guest on esx server,use virt-v2v to convert it:
# virt-v2v  -ic vpx://root.4.103/tzheng-demo/10.66.106.63/?no_verify=1 -o rhev -os 10.66.90.115:/vol/v2v_auto/fc_export -of raw -b rhevm -n rhevm esx5.5-win7-i386 --password-file /tmp/v2v_passwd 
[   0.0] Opening the source -i libvirt -ic vpx://root.4.103/tzheng-demo/10.66.106.63/?no_verify=1 esx5.5-win7-i386
[   2.0] Creating an overlay to protect the source from being modified
[   3.0] Opening the overlay
[ 104.0] Initializing the target -o rhev -os 10.66.90.115:/vol/v2v_auto/fc_export
virt-v2v: warning: cannot write files to the NFS server as 36:36, even 
though we appear to be running as root. This probably means the NFS client 
or idmapd is not configured properly.

You will have to chown the files that virt-v2v creates after the run, 
otherwise RHEV-M will not be able to import the VM.
[ 104.0] Inspecting the overlay
virt-v2v: error: libguestfs error: umount_all: umount: /sysroot: umount: 
/sysroot: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Result: I can also reproduce bug issue, so move this bug from ON_QA to ASSIGNED.

Comment 9 zhoujunqin 2015-07-27 09:47:50 UTC
Created attachment 1056533 [details]
virt-v2v debug log

Comment 10 Richard W.M. Jones 2015-07-28 13:03:18 UTC
From the build log:

  sh: lsof: command not found

There was a mistake in my previous commit, which meant that the
workaround was not invoked.

The -1.48 package, building now, should fix it.

Comment 11 zhoujunqin 2015-07-29 04:01:19 UTC
Try to verify this bug again with new build:
libguestfs-1.28.1-1.48.el7.x86_64
virt-v2v-1.28.1-1.48.el7.x86_64

steps:

# virt-v2v  -ic vpx://root.4.103/tzheng-demo/10.66.106.63/?no_verify=1 -o rhev -os 10.66.90.115:/vol/v2v_auto/fc_export -of raw -b rhevm -n rhevm esx5.5-win7-i386 -on esx5.5-win7-i386-juzhou  --password-file /tmp/v2v_passwd 
[   0.0] Opening the source -i libvirt -ic vpx://root.4.103/tzheng-demo/10.66.106.63/?no_verify=1 esx5.5-win7-i386
[   1.0] Creating an overlay to protect the source from being modified
[   2.0] Opening the overlay
[  76.0] Initializing the target -o rhev -os 10.66.90.115:/vol/v2v_auto/fc_export
virt-v2v: warning: cannot write files to the NFS server as 36:36, even 
though we appear to be running as root. This probably means the NFS client 
or idmapd is not configured properly.

You will have to chown the files that virt-v2v creates after the run, 
otherwise RHEV-M will not be able to import the VM.
[  76.0] Inspecting the overlay
[ 122.0] Checking for sufficient free disk space in the guest
[ 122.0] Estimating space required on target for each disk
[ 122.0] Converting Windows 7 Ultimate to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 146.0] Mapping filesystem data to avoid copying unused and blank areas
[ 147.0] Closing the overlay
[ 147.0] Checking if the guest needs BIOS or UEFI to boot
[ 147.0] Copying disk 1/1 to /tmp/v2v.J46wn1/f58033a8-ba4f-491b-b96e-28fe03792e1e/images/c37c1d03-02d7-4ee7-ba11-ec74817b1668/b55f3203-a473-4ad3-8035-7f58ec6e78e1 (raw)
    (100.00/100%)
[1093.0] Creating output metadata
[1093.0] Finishing off

Result: Conversion finished with no error, and can import and boot up on rhevm server, so move this bug from ON_QA to VERIFIED.

Comment 13 errata-xmlrpc 2015-11-19 07:02:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2183.html


Note You need to log in before you can comment on or make changes to this bug.