Red Hat Bugzilla – Bug 1465849
v2v hangs on removing vmware-tool
Last modified: 2017-10-10 23:40:51 EDT
Description of problem:
During the import from Vmware(Suse,RHEL VMs) the import hangs on the
it stay 10-30 minutes on this tasks, after that once returned Segmentation fault, second time we have to kill v2v.
We tried 3 different VM and it hangs on the same
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. run virt-v2v
2. hang on "/usr/bin/vmware-uninstall-tools.pl"
One Segmentation fault, one kill, one hang for long time
It is possible to handle this better ? We never go to the error
warning (f_"VMware tools was detected, but uninstallation failed. The error message was: %s (ignored)")
PoC for Customer, it failed on Suse and RHEL VM's
I will try generate dumps as soon as will be possible
strace: Process 16062 attached
restart_syscall(<... resuming interrupted poll ...>
Did you see something similar ?
Can you please run virt-v2v with the -v -x parameters added, and provide the full log? I.e. something like:
$ virt-v2v -v -x [all the other parameters already used] 2>&1 | tee v2v.log
Created attachment 1292613 [details]
v2v hangs log
Ehm you forgot -x too...
Also I don't think we have the Perl script. Could you download it
and attach it to the bug:
virt-copy-out -a guest /usr/bin/vmware-uninstall-tools.pl /tmp
& upload /tmp/vmware-uninstall-tools.pl
See also comment 5.
I am not able to provide the /usr/bin/vmware-uninstall-tools.pl now, I will try to reproduce this as we faced this during remote session with cu every time.
Attached the pl script, I am trying to reproduce it but I'm not sure as till now with no success.
Created attachment 1299845 [details]
I tried to reproduce this using the following steps:
(1) Download VMwaretools-10.0.6-3560309.zip (or any other version) from
(2) Unpack the ZIP file, find linux.iso, open it [eg using guestfish]
and extract VMwareTools-10.0.6-3560309.tar.gz (exact version number
may be different).
(3) Install a new RHEL 7.3 guest with VMware tools enabled:
$ virt-builder rhel-7.3 \
--install bash,perl,net-tools \
--copy-in /var/tmp/VMwareTools-10.0.6-3560309.tar.gz:/tmp \
--run-command 'cd /tmp && tar zxf /tmp/VMwareTools-10.0.6-3560309.tar.gz' \
--run-command 'cd /tmp/vmware-tools-distrib &&
./vmware-install.pl -d default --force-install'
(4) Perform a conversion:
$ virt-v2v -i disk rhel-7.3.img -o null
However I could not reproduce the hang (which is possibly not surprising
because VMware tools isn't "really" installed here).
I had a closer look at the log provided and (1) it's a SUSE guest and
(2) the guest is hanging when rebuilding the kdump initrd.
I've heard that one before ...
I wonder if we could set rootdev as we do in this patch:
A speculative patch would look like this:
diff --git a/v2v/convert_linux.ml b/v2v/convert_linux.ml
index c34bf3e91..5d650561f 100644
@@ -304,7 +304,11 @@ let rec convert (g : G.guestfs) inspect source output rcaps =
let uninstaller = "/usr/bin/vmware-uninstall-tools.pl" in
if g#is_file ~followsymlinks:true uninstaller then (
- ignore (g#command [| uninstaller |]);
+ if family = `SUSE_family then
+ ignore (g#sh (sprintf "/usr/bin/env rootdev=%s %s"
+ inspect.i_root uninstaller))
+ ignore (g#command [| uninstaller |]);
(* Reload Augeas to detect changes made by vbox tools uninst. *)
Pino, do we have any SUSE Enterprise templates we can use?
(In reply to Richard W.M. Jones from comment #12)
> I had a closer look at the log provided and (1) it's a SUSE guest and
> (2) the guest is hanging when rebuilding the kdump initrd.
> I've heard that one before ...
> I wonder if we could set rootdev as we do in this patch:
> A speculative patch would look like this:
> diff --git a/v2v/convert_linux.ml b/v2v/convert_linux.ml
> index c34bf3e91..5d650561f 100644
> --- a/v2v/convert_linux.ml
> +++ b/v2v/convert_linux.ml
> @@ -304,7 +304,11 @@ let rec convert (g : G.guestfs) inspect source output
> rcaps =
> let uninstaller = "/usr/bin/vmware-uninstall-tools.pl" in
> if g#is_file ~followsymlinks:true uninstaller then (
> - ignore (g#command [| uninstaller |]);
> + if family = `SUSE_family then
> + ignore (g#sh (sprintf "/usr/bin/env rootdev=%s %s"
> + inspect.i_root uninstaller))
Better use g#command here too, as done when mkinitrd is run:
ignore (g#command [| "/usr/bin/env";
"rootdev=" ^ inspect.i_root;
this way quoting issues are avoided.
Also, theoretically this could be done regardless of the distro, I don't think on other distros the rootdev environment variable should do much. (Although it can be avoided, to be safe.)
> Pino, do we have any SUSE Enterprise templates we can use?
Nope, neither on VMware.
Also, I recall being told the hang happened also on other guests than SUSE, i.e. RHEL, but I cannot find references now. Jaroslav, do you remember more?
> Also, I recall being told the hang happened also on other guests than SUSE,
> i.e. RHEL, but I cannot find references now. Jaroslav, do you remember more?
Yes it happened for SLES 11 and RHEL 6.x. I attached screenshot from the remote sessions, unfortunately is all what i have for now.
There are 3 pictures where the conversion hanged for ~20-30 minutes, I am not sure if it was related to CU environment but was no related to one distro.
Thanks a lot !
Created attachment 1300309 [details]
hang for 20 minutes
Created attachment 1300310 [details]
Seg fault in one case
Created attachment 1300311 [details]
(In reply to Jaroslav Spanko from comment #16)
> Created attachment 1300310 [details]
> Seg fault in one case
VMware is running a tool which communicates with its own hypervisor.
However there's no VMware hypervisor because everything is run under
qemu, so instead of doing the right thing it crashes. I actually
talked to VMware about this a while back and they fixed it.
(In reply to Jaroslav Spanko from comment #17)
> Created attachment 1300311 [details]
> RHEL hang
The RHEL hang is distinctly different from the SUSE hang, although
with less information available.
Unfortunately I wasn't able to reproduce the RHEL hang using a RHEL
6.8 guest and the same steps as in comment 11.
I spend a couple of days on this and read a lot of documentation, but
I still cannot work out how to enable kdump on OpenSUSE. I think a
better approach is that I prepare a package containing the proposed
patch and the reporter can see if it makes any difference.
I just wonder whether the hang, and the crash are two different results of the same issue, i.e. the vmware-uninstall-tools.pl script trying to communicate with VMX, and failing.
(In reply to Pino Toscano from comment #20)
> I just wonder whether the hang, and the crash are two different results of
> the same issue, i.e. the vmware-uninstall-tools.pl script trying to
> communicate with VMX, and failing.
Likely there are several different things going on, the only
common theme being that they are all caused by running
#1 The log posted in comment 4 looks almost certainly as if
the SUSE mkdumprd script is hanging (see my full analysis in
comment 12). This is what the proposed patch may fix.
#2 There is also a hang in a RHEL 6 guest, but that cannot be the
same thing as above.
#3 There is also a segfault in a VMware tool, which is caused by
the VMware port not being available.
It could be as you say that #2 and #3 are different aspects of
the same thing, or not. I'm unable to reproduce any of them.
As a bit of clarification, it seems like #1 is a problem, but would
not cause a hang.
Note that in the short term (i.e. for the whole 7.4.z) we will disable the execution of vmware-uninstall-tools.pl, since it turning out to be problematic -- see also bug #1480623.
Regarding 7.5, so far it will be disabled too (bug #1477905), unless we find out what are the exact issues the uninstallation script is hitting, and we fix them somehow.
Self-note: fixing this for RHEL 7.5 will make bug 1477905 and bug 1481930 obsolete.