Bug 1465849 - v2v hangs on removing vmware-tool
v2v hangs on removing vmware-tool
Status: NEW
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs (Show other bugs)
7.4
Unspecified Unspecified
high Severity high
: rc
: ---
Assigned To: Richard W.M. Jones
Virtualization Bugs
V2V
:
Depends On: 1477905
Blocks: 1420851
  Show dependency treegraph
 
Reported: 2017-06-28 07:05 EDT by Jaroslav Spanko
Modified: 2018-06-20 22:53 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
uninstall script (200.75 KB, application/x-perl)
2017-07-17 08:40 EDT, Jaroslav Spanko
no flags Details
hang for 20 minutes (198.99 KB, image/png)
2017-07-18 04:07 EDT, Jaroslav Spanko
no flags Details
Seg fault in one case (216.35 KB, image/png)
2017-07-18 04:08 EDT, Jaroslav Spanko
no flags Details
RHEL hang (233.94 KB, image/png)
2017-07-18 04:08 EDT, Jaroslav Spanko
no flags Details

  None (edit)
Description Jaroslav Spanko 2017-06-28 07:05:09 EDT
Description of problem:
During the import from Vmware(Suse,RHEL VMs) the import hangs on the 
"/usr/bin/vmware-uninstall-tools.pl"
it stay 10-30 minutes on this tasks, after that once returned Segmentation fault, second time we have to kill v2v.
We tried 3 different VM and it hangs on the same

Version-Release number of selected component (if applicable):
7.4 beta
libguestfs-1.36.3-5.el7.suse_1.x86_64.rpm
libguestfs-1.36.3-5.el7.x86_64

How reproducible:
100 %

Steps to Reproduce:
1. run virt-v2v
2. hang on "/usr/bin/vmware-uninstall-tools.pl"

Actual results:
One Segmentation fault, one kill, one hang for long time

Expected results:
It is possible to handle this better ? We never go to the error
warning (f_"VMware tools was detected, but uninstallation failed.  The error message was: %s (ignored)")

Additional info:
PoC for Customer, it failed on Suse and RHEL VM's
I will try generate dumps as soon as will be possible
strace
strace: Process 16062 attached
restart_syscall(<... resuming interrupted poll ...>
Did you see something similar ?
Comment 2 Pino Toscano 2017-06-28 07:15:30 EDT
Can you please run virt-v2v with the -v -x parameters added, and provide the full log? I.e. something like:

  $ virt-v2v -v -x [all the other parameters already used] 2>&1 | tee v2v.log
Comment 3 Jaroslav Spanko 2017-06-28 07:24:51 EDT
uploaded
thanks
Comment 4 Jaroslav Spanko 2017-06-28 07:25 EDT
Created attachment 1292613 [details]
v2v hangs log
Comment 5 Pino Toscano 2017-06-28 08:06:34 EDT
Ehm you forgot -x too...
Comment 6 Richard W.M. Jones 2017-06-29 05:17:25 EDT
Also I don't think we have the Perl script.  Could you download it
and attach it to the bug:

virt-copy-out -a guest /usr/bin/vmware-uninstall-tools.pl /tmp
& upload /tmp/vmware-uninstall-tools.pl

See also comment 5.
Comment 8 Jaroslav Spanko 2017-06-29 09:18:35 EDT
I am not able to provide the /usr/bin/vmware-uninstall-tools.pl now, I will try to reproduce this as we faced this during remote session with cu every time.
Thanks
Comment 9 Jaroslav Spanko 2017-07-17 08:39:40 EDT
Attached the pl script, I am trying to reproduce it but I'm not sure as till now with no success.
Thanks !
Comment 10 Jaroslav Spanko 2017-07-17 08:40 EDT
Created attachment 1299845 [details]
uninstall script
Comment 11 Richard W.M. Jones 2017-07-17 10:33:54 EDT
I tried to reproduce this using the following steps:

(1) Download VMwaretools-10.0.6-3560309.zip (or any other version) from
VMware's site.

(2) Unpack the ZIP file, find linux.iso, open it [eg using guestfish]
and extract VMwareTools-10.0.6-3560309.tar.gz (exact version number
may be different).

(3) Install a new RHEL 7.3 guest with VMware tools enabled:

$ virt-builder rhel-7.3 \
    --install bash,perl,net-tools \
    --copy-in /var/tmp/VMwareTools-10.0.6-3560309.tar.gz:/tmp \
    --run-command 'cd /tmp && tar zxf /tmp/VMwareTools-10.0.6-3560309.tar.gz' \
    --run-command 'cd /tmp/vmware-tools-distrib &&
                   ./vmware-install.pl -d default --force-install'

(4) Perform a conversion:

$ virt-v2v -i disk rhel-7.3.img -o null

However I could not reproduce the hang (which is possibly not surprising
because VMware tools isn't "really" installed here).
Comment 12 Richard W.M. Jones 2017-07-17 10:41:24 EDT
I had a closer look at the log provided and (1) it's a SUSE guest and
(2) the guest is hanging when rebuilding the kdump initrd.

I've heard that one before ...

I wonder if we could set rootdev as we do in this patch:

https://www.redhat.com/archives/libguestfs/2017-June/msg00000.html

A speculative patch would look like this:

diff --git a/v2v/convert_linux.ml b/v2v/convert_linux.ml
index c34bf3e91..5d650561f 100644
--- a/v2v/convert_linux.ml
+++ b/v2v/convert_linux.ml
@@ -304,7 +304,11 @@ let rec convert (g : G.guestfs) inspect source output rcaps =
     let uninstaller = "/usr/bin/vmware-uninstall-tools.pl" in
     if g#is_file ~followsymlinks:true uninstaller then (
       try
-        ignore (g#command [| uninstaller |]);
+        if family = `SUSE_family then
+          ignore (g#sh (sprintf "/usr/bin/env rootdev=%s %s"
+                                inspect.i_root uninstaller))
+        else
+          ignore (g#command [| uninstaller |]);
 
         (* Reload Augeas to detect changes made by vbox tools uninst. *)
         Linux.augeas_reload g


Pino, do we have any SUSE Enterprise templates we can use?
Comment 13 Pino Toscano 2017-07-17 12:52:13 EDT
(In reply to Richard W.M. Jones from comment #12)
> I had a closer look at the log provided and (1) it's a SUSE guest and
> (2) the guest is hanging when rebuilding the kdump initrd.
> 
> I've heard that one before ...
> 
> I wonder if we could set rootdev as we do in this patch:
> 
> https://www.redhat.com/archives/libguestfs/2017-June/msg00000.html
> 
> A speculative patch would look like this:
> 
> diff --git a/v2v/convert_linux.ml b/v2v/convert_linux.ml
> index c34bf3e91..5d650561f 100644
> --- a/v2v/convert_linux.ml
> +++ b/v2v/convert_linux.ml
> @@ -304,7 +304,11 @@ let rec convert (g : G.guestfs) inspect source output
> rcaps =
>      let uninstaller = "/usr/bin/vmware-uninstall-tools.pl" in
>      if g#is_file ~followsymlinks:true uninstaller then (
>        try
> -        ignore (g#command [| uninstaller |]);
> +        if family = `SUSE_family then
> +          ignore (g#sh (sprintf "/usr/bin/env rootdev=%s %s"
> +                                inspect.i_root uninstaller))

Better use g#command here too, as done when mkinitrd is run:

         ignore (g#command [| "/usr/bin/env";
                              "rootdev=" ^ inspect.i_root;
                              uninstaller |]);

this way quoting issues are avoided.

Also, theoretically this could be done regardless of the distro, I don't think on other distros the rootdev environment variable should do much. (Although it can be avoided, to be safe.)

> Pino, do we have any SUSE Enterprise templates we can use?

Nope, neither on VMware.

Also, I recall being told the hang happened also on other guests than SUSE, i.e. RHEL, but I cannot find references now.  Jaroslav, do you remember more?
Comment 14 Jaroslav Spanko 2017-07-18 04:06:29 EDT
> Also, I recall being told the hang happened also on other guests than SUSE,
> i.e. RHEL, but I cannot find references now.  Jaroslav, do you remember more?

Hi Pino,Rich
Yes it happened for SLES 11 and RHEL 6.x. I attached screenshot from the remote sessions, unfortunately is all what i have for now.
There are 3 pictures where the conversion hanged for ~20-30 minutes, I am not sure if it was related to CU environment but was no related to one distro.
 
Thanks a lot !
Comment 15 Jaroslav Spanko 2017-07-18 04:07 EDT
Created attachment 1300309 [details]
hang for 20 minutes
Comment 16 Jaroslav Spanko 2017-07-18 04:08 EDT
Created attachment 1300310 [details]
Seg fault in one case
Comment 17 Jaroslav Spanko 2017-07-18 04:08 EDT
Created attachment 1300311 [details]
RHEL hang
Comment 18 Richard W.M. Jones 2017-07-18 05:09:43 EDT
(In reply to Jaroslav Spanko from comment #16)
> Created attachment 1300310 [details]
> Seg fault in one case

VMware is running a tool which communicates with its own hypervisor.
However there's no VMware hypervisor because everything is run under
qemu, so instead of doing the right thing it crashes.  I actually
talked to VMware about this a while back and they fixed it.

(In reply to Jaroslav Spanko from comment #17)
> Created attachment 1300311 [details]
> RHEL hang

The RHEL hang is distinctly different from the SUSE hang, although
with less information available.

Unfortunately I wasn't able to reproduce the RHEL hang using a RHEL
6.8 guest and the same steps as in comment 11.
Comment 19 Richard W.M. Jones 2017-07-18 07:47:26 EDT
I spend a couple of days on this and read a lot of documentation, but
I still cannot work out how to enable kdump on OpenSUSE.  I think a
better approach is that I prepare a package containing the proposed
patch and the reporter can see if it makes any difference.
Comment 20 Pino Toscano 2017-07-18 08:45:54 EDT
I just wonder whether the hang, and the crash are two different results of the same issue, i.e. the vmware-uninstall-tools.pl script trying to communicate with VMX, and failing.
Comment 21 Richard W.M. Jones 2017-07-18 08:52:50 EDT
(In reply to Pino Toscano from comment #20)
> I just wonder whether the hang, and the crash are two different results of
> the same issue, i.e. the vmware-uninstall-tools.pl script trying to
> communicate with VMX, and failing.

Likely there are several different things going on, the only
common theme being that they are all caused by running
vmware-uninstall-tools.pl.

#1 The log posted in comment 4 looks almost certainly as if
the SUSE mkdumprd script is hanging (see my full analysis in
comment 12).  This is what the proposed patch may fix.

#2 There is also a hang in a RHEL 6 guest, but that cannot be the
same thing as above.

#3 There is also a segfault in a VMware tool, which is caused by
the VMware port not being available.

It could be as you say that #2 and #3 are different aspects of
the same thing, or not.  I'm unable to reproduce any of them.
Comment 22 Richard W.M. Jones 2017-07-18 09:12:56 EDT
As a bit of clarification, it seems like #1 is a problem, but would
not cause a hang.
Comment 25 Pino Toscano 2017-08-14 05:25:22 EDT
Note that in the short term (i.e. for the whole 7.4.z) we will disable the execution of vmware-uninstall-tools.pl, since it turning out to be problematic -- see also bug #1480623.

Regarding 7.5, so far it will be disabled too (bug #1477905), unless we find out what are the exact issues the uninstallation script is hitting, and we fix them somehow.
Comment 26 Pino Toscano 2017-08-22 02:56:38 EDT
Self-note: fixing this for RHEL 7.5 will make bug 1477905 and bug 1481930 obsolete.

Note You need to log in before you can comment on or make changes to this bug.