Bug 1477905

Summary: Disable uninstallation of VMware tools (again)
Product: Red Hat Enterprise Linux 7 Reporter: Richard W.M. Jones <rjones>
Component: libguestfsAssignee: Pino Toscano <ptoscano>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.4CC: jherrman, juzhou, kuwei, mtessun, mxie, ptoscano, rjones, tzheng, xiaodwan
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: V2V
Fixed In Version: libguestfs-1.36.5-1.el7 Doc Type: Bug Fix
Doc Text:
Red Hat Enterprise Linux 7.4 implemented a feature that enabled the virt-v2v utility to automatically remove VMware tools from Linux guest virtual machines that virt-v2v is converting. However, when the VMware tools on the source guest were installed from a tarball, this feature did not work correctly. As a consequence, the virt-v2v conversion took significantly longer and, in some cases, became unresponsive. With this update, virt-v2v does not attempt to remove VMware tools that were installed from a tarball, which prevents the described problem from occurring. Note that VMware tools installed using RPM packages are still removed during the conversion.
Story Points: ---
Clone Of:
: 1480623 (view as bug list) Environment:
Last Closed: 2018-04-10 09:18:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1465849, 1480623    

Description Richard W.M. Jones 2017-08-03 07:48:11 UTC
Description of problem:

Upstream virt-v2v can run /usr/bin/vmware-uninstall-tools.pl from
the guest to uninstall VMware drivers.  We enabled this feature in
RHEL 7.4 but there have been numerous reports of problems, so we
should disable it again.

In bug 1155610 we disabled uninstallation in RHEL 7.1.

In bug 1155150 we reenabled uninstallation in RHEL 7.4.

Version-Release number of selected component (if applicable):

libguestfs-1.36.3-6.el7_4.2

Comment 2 Richard W.M. Jones 2017-08-03 07:52:37 UTC
Issues come in two categories:

(1) It's slow, taking around 20 minutes per guest.

(2) It hangs, causing virt-v2v to never complete.

Comment 11 mxie@redhat.com 2017-09-20 12:35:02 UTC
Verify the bug with below builds:
virt-v2v-1.36.6-1.el7.x86_64
libguestfs-1.36.6-1.el7.x86_64
libvirt-3.7.0-2.el7.x86_64
qemu-kvm-rhev-2.9.0-16.el7_4.8.x86_64


Steps:

Scenario1:Use v2v to convert a rhel6 guest whose vmware-tools is installed by rpm packages 
1.1 Install vmware-tools in rhel6.8 guest with rpm packages but there is no vmware-tools process in guest if vmware-tools is installed with rpm packages

# rpm -qa |grep vmware
xorg-x11-drv-vmware-13.1.0-2.el6.x86_64
vmware-tools-libraries-nox-8.6.17-1.el6.x86_64
vmware-tools-foundation-8.6.17-1.el6.x86_64
vmware-tools-libraries-x-8.6.17-1.el6.x86_64

# ps -ef |grep vmware
 root      3226  3205  0 02:23 pts/0    00:00:00 grep vmware


1.2 Convert guest from vmware by virt-v2v and there is no error about vmware-tools during conversion
# virt-v2v -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 Auto-esx6.0-rhel6.8-vmware-tool --password-file /tmp/passwd
[   0.0] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 Auto-esx6.0-rhel6.8-vmware-tool
[   1.3] Creating an overlay to protect the source from being modified
[   2.1] Initializing the target -o libvirt -os default
[   2.1] Opening the overlay
[  14.9] Inspecting the overlay
[ 103.5] Checking for sufficient free disk space in the guest
[ 103.5] Estimating space required on target for each disk
[ 103.5] Converting Red Hat Enterprise Linux Server release 6.8 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 855.2] Mapping filesystem data to avoid copying unused and blank areas
[ 856.3] Closing the overlay
[ 856.6] Checking if the guest needs BIOS or UEFI to boot
[ 856.6] Assigning disks to buses
[ 856.6] Copying disk 1/1 to /var/lib/libvirt/images/Auto-esx6.0-rhel6.8-vmware-tool-sda (raw)
    (100.00/100%)
[1899.3] Creating output metadata
Pool default refreshed

Domain Auto-esx6.0-rhel6.8-vmware-tool defined from /tmp/v2vlibvirt80ed96.xml

[1899.8] Finishing off


1.3 After finishing v2v conversion, find related vmware-tools packages have been removed in guest
# rpm -qa |grep vmware
xorg-x11-drv-vmware-13.1.0-2.el6.x86_64

Result:
  Related vmware-tools packages could be removed successfully in rhel6 guest if vmware-tools is installed with rpm packages during virt-v2v conversion 



Scenario2:Use v2v to convert a rhel6 guest which has installed vmware-tools via vsphere client 
2.1 If install vmware-tools in rhel6 guest via vsphere client,there is no vmware-tools package intalled but there is vmware-tools process running in guest

# ps -ef |grep vmware
root      1856     1  0 02:21 ?        00:00:00 /usr/lib/vmware-vgauth/VGAuthService -s
root      2006     1  0 02:21 ?        00:00:00 //usr/lib/vmware-caf/pme/bin/ManagementAgentHost
root      2953     1  0 02:22 ?        00:00:00 /usr/lib/vmware-tools/sbin64/vmtoolsd -n vmusr
root      3226  3205  0 02:23 pts/0    00:00:00 grep vmware


2.2 Convert guest from vmware to rhv by virt-v2v and there is no error about vmware-tools during conversion
# virt-v2v -ic vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/?no_verify=1 esx6.5-rhel6.9-x64-vmware-tools --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
[   0.0] Opening the source -i libvirt -ic vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/?no_verify=1 esx6.5-rhel6.9-x64-vmware-tools
[   1.2] Creating an overlay to protect the source from being modified
[   2.2] Initializing the target -o rhv -os 10.73.131.93:/home/nfs_export
[   2.5] Opening the overlay
[  19.8] Inspecting the overlay
[ 129.9] Checking for sufficient free disk space in the guest
[ 129.9] Estimating space required on target for each disk
[ 129.9] Converting Red Hat Enterprise Linux Server release 6.9 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[1065.3] Mapping filesystem data to avoid copying unused and blank areas
[1068.5] Closing the overlay
[1068.7] Checking if the guest needs BIOS or UEFI to boot
[1068.7] Assigning disks to buses
[1068.7] Copying disk 1/1 to /tmp/v2v.qv9T0I/bdf9c90b-e6f0-439c-aa9f-6305fd5fad7e/images/b276ed39-39fb-4eb7-b0ca-aeb68c965251/9707d23f-e7a1-4ff8-bc44-70c39a1e8600 (raw)
    (100.00/100%)
[2796.8] Creating output metadata
[2796.9] Finishing off


2.3 After finishing conversion,check vmware-tools process
# ps -ef |grep vmware
root      3226  3205  0 02:23 pts/0    00:00:00 grep vmware


2.4 Try to uninstall vmware tools manually in guest
---------------------------------
#./vmware-uninstall-tools.pl 
Uninstalling the tar installation of VMware Tools.

Stopping services for vmware-tools

vmware-tools stop/waiting
File /etc/vmware-caf/pme/config/cafenv-appconfig is backed up to 
/etc/vmware-caf/pme/config/cafenv-appconfig.old.0.

This program previously created the directory /usr/lib/vmware-caf/pme/lib, and 
was about to remove it. Since there are files in that directory that this 
program did not create, it will not be removed.

The removal of VMware Tools 10.1.7 build-5541682 for Linux completed 
successfully.  Thank you for having tried this software.
-------------------------------

Result:
  Although related vmware-tools process has disappeared in rhel6 guest after v2v conversion, but vmware-tools still could be uninstalled manually.so virt-v2v didn't uninstall vmware-tools completely for rhel6 guest during conversion 



Scenario3:Use v2v to convert rhel7.4 guest which has installed vmware tools via vsphere client
3.1 Install vmware tools in rhel7 guest via vsphere client, then vmware tools service will be running in guest and check vmware-tools process as below

#service vmware-tools status
vmtoolsd is running

#ps -ef |grep vmware
root      1042     1  0 16:26 ?        00:00:00 /usr/lib/vmware-vgauth/VGAuthService -s
root      1121     1  0 16:26 ?        00:00:03 //usr/lib/vmware-caf/pme/bin/ManagementAgentHost
root      2710     1  0 16:28 ?        00:00:10 /usr/lib/vmware-tools/sbin64/vmtoolsd -n vmusr
root      2741  2937  0 18:17 pts/0    00:00:00 grep --color=auto vmware

#systemctl status vmware-tools
● vmware-tools.service - SYSV: Manages the services needed to run VMware software
   Loaded: loaded (/etc/rc.d/init.d/vmware-tools; bad; vendor preset: disabled)
   Active: active (running) since Wed 2017-09-20 20:14:52 CST; 4min 36s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 755 ExecStart=/etc/rc.d/init.d/vmware-tools start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/vmware-tools.service
           ├─1015 /usr/sbin/vmtoolsd
           ├─1043 /usr/lib/vmware-vgauth/VGAuthService -s
           └─1130 //usr/lib/vmware-caf/pme/bin/ManagementAgentHost

Sep 20 20:14:41 localhost.localdomain systemd[1]: Starting SYSV: Manages the ...
Sep 20 20:14:42 localhost.localdomain vmware-tools[755]: Checking acpi hot pl...
Sep 20 20:14:42 localhost.localdomain vmware-tools[755]: Starting VMware Tool...
Sep 20 20:14:43 localhost.localdomain vmware-tools[755]: Switching to guest c...
Sep 20 20:14:44 localhost.localdomain vmware-tools[755]: VMware Automatic Kmo...
Sep 20 20:14:45 localhost.localdomain vmware-tools[755]: Guest operating syst...
Sep 20 20:14:47 localhost.localdomain vmware-tools[755]: VGAuthService:[  OK  ]
Sep 20 20:14:52 localhost.localdomain ManagementAgentHost[1081]: Initializing...
Sep 20 20:14:52 localhost.localdomain vmware-tools[755]: Common Agent:[  OK  ]
Sep 20 20:14:52 localhost.localdomain systemd[1]: Started SYSV: Manages the s...
Hint: Some lines were ellipsized, use -l to show in full.



3.2 Convert guest from vmware to rhv by virt-v2v and there is no error about vmware-tools during conversion
# virt-v2v -ic vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/?no_verify=1 esx6.5-rhel7.4-x64-vmware-tools --password-file /tmp/passwd -o rhv -os 10.73.131.93:/home/nfs_export 
[   0.0] Opening the source -i libvirt -ic vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/?no_verify=1 esx6.5-rhel7.4-x64-vmware-tools
[   1.4] Creating an overlay to protect the source from being modified
[   2.2] Initializing the target -o rhv -os 10.73.131.93:/home/nfs_export
[   2.5] Opening the overlay
[  22.5] Inspecting the overlay
[ 153.7] Checking for sufficient free disk space in the guest
[ 153.7] Estimating space required on target for each disk
[ 153.7] Converting Red Hat Enterprise Linux Server 7.4 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[1230.7] Mapping filesystem data to avoid copying unused and blank areas
[1233.0] Closing the overlay
[1233.2] Checking if the guest needs BIOS or UEFI to boot
[1233.2] Assigning disks to buses
[1233.2] Copying disk 1/1 to /tmp/v2v.9afQMv/bdf9c90b-e6f0-439c-aa9f-6305fd5fad7e/images/d1fa74ce-4ee7-41d2-8d01-7d1586214d49/8a56ff44-c99a-4ca7-b4c1-31f7c2794230 (raw)
    (100.00/100%)
[3803.6] Creating output metadata
[3803.9] Finishing off


3.3 After finishing conversion, check status of vmware-tools service, vmware-tools process in guest
#service vmware-tools status
vmtoolsd is not running

#ps -ef |grep vmware
root      2741  2937  0 18:17 pts/0    00:00:00 grep --color=auto vmware

# systemctl status vmware-tools
● vmware-tools.service - SYSV: Manages the services needed to run VMware software
   Loaded: loaded (/etc/rc.d/init.d/vmware-tools; bad; vendor preset: disabled)
   Active: active (exited) since Wed 2017-09-20 16:13:58 CST; 3min 32s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 691 ExecStart=/etc/rc.d/init.d/vmware-tools start (code=exited, status=0/SUCCESS)

Sep 20 16:13:57 localhost.localdomain systemd[1]: Starting SYSV: Manages the ...
Sep 20 16:13:58 localhost.localdomain vmware-tools[691]: Starting VMware Tool...
Sep 20 16:13:58 localhost.localdomain vmware-tools[691]: Switching to host co...
Sep 20 16:13:58 localhost.localdomain systemd[1]: Started SYSV: Manages the s...
Hint: Some lines were ellipsized, use -l to show in full.



3.4 Try to uninstall vmware-tools manually in rhel7 guest at rhv
--------------
# ./vmware-uninstall-tools.pl 
Uninstalling the tar installation of VMware Tools.

Stopping services for VMware Tools


This program previously created the directory /usr/lib/vmware-caf/pme/bin, and 
was about to remove it. Since there are files in that directory that this 
program did not create, it will not be removed.

This program previously created the directory /usr/lib/vmware-caf/pme, and was 
about to remove it. Since there are files in that directory that this program 
did not create, it will not be removed.

This program previously created the directory /usr/lib/vmware-caf, and was 
about to remove it. Since there are files in that directory that this program 
did not create, it will not be removed.

The removal of VMware Tools 10.1.7 build-5541682 for Linux completed 
successfully.  Thank you for having tried this software.
----------------

3.5 check status of vmware tools service in guest again
# systemctl status vmware-tools
Unit vmware-tools.service could not be found.


Result:
   Although vmtoolsd service has been stopped and related vmware-tools process has disappeared , some vmware-tools service are still running in rhel7 guest after v2v conversion, so vmware-tools is not uninstalled completely for rhel7 guest during virt-v2v conversion


Hi rjones,
   Pls help to check the scenario2 and scenario3, maybe v2v didn't uninstalled vmware-tools completely during conversion,thanks!

Comment 12 Richard W.M. Jones 2017-09-20 12:43:19 UTC
All three scenarios are correct.  In scenario 1 we are still
removing the RPMs.  In scenarios 2 & 3 we no longer attempt to
run /usr/bin/vmware-uninstall-tools.pl (because it sometimes
hung for reasons which were not understood fully), but the script
is still there for users to run after conversion if they want.

Comment 13 mxie@redhat.com 2017-09-21 01:38:48 UTC
According to comment 11 and comment12, move this bug from ON_QA to VERIFIED

Comment 16 errata-xmlrpc 2018-04-10 09:18:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0677