Bug 1850000 - virt-v2v contains broken rhev-apt
Summary: virt-v2v contains broken rhev-apt
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: virt-v2v
Version: 8.3
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.3
Assignee: Pino Toscano
QA Contact: mxie@redhat.com
URL:
Whiteboard: V2V
Depends On: 1789327
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-23 11:50 UTC by Pino Toscano
Modified: 2020-11-17 17:50 UTC (History)
11 users (show)

Fixed In Version: virt-v2v-1.42.0-5.module+el8.3.0+7152+ab3787c3
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1849999
Environment:
Last Closed: 2020-11-17 17:49:16 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
screenshots-for-comment4 (1.63 MB, application/x-xz)
2020-07-02 14:45 UTC, mxie@redhat.com
no flags Details
win8.1-x86-scripts-done.png (268.21 KB, image/png)
2020-07-07 15:22 UTC, mxie@redhat.com
no flags Details
win8.1-x86-qemu-ga-i386.msi.log (116.37 KB, application/octet-stream)
2020-07-09 10:38 UTC, mxie@redhat.com
no flags Details

Description Pino Toscano 2020-06-23 11:50:54 UTC
+++ This bug was initially created as a clone of Bug #1849999 +++

+++ This bug was initially created as a clone of Bug #1849997 +++

+++ This bug was initially created as a clone of Bug #1753914 +++

Since RHV 4.3 we have a bug in rhev-apt which prevent it from wworking correctly. Unfortunately with RHEL 7.7 you pulled in this broken. Because we don't have a build with the fix yet, could you revert to the previous version?

-----------------------------------

This was fixed in a new RHV Tools ISO build -- see bug 1753914 for all the details.

Comment 4 mxie@redhat.com 2020-07-02 14:41:26 UTC
Verify the bug with below builds:
virt-v2v-1.42.0-5.module+el8.3.0+7152+ab3787c3.x86_64
libguestfs-1.42.0-2.module+el8.3.0+6798+ad6e66be.x86_64
libvirt-6.4.0-1.module+el8.3.0+6881+88468c00.x86_64
qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64
virtio-win-1.9.11-1.el8.noarch
nbdkit-1.20.4-2.module+el8.3.0+7165+edc86943.x86_64


Steps:
1.Convert different versions of windows guests from VMware to rhv by virt-v2v
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-cafile=/root/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt $windows_guest

2.Power on guest on rhv after v2v finishing conversion, check guests if firstboot, rhev-apt, qemu-ga is running normally, if there is rhev-apt error in windows log and if certificate of rhev-apt is valid. Summary test result as below:

Guest	        Firstboot               RHEV-apt      QEMU-ga    "No rhev-apt/qemu-ga error   Cert valid   Show IP info on rhv
                                                                   in windows log"                
win2019		Yes			  Yes           Yes             Yes             	 Yes               Yes 
win2016		Yes			  Yes           Yes             Yes             	 Yes               Yes
win10 x86	Yes                       Yes           Yes             Yes             	 Yes               Yes
win10 x64	Yes(reboot manually)      Yes           Yes             Yes             	 Yes               Yes
win2012r2	Yes                       No	        No	        Yes	        	 Yes               No
win2012		Yes                       Yes           Yes             Yes             	 Yes               Yes 
win8.1 x86	Yes	                  No	        Yes	        qemu-ga error	         Yes               Yes
win8.1 x64	Yes	                  Yes	        Yes	        Yes	      		 Yes               No
win8 x86	Yes	     		  Yes	 	Yes	        Yes	              	 Yes               Yes
win8 x64	Yes	     		  Yes	 	Yes	        Yes                 	 Yes	           Yes
win2008r2	Yes	     		  Yes           No    	  	Yes                 	 Yes               No


Below is the conclusion for the test result:

  (1)For win10-x64 guest, firstboot service failed to start when guest is started for the first time, but firstboot service can start normally after rebooting guest manually, there is firstboot error info in windows log, pls refer to screenshot "win10-x64-firstboot-error", I also met the problem when testing win2016 and win2019 sometimes.

 (2)For win2012r2, rhev-apt and qemu-ga can't be installed with 100% reproducible rate. For win8.1-x64, rhev-apt and qemu-ga can't be installed sometimes and reproducible rate is about 30%, please refer to "win8.1-x64-different-result.png". I think the problem of win2012r2 and win8.1-x64 is already tracked by bug1584678 and bug1820152. 

  (3) For win8.1-x86, it's strange rhev-apt can't installed but qemu-ga can start normally, find a qemu-ga error in windows log, pls refer to "win8.1-x86-screenshots"

  (4) For win2008r2, rhev-apt can start normally but qemu-ga can't installed, I think the problem is already tracked by bug1820144



Hi Pino,

   For reproducible rate of problem1 , test win10-x64 twice, failed both times; win2019 and win2016 are tested twice, each guest failed once; didn't met the problem on the other guests. Overall,the reproducible rate of firstboot problem is not high, do you think the result is acceptable?


Hi Tomas,

   Pls help to confirm the problem3, thanks!

Comment 5 mxie@redhat.com 2020-07-02 14:45:22 UTC
Created attachment 1699646 [details]
screenshots-for-comment4

Comment 6 Pino Toscano 2020-07-07 09:28:24 UTC
(In reply to mxie from comment #4)
> Verify the bug with below builds:
> virt-v2v-1.42.0-5.module+el8.3.0+7152+ab3787c3.x86_64
> libguestfs-1.42.0-2.module+el8.3.0+6798+ad6e66be.x86_64
> libvirt-6.4.0-1.module+el8.3.0+6881+88468c00.x86_64
> qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64
> virtio-win-1.9.11-1.el8.noarch
> nbdkit-1.20.4-2.module+el8.3.0+7165+edc86943.x86_64
> 
> 
> Steps:
> 1.Convert different versions of windows guests from VMware to rhv by virt-v2v
> # virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it
> vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io 
> vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 
> -ip /home/passwd  -o rhv-upload -oo rhv-cafile=/root/ca.pem -oo rhv-direct
> -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op
> /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt $windows_guest
> 
> 2.Power on guest on rhv after v2v finishing conversion, check guests if
> firstboot, rhev-apt, qemu-ga is running normally, if there is rhev-apt error
> in windows log and if certificate of rhev-apt is valid. Summary test result
> as below:
> 
> Guest	        Firstboot               RHEV-apt      QEMU-ga    "No
> rhev-apt/qemu-ga error   Cert valid   Show IP info on rhv
>                                                                    in
> windows log"                
> win2019		Yes			  Yes           Yes             Yes             	 Yes        
> Yes 
> win2016		Yes			  Yes           Yes             Yes             	 Yes        
> Yes
> win10 x86	Yes                       Yes           Yes             Yes       
> Yes               Yes
> win10 x64	Yes(reboot manually)      Yes           Yes             Yes       
> Yes               Yes
> win2012r2	Yes                       No	        No	        Yes	        	 Yes 
> No
> win2012		Yes                       Yes           Yes             Yes        
> Yes               Yes 
> win8.1 x86	Yes	                  No	        Yes	        qemu-ga error	      
> Yes               Yes
> win8.1 x64	Yes	                  Yes	        Yes	        Yes	      		 Yes   
> No
> win8 x86	Yes	     		  Yes	 	Yes	        Yes	              	 Yes             
> Yes
> win8 x64	Yes	     		  Yes	 	Yes	        Yes                 	 Yes	          
> Yes
> win2008r2	Yes	     		  Yes           No    	  	Yes                 	 Yes    
> No
> 
> 
> Below is the conclusion for the test result:

One thing that I'm not sure whether it is noted anywhere: in this version (i.e. virt-v2v >= 1.42.0, so it applies only to AV 8.3.0 as of today) the installation of qemu-ga is not started directly by the firstboot script, but it is scheduled with a 120 seconds delay.
So make sure that you keep Windows running for at least 5 minutes, just to be sure.

>   (1)For win10-x64 guest, firstboot service failed to start when guest is
> started for the first time, but firstboot service can start normally after
> rebooting guest manually, there is firstboot error info in windows log, pls
> refer to screenshot "win10-x64-firstboot-error", I also met the problem when
> testing win2016 and win2019 sometimes.

No clue about this one. Does it happen everytime? Often? Or rarely?

>   (3) For win8.1-x86, it's strange rhev-apt can't installed but qemu-ga can
> start normally, find a qemu-ga error in windows log, pls refer to
> "win8.1-x86-screenshots"

Looking at win8.1-x86-qemu-ga-error.png it seems that something in the system is not installed correctly, so the installation cannot be done.

>    For reproducible rate of problem1 , test win10-x64 twice, failed both
> times; win2019 and win2016 are tested twice, each guest failed once; didn't
> met the problem on the other guests. Overall,the reproducible rate of
> firstboot problem is not high, do you think the result is acceptable?

Looks so; it is better or worse than with older versions of rhev-apt?

Comment 7 mxie@redhat.com 2020-07-07 10:27:23 UTC
> One thing that I'm not sure whether it is noted anywhere: in this version
> (i.e. virt-v2v >= 1.42.0, so it applies only to AV 8.3.0 as of today) the
> installation of qemu-ga is not started directly by the firstboot script, but
> it is scheduled with a 120 seconds delay.
> So make sure that you keep Windows running for at least 5 minutes, just to
> be sure.

I'm sure every guest had powered on more than half an hour when I checked them

> >   (1)For win10-x64 guest, firstboot service failed to start when guest is
> > started for the first time, but firstboot service can start normally after
> > rebooting guest manually, there is firstboot error info in windows log, pls
> > refer to screenshot "win10-x64-firstboot-error", I also met the problem when
> > testing win2016 and win2019 sometimes.
> 
> No clue about this one. Does it happen everytime? Often? Or rarely?

I only met the firstboot problem when testing win10-x64, win2016 and win2019 guest on rhel8.3 AV and reproducible rate of the is only about 50%. Besides, it's strange that I didn't meet the problem when testing rhev-apt on rhel7.9, pls refer to bug1753914, and zili said she didn't meet this problem on rhel8.2.1 (bug1849999)


> Looks so; it is better or worse than with older versions of rhev-apt?

I think this version of rhev-apt is better than older one, At least this version has fixed bug1789327 and most of windows guest works well with this version of rhev-apt

Comment 8 Tomáš Golembiovský 2020-07-07 11:12:18 UTC
> 
> >   (3) For win8.1-x86, it's strange rhev-apt can't installed but qemu-ga can
> > start normally, find a qemu-ga error in windows log, pls refer to
> > "win8.1-x86-screenshots"
> 
> Looking at win8.1-x86-qemu-ga-error.png it seems that something in the
> system is not installed correctly, so the installation cannot be done.
> 

I wonder if the error is only related to qemu-ga installation or is maybe a consequence of the failed/aborted rhev-apt installation before. You should find the log from MSI installed in `scripts-done` directory, ould you please attach it?

Either way this is unrelated to any rhev-apt changes so should not block validation of this bug.

Comment 9 mxie@redhat.com 2020-07-07 15:21:39 UTC
(In reply to Tomáš Golembiovský from comment #8)
> > 
> > >   (3) For win8.1-x86, it's strange rhev-apt can't installed but qemu-ga can
> > > start normally, find a qemu-ga error in windows log, pls refer to
> > > "win8.1-x86-screenshots"
> > 
> > Looking at win8.1-x86-qemu-ga-error.png it seems that something in the
> > system is not installed correctly, so the installation cannot be done.
> > 
> 
> I wonder if the error is only related to qemu-ga installation or is maybe a
> consequence of the failed/aborted rhev-apt installation before. You should
> find the log from MSI installed in `scripts-done` directory, ould you please
> attach it?
> 
> Either way this is unrelated to any rhev-apt changes so should not block
> validation of this bug.

I didn't find log in scripts-done folder of win8.1-x86 guest and I think it's because rhev-apt is not installed properly, pls refer to 'win8.1-x86-scripts-done.png'

Comment 10 mxie@redhat.com 2020-07-07 15:22:17 UTC
Created attachment 1700180 [details]
win8.1-x86-scripts-done.png

Comment 11 Tomáš Golembiovský 2020-07-09 07:11:47 UTC
(In reply to mxie from comment #9)
> I didn't find log in scripts-done folder of win8.1-x86 guest and I think
> it's because rhev-apt is not installed properly, pls refer to
> 'win8.1-x86-scripts-done.png'

Sorry, my mistake. The log should actually be in C:\ where the MSI is.

Comment 12 mxie@redhat.com 2020-07-09 10:38:22 UTC
(In reply to Tomáš Golembiovský from comment #11)
> (In reply to mxie from comment #9)
> > I didn't find log in scripts-done folder of win8.1-x86 guest and I think
> > it's because rhev-apt is not installed properly, pls refer to
> > 'win8.1-x86-scripts-done.png'
> 
> Sorry, my mistake. The log should actually be in C:\ where the MSI is.

Pls check 'win8.1-x86-qemu-ga-i386.msi.log'

Comment 13 mxie@redhat.com 2020-07-09 10:38:59 UTC
Created attachment 1700417 [details]
win8.1-x86-qemu-ga-i386.msi.log

Comment 14 mxie@redhat.com 2020-07-09 12:16:30 UTC
As firstboot problem of win10-x64 and qemu-ga error of win8.1-x86 should not block validation of this bug, move the bug from ON_QA to VERIFIED

Comment 15 Tomáš Golembiovský 2020-07-09 14:50:00 UTC
Thank, I see the following in the logs:

MSI (s) (CC:F8) [12:08:01:424]: Checking in-progress install: install for different product.
MSI (s) (CC:F8) [12:08:01:424]: Checking in-progress install: install for same configuration.
MSI (s) (CC:F8) [12:08:01:424]: Note: 1: 1704 2: Microsoft Visual C++ 2008 Redistributable - x86 9.0.30729.17 
MSI (s) (CC:F8) [12:08:01:424]: Note: 1: 2262 2: Error 3: -2147287038 
Action start 12:08:01: InstallInitialize.
Error 1704. An installation for Microsoft Visual C++ 2008 Redistributable - x86 9.0.30729.17 is currently suspended.  You must undo the changes made by that installation to continue.  Do you want to undo those changes?
MSI (s) (CC:F8) [12:08:01:424]: Note: 1: 2262 2: Error 3: -2147287038 
MSI (s) (CC:F8) [12:08:01:424]: Product: QEMU guest agent -- Error 1704. An installation for Microsoft Visual C++ 2008 Redistributable - x86 9.0.30729.17 is currently suspended.  You must undo the changes made by that installation to continue.  Do you want to undo those changes?

MSI (s) (CC:F8) [12:08:01:518]: Note: 1: 2265 2:  3: -2147287035 
MSI (s) (CC:F8) [12:08:01:518]: Machine policy value 'DisableRollback' is 0
MSI (s) (CC:F8) [12:08:01:518]: Executing op: Header(Signature=1397708873,Version=500,Timestamp=1357471537,LangId=1033,Platform=0,ScriptType=2,ScriptMajorVersion=21,ScriptMinorVersion=4,ScriptAttributes=1)


So the issue is not caused by qemu-ga installation. It is either caused by the incomplete rhev-apt installation (do we have logs?) or it was already in corrupted state before virt-v2v conversion.

Comment 18 errata-xmlrpc 2020-11-17 17:49:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5137


Note You need to log in before you can comment on or make changes to this bug.