Bug 1753914

Note that working rhev-apt is crucial for IMS with RHV

(In reply to Tomáš Golembiovský from comment #0)
> Since RHV 4.3 we have a bug in rhev-apt which prevent it from wworking
> correctly. Unfortunately with RHEL 7.7 you pulled in this broken.

That was the latest version available when I built libguestfs with the fix for bug 1688155.

> Because we don't have a build with the fix yet, could you revert to the previous version?

Do we have a bug against rhev-apt for its broken behaviour?
What is the actual problem, anyway? Our QE tested that version successfully, so I'm puzzled to hear that it's broken.

I can revert to a previous version, sure; however, I need to know:
- what's the actual issue, and how our QE can periodically check that it does not happen
- that there is a short-term plan for fixing rhev-apt itself

(In reply to Pino Toscano from comment #3)
> (In reply to Tomáš Golembiovský from comment #0)
> > Since RHV 4.3 we have a bug in rhev-apt which prevent it from wworking
> > correctly. Unfortunately with RHEL 7.7 you pulled in this broken.
> 
> That was the latest version available when I built libguestfs with the fix
> for bug 1688155.

You already update rhev-apt to close bug 1688056 in 7.6.

> 
> > Because we don't have a build with the fix yet, could you revert to the previous version?
> 
> Do we have a bug against rhev-apt for its broken behaviour?

Gal, we don't have a bug, do we?

> What is the actual problem, anyway? Our QE tested that version successfully,
> so I'm puzzled to hear that it's broken.
> 

From the comments in the bug it seems they just verified that the signature cert is valid.

> I can revert to a previous version, sure; however, I need to know:
> - what's the actual issue, and how our QE can periodically check that it
> does not happen

You have to run the guest in RhV. RHV should automatically attach rhv tools ISO (or attach the ISO manually). rhv-apt should step in during the boot and install drivers and tools. QE should then verify that the guest tools (qemu-ga, vdagent, ovirt-guest-agent) were installed automatically.

> - that there is a short-term plan for fixing rhev-apt itself

Gal?

(In reply to Tomáš Golembiovský from comment #4)
> (In reply to Pino Toscano from comment #3)
> > (In reply to Tomáš Golembiovský from comment #0)
> > > Since RHV 4.3 we have a bug in rhev-apt which prevent it from wworking
> > > correctly. Unfortunately with RHEL 7.7 you pulled in this broken.
> > 
> > That was the latest version available when I built libguestfs with the fix
> > for bug 1688155.
> 
> You already update rhev-apt to close bug 1688056 in 7.6.

No -- bug 1688056 is the RHV bug, while bug 1688155 is the libguestfs bug in RHEL 7.7 to update rhev-apt after bug 1688056 was fixed.
AFAICT, bug 1688155 was not fixed in RHEL 7.6.z.

(In reply to Tomáš Golembiovský from comment #4)
> (In reply to Pino Toscano from comment #3)
> > (In reply to Tomáš Golembiovský from comment #0)
> > > Since RHV 4.3 we have a bug in rhev-apt which prevent it from wworking
> > > correctly. Unfortunately with RHEL 7.7 you pulled in this broken.
> > 
> > That was the latest version available when I built libguestfs with the fix
> > for bug 1688155.
> 
> You already update rhev-apt to close bug 1688056 in 7.6.
> 
> > 
> > > Because we don't have a build with the fix yet, could you revert to the previous version?
> > 
> > Do we have a bug against rhev-apt for its broken behaviour?
> 
> Gal, we don't have a bug, do we?
> 
> > What is the actual problem, anyway? Our QE tested that version successfully,
> > so I'm puzzled to hear that it's broken.
> > 
> 
> From the comments in the bug it seems they just verified that the signature
> cert is valid.

For bug 1688155,QE only use it to track the Certificate of rhev-apt.exe is valid.

> 
> > I can revert to a previous version, sure; however, I need to know:
> > - what's the actual issue, and how our QE can periodically check that it
> > does not happen
> 
> You have to run the guest in RhV. RHV should automatically attach rhv tools
> ISO (or attach the ISO manually). rhv-apt should step in during the boot and
> install drivers and tools. QE should then verify that the guest tools
> (qemu-ga, vdagent, ovirt-guest-agent) were installed automatically.


For the rhv tools,QE has verified linux guests on Bug 1619665 which is fixed since libguestfs-1.40.2-3.el7.
There is still RFE bug open for windows guest. 
Bug 1571239 - RFE: virt-v2v should be able to use rhev-apt from tools ISO

> 
> > - that there is a short-term plan for fixing rhev-apt itself
> 
> Gal?

(In reply to Tomáš Golembiovský from comment #4)
> (In reply to Pino Toscano from comment #3)
> > (In reply to Tomáš Golembiovský from comment #0)
> > > Since RHV 4.3 we have a bug in rhev-apt which prevent it from wworking
> > > correctly. Unfortunately with RHEL 7.7 you pulled in this broken.
> > 
> > That was the latest version available when I built libguestfs with the fix
> > for bug 1688155.
> 
> You already update rhev-apt to close bug 1688056 in 7.6.
> 
> > 
> > > Because we don't have a build with the fix yet, could you revert to the previous version?
> > 
> > Do we have a bug against rhev-apt for its broken behaviour?
> 
> Gal, we don't have a bug, do we?
> 
> > What is the actual problem, anyway? Our QE tested that version successfully,
> > so I'm puzzled to hear that it's broken.
> > 
> 
> From the comments in the bug it seems they just verified that the signature
> cert is valid.
> 
> > I can revert to a previous version, sure; however, I need to know:
> > - what's the actual issue, and how our QE can periodically check that it
> > does not happen
> 
> You have to run the guest in RhV. RHV should automatically attach rhv tools
> ISO (or attach the ISO manually). rhv-apt should step in during the boot and
> install drivers and tools. QE should then verify that the guest tools
> (qemu-ga, vdagent, ovirt-guest-agent) were installed automatically.
> 
> > - that there is a short-term plan for fixing rhev-apt itself
> 
> Gal?

The apt service has been broken for almost a year now (at least) and we plan to deplicate it on 4.4,
I hope this is not news to anyone :)

On the other hand, I have your patch Tomas, does it fixes apt for v2v?
I merged it and I can build it for the next release of tools

(In reply to Gal Zaidman from comment #7)
> The apt service has been broken for almost a year now (at least)
> [...]
> On the other hand, I have your patch Tomas, does it fixes apt for v2v?
> I merged it and I can build it for the next release of tools

Again: what is the actual issue? Any RHV bug? What's the ETA for fixing it?
Let me stress this once more: getting a detailed answer to the above questions, all of them, is a strong prerequisite to doing any changes in libguestfs.

> and we plan to deplicate it on 4.4,
> I hope this is not news to anyone :)

Yes, this is news to me, although topic for a different bug.

(In reply to Tomáš Golembiovský from comment #4)
> (In reply to Pino Toscano from comment #3)
> > (In reply to Tomáš Golembiovský from comment #0)
> > > Since RHV 4.3 we have a bug in rhev-apt which prevent it from wworking
> > > correctly. Unfortunately with RHEL 7.7 you pulled in this broken.
> > 
> > That was the latest version available when I built libguestfs with the fix
> > for bug 1688155.
> 
> You already update rhev-apt to close bug 1688056 in 7.6.
> 
> > 
> > > Because we don't have a build with the fix yet, could you revert to the previous version?
> > 
> > Do we have a bug against rhev-apt for its broken behaviour?
> 
> Gal, we don't have a bug, do we?
> 

No,
we do have a deprecation bug for it on 4.4:
https://bugzilla.redhat.com/show_bug.cgi?id=1746354
We also have liran bug fix:
https://bugzilla.redhat.com/show_bug.cgi?id=1730538

> > What is the actual problem, anyway? Our QE tested that version successfully,
> > so I'm puzzled to hear that it's broken.
> > 
> 
> From the comments in the bug it seems they just verified that the signature
> cert is valid.
> 
> > I can revert to a previous version, sure; however, I need to know:
> > - what's the actual issue, and how our QE can periodically check that it
> > does not happen
> 
> You have to run the guest in RhV. RHV should automatically attach rhv tools
> ISO (or attach the ISO manually). rhv-apt should step in during the boot and
> install drivers and tools. QE should then verify that the guest tools
> (qemu-ga, vdagent, ovirt-guest-agent) were installed automatically.
> 
> > - that there is a short-term plan for fixing rhev-apt itself
> 
> Gal?

Not on my side,
There is the patch that you sent to apt, together with Liran patch, does it fixes the issue?

> > > - that there is a short-term plan for fixing rhev-apt itself
> > 
> > Gal?
> 
> Not on my side,
> There is the patch that you sent to apt, together with Liran patch, does it
> fixes the issue?

Yes, the patch I sent for apt is exactly what is needed.

(In reply to Tomáš Golembiovský from comment #10)
> > > > - that there is a short-term plan for fixing rhev-apt itself
> > > 
> > > Gal?
> > 
> > Not on my side,
> > There is the patch that you sent to apt, together with Liran patch, does it
> > fixes the issue?
> 
> Yes, the patch I sent for apt is exactly what is needed.

OK so I will build a new version of apt with your patch then and add it to the installer

This bug will be addressed in next major release.

Friendly remember that this bug still lacks:
- proper description of the issue
- any link to the issues/patches/etc reported for rhv-apt

(In reply to Pino Toscano from comment #14)
> Friendly remember that this bug still lacks:
> - proper description of the issue

During VM conversion virt-v2v installs the rhev-apt into the Windows guest. Later when the guest is started in RHV and guest tools is (automatically) attached, the installation of tools and drivers does not start because the checksum file on the tools ISO is not recognized. The problem is not withe checksum file but with the faulty code in rhev-apt.

> - any link to the issues/patches/etc reported for rhv-apt

I have opened a bug on RHV here: 1789327

Hi Tomas,

    Virt-v2v will install storage, network, video drivers etc from virtio-win.iso and copy rhev-apt.exe to windows guest during conversion, we just check the certificate validity time of rhev-apt.exe and check if RHEV-apt service can be started normally after v2v conversion, we didn't install/update any drivers and tools from rhv tools ISO in windows guest as comment6 said, I try to reproduce the bug with below steps but didn't find problem, could you please provide some steps how to verify if the rhev-apt we used is broken?

Packages:
virt-v2v-1.40.2-5.el7_7.3.x86_64
libguestfs-1.40.2-5.el7_7.3.x86_64
RHEV-apt: 4.43.3

Steps:
1. Convert a windows guest from vmware to rhv by virt-v2v
2. Check guest after finishing conversion, found certificate of rhev-apt.exe is in validity time and RHEV-apt service can be started normally 
3. Copy RHV-toolsSetup_4.3_10.iso to guest and run RHEV-toolsSetup.exe by manual, the installation can be finished without error, pls check the screenshot"rhev-toolsSetup"

Created attachment 1651205 [details]
rhev-toolsSetup.png

(In reply to mxie from comment #16)

> 3. Copy RHV-toolsSetup_4.3_10.iso to guest and run RHEV-toolsSetup.exe by
> manual, the installation can be finished without error, pls check the
> screenshot"rhev-toolsSetup"

When rhev-apt is working properly it is not necessary to start the installer manually. The CD with tools should be detected automatically and installation/update triggered.

Please check Event Log for messages about success/errors from rhev-apt and WSH (Windows Script Host).

(In reply to Tomáš Golembiovský from comment #18)
> (In reply to mxie from comment #16)
> 
> > 3. Copy RHV-toolsSetup_4.3_10.iso to guest and run RHEV-toolsSetup.exe by
> > manual, the installation can be finished without error, pls check the
> > screenshot"rhev-toolsSetup"
> 
> When rhev-apt is working properly it is not necessary to start the installer
> manually. The CD with tools should be detected automatically and
> installation/update triggered.
> 
> Please check Event Log for messages about success/errors from rhev-apt and
> WSH (Windows Script Host).

Just heads up that rhev-apt has been removed in RHV 4.4 and deprecated in RHV 4.3.
If an alternative to its use is possible I would consider looking in that direction instead.

It looks like a new RHV Tools ISO build ships with a fixed rhev-apt.exe, and so far it has been tested "generally OK" by our QE.
Hence acking.

Created attachment 1698068 [details]
win2008-spice-agent-not-start.png

Created attachment 1698070 [details]
win2012r2-installed-virtio-driver.png

Created attachment 1698071 [details]
win2012r2-no-virtio-drivers.png

Created attachment 1698072 [details]
win2019-agent-status.png

First of all - great bit of testing, thanks mxie.

This updated package is an improvement on the current situation even if
it doesn't work absolutely everywhere, and therefore we should take it,
and defer the W2K12R2 issues for another time (I think there's already a
separate bug for that).

Verify the bug with below builds:
virt-v2v-1.40.2-10.el7.x86_64
libguestfs-1.40.2-10.el7.x86_64
libvirt-4.5.0-36.el7.x86_64
qemu-kvm-rhev-2.12.0-48.el7.x86_64
nbdkit-1.8.0-4.el7.x86_64
virtio-win-1.9.6-1.el7.noarch
kernel-3.10.0-1149.el7.x86_64
VMware-vix-disklib-6.7.3-14389676.x86_64.tar.gz
rhv-guest-tools-iso-4.3-12.el7ev

Steps:
1.Install latest rhv-guest-tools-iso package and copy rhv-tools-setup.iso to iso domain of rhv env
1.1 #yum install http://download.eng.bos.redhat.com/brewroot/packages/rhv-guest-tools-iso/4.3/12/noarch/rhv-guest-tools-iso-4.3-12.el7ev.noarch.rpm

1.2 # scp -r /usr/share/rhv-guest-tools-iso//RHV-toolsSetup_4.3_12.iso 10.66.xx.xx:/home/nfs_iso/d1e6f0e1-e8d0-4082-90fc-3d3d086b33f7/images/11111111-1111-1111-1111-111111111111


2.Convert different versions of windows guests to rhv4.3 by virt-v2v
# virt-v2v -ic vpx://root.73.141/data/10.73.196.89/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib/ -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -oo rhv-cafile=/home/ca.pem -oo rhv-verifypeer=true -of raw -os nfs_data -b ovirtmgmt  $windows_guest


3.Power on guest on rhv after v2v conversion, check if rhev-apt, guest agents are installed normally, if there is error info about rhev-apt in windows log, and if certificate is valid

Test Result：        
Guest	        Firstboot              RHEV-apt   Certificate valid    QEMU-ga   SPICE-ga     Show IP info rhv
win2019		Yes(reboot manually)     Yes           Yes               No        No            No 
win2016		Yes(reboot manually)     Yes           Yes               No        No            No
win10 x86	Yes                      Yes	       Yes               No        No            No
win10 x64	Yes(reboot manually)     Yes           Yes               No        No            No
win2012r2	Yes                      No	       Yes	         No	   No            No
win2012		Yes                      Yes	       Yes               No        No            No
win8.1 x86	Yes	                 No	       Yes	         No	   No            No
win8.1 x64	Yes	                 No	       Yes	         No	   No            No


Hi Pino,
   Please check above test result, found below problem:

   (1)Found firstboot service failed to start in some windows guests when the guests start for the first time, but firstboot service can start normally after rebooting guest manually, find some error info in windows log, pls refer to screenshots "firstboot-error"

   (2)Found windows guests can't install qemu-ga, spice-ga from RHV-toolsSetup_4.3_12.iso automatically after rhev-apt service running , find error log about qemu-ga in windows log, pls refer to screenshot"qemu-ga-error-RHV-toolsSetup_4.3_12.iso"
      
      I think the qemu-ga error is caused by RHV-toolsSetup_4.3_12.iso, if copy RHV-toolsSetup_4.3_11.iso to iso domain of rhv env and delete RHV-toolsSetup_4.3_12.iso, then convert a windows guest to rhv by v2v, found qemu-ga and spice-agent can start normally in guest after rhev-apt running, pls refer to screenshot"win2019-rhv-tools-iso-4.3-11.png"

      Compare RHV-toolsSetup_4.3_12.iso and RHV-toolsSetup_4.3_11.iso, found RHV-toolsSetup_4.3_11.iso doesn't have el8 folder in linux path but RHV-toolsSetup_4.3_12.iso has el8 folder and there is a qemu-ga rhel8 package
       # mount /usr/share/rhv-guest-tools-iso/RHV-toolsSetup_4.3_11.iso /media
       # ls /media/linux/
         debian  el6  el7  fc28  lp151

       # mount /usr/share/rhv-guest-tools-iso/RHV-toolsSetup_4.3_12.iso /media
       # ls /mnt/linux/el8/
         qemu-guest-agent-2.12.0-99.module+el8.2.0+5827+8c39933c.x86_64.rpm


  (3)Found rhev-apt can't be installed normally in win8.1 x64/x86 guest sometimes, pls refer to screenshot"win8.1-rhev-apt", I think the problem is realted to the virtio drivers of virtio-win because rhev-apt can install in win8.1 guests normally when copying virtio drivers from  RHV-toolsSetup_4.3_11.iso during v2v conversion (pls refer to the result of comment30)

Created attachment 1698691 [details]
firstboot-error.zip

Created attachment 1698692 [details]
qemu-ga-error-RHV-toolsSetup_4.3_12.iso.zip

Created attachment 1698693 [details]
win2019-rhv-tools-iso-4.3-11.png

Created attachment 1698694 [details]
win8.1-rhev-apt.zip

Hi Pino,

   I have filed bug1850963 to track the qemu-ga problem (question2 of comment44), may I verify the bug with rhv-guest-tools-iso-4.3-11?

(In reply to mxie from comment #44)
> 1.2 # scp -r /usr/share/rhv-guest-tools-iso//RHV-toolsSetup_4.3_12.iso
> 10.66.xx.xx:/home/nfs_iso/d1e6f0e1-e8d0-4082-90fc-3d3d086b33f7/images/
> 11111111-1111-1111-1111-111111111111

IIRC files under an ISO domain require a precise UID/GID, and potentially also SELinux labels.
I'd rather upload ISOs using the proper tools: the web UI or the CLI tool engine-iso-uploader:
https://www.ovirt.org/documentation/administration_guide/#Uploading_the_VirtIO_and_Guest_Tool_Image_Files_to_an_ISO_Storage_Domain

Are there any errors/messages in either engine.log or vdsm.log related to the ISO?

>    (1)Found firstboot service failed to start in some windows guests when
> the guests start for the first time, but firstboot service can start
> normally after rebooting guest manually, find some error info in windows
> log, pls refer to screenshots "firstboot-error"

A possible reason might be that rhev-apt cannot detect the ISO.

>    (2)Found windows guests can't install qemu-ga, spice-ga from
> RHV-toolsSetup_4.3_12.iso automatically after rhev-apt service running ,
> find error log about qemu-ga in windows log, pls refer to
> screenshot"qemu-ga-error-RHV-toolsSetup_4.3_12.iso"

Again, it lools like the ISO was not available.

TBH I don't understand why it is even trying to use the RHEL 8 qemu-guest-agent package on Windows...

>       I think the qemu-ga error is caused by RHV-toolsSetup_4.3_12.iso, if
> copy RHV-toolsSetup_4.3_11.iso to iso domain of rhv env and delete
> RHV-toolsSetup_4.3_12.iso, then convert a windows guest to rhv by v2v, found
> qemu-ga and spice-agent can start normally in guest after rhev-apt running,
> pls refer to screenshot"win2019-rhv-tools-iso-4.3-11.png"


I checked the differences between 4.3-11 and 4.3-12, and I don't see anything relevant to this issue.
All the files that are related to a conversion (rhev-apt, drivers, etc) are bit-by-bit identical.
The only relevant change in 4.3-12 is the addition of the RHEL 8 qemu-guest-agent package, which I definitely do not expect to matter for Windows conversions :)
(And even more so, virt-v2v in RHEL 7 cannot install it anyway.)

(In reply to mxie from comment #49)
>    I have filed bug1850963 to track the qemu-ga problem (question2 of
> comment44), may I verify the bug with rhv-guest-tools-iso-4.3-11?

IMHO it is not the issue, see my comments above.

> https://www.ovirt.org/documentation/administration_guide/
> #Uploading_the_VirtIO_and_Guest_Tool_Image_Files_to_an_ISO_Storage_Domain
> 
> Are there any errors/messages in either engine.log or vdsm.log related to
> the ISO?

Didn't find any error info about ISO in engine.log or vdsm.log, and RHV-toolsSetup_4.3_12.iso can be mounted to D:\ automatically in windows guest during my testing, pls refer to ""


# cat engine.log |grep rhv-tools
2020-06-25 13:25:09,732+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-1228) [e54bb402-ce40-49f3-8bd0-48b22247ec0a] FINISH, GetFileStatsVDSCommand, return: {RHEL-8.2.0-20200404.0-BaseOS-x86_64-boot.iso={status=0, ctime=1592906231.0, size=654311424}, rhv-tools-setup.iso={status=0, ctime=1593012847.0, size=411856896}, RHV-toolsSetup_4.3_12.iso={status=0, ctime=1593061144.0, size=412086272}, RHV-toolsSetup_4.3_11.iso={status=0, ctime=1593012258.0, size=411856896}}, log id: 5593696a
2020-06-25 13:43:36,790+08 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (default task-1228) [5dbfffb0-9698-4122-88ac-edbc9345aa99] FINISH, GetFileStatsVDSCommand, return: {RHEL-8.2.0-20200404.0-BaseOS-x86_64-boot.iso={status=0, ctime=1592906231.0, size=654311424}, rhv-tools-setup.iso={status=0, ctime=1593012847.0, size=411856896}, RHV-toolsSetup_4.3_12.iso={status=0, ctime=1593062713.0, size=412086272}, RHV-toolsSetup_4.3_11.iso={status=0, ctime=1593012258.0, size=411856896}}, log id: 7ddba6aa

# cat vdsm.log |grep rhv-tools
nothing


Besides, I used the same way to upload the RHV-toolsSetup_4.3_11.iso to iso domain of rhv when test the rhev-apt of RHV-toolsSetup_4.3_11.iso(pls see comment30), all windows guests can install guest agents from rhv-guest-tools iso automatically as long as rhev-apt is running, so I don't think qemu-ga problem has relationship with the way of uploading rhv-guest-tools-iso


> >    (1)Found firstboot service failed to start in some windows guests when
> > the guests start for the first time, but firstboot service can start
> > normally after rebooting guest manually, find some error info in windows
> > log, pls refer to screenshots "firstboot-error"
> 
> A possible reason might be that rhev-apt cannot detect the ISO.


I think the firstboot problem has no relationship with rhv-guest-tools.iso because rhev-apt is still not installed at this time


> TBH I don't understand why it is even trying to use the RHEL 8
> qemu-guest-agent package on Windows...

rhv-guest-tools-iso added this package because of bug1844971


> (In reply to mxie from comment #49)
> >    I have filed bug1850963 to track the qemu-ga problem (question2 of
> > comment44), may I verify the bug with rhv-guest-tools-iso-4.3-11?
> 
> IMHO it is not the issue, see my comments above.

But current problem is guest agents can't be installed if test bug with rhv-guest-tools-iso-4.3-12. Good news is I didn't find any error info about checksum of rhev-apt in windows log, do you think it's enough to verify the bug?

Created attachment 1698772 [details]
RHV-toolsSetup_4.3_12.iso-mount-automatically.png

Hi Pino,

   I have uploaded iso file to rhv env with the supported method(pls check below step1), but windows guest still can't install guest agent from rhv-rhv-guest-tools-4.3-12 iso automatically because of bug1850963. So I will not check if windows guest can install guest-agent from rhv-gust-tool iso automatically, will just verify if rhev-apt service can start normally in guest, if certificate of rhev-apt is valid, and if there is checksum error of rhev-apt in windows log for the bug, do you agree?


Steps:
1.Log into ovirt-engine server to upload Guest Tool Image Files to ISO Storage Domain
1.1 # yum install http://download.eng.bos.redhat.com/brewroot/packages/rhv-guest-tools-iso/4.3/12/noarch/rhv-guest-tools-iso-4.3-12.el7ev.noarch.rpm -y

1.2 # engine-iso-uploader --iso-domain=nfs_iso upload /usr/share/rhv-guest-tools-iso/RHV-toolsSetup_4.3_12.iso
Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort): 
Uploading, please wait...
INFO: Start uploading /usr/share/rhv-guest-tools-iso/RHV-toolsSetup_4.3_12.iso 
Uploading: [########################################] 100%
INFO: /usr/share/rhv-guest-tools-iso/RHV-toolsSetup_4.3_12.iso uploaded successfully

2.Convert a windows guest to rhv4.3 by virt-v2v on rhel7.9 v2v conversion server
# virt-v2v -ic vpx://root.73.141/data/10.73.196.89/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib/ -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -oo rhv-cafile=/home/ca.pem -oo rhv-verifypeer=true -of raw -os nfs_data -b ovirtmgmt 

3.Power on guest after v2v conversion on rhv, found windows guest can't install guest-agent from rhv-gust-tool iso automatically after rhev-apt running in guest, then found same qemu-ga error as comment44 in windows log

This is likely a problem with the new ISO. Possibly the '+' in filename is confusing the hash-checking part of rhev-apt. Please use ISO 4.3-11 to verify this bug. It does not matter which ISO you use what is important is the version of rhev-apt bundled in virt-v2v RPM. Not the one on the ISO.

Please try with RHV-toolsSetup_4.3_13.iso

Verify the bug with below builds again:
virt-v2v-1.40.2-10.el7.x86_64
libguestfs-1.40.2-10.el7.x86_64
libvirt-4.5.0-36.el7.x86_64
qemu-kvm-rhev-2.12.0-48.el7.x86_64
nbdkit-1.8.0-4.el7.x86_64
virtio-win-1.9.6-1.el7.noarch
rhv-guest-tools-iso-4.3-13.el7ev

Steps:
1.Log into engine server to upload Guest Tool Image Files to ISO Storage Domain
1.1 # http://download.eng.bos.redhat.com/brewroot/packages/rhv-guest-tools-iso/4.3/13/noarch/rhv-guest-tools-iso-4.3-13.el7ev.noarch.rpm -y

1.2 # engine-iso-uploader --iso-domain=iscsi_iso upload /usr/share/rhv-guest-tools-iso/RHV-toolsSetup_4.3_13.iso
Please provide the REST API password for the admin@internal oVirt Engine user (CTRL+D to abort): 
Uploading, please wait...
INFO: Start uploading /usr/share/rhv-guest-tools-iso/RHV-toolsSetup_4.3_13.iso 
Uploading: [########################################] 100%
INFO: /usr/share/rhv-guest-tools-iso/RHV-toolsSetup_4.3_13.iso uploaded successfully

2.Convert different versions of windows guests to rhv4.3 by virt-v2v
#  virt-v2v -ic vpx://root.73.141/data/10.73.196.89/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib/ -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA  -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -oo rhv-verifypeer=true -of raw -oo rhv-cluster=ISCSI  -os iscsi_data -b ovirtmgmt -oo rhv-cafile=/home/ca.pem -oa preallocated $windows_guest


3.Power on guest on rhv after v2v conversion, check if rhev-apt is installed normally,if certificate of rhev-apt is valid, if guest agents can be installed from RHV-toolsSetup_4.3_13.iso automatically, if there is error info about rhev-apt and guest agents in windows log

Test Result： 
       
Guest	        Firstboot              RHEV-apt   Certificate valid    QEMU-ga   SPICE-ga     Error in windows log
win2019		pass        		 pass          pass        	pass      pass 		 None	
win2016		pass        		 pass          pass        	pass      pass 		 rhev-apt error
win10 x86	pass        		 pass          pass        	pass      pass 		 None	
win10 x64	pass        		 pass          pass        	pass      pass 		 None	
win2012r2	pass                     fail	       pass	        fail	  fail           None
win2012		pass        		 pass          pass        	pass      pass 		 None	
win8.1 x86	pass        		 pass          pass        	pass      pass 		 rhev-apt error
win8.1 x64	pass        		 pass          pass        	pass      pass 		 rhev-apt error
win8 x86	pass        		 pass          pass        	pass      pass 		 rhev-apt error
win8 x64	pass                     pass          pass        	pass      pass 		 rhev-apt error
win2008r2	pass        		 pass          pass        	pass      pass 		 None	
win2008 x86	pass        		 pass	       pass        	pass	  fail 		 None
win2008 x64	pass        		 pass	       pass        	pass	  pass 		 None


Below is the conclusion for the test result:

 （1）For win2012r2 and win8-x64，rhev-apt can't be installed and this problem is already tracked by bug1584678

  (2) Although guest agents can be installed from rhev-guest-tools-iso normally, found a rhev-apt error in windows log of win2016, win8.1-x86/x64 and win8-x86/x64, pls refer to related screenshots
  
  (3) Spice agent can't start in win2008 x86 guest but spice agent runs normally in win2008 x64, according to the result of win2008 of comment30 and current result of win2008 x86, I think spice agent can't start in win2008 guest sometimes, pls refer to related screenshots


Hi Tomas,
   
    Please help to check above problem (1) and (2), thanks!

Created attachment 1699802 [details]
screenshots-for-comment56

>  （1）For win2012r2 and win8-x64，rhev-apt can't be installed and this problem
> is already tracked by bug1584678

Yes, this looks like the old issue we keep seeing on Win 2012R2.

> 
>   (2) Although guest agents can be installed from rhev-guest-tools-iso
> normally, found a rhev-apt error in windows log of win2016, win8.1-x86/x64
> and win8-x86/x64, pls refer to related screenshots

This the first time I see this. I wonder if the error may have always been there or it exhibits only in certain circumstances. Based on the documentation the error 1641 is not really an error -- it means that the installation finished successfully and OS is going to reboot. From screenshots it also seems all the agents were installed which confirms it. This is only a cosmetic issue and should not block validation of this bug.

Hi Tomas,
   
    Thanks for your confirmation, could you please also check the problem(3),thanks

(In reply to mxie from comment #59)
> Hi Tomas,
>    
>     Thanks for your confirmation, could you please also check the
> problem(3),thanks

It looks like the virtio drivers were not properly installed or don't work. This may be problem with the RHV tools installer or virtio drivers. But it also looks like unrelated issue.

Created attachment 1700272 [details]
win 2008 x86 -- problem with virtio drivers

Thanks Tomas for confirmation, file bug1855176 to track spice agent problem of win2008 guest, so move the bug from ON_QA to VERIFIED status.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (libguestfs bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4044