Bug 1584678

Created attachment 1446265 [details]
virt-v2v-rhev-apt.log

Could you see if there is a file called (I think)

c:\program files\guestfs/firstboot/log.txt

It should contain the log from the firstboot script.  It seems as
if the firstboot script ran (because it was moved to scripts-done)
so the log should exist, and there may be an error message in it.

Created attachment 1446496 [details]
firstboot-log

So the error seems to be:

  starting rhev-apt
  The system cannot execute the specified program.
  .... exit code 1

The firstboot script we wrote contains:

  @echo off

  echo installing rhev-apt
  "\rhev-apt.exe" /S /v /qn
  echo starting rhev-apt
  net start rhev-apt

It looks as if the \rhev-apt.exe command didn't give any visible
error, but the net start rhev-apt command gave the error
"The system cannot execute the specified program."  Why that
happened is one of those Windows mysteries.

This happens on Windows Server 2012 R2 & Windows Server 2008 R2,
so it's probably not specific to any particular version of Windows.

Created attachment 1446562 [details]
win2008r2-rhev-apt

Looking at comment 8, it seems as if rhev-apt started successfully
(on Windows 2008 R2).  If there's still a bug in Windows 2008 R2
then it is something different.

Hi rjones,

   There is rhev-apt serivce running in win2008r2 guest after v2v converting to rhv no matter source is vmware or xen, but it just needs a long time to finish the installation that's why I didn't find rhev-apt service in win2008r2 yesterday,anyway,sorry for providing wrong information to you

   I summarize status of rhev-apt installation in windows guest as below, so just win2012r2 has problem to install rhev-apt
  
   Guest           rhev-apt serivce running
   Win7                 Yes
   Win2008              Yes
   Win2008r2            Yes
   Win2012              Yes
   Win2012r2            No
   Win8                 Yes
   Win8.1               Yes
   Win10                Yes
   Win2016              Yes

What happens is very odd.

The two commands we run as:

  rhev-apt /s /v /qn
  net start rhev-apt

The first command on Windows Server 2012 R2 opens an interactive
"InstallShield"-style window.  However we asked for it to run
non-interactively, so this is a bug.

The second command either fails or succeeds.  If the InstallShield
thing hasn't finished running then it fails with the error described
previously.  if the InstallShield thing has finished then the second
command succeeds.

Hi rjones, 

   It's strange I can't reproduce the bug with below builds now

virt-v2v-1.38.2-11.el7.x86_64
libguestfs-1.38.2-11.el7.x86_64
libvirt-4.5.0-7.el7.x86_64
qemu-kvm-rhev-2.12.0-12.el7.x86_64
virtio-win-1.9.6-1.el7.noarch

There shouldn't be any difference.  The embedded copy of RHEV-APT
was last updated on 15 May 2018 which was before this bug was
reported, and both versions of virt-v2v have the same embedded
version.  However we suspect the bug might not be 100% reproducible.

(In reply to Richard W.M. Jones from comment #11)
> What happens is very odd.
> 
> The two commands we run as:
> 
>   rhev-apt /s /v /qn
>   net start rhev-apt
> 
> The first command on Windows Server 2012 R2 opens an interactive
> "InstallShield"-style window.  However we asked for it to run
> non-interactively, so this is a bug.
> 
> The second command either fails or succeeds.  If the InstallShield
> thing hasn't finished running then it fails with the error described
> previously.  if the InstallShield thing has finished then the second
> command succeeds.

There are actually 2 issues here.

First issue - you run it incorrectly.
It supposed to be: /S /v/qn - note the big S in /S and that /v/qn options have no space between them.

Second issue - the service start in silent RHEV-Apt installation seems to be broken (and looking at the code may never work), not sure how/why QA never noticed it, probably because it's only relevant in such edge cases, as after the reboot the service will be started normally, so it will be irrelevant.

If/once the second issue is fixed you won't have to run manually the "net start" command, unless if that what you actually want (in which case please note this in the BZ).

If you need the service start in silent mode to be fixed, please open BZ.

Adding Sandro.

(In reply to Richard W.M. Jones from comment #14)
> There shouldn't be any difference.  The embedded copy of RHEV-APT
> was last updated on 15 May 2018 which was before this bug was
> reported, and both versions of virt-v2v have the same embedded
> version.  However we suspect the bug might not be 100% reproducible.

Just one more note/clarification - the logic/flow is supposed to be same under all OS versions (and since you're not actually running it in silent mode due to the wrong options, the service should actually be started on all OS versions), so not sure why you only have issues under Windows 2K12R2, that needs to be further investigated.

The fact that you get issue running "net start" before the setup finishes is OK - you can't really work with the service before it's installed.

Actually comment 11 is slightly wrong.  The actual command we are
running now is:

  \rhev-apt.exe /S /v /qn

I take it that the command we should be running is:

  \rhev-apt.exe /S /v/qn                                                    

(ie. removing a single space).  Is this correct?

> If you need the service start in silent mode to be fixed, please open BZ.

If I understand correct, there is a proposed change to make
the service start up without needing an explicit "net start rhev-apt"
command.  Is it safe to leave the net start command there, so that we
can handle rhev-apt before and after the future change?

And yes we need this to be fixed - can we use the current BZ for
that?

(In reply to Richard W.M. Jones from comment #19)
> Actually comment 11 is slightly wrong.  The actual command we are
> running now is:
> 
>   \rhev-apt.exe /S /v /qn
> 
> I take it that the command we should be running is:
> 
>   \rhev-apt.exe /S /v/qn                                                    
> 
> (ie. removing a single space).  Is this correct?
> 

Yes, /S /v/qn is the correct way to run it.

> > If you need the service start in silent mode to be fixed, please open BZ.
> 
> If I understand correct, there is a proposed change to make
> the service start up without needing an explicit "net start rhev-apt"
> command.  Is it safe to leave the net start command there, so that we
> can handle rhev-apt before and after the future change?
> 
> And yes we need this to be fixed - can we use the current BZ for
> that?

Yes, I think that this is what needs to be done, hopefully it won't be too complicated.

Still, I'm a bit curious why it hasn't worked for you only on 2K12R2, since it's supposed to work the same on all OS versions, and since you haven't actually ran it in silent mode it was supposed to start the service.

It's better not to try starting this particular service before the installer completes, in order to avoid race conditions, but otherwise it should be safe to run the "net start" command on the running service.

We've now changed the command to do \rhev-apt.exe /S /v/qn
and retested all versions of Windows, and it still only fails
on Win2012R2.  See:
https://bugzilla.redhat.com/show_bug.cgi?id=1625216#c3

Created attachment 1519423 [details]
win2012 install on iscsi storage

We have experienced similar issue recently. Again it was Windows Server 2012 R2. The installation in firstboot was performed with the fixed arguments to the installer (as suggested in comment 20). Note that rhev-apt was *not* installed at all. I am attaching the firstboot log for reference. Interesting thing is that no error is reported and it didn't even come to starting the service using "net start rhev-apt" (seems like batch script died in the middle?). 

I have posted a patch to virt-v2v that would allow us to collect log from MSI installer[1]. Hopefully this can help us in the future should we hit it again. But that is just the MSI Installer and not the rhev-apt.exe package.

Lev, is there some hidden option to log what rhev-apt.exe is doing before it fires the MSI Installer?

https://www.redhat.com/archives/libguestfs/2019-February/msg00029.html

Created attachment 1534813 [details]
failed installation on Windows 2012 R2

I think the issue here is the incorrect command line parameters.

Specifically due to the change that you mentioned:
\"\\rhev-apt.exe\" /S /v/qn /v/l*vx \"/v\\\"%cd%\\rhev-apt.log\\\"\"


The actual command (before you do the escaping like you did) should look like similar to this:
\rhev-apt.exe /S /v/qn /V"/log C:\logs\rhev-apt.log"

Not sure why you require this escaping but, in your case it should be probably something like this:
\"\\rhev-apt.exe\" /S /v/qn /V\"//log \'%cd%\\rhev-apt.log\'\"

Please note the nested quotes.

(In reply to Tomáš Golembiovský from comment #26)
> We have experienced similar issue recently. Again it was Windows Server 2012
> R2. The installation in firstboot was performed with the fixed arguments to
> the installer (as suggested in comment 20). Note that rhev-apt was *not*
> installed at all. I am attaching the firstboot log for reference.
> Interesting thing is that no error is reported and it didn't even come to
> starting the service using "net start rhev-apt" (seems like batch script
> died in the middle?). 
> 
> I have posted a patch to virt-v2v that would allow us to collect log from
> MSI installer[1]. Hopefully this can help us in the future should we hit it
> again. But that is just the MSI Installer and not the rhev-apt.exe package.
> 
> Lev, is there some hidden option to log what rhev-apt.exe is doing before it
> fires the MSI Installer?

I'm not aware of the option to enable the logging of the loader itself.

(In reply to Lev Veyde from comment #29)
> I think the issue here is the incorrect command line parameters.
> 
> Specifically due to the change that you mentioned:
> \"\\rhev-apt.exe\" /S /v/qn /v/l*vx \"/v\\\"%cd%\\rhev-apt.log\\\"\"

We experienced the issue I described before this change. The command line was:
"\rhev-apt.exe" /S /v/qn


> 
> The actual command (before you do the escaping like you did) should look
> like similar to this:
> \rhev-apt.exe /S /v/qn /V"/log C:\logs\rhev-apt.log"

Despite what the help for rhev-apt and msiexec suggest the correct command line really is:

\rhev-apt.exe /S /v/qn /v/log /vC:\some\path.log ... meaning /v must be prefixed to every argument for msiexec.

> 
> Not sure why you require this escaping but, in your case it should be
> probably something like this:
> \"\\rhev-apt.exe\" /S /v/qn /V\"//log \'%cd%\\rhev-apt.log\'\"
> 
> Please note the nested quotes.

The path has to be quoted twice otherwise spaces in path cause the arguments to be broken into more than one when passed to msiexec. Don't ask me why this happens. But you are right, using ' instead of second " might also work. It just did not occur to me at the time of writing.


(In reply to Lev Veyde from comment #30)
> (In reply to Tomáš Golembiovský from comment #26)
> > Lev, is there some hidden option to log what rhev-apt.exe is doing before it
> > fires the MSI Installer?
> 
> I'm not aware of the option to enable the logging of the loader itself.

Oh well, then let's hope that whatever is causing the installation to fail happens in msiexec. Thanks for the info.

(In reply to Tomáš Golembiovský from comment #31)
> (In reply to Lev Veyde from comment #29)
> > 
> > The actual command (before you do the escaping like you did) should look
> > like similar to this:
> > \rhev-apt.exe /S /v/qn /V"/log C:\logs\rhev-apt.log"
> 
> Despite what the help for rhev-apt and msiexec suggest the correct command
> line really is:
> 
> \rhev-apt.exe /S /v/qn /v/log /vC:\some\path.log ... meaning /v must be
> prefixed to every argument for msiexec.
> 

One more note that may be useful to whomever who comes across this in the distant future:

- case is ignored for rhev-apt.exe arguments, i.e. /s is same as /S just like /v is same as /V

(despite what the help suggest)

Moving to 4.3.2 not being identified as blocker for 4.3.1.

Closing this as wontfix since the bug is targeted to 4.4 and we are deprecating the APT service on 4.4.

So what are we supposed to do to make guest drivers update?

On 4.4 we are moving the driver installer to virtio-win,
The virtio-win team is working to get the drivers updated through windows update service.
There is a discussion about marking and notifying Windows VMs that have an old version of the drivers installed.

After debugging, found rhev-apt can install successfully if don't install virtio driver for win2012r2 guest during v2v conversion

Packages:
virt-v2v-1.40.2-23.module+el8.2.1+6797+0db00a40.x86_64
libguestfs-1.40.2-23.module+el8.2.1+6797+0db00a40.x86_64
libvirt-6.0.0-22.module+el8.2.1+6815+1c792dc8.x86_64
qemu-kvm-4.2.0-23.module+el8.2.1+6917+927fbb44.x86_64
nbdkit-1.16.2-4.module+el8.2.1+6710+effcb1df.x86_64


Steps:
1. Set value of VIRTIO_WIN is empty.
# export VIRTIO_WIN=

2.Convert a win2012r2 guest from VMware to rhv by virt-v2v
#virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78  esx7.0-win2012r2-x86_64 -ip /home/passwd  -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt -v -x |& tee > esx7.0-win2012r2-no-virtio-drivers.log

3.Found rhev-apt can install successfully, pls refer to screenshot 'win2012r2-rhev-apt-no-virtio-drivers.png' and log'esx7.0-win2012r2-no-virtio-drivers'

Created attachment 1695767 [details]
win2012r2-rhev-apt-no-virtio-drivers.png

Created attachment 1695768 [details]
esx7.0-win2012r2-no-virtio-drivers.log