Bug 1584678 - On W2K12r2 rhev-apt does not run non-interactively, causing race when starting rhev-apt service from the command line
Summary: On W2K12r2 rhev-apt does not run non-interactively, causing race when startin...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: rhv-guest-tools-iso
Version: unspecified
Hardware: x86_64
OS: Unspecified
medium
high
Target Milestone: ovirt-4.4.0
: ---
Assignee: Gal Zaidman
QA Contact: Virtualization Bugs
URL:
Whiteboard: V2V
Depends On:
Blocks: TRACKER-bugs-affecting-libguestfs
TreeView+ depends on / blocked
 
Reported: 2018-05-31 12:38 UTC by mxie@redhat.com
Modified: 2020-06-07 06:31 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1624902 (view as bug list)
Environment:
Last Closed: 2019-08-26 11:40:43 UTC
oVirt Team: Integration
Target Upstream Version:


Attachments (Terms of Use)
rhev-apt (66.09 KB, image/png)
2018-05-31 12:38 UTC, mxie@redhat.com
no flags Details
virt-v2v-rhev-apt.log (1.22 MB, text/plain)
2018-05-31 12:40 UTC, mxie@redhat.com
no flags Details
firstboot-log (63.65 KB, image/png)
2018-06-01 02:45 UTC, mxie@redhat.com
no flags Details
win2008r2-rhev-apt (60.81 KB, image/png)
2018-06-01 08:11 UTC, mxie@redhat.com
no flags Details
win2012 install on iscsi storage (301.66 KB, image/png)
2019-01-09 10:08 UTC, liuzi
no flags Details
failed installation on Windows 2012 R2 (40.81 KB, image/png)
2019-02-14 13:54 UTC, Tomáš Golembiovský
no flags Details
win2012r2-rhev-apt-no-virtio-drivers.png (104.19 KB, image/png)
2020-06-07 06:29 UTC, mxie@redhat.com
no flags Details
esx7.0-win2012r2-no-virtio-drivers.log (2.86 MB, text/plain)
2020-06-07 06:31 UTC, mxie@redhat.com
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1620313 'unspecified' 'CLOSED' 'firstboot commands do not run properly on Windows 2012 guest' 2019-11-26 18:00:24 UTC

Internal Links: 1620313

Description mxie@redhat.com 2018-05-31 12:38:49 UTC
Created attachment 1446264 [details]
rhev-apt

Description of problem:
There is no rhev-apt serivce running by executing command "net start" in windows guest after v2v converting to rhv

Version-Release number of selected component (if applicable):
virt-v2v-1.38.2-3.el7.x86_64
libguestfs-1.38.2-3.el7.x86_64
libvirt-4.3.0-1.el7.x86_64
qemu-kvm-rhev-2.12.0-2.el7.x86_64
virtio-win-1.9.4-2.el7.noarch
RHV:4.2.4-0.1.el7

How reproducible:
100%

Steps to Reproduce:
1.Convert a windows guest to rhv by virt-v2v

2.Import the guest from export domain to data domain after v2v finishing conversion

3 Check rhev-apt.exe status in guest,details pls refer to screenshot"rhev-apt"

3.1 There is rhev-apt.exe under C:\ drive
3.2 There is no rhev-apt service running by executing command "net start" 
3.3 There is no 0001-configure-rhev-apt.bat in c:/Program Files/Guestfs/Firstboot/scripts/
3.4 There is 0001-configure-rhev-apt.bat in c:/Program Files/Guestfs/Firstboot/scripts-done

Actual results:
As above description

Expected results:
There is rhev-apt serivce running by executing command "net start" in windows guest after v2v converting to rhv

Additional info:
1.There is rhev-apt serivce running by executing command "net start" in windows guest if importing the guest from vmware on rhv4.2

Comment 2 mxie@redhat.com 2018-05-31 12:40:37 UTC
Created attachment 1446265 [details]
virt-v2v-rhev-apt.log

Comment 3 Richard W.M. Jones 2018-05-31 13:35:26 UTC
Could you see if there is a file called (I think)

c:\program files\guestfs/firstboot/log.txt

It should contain the log from the firstboot script.  It seems as
if the firstboot script ran (because it was moved to scripts-done)
so the log should exist, and there may be an error message in it.

Comment 4 mxie@redhat.com 2018-06-01 02:45:44 UTC
Created attachment 1446496 [details]
firstboot-log

Comment 5 Richard W.M. Jones 2018-06-01 07:55:22 UTC
So the error seems to be:

  starting rhev-apt
  The system cannot execute the specified program.
  .... exit code 1

Comment 6 Richard W.M. Jones 2018-06-01 08:00:47 UTC
The firstboot script we wrote contains:

  @echo off

  echo installing rhev-apt
  "\rhev-apt.exe" /S /v /qn
  echo starting rhev-apt
  net start rhev-apt

It looks as if the \rhev-apt.exe command didn't give any visible
error, but the net start rhev-apt command gave the error
"The system cannot execute the specified program."  Why that
happened is one of those Windows mysteries.

Comment 7 Richard W.M. Jones 2018-06-01 08:07:50 UTC
This happens on Windows Server 2012 R2 & Windows Server 2008 R2,
so it's probably not specific to any particular version of Windows.

Comment 8 mxie@redhat.com 2018-06-01 08:11:29 UTC
Created attachment 1446562 [details]
win2008r2-rhev-apt

Comment 9 Richard W.M. Jones 2018-06-01 08:22:20 UTC
Looking at comment 8, it seems as if rhev-apt started successfully
(on Windows 2008 R2).  If there's still a bug in Windows 2008 R2
then it is something different.

Comment 10 mxie@redhat.com 2018-06-01 16:14:57 UTC
Hi rjones,

   There is rhev-apt serivce running in win2008r2 guest after v2v converting to rhv no matter source is vmware or xen, but it just needs a long time to finish the installation that's why I didn't find rhev-apt service in win2008r2 yesterday,anyway,sorry for providing wrong information to you

   I summarize status of rhev-apt installation in windows guest as below, so just win2012r2 has problem to install rhev-apt
  
   Guest           rhev-apt serivce running
   Win7                 Yes
   Win2008              Yes
   Win2008r2            Yes
   Win2012              Yes
   Win2012r2            No
   Win8                 Yes
   Win8.1               Yes
   Win10                Yes
   Win2016              Yes

Comment 11 Richard W.M. Jones 2018-06-04 13:45:39 UTC
What happens is very odd.

The two commands we run as:

  rhev-apt /s /v /qn
  net start rhev-apt

The first command on Windows Server 2012 R2 opens an interactive
"InstallShield"-style window.  However we asked for it to run
non-interactively, so this is a bug.

The second command either fails or succeeds.  If the InstallShield
thing hasn't finished running then it fails with the error described
previously.  if the InstallShield thing has finished then the second
command succeeds.

Comment 13 mxie@redhat.com 2018-08-31 06:34:40 UTC
Hi rjones, 

   It's strange I can't reproduce the bug with below builds now

virt-v2v-1.38.2-11.el7.x86_64
libguestfs-1.38.2-11.el7.x86_64
libvirt-4.5.0-7.el7.x86_64
qemu-kvm-rhev-2.12.0-12.el7.x86_64
virtio-win-1.9.6-1.el7.noarch

Comment 14 Richard W.M. Jones 2018-08-31 08:53:52 UTC
There shouldn't be any difference.  The embedded copy of RHEV-APT
was last updated on 15 May 2018 which was before this bug was
reported, and both versions of virt-v2v have the same embedded
version.  However we suspect the bug might not be 100% reproducible.

Comment 15 Lev Veyde 2018-09-03 14:01:01 UTC
(In reply to Richard W.M. Jones from comment #11)
> What happens is very odd.
> 
> The two commands we run as:
> 
>   rhev-apt /s /v /qn
>   net start rhev-apt
> 
> The first command on Windows Server 2012 R2 opens an interactive
> "InstallShield"-style window.  However we asked for it to run
> non-interactively, so this is a bug.
> 
> The second command either fails or succeeds.  If the InstallShield
> thing hasn't finished running then it fails with the error described
> previously.  if the InstallShield thing has finished then the second
> command succeeds.

There are actually 2 issues here.

First issue - you run it incorrectly.
It supposed to be: /S /v/qn - note the big S in /S and that /v/qn options have no space between them.

Second issue - the service start in silent RHEV-Apt installation seems to be broken (and looking at the code may never work), not sure how/why QA never noticed it, probably because it's only relevant in such edge cases, as after the reboot the service will be started normally, so it will be irrelevant.

If/once the second issue is fixed you won't have to run manually the "net start" command, unless if that what you actually want (in which case please note this in the BZ).

If you need the service start in silent mode to be fixed, please open BZ.

Adding Sandro.

Comment 16 Lev Veyde 2018-09-03 14:16:48 UTC
(In reply to Richard W.M. Jones from comment #14)
> There shouldn't be any difference.  The embedded copy of RHEV-APT
> was last updated on 15 May 2018 which was before this bug was
> reported, and both versions of virt-v2v have the same embedded
> version.  However we suspect the bug might not be 100% reproducible.

Just one more note/clarification - the logic/flow is supposed to be same under all OS versions (and since you're not actually running it in silent mode due to the wrong options, the service should actually be started on all OS versions), so not sure why you only have issues under Windows 2K12R2, that needs to be further investigated.

The fact that you get issue running "net start" before the setup finishes is OK - you can't really work with the service before it's installed.

Comment 19 Richard W.M. Jones 2018-09-03 14:22:51 UTC
Actually comment 11 is slightly wrong.  The actual command we are
running now is:

  \rhev-apt.exe /S /v /qn

I take it that the command we should be running is:

  \rhev-apt.exe /S /v/qn                                                    

(ie. removing a single space).  Is this correct?

> If you need the service start in silent mode to be fixed, please open BZ.

If I understand correct, there is a proposed change to make
the service start up without needing an explicit "net start rhev-apt"
command.  Is it safe to leave the net start command there, so that we
can handle rhev-apt before and after the future change?

And yes we need this to be fixed - can we use the current BZ for
that?

Comment 20 Lev Veyde 2018-09-03 14:45:31 UTC
(In reply to Richard W.M. Jones from comment #19)
> Actually comment 11 is slightly wrong.  The actual command we are
> running now is:
> 
>   \rhev-apt.exe /S /v /qn
> 
> I take it that the command we should be running is:
> 
>   \rhev-apt.exe /S /v/qn                                                    
> 
> (ie. removing a single space).  Is this correct?
> 

Yes, /S /v/qn is the correct way to run it.

> > If you need the service start in silent mode to be fixed, please open BZ.
> 
> If I understand correct, there is a proposed change to make
> the service start up without needing an explicit "net start rhev-apt"
> command.  Is it safe to leave the net start command there, so that we
> can handle rhev-apt before and after the future change?
> 
> And yes we need this to be fixed - can we use the current BZ for
> that?

Yes, I think that this is what needs to be done, hopefully it won't be too complicated.

Still, I'm a bit curious why it hasn't worked for you only on 2K12R2, since it's supposed to work the same on all OS versions, and since you haven't actually ran it in silent mode it was supposed to start the service.

It's better not to try starting this particular service before the installer completes, in order to avoid race conditions, but otherwise it should be safe to run the "net start" command on the running service.

Comment 21 Richard W.M. Jones 2018-10-17 09:37:25 UTC
We've now changed the command to do \rhev-apt.exe /S /v/qn
and retested all versions of Windows, and it still only fails
on Win2012R2.  See:
https://bugzilla.redhat.com/show_bug.cgi?id=1625216#c3

Comment 24 liuzi 2019-01-09 10:08:23 UTC
Created attachment 1519423 [details]
win2012 install on iscsi storage

Comment 26 Tomáš Golembiovský 2019-02-14 13:53:25 UTC
We have experienced similar issue recently. Again it was Windows Server 2012 R2. The installation in firstboot was performed with the fixed arguments to the installer (as suggested in comment 20). Note that rhev-apt was *not* installed at all. I am attaching the firstboot log for reference. Interesting thing is that no error is reported and it didn't even come to starting the service using "net start rhev-apt" (seems like batch script died in the middle?). 

I have posted a patch to virt-v2v that would allow us to collect log from MSI installer[1]. Hopefully this can help us in the future should we hit it again. But that is just the MSI Installer and not the rhev-apt.exe package.

Lev, is there some hidden option to log what rhev-apt.exe is doing before it fires the MSI Installer?

Comment 28 Tomáš Golembiovský 2019-02-14 13:54:47 UTC
Created attachment 1534813 [details]
failed installation on Windows 2012 R2

Comment 29 Lev Veyde 2019-02-14 16:17:02 UTC
I think the issue here is the incorrect command line parameters.

Specifically due to the change that you mentioned:
\"\\rhev-apt.exe\" /S /v/qn /v/l*vx \"/v\\\"%cd%\\rhev-apt.log\\\"\"


The actual command (before you do the escaping like you did) should look like similar to this:
\rhev-apt.exe /S /v/qn /V"/log C:\logs\rhev-apt.log"

Not sure why you require this escaping but, in your case it should be probably something like this:
\"\\rhev-apt.exe\" /S /v/qn /V\"//log \'%cd%\\rhev-apt.log\'\"

Please note the nested quotes.

Comment 30 Lev Veyde 2019-02-14 16:21:33 UTC
(In reply to Tomáš Golembiovský from comment #26)
> We have experienced similar issue recently. Again it was Windows Server 2012
> R2. The installation in firstboot was performed with the fixed arguments to
> the installer (as suggested in comment 20). Note that rhev-apt was *not*
> installed at all. I am attaching the firstboot log for reference.
> Interesting thing is that no error is reported and it didn't even come to
> starting the service using "net start rhev-apt" (seems like batch script
> died in the middle?). 
> 
> I have posted a patch to virt-v2v that would allow us to collect log from
> MSI installer[1]. Hopefully this can help us in the future should we hit it
> again. But that is just the MSI Installer and not the rhev-apt.exe package.
> 
> Lev, is there some hidden option to log what rhev-apt.exe is doing before it
> fires the MSI Installer?

I'm not aware of the option to enable the logging of the loader itself.

Comment 31 Tomáš Golembiovský 2019-02-15 09:11:02 UTC
(In reply to Lev Veyde from comment #29)
> I think the issue here is the incorrect command line parameters.
> 
> Specifically due to the change that you mentioned:
> \"\\rhev-apt.exe\" /S /v/qn /v/l*vx \"/v\\\"%cd%\\rhev-apt.log\\\"\"

We experienced the issue I described before this change. The command line was:
"\rhev-apt.exe" /S /v/qn


> 
> The actual command (before you do the escaping like you did) should look
> like similar to this:
> \rhev-apt.exe /S /v/qn /V"/log C:\logs\rhev-apt.log"

Despite what the help for rhev-apt and msiexec suggest the correct command line really is:

\rhev-apt.exe /S /v/qn /v/log /vC:\some\path.log ... meaning /v must be prefixed to every argument for msiexec.

> 
> Not sure why you require this escaping but, in your case it should be
> probably something like this:
> \"\\rhev-apt.exe\" /S /v/qn /V\"//log \'%cd%\\rhev-apt.log\'\"
> 
> Please note the nested quotes.

The path has to be quoted twice otherwise spaces in path cause the arguments to be broken into more than one when passed to msiexec. Don't ask me why this happens. But you are right, using ' instead of second " might also work. It just did not occur to me at the time of writing.


(In reply to Lev Veyde from comment #30)
> (In reply to Tomáš Golembiovský from comment #26)
> > Lev, is there some hidden option to log what rhev-apt.exe is doing before it
> > fires the MSI Installer?
> 
> I'm not aware of the option to enable the logging of the loader itself.

Oh well, then let's hope that whatever is causing the installation to fail happens in msiexec. Thanks for the info.

Comment 32 Tomáš Golembiovský 2019-02-15 09:15:06 UTC
(In reply to Tomáš Golembiovský from comment #31)
> (In reply to Lev Veyde from comment #29)
> > 
> > The actual command (before you do the escaping like you did) should look
> > like similar to this:
> > \rhev-apt.exe /S /v/qn /V"/log C:\logs\rhev-apt.log"
> 
> Despite what the help for rhev-apt and msiexec suggest the correct command
> line really is:
> 
> \rhev-apt.exe /S /v/qn /v/log /vC:\some\path.log ... meaning /v must be
> prefixed to every argument for msiexec.
> 

One more note that may be useful to whomever who comes across this in the distant future:

- case is ignored for rhev-apt.exe arguments, i.e. /s is same as /S just like /v is same as /V

(despite what the help suggest)

Comment 33 Sandro Bonazzola 2019-02-18 07:54:54 UTC
Moving to 4.3.2 not being identified as blocker for 4.3.1.

Comment 34 Gal Zaidman 2019-08-26 11:40:43 UTC
Closing this as wontfix since the bug is targeted to 4.4 and we are deprecating the APT service on 4.4.

Comment 35 Richard W.M. Jones 2019-08-27 10:52:01 UTC
So what are we supposed to do to make guest drivers update?

Comment 36 Gal Zaidman 2019-08-27 14:31:41 UTC
On 4.4 we are moving the driver installer to virtio-win,
The virtio-win team is working to get the drivers updated through windows update service.
There is a discussion about marking and notifying Windows VMs that have an old version of the drivers installed.

Comment 37 mxie@redhat.com 2020-06-07 06:28:58 UTC
After debugging, found rhev-apt can install successfully if don't install virtio driver for win2012r2 guest during v2v conversion

Packages:
virt-v2v-1.40.2-23.module+el8.2.1+6797+0db00a40.x86_64
libguestfs-1.40.2-23.module+el8.2.1+6797+0db00a40.x86_64
libvirt-6.0.0-22.module+el8.2.1+6815+1c792dc8.x86_64
qemu-kvm-4.2.0-23.module+el8.2.1+6917+927fbb44.x86_64
nbdkit-1.16.2-4.module+el8.2.1+6710+effcb1df.x86_64


Steps:
1. Set value of VIRTIO_WIN is empty.
# export VIRTIO_WIN=

2.Convert a win2012r2 guest from VMware to rhv by virt-v2v
#virt-v2v -ic vpx://root@10.73.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78  esx7.0-win2012r2-x86_64 -ip /home/passwd  -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt -v -x |& tee > esx7.0-win2012r2-no-virtio-drivers.log

3.Found rhev-apt can install successfully, pls refer to screenshot 'win2012r2-rhev-apt-no-virtio-drivers.png' and log'esx7.0-win2012r2-no-virtio-drivers'

Comment 38 mxie@redhat.com 2020-06-07 06:29:38 UTC
Created attachment 1695767 [details]
win2012r2-rhev-apt-no-virtio-drivers.png

Comment 39 mxie@redhat.com 2020-06-07 06:31:27 UTC
Created attachment 1695768 [details]
esx7.0-win2012r2-no-virtio-drivers.log


Note You need to log in before you can comment on or make changes to this bug.