Bug 1895323 - After virt-v2v, qemu-ga does not run at Windows firstboot if the timezone is not M/d/yyyy format
Summary: After virt-v2v, qemu-ga does not run at Windows firstboot if the timezone is ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: virt-v2v
Version: 8.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.3
Assignee: Richard W.M. Jones
QA Contact: mxie@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-06 10:40 UTC by Mark Zealey
Modified: 2024-10-01 17:02 UTC (History)
13 users (show)

Fixed In Version: virt-v2v-1.42.0-10.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-16 07:51:01 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
screenshot of the log and script (47.29 KB, image/png)
2020-11-06 10:40 UTC, Mark Zealey
no flags Details
win2016-virt-v2v-1.42.0-6.module+el8.4.0.png (123 bytes, image/png)
2020-12-01 13:41 UTC, mxie@redhat.com
no flags Details
win2016-virt-v2v-1.42.0-6.module+el8.3.0.png (104.29 KB, image/png)
2020-12-01 13:43 UTC, mxie@redhat.com
no flags Details
timeformat_yy-mm-dd_mxie.png (92.15 KB, image/png)
2020-12-02 07:21 UTC, mxie@redhat.com
no flags Details
timeformat_dd-mm-yy_mxie.png (79.94 KB, image/png)
2020-12-02 07:26 UTC, mxie@redhat.com
no flags Details
windows-note-sync-internet-time-before-after.png (202.88 KB, image/png)
2020-12-15 05:14 UTC, mxie@redhat.com
no flags Details
time (39.57 KB, image/png)
2021-01-05 12:57 UTC, Mark Zealey
no flags Details
date (23.88 KB, image/png)
2021-01-05 12:59 UTC, Mark Zealey
no flags Details
log-info-for-qemu-ga-not-install-rhel8.4.png (96.05 KB, image/png)
2021-05-17 17:08 UTC, mxie@redhat.com
no flags Details
msiexec-error-log-rhel8.4.png (115.93 KB, image/png)
2021-05-18 04:00 UTC, mxie@redhat.com
no flags Details
log-info-for-qemu-ga-not-install-v2v-1.42.0-9-have-rhev-apt (123.87 KB, image/png)
2021-05-20 14:05 UTC, mxie@redhat.com
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2021:4684 0 None None None 2021-11-16 07:51:22 UTC

Description Mark Zealey 2020-11-06 10:40:49 UTC
Created attachment 1727087 [details]
screenshot of the log and script

Description of problem:

I am converting a Win 2012 guest with the latest virt-v2v from centos8 stream (libguestfs 1.40.2rhel=8,release=25.module_el8.4.0+547+a85d02ba,libvirt)

virtio drivers are being installed fine, but qemu-ga is not being installed correctly. v2v installs the firstboot script:

libguestfs: trace: v2v: internal_write "/Program Files/Guestfs/Firstboot/scripts/0001-install-qemu-ga-x86_64-msi.bat" "echo Removing any previously scheduled qemu-ga installation\x0d\x0aschtasks.exe /Delete /TN Firstboot-qemu-ga /F\x0d\x0aecho Scheduling delayed installation of qemu-ga from qemu-ga-x86_64.msi\x0d\x0apowershell.exe -command "$d = (get-date).AddSeconds(120); schtasks.exe /Cre"<truncated, original size 446 bytes>


But as the attached screenshot shows, this script has an issue.

It appears that $d.ToString('MM/dd/yyyy') is not correct (it produces MM-dd-yyyy output in my powershell test) and according to https://stackoverflow.com/a/40567310/1562400 it needs some special escaping. I believe it should be changed to $d.ToString("MM'/'dd'/'yyyy") in order to work correctly.

Comment 1 Bryan Kinney 2020-11-30 06:51:16 UTC
This is affecting a Windows Server 2016 guest as well.
It is on a RHVM 4.4.3 environment.

0002-install-qemu-ga-x86_64-msi.bat need to be changed so it recognizes time as dd/MM/yyyy (currently its using time as MM/dd/yyy).
Or changed to accommodate different time date formats.

Original:
powershell.exe -command "$d = (get-date).AddSeconds(120); schtasks.exe /Create /SC ONCE /ST $d.ToString('HH:mm') /SD $d.ToString('MM/dd/yyyy') /RU SYSTEM /TN Firstboot-qemu-ga /TR \"C:\qemu-ga-x86_64.msi /forcerestart /qn /l+*vx C:\qemu-ga-x86_64.msi.log\""

Work-around is to edit it directly:
powershell.exe -command "$d = (get-date).AddSeconds(120); schtasks.exe /Create /SC ONCE /ST $d.ToString('HH:mm') /SD $d.ToString('dd/MM/yyyy') /RU SYSTEM /TN Firstboot-qemu-ga /TR \"C:\qemu-ga-x86_64.msi /forcerestart /qn /l+*vx C:\qemu-ga-x86_64.msi.log\"" 

Doing it manually, for a large number of Windows servers isn't feasible.

I am also raising the urgency of this to match case.

Comment 2 Richard W.M. Jones 2020-11-30 08:55:19 UTC
I think it actually needs to be

/SD $d.ToString('MM'/'dd'/'yyyy')

according to the schtasks documentation here and comment 0.

https://docs.microsoft.com/en-us/windows/win32/taskschd/schtasks

I'll make a package with only this change for the customer to test.

Comment 3 Mark Zealey 2020-11-30 09:00:02 UTC
I believe the initial issue that I reported regarding quoting / characters is a bit different from the newly reported one but they are probably related.

Comment 4 Richard W.M. Jones 2020-11-30 09:09:33 UTC
Actually, ignore that previous comment.  Windows is a trashfire as usual.

Comment 5 Richard W.M. Jones 2020-11-30 09:11:25 UTC
So schtasks is a locale-sensitive.  From a stackoverflow comment:

  Note that the "schtasks.exe" has some localization issues, if you
  want to create a daily trigger with it, on an English Windows you'd
  have to use "daily", on a German one (for example) you'd have to use
  "täglich" instead.

The correct solution seems to be: https://stackoverflow.com/a/18730884

Comment 6 Richard W.M. Jones 2020-11-30 09:22:01 UTC
Proposed patch, but I've not tested it yet:
https://www.redhat.com/archives/libguestfs/2020-November/msg00026.html

Comment 7 Bryan Kinney 2020-12-01 06:13:50 UTC
Is it possible to have that fix in comment #6 verified?

The customer is looking at December 8th to run the live migration.

Comment 8 Bryan Kinney 2020-12-01 07:52:35 UTC
The diff provided is for the convert_windows.ml source code.
   https://github.com/libguestfs/virt-v2v/blob/master/v2v/convert_windows.ml

For testing, should the msi instead be changed instead?

Comment 9 Bryan Kinney 2020-12-01 09:37:31 UTC
The customer has also confirmed that the fixed rpm provided,
corrected the date format issue.

Thank you for providing the correction to the code.

Comment 10 Richard W.M. Jones 2020-12-01 10:05:50 UTC
Fixed upstream in
https://github.com/libguestfs/virt-v2v/commit/b65711c9293415f671d35d0e2f9b55a74343da45

Comment 11 mxie@redhat.com 2020-12-01 13:40:53 UTC
Can't reproduce the bug with rhel8.3 AV builds:
virt-v2v-1.42.0-6.module+el8.3.0+7898+13f907d5.x86_64
libguestfs-1.42.0-2.module+el8.3.0+6798+ad6e66be.x86_64
libguestfs-winsupport-8.2-1.module+el8.3.0+6124+819ee737.x86_64
virtio-win-1.9.14-4.el8.noarch

Steps:
1. Convert a win2016 guest from VMware to rhv4.4.4 by virt-v2v
#virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt esx7.0-win2016-x86_64

2.Check the guest after v2v conversion, qemu guest agent can be installed in guest successfully and 0002-install-qemu-ga-x86_64-msi.bat is using time as MM/dd/yyyy, pls refer to "win2016-virt-v2v-1.42.0-6.module+el8.3.0.png"
 

Also can't reproduce the bug with rhel8.4 AV builds:
virt-v2v-1.42.0-6.module+el8.4.0+8855+a9e237a9.x86_64
libguestfs-1.42.0-2.module+el8.4.0+8855+a9e237a9.x86_64
virtio-win-1.9.14-4.el8.noarch
libguestfs-winsupport-8.2-1.module+el8.4.0+8855+a9e237a9.x86_64


Steps:
1. Convert a win2016 guest from VMware to rhv4.4.4 by virt-v2v
#virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt esx6.7-win2016-x86_64


2.Check the guest after v2v conversion, qemu guest agent can be installed in guest successfully and 0002-install-qemu-ga-x86_64-msi.bat is using time as MM/dd/yyyy, pls refer to "win2016-virt-v2v-1.42.0-6.module+el8.4.0.png"

Comment 12 mxie@redhat.com 2020-12-01 13:41:19 UTC
Created attachment 1735234 [details]
win2016-virt-v2v-1.42.0-6.module+el8.4.0.png

Comment 13 mxie@redhat.com 2020-12-01 13:43:49 UTC
Created attachment 1735235 [details]
win2016-virt-v2v-1.42.0-6.module+el8.3.0.png

Comment 14 Richard W.M. Jones 2020-12-01 13:50:16 UTC
mxie: The problem only affects Windows guests which use the d/M/yyyy-style timezone.
You can find out if your Windows guest is using this by doing:

(1) Start up the guest.

(2) Start up CMD prompt.

(3) Run the following command:

powershell -command "echo [System.Globalization.DateTimeFormatInfo]::CurrentInfo.ShortDatePattern"

It will print something like d/M/yyyy or yyyy/MM/dd (there are other possibilities).

If this prints M/d/yyyy then the bug will NOT affect the guest.  We know that
d/M/yyyy is affected.  We do not know about other timezones.

You'll probably have to change the timezone of the guest (somehow - I'm not sure
how, Control Panel?) to a European or Australian timezone in order to reproduce
the bug.

Some more information in this page:
https://docs.microsoft.com/en-us/dotnet/api/system.globalization.datetimeformatinfo.shortdatepattern?view=net-5.0

Comment 15 mxie@redhat.com 2020-12-02 07:20:58 UTC
It's strange that I still can't reproduce the bug after changing time format to dd/mm/yy or yy/mm/dd, details pls refer to below steps and attached screenshots

Package versions:
virt-v2v-1.42.0-6.module+el8.4.0+8855+a9e237a9.x86_64
libguestfs-1.42.0-2.module+el8.4.0+8855+a9e237a9.x86_64
libvirt-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64
qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9.x86_64
nbdkit-1.22.0-2.module+el8.4.0+8855+a9e237a9.x86_64
libguestfs-winsupport-8.2-1.module+el8.4.0+8855+a9e237a9.x86_64
virtio-win-1.9.14-4.el8.noarch


Steps:
1.Prepare a windows guest and change the time format of guest to dd/mm/yy with below steps on VMware

2.Convert the guest to rhv4.4.4 by v2v
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt esx7.0-win2016-x86_64-time_dd-mm-yy

3.Check the guest after v2v conversion, qemu guest agent can be installed in guest successfully, pls check screenshot"timeformat_dd-mm-yy_mxie"

4.Prepare a windows guest and change the time format of guest to yy/mm/dd with below steps on VMware

5.Convert the guest to rhv4.4.4 by v2v
# # virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt esx7.0-win2016-x86_64-time_yy-mm-dd

6.Check the guest after v2v conversion, qemu guest agent can be installed in guest successfully, pls check screenshot"timeformat_yy-mm-dd_mxie"


Hi Richard,

   The command you mentioned in comment14 can't show time format of guest. I changed the time format in windows guests using method https://www.moore.co.uk/msuk/mooresouth/news/may-2016/has-upgrading-to-windows-10-affected-your-date-for, anyway, I still can't reproduce the bug, do you have any other suggestions?

Comment 16 mxie@redhat.com 2020-12-02 07:21:48 UTC
Created attachment 1735499 [details]
timeformat_yy-mm-dd_mxie.png

Comment 17 mxie@redhat.com 2020-12-02 07:26:22 UTC
Created attachment 1735500 [details]
timeformat_dd-mm-yy_mxie.png

Comment 18 Richard W.M. Jones 2020-12-02 09:34:24 UTC
Does QEMU-GA really get installed?

The problem could be that now we're in December, 12/02 and 02/12 are both
valid dates.  At the end of last month when the failure occurred, either
28/11 or 11/28 would have been an invalid date.

It may be that QEMU-GA installation is being scheduled now for 12th February 2020
instead of 120 seconds after installation.  I'm not sure what happens then -
maybe Windows runs the task immediately (since it's scheduled for a past date)
or maybe it gets stuck in some way.

There is a way to get SCHTASKS to list the scheduled commands:

  schtasks /query

Comment 19 Tomáš Golembiovský 2020-12-02 09:53:41 UTC
(In reply to Richard W.M. Jones from comment #18)
> It may be that QEMU-GA installation is being scheduled now for 12th February
> 2020
> instead of 120 seconds after installation.  I'm not sure what happens then -
> maybe Windows runs the task immediately (since it's scheduled for a past
> date)
> or maybe it gets stuck in some way.

In my experience scheduling fails when the date/time is in the past. (The error is different from what we saw in this case.)

Comment 20 Richard W.M. Jones 2020-12-02 12:16:56 UTC
I seem to be able to confirm this (on W2K19).  It looks as if the
task gets created successfully, albeit with a warning:

WARNING: Task may not run because /ST is earlier than current time.
SUCCESS: The scheduled task "test" has successfully been created

As far as I can tell the task does NOT run.

So I don't understand how it was that qemu-ga was installed even
though the date would have referred to some time in February.

Comment 23 mxie@redhat.com 2020-12-15 05:13:28 UTC
I still can't reproduce the bug even if set timezone as Australia, set time format as dd/mm/yy or yy/mm/dd, also set the date /time in the past time, such as 28/11/20 in VMware win2016 guest before v2v conversion. The qemu-ga still can install successfully in guest after v2v converting guest to rhv4.4

   Hi Mark,

     Could you please provide more info about how did you set the date in your VMware windows guest? Such as timezone, time format and what was the date etc.
 
   Hi Tomas,

     It's strange that I set date as 28/11/20 and set the computer is not automatically sync with internet time in VMware win2016 guest before v2v conversion(windows guest can keep the time as 28/11/20 after rebooting on VMware), but the date of guest will be changed to today(12/15/20) after v2v conversion, pls see screenshot "windows-not-sync-internet-time-before-after" I think qemu-ga make the setting "computer is not automatically sync with internet time" invalid, so the date will be changed in guest after v2v conversion, is it bug? 

   Windows guest can't keep time in the past because of above problem, so I can't reproduce the bug so far, do you have any other suggestions?

Comment 24 mxie@redhat.com 2020-12-15 05:14:13 UTC
Created attachment 1739235 [details]
windows-note-sync-internet-time-before-after.png

Comment 25 Mark Zealey 2021-01-05 12:57:55 UTC
Created attachment 1744564 [details]
time

Comment 26 Mark Zealey 2021-01-05 12:59:13 UTC
Created attachment 1744565 [details]
date

Comment 27 Mark Zealey 2021-01-05 13:00:15 UTC
attached screen shots of date+time, they are set from the host so should be fine but perhaps localization is not working properly. this is w2k12 & I believe my proposal at the very beginning works correctly for this case

Comment 28 Martin Kletzander 2021-02-10 13:37:29 UTC
(In reply to mxie from comment #23)
My understanding (and I am guessing here) is that you need to do those things in a special way:

 1) Set the format in as M/d/yyyy (which is different from MM/d/yyyy and M/d/yy)

 2) Set the date in such a way that it triggers what I would describe as "bug in schtasks" which, according to sources in this BZ might be 10th of September or probably others, not sure whent he customer was actually migrating.  What I am thinking is it relies on two things:

    a) it needs to be a valid date when you switch day and month (possibly it might need to be in the past when switched, but I'm not sure, see point (b)) and

    b) when you switch it the first number needs to be one digit (so the month in the original proper date needs to be lesser than 10) as the initial 0 probably also breaks some additional logic,

    but as I said that is what I *think*, it does not necessarily have to be correct.  It would be easier to test some date like that in a real Windows environment.

 3) Probably make sure that this is not "fixed" by ntp or some other time fix.  I'm guessing all the hosts that the VMs run on (conversion host as well as the destination) need to be running in a timezone that has that particular time set.  The reason behind that is that the HW clock (RTC) time is probably not transferred as nobody really wants that to happen and upon boot the VM needs to figure out what time it actually is and takes the time from RTC which for Windows will be probably set to local time of the host.

Given the above, I think the easiest way to confirm is to test this at the start of the next month, but that is, of course, not something that would help us in the long run, just one thing to try in case we won't find a way to reproduce it on another day.

I might have misunderstood some of the details, but we would need more details from the customers who faced the issue.

Comment 29 Martin Kletzander 2021-02-11 09:01:09 UTC
@Bryan: I see the customer case is closed, would it be OK to dial the priority back a little bit?  Just se we are sure we plan accordingly.  Thank you.

Comment 30 Bryan Kinney 2021-02-11 23:56:21 UTC
Yes. I have changed the severity on the customer case.
The newer version fixed the original issue for the customer.

Comment 32 Richard W.M. Jones 2021-04-23 10:03:54 UTC
Let's try to put this in RHEL AV 8.5 since it is affecting customers
(see also bug 1929158).

Comment 36 mxie@redhat.com 2021-05-17 17:06:59 UTC
Reproduce the bug with below builds:
virt-v2v-1.42.0-9.module+el8.4.0+9561+069bb9c1.x86_64
libguestfs-1.44.0-2.module+el8.4.0+10146+75917d2f.x86_64
libvirt-client-7.0.0-14.module+el8.4.0+10886+79296686.x86_64
qemu-kvm-5.2.0-16.module+el8.4.0+10806+b7d97207.x86_64
nbdkit-1.24.0-1.module+el8.4.0+9341+96cf2672.x86_64
virtio-win-1.9.16-2.el8.noarch


Steps to reproduce:
1.Prepare win2016 guests and set different date format in guest on VMware, then convert guests from VMware to rhv4.4 by v2v
#virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -oo rhv-cluster=NFS  -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt $win2016_guest_set_different_date_format

Check if qemu-ga can be installed automatically after after v2v conversion, get below result:

Short_date      Long_date               If qemu-ga is running 
M/d/yyyy	d MMMM,yyyy		NO
MM/dd/yy	MMMM d,yyyy		NO
yyyy-MM-dd	dddd,MMMM d,yyyy        NO
yyyy/MM/dd	dd MMMM yyyy	        NO
d.M.yy		dd MMMM yyyy	        NO
dd.MMM.yyyy	dddd,d.MMMM yyyy	NO
dd/MM/yyyy	dddd, MMMM dd,yyyy	NO

Try to find the reason why qemu-ga can't be installed automatically, there is no error about date in firstboot log but found message 'Failed to grab execution mutex. Sytem error 258 ' in qemu-ga.msi log, please refer to screenshot'log-info-for-qemu-ga-not-install-rhel8.4'

2.Check powershell command in 0001-install-qemu-ga-x86_64-msi, the command is 'powershell.exe -command \"$d = (get-date).AddSeconds(120); schtasks.exe /Create /SC ONCE /ST $d.ToString('HH:mm') /SD $d.ToString('MM/dd/yyyy') /RU SYSTEM /TN Firstboot-qemu-ga /TR \"C:\qemu-ga-x86_64.msi /forcerestart /qn /l+*vx C:\qemu-ga-x86_64.msi.log\""'



Verify the bug with below builds:
virt-v2v-1.42.0-12.module+el8.5.0+10976+d40a93e9.x86_64
libguestfs-1.44.0-3.module+el8.5.0+10681+17a9b157.x86_64
libvirt-client-7.3.0-1.module+el8.5.0+11004+f4810536.x86_64
qemu-kvm-6.0.0-16.module+el8.5.0+10848+2dccc46d.x86_64
nbdkit-1.24.0-1.module+el8.4.0+9341+96cf2672.x86_64
virtio-win-1.9.16-2.el8.noarch


1.Prepare win2016 guests and set different date format in guest on VMware, then convert guests from VMware to rhv4.4 by v2v
#virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -oo rhv-cluster=NFS  -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt $win2016_guest_set_different_date_format

Check if qemu-ga can be installed automatically after after v2v conversion, get below result:

Short_date      Long_date               If qemu-ga is running 
M/d/yyyy	d MMMM,yyyy		YES
MM/dd/yy	MMMM d,yyyy		YES
yyyy-MM-dd	dddd,MMMM d,yyyy        YES
yyyy/MM/dd	dd MMMM yyyy	        YES
d.M.yy		dd MMMM yyyy	        YES
dd.MMM.yyyy	dddd,d.MMMM yyyy	YES
dd/MM/yyyy	dddd, MMMM dd,yyyy	YES

2.Check powershell command in 0001-install-qemu-ga-x86_64-msi, the command has been changed as 'powershell.exe -command \"$d = (get-date).AddSeconds(120); $FormatHack = ($([System.Globalization.DateTimeFormatInfo]::CurrentInfo.ShortDatePattern) -replace 'M+/', 'MM/') -replace 'd+/', 'dd/'; schtasks.exe /Create /SC ONCE /ST $d.ToString('HH:mm') /SD $d.ToString($FormatHack) /RU SYSTEM /TN Firstboot-qemu-ga /TR \\"C:\qemu-ga-x86_64.msi /forcerestart /qn /l+*vx C:\qemu-ga-x86_64.msi.log\""'


Result:
   According to above test result, qemu-ga can be installed automatically in windows guest after v2v conversion even if original guest has set some specific date format, so move the bug from ON_QA to VERIFIED

Comment 37 mxie@redhat.com 2021-05-17 17:08:12 UTC
Created attachment 1784168 [details]
log-info-for-qemu-ga-not-install-rhel8.4.png

Comment 38 mxie@redhat.com 2021-05-18 04:00:05 UTC
Created attachment 1784292 [details]
msiexec-error-log-rhel8.4.png

Comment 39 mxie@redhat.com 2021-05-20 13:53:49 UTC
Please ignore comment36 because I forgot I have deleted rhev-apt when I reproduced the bug on rhel8.4 v2v server, so the results of reproduce steps weren't inaccurate, so I retest the bug as below:


Reproduce the bug with below builds:
virt-v2v-1.42.0-9.module+el8.4.0+9561+069bb9c1.x86_64
libguestfs-1.44.0-2.module+el8.4.0+10146+75917d2f.x86_64
libvirt-client-7.0.0-14.module+el8.4.0+10886+79296686.x86_64
qemu-kvm-5.2.0-16.module+el8.4.0+10806+b7d97207.x86_64
nbdkit-1.24.0-1.module+el8.4.0+9341+96cf2672.x86_64
virtio-win-1.9.16-2.el8.noarch


Steps to reproduce:
1.Prepare win2016 guests and set different date format in guest on VMware, then convert guests from VMware to rhv4.4 by v2v.
#virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -oo rhv-cluster=NFS  -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt $win2016_guest_set_different_date_format

Check if qemu-ga can be installed automatically after after v2v conversion, get below result:

Short_date      Long_date               Qemu-ga running 
M/d/yyyy	MMMM d,yyyy		YES
MM/dd/yy	MMMM d,yyyy		YES
d/M/y		dddd d MMMM yyyy	YES
dd.MM.yyyy	dddd,d.MMMM yyyy	NO
dd-MMM-yy	dddd, MMMM d,yyyy	YES
yyyy/MM/dd	dd MMMM yyyy	        YES
yy/MM/dd	MMMM d,yyyy	        NO
yyyy/M/d	yyyy'年'M'月'd'日'       YES

Try to find the reason why qemu-ga can't be installed automatically, there is no error about date in firstboot log but found message 'Failed to grab execution mutex. Sytem error 258 ' in qemu-ga.msi log, please refer to screenshot'log-info-for-qemu-ga-not-install-v2v-1.42.0-9-have-rhev-apt'

2.Check powershell command in 0001-install-qemu-ga-x86_64-msi, the command is 'powershell.exe -command \"$d = (get-date).AddSeconds(120); schtasks.exe /Create /SC ONCE /ST $d.ToString('HH:mm') /SD $d.ToString('MM/dd/yyyy') /RU SYSTEM /TN Firstboot-qemu-ga /TR \"C:\qemu-ga-x86_64.msi /forcerestart /qn /l+*vx C:\qemu-ga-x86_64.msi.log\""'



Verify the bug with below builds:
virt-v2v-1.42.0-12.module+el8.5.0+10976+d40a93e9.x86_64
libguestfs-1.44.0-3.module+el8.5.0+10681+17a9b157.x86_64
libvirt-client-7.3.0-1.module+el8.5.0+11004+f4810536.x86_64
qemu-kvm-6.0.0-16.module+el8.5.0+10848+2dccc46d.x86_64
nbdkit-1.24.0-1.module+el8.4.0+9341+96cf2672.x86_64
virtio-win-1.9.16-2.el8.noarch


1.Prepare win2016 guests and set different date format in guest on VMware, then convert guests from VMware to rhv4.4 by v2v
#virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -oo rhv-cluster=NFS  -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt $win2016_guest_set_different_date_format

Check if qemu-ga can be installed automatically after after v2v conversion, get below result:

Short_date      Long_date               Qemu-ga running 
M/d/yyyy	MMMM d,yyyy		YES
MM/dd/yy	MMMM d,yyyy		YES
d/M/y		dddd d MMMM yyyy	YES
dd.MM.yyyy	dddd,d.MMMM yyyy	YES
dd-MMM-yy	dddd, MMMM d,yyyy	YES
yyyy/MM/dd	dd MMMM yyyy	        YES
yy/MM/dd	MMMM d,yyyy	        YES
yyyy/M/d	yyyy'年'M'月'd'日'       YES

2.Check powershell command in 0001-install-qemu-ga-x86_64-msi, the command has been changed as 'powershell.exe -command \"$d = (get-date).AddSeconds(120); $FormatHack = ($([System.Globalization.DateTimeFormatInfo]::CurrentInfo.ShortDatePattern) -replace 'M+/', 'MM/') -replace 'd+/', 'dd/'; schtasks.exe /Create /SC ONCE /ST $d.ToString('HH:mm') /SD $d.ToString($FormatHack) /RU SYSTEM /TN Firstboot-qemu-ga /TR \\"C:\qemu-ga-x86_64.msi /forcerestart /qn /l+*vx C:\qemu-ga-x86_64.msi.log\""'


Result:
   According to above test results, qemu-ga can't installed automatically in windows guest sometimes after virt-v2v-1.42.0-9 conversion, but qemu-ga can always installed automatically in windows guest after virt-v2v-1.42.0-12 conversion, virt-v2v-1.42.0-12 is more stable than virt-v2v-1.42.0-9 to install qemu-ga for windows guest when original guest has set some specific date format, so I think the bug has been fixed.

Comment 40 mxie@redhat.com 2021-05-20 14:05:23 UTC
Created attachment 1785186 [details]
log-info-for-qemu-ga-not-install-v2v-1.42.0-9-have-rhev-apt

Comment 42 Xiaodai Wang 2021-11-10 10:02:02 UTC
Based this patch: https://github.com/libguestfs/virt-v2v/commit/b65711c9293415f671d35d0e2f9b55a74343da45
I customized two formats and got same error of schtasks in powershell:
1) Set Date and Time Format to 'd/M/yy', which the 'yy' is not replaced to 'yyyy'.
2) Set the format to 'd.M.yy', the patch can't match and replace the format correctly.

But when I tried it by virt-v2v, there is no any errors, I've no idea.
Is there any difference between virt-v2v and manually run it in powershell?

Comment 43 Richard W.M. Jones 2021-11-11 10:08:14 UTC
Hi Xiaodai, I'm a bit unclear on what you're trying to do.  What commands
did you use and what didn't work?

Comment 44 Xiaodai Wang 2021-11-11 10:50:35 UTC
I was trying to setup a guest which can reproduce the problem 100%.

In comment 5, you pointed out the error comes from schtasks.exe.
I can reproduce it in powershell easily. And the fix can work well
with 'M/d/yyyy'.

But I checked the time format, there are lots styles, example, 'M/d/yy',
'yy.M.d'. The '$FormatHack' only replaces the 'M' and 'd', but when it
is 'yy' instead of 'yyyy', the schtasks still reports errors. And if the 
format is 'yy.M.d', the replace ''M+/' will not work.

I assume the 'yy.M.d' setting will make the problem be reproduced in v2v
but after the conversion, everything goes well, I'm confused.

I attached the screenshot of testing in powershell.

Comment 46 Richard W.M. Jones 2021-11-11 12:00:12 UTC
Ok as I understand the problem:

(1) You have a machine where ShortDatePattern is yy.M.d

=> Xiaodai, what locale is this?  Did you set the locale after the guest booted?
Does the change persist across reboots?

(2) You're quite correct that the -replace expression we use ('M+/' and 'd+/')
will not match this, and will therefore not work to replace M with MM and d with dd
in the date we are trying to produce.

(Reference:
https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_regular_expressions?view=powershell-7.2)

(3) When run by hand, the -SD parameter (start date) is not set to the expected
yyyy/mm/dd format and schtasks prints:

ERROR: Invalid Start Date (Date should be in "yyyy/mm/dd" format

(4) Using M+ instead of M+/ and d+ instead of d+/ as the regexps would be part of
a fix.  However we would also need to replace the "." with "/".  mxie reported
(comment 39) that locales with . in the short date worked fine anyway.

(5) The bug cannot actually be reproduced by virt-v2v for some other reason.  Could
be that the locale is reset to another value when the guest boots up?

Comment 48 Xiaodai Wang 2021-11-12 03:00:21 UTC
From my observation, The valid start date format must be same order
as the shortdatepattern. Besides that, all 'M', 'd', 'yy' must be changed
to 'MM', 'dd', 'yyyy'. All separators are valid.

for example:

shortdatepattern   errorinschtasks   invalidSD              validSD
dd/MM/yy           dd/mm/yyyy        12.10.21             12/10/2021
                                     2021/10/12           12.10-2021
yy.M.d             yyyy/mm/dd        21.10.10             2021/10/12
                                     10/12/2021           2021/10-12

Comment 49 Richard W.M. Jones 2021-11-12 08:51:31 UTC
I think something like this may fix it:

https://listman.redhat.com/archives/libguestfs/2021-November/msg00139.html

Comment 50 Xiaodai Wang 2021-11-12 10:03:37 UTC
(In reply to Richard W.M. Jones from comment #49)
> I think something like this may fix it:
> 
> https://listman.redhat.com/archives/libguestfs/2021-November/msg00139.html

and the 'y+' should also be replaced with 'yyyy'.

Comment 51 Richard W.M. Jones 2021-11-12 10:40:48 UTC
v2 posted:
https://listman.redhat.com/archives/libguestfs/2021-November/msg00140.html

(These are totally untested BTW)

Comment 54 errata-xmlrpc 2021-11-16 07:51:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4684


Note You need to log in before you can comment on or make changes to this bug.