This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1963032 - qemu-ga sometimes fails to install at firstboot: Failed to grab execution mutex. System error 258.
Summary: qemu-ga sometimes fails to install at firstboot: Failed to grab execution mut...
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: virt-v2v
Version: 9.0
Hardware: x86_64
OS: Unspecified
medium
medium
Target Milestone: beta
: ---
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-21 08:08 UTC by mxie@redhat.com
Modified: 2023-09-22 17:36 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-09-22 17:36:12 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
log-info-for-qemu-ga-not-install-after-rhel9-v2v.png (99.65 KB, image/png)
2021-05-21 08:08 UTC, mxie@redhat.com 		no flags Details
msiexec-error-windows-log.png (109.60 KB, image/png)
2021-05-21 08:11 UTC, mxie@redhat.com 		no flags Details
esx7.0-win2016-date-yyyy-m-d-china-rhel9.0.log (2.12 MB, text/plain)
2021-05-21 08:13 UTC, mxie@redhat.com 		no flags Details
esx7.0-win2016-date-yyyy-m-d-china-v2v-libguestfs-1.42.0-6-virtio-win-1.9.16-2.log (2.36 MB, text/plain)
2021-05-25 11:58 UTC, mxie@redhat.com 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker   RHEL-7482 0 None Migrated None 2023-09-22 17:32:36 UTC

Description mxie@redhat.com 2021-05-21 08:08:08 UTC 
Created attachment 1785440 [details]
log-info-for-qemu-ga-not-install-after-rhel9-v2v.png

Description of problem:
windows guest fails to install qemu-ga sometimes because of system error 258 after v2v conversion 

Version-Release number of selected component (if applicable):
virt-v2v-1.44.0-1.el9.1.x86_64
libguestfs-1.45.5-1.el9.x86_64
guestfs-tools-1.46.1-2.el9.x86_64
libvirt-7.0.0-6.el9.x86_64
qemu-kvm-6.0.0-2.el9.x86_64
virtio-win-1.9.15-3.el9.noarch
libguestfs-winsupport-8.2-1.el9.x86_64
rhv4.4.6.3-0.8.el8ev

How reproducible:
80%

Steps to Reproduce:
1.Prepare win2016 guests and set different date format in guest on VMware, then convert guests from VMware to rhv4.4 by v2v
#virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -oo rhv-cluster=NFS  -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt $win2016_guest_set_different_date_format

Check if qemu-ga can be installed automatically after after v2v conversion, get below result:

Short_date      Long_date               Qemu-ga running 
M/d/yyyy	MMMM d,yyyy		NO
MM/dd/yy	MMMM d,yyyy		NO 50% YES 50%
d/M/y		dddd d MMMM yyyy	NO
dd.MM.yyyy	dddd,d.MMMM yyyy	NO
dd-MMM-yy	dddd, MMMM d,yyyy	NO
yyyy/MM/dd	dd MMMM yyyy	        NO
yy/MM/dd	MMMM d,yyyy	        NO
yyyy/M/d	yyyy'年'M'月'd'日'       NO


2.Check firstboot log and qemu-ga log, found there is no error about date in firstboot log, but found error message 'Failed to grab execution mutex. Sytem error 258 ' in qemu-ga.msi log, please refer to screenshot'log-info-for-qemu-ga-not-install-after-rhel9-v2v'

3.Check powershell command in 0001-install-qemu-ga-x86_64-msi, the command is 'powershell.exe -command \"$d = (get-date).AddSeconds(120); $FormatHack = ($([System.Globalization.DateTimeFormatInfo]::CurrentInfo.ShortDatePattern) -replace 'M+/', 'MM/') -replace 'd+/', 'dd/'; schtasks.exe /Create /SC ONCE /ST $d.ToString('HH:mm') /SD $d.ToString($FormatHack) /RU SYSTEM /TN Firstboot-qemu-ga /TR \\"C:\qemu-ga-x86_64.msi /forcerestart /qn /l+*vx C:\qemu-ga-x86_64.msi.log\""'


Actual results:
As above description


Expected results:
Qemu-ga can install in windows guest automatically after v2v conversion when original guest has set some specific date format


Additional info:
   I think the problem has relationship with rhev-apt, because qemu-ga also can't installed automatically in windows guest sometimes when I converted the windows guest without rhev-apt installed on rhel8.4 v2v server,details pls refer to the steps to reproduce in https://bugzilla.redhat.com/show_bug.cgi?id=1895323#c36.    
   But the installation of qemu-ga will be more stable if rhev-apt is existing on rhel8.4 v2v server during conversion, details pls refer to the steps to reproduce in https://bugzilla.redhat.com/show_bug.cgi?id=1895323#c39
Comment 1 mxie@redhat.com 2021-05-21 08:11:04 UTC 
Created attachment 1785441 [details]
msiexec-error-windows-log.png
Comment 2 mxie@redhat.com 2021-05-21 08:13:11 UTC 
Created attachment 1785442 [details]
esx7.0-win2016-date-yyyy-m-d-china-rhel9.0.log
Comment 3 Richard W.M. Jones 2021-05-24 08:58:07 UTC 
Apparently the problem means that another instance of msiexec is
running at the same time, and rather than do anything sensible
only one is allowed to run and the other fails.  Of course the question
then become what other instance of msiexec is running?

Is this a regression over RHEL 8?  I can't imagine what has changed
that would stop this from working between RHEL 8 and RHEL 9.
Comment 4 Yvugenfi@redhat.com 2021-05-25 08:15:21 UTC 
What kind of installer runs during v2v? Is it https://github.com/virtio-win/virtio-win-guest-tools-installer or qemu-ga installer separately?
Comment 5 Richard W.M. Jones 2021-05-25 08:30:39 UTC 
It's run via virt-v2v, and the command run is:

  qemu-ga-x86_64.msi /forcerestart /qn /l+*vx C:\qemu-ga.log

qemu-ga-x86_64.msi comes from virtio-win-1.9.15-3.el9.noarch
(filename: /guest-agent/qemu-ga-x86_64.msi)

Looking at the log file, I suspect what could be happening here
is that the instance of MsiExec that we started running 60 seconds
earlier to uninstall Vmware-Tools is still running.

I wonder if there's a way to "chain" MsiExec jobs so that one
runs after another, and the whole lot only runs after the network
has come up.
Comment 6 Richard W.M. Jones 2021-05-25 08:34:27 UTC 
> It's run via virt-v2v,

What I mean by this is virt-v2v sets it up, and it runs once the first
time the guest is booted on KVM.
Comment 7 Yvugenfi@redhat.com 2021-05-25 09:27:04 UTC 
(In reply to Richard W.M. Jones from comment #6)
> > It's run via virt-v2v,
> 
> What I mean by this is virt-v2v sets it up, and it runs once the first
> time the guest is booted on KVM.
Sure. 

(In reply to Richard W.M. Jones from comment #5)
> It's run via virt-v2v, and the command run is:
> 
>   qemu-ga-x86_64.msi /forcerestart /qn /l+*vx C:\qemu-ga.log
> 
> qemu-ga-x86_64.msi comes from virtio-win-1.9.15-3.el9.noarch
> (filename: /guest-agent/qemu-ga-x86_64.msi)
> 
> Looking at the log file, I suspect what could be happening here
> is that the instance of MsiExec that we started running 60 seconds
> earlier to uninstall Vmware-Tools is still running.
> 
> I wonder if there's a way to "chain" MsiExec jobs so that one
> runs after another, and the whole lot only runs after the network
> has come up.
You can try to create a wrapper batch file and use "start /wait" for each command that triggers the execution of MsiExec.
Comment 8 mxie@redhat.com 2021-05-25 11:55:50 UTC 
BTW, I didn't meet the bug when I test (In reply to Richard W.M. Jones from comment #3)
> Apparently the problem means that another instance of msiexec is
> running at the same time, and rather than do anything sensible
> only one is allowed to run and the other fails.  Of course the question
> then become what other instance of msiexec is running?
> 
> Is this a regression over RHEL 8?  I can't imagine what has changed
> that would stop this from working between RHEL 8 and RHEL 9.

I didn't meet the bug when I tested bug1895323 with virt-v2v-1.42.0-6.module+el8.4.0+8855, please refer to https://bugzilla.redhat.com/show_bug.cgi?id=1895323#c15

I downgrade virt-v2v and libguestfs to 1.42.0-6.module+el8.4.0+8855 and convert the same guest of this bug, qemu-ga can installed normally after v2v conversion, compare the debug logs of virt-v2v-1.42.0-6 and virt-v2v-1.44.0-1.el9.1, found info of install-qemu-ga-x86_64-msi.bat is a little different between them


# cat esx7.0-win2016-date-yyyy-m-d-china-v2v-libguestfs-1.42.0-6-virtio-win-1.9.16-2.log |grep install-qemu-ga-x86_64-msi.bat
libguestfs: trace: v2v: write "/Program Files/Guestfs/Firstboot/scripts/0002-install-qemu-ga-x86_64-msi.bat" "echo Removing any previously scheduled qemu-ga installation\x0d\x0aschtasks.exe /Delete /TN Firstboot-qemu-ga /F\x0d\x0aecho Scheduling delayed installation of qemu-ga from qemu-ga-x86_64.msi\x0d\x0apowershell.exe -command "$d = (get-date).AddSeconds(120); schtasks.exe /Cre"<truncated, original size 446 bytes>
libguestfs: trace: v2v: internal_write "/Program Files/Guestfs/Firstboot/scripts/0002-install-qemu-ga-x86_64-msi.bat" "echo Removing any previously scheduled qemu-ga installation\x0d\x0aschtasks.exe /Delete /TN Firstboot-qemu-ga /F\x0d\x0aecho Scheduling delayed installation of qemu-ga from qemu-ga-x86_64.msi\x0d\x0apowershell.exe -command "$d = (get-date).AddSeconds(120); schtasks.exe /Cre"<truncated, original size 446 bytes>


#  cat esx7.0-win2016-date-yyyy-m-d-china-rhel9.0.log |grep install-qemu-ga-x86_64-msi.bat
libguestfs: trace: v2v: write "/Program Files/Guestfs/Firstboot/scripts/0001-install-qemu-ga-x86_64-msi.bat" "echo Removing any previously scheduled qemu-ga installation\x0d\x0aschtasks.exe /Delete /TN Firstboot-qemu-ga /F\x0d\x0aecho Scheduling delayed installation of qemu-ga from qemu-ga-x86_64.msi\x0d\x0apowershell.exe -command "$d = (get-date).AddSeconds(120); $FormatHack = ($("<truncated, original size 581 bytes>
libguestfs: trace: v2v: internal_write "/Program Files/Guestfs/Firstboot/scripts/0001-install-qemu-ga-x86_64-msi.bat" "echo Removing any previously scheduled qemu-ga installation\x0d\x0aschtasks.exe /Delete /TN Firstboot-qemu-ga /F\x0d\x0aecho Scheduling delayed installation of qemu-ga from qemu-ga-x86_64.msi\x0d\x0apowershell.exe -command "$d = (get-date).AddSeconds(120); $FormatHack = ($("<truncated, original size 581 bytes>
Comment 9 mxie@redhat.com 2021-05-25 11:58:16 UTC 
Created attachment 1786835 [details]
esx7.0-win2016-date-yyyy-m-d-china-v2v-libguestfs-1.42.0-6-virtio-win-1.9.16-2.log
Comment 10 Richard W.M. Jones 2021-05-25 12:22:16 UTC 
mxie: I suspect there is simply a race between the two runs of MsiExec.

Yan is correct here that we need to go about this differently and his
suggestion of using start /wait is what I'll try.  Still looking for a
way to wait until the network is available however.

FWIW my analysis of the two logs ...

In your log (virt-v2v 1.42):

0001-configure-rhev-apt.bat - installs rhev-apt immediately
0002-install-qemu-ga-x86_64-msi.bat - after 120 seconds installs qemu-ga via msiexec
0003-uninstall-VMware-Tools.bat - removes VMware tools immediately via msiexec

In the current bug (virt-v2v 1.44):

0001-install-qemu-ga-x86_64-msi.bat - after 120 seconds installs qemu-ga via msiexec
0002-uninstall-VMware-Tools.bat - removes VMware tools immediately via msiexec

The difference is because in virt-v2v 1.44 we no longer try to install
rhev-apt, since it's obsolete.
Comment 12 Klaus Heinrich Kiwi 2021-08-09 20:47:20 UTC 
(In reply to Richard W.M. Jones from comment #10)
> mxie: I suspect there is simply a race between the two runs of MsiExec.
> 
> Yan is correct here that we need to go about this differently and his
> suggestion of using start /wait is what I'll try.  Still looking for a
> way to wait until the network is available however.
> 
> FWIW my analysis of the two logs ...
> 
> In your log (virt-v2v 1.42):
> 
> 0001-configure-rhev-apt.bat - installs rhev-apt immediately
> 0002-install-qemu-ga-x86_64-msi.bat - after 120 seconds installs qemu-ga via
> msiexec
> 0003-uninstall-VMware-Tools.bat - removes VMware tools immediately via
> msiexec
> 
> In the current bug (virt-v2v 1.44):
> 
> 0001-install-qemu-ga-x86_64-msi.bat - after 120 seconds installs qemu-ga via
> msiexec
> 0002-uninstall-VMware-Tools.bat - removes VMware tools immediately via
> msiexec
> 
> The difference is because in virt-v2v 1.44 we no longer try to install
> rhev-apt, since it's obsolete.

is this something we can mitigate (perhaps with a delay /sleep), or perhaps something for us to document and move on?
Comment 13 Richard W.M. Jones 2021-08-09 21:02:10 UTC 
msiexec is a piece of junk that causes these kinds of random problems (see
also uninstall VMware Tools not working).  But also we need to rewrite
firstboot so it can run scripts in sequence and with dependencies between
scripts (bug 1788823) which would help a lot here.
Comment 14 Klaus Heinrich Kiwi 2021-08-11 20:33:25 UTC 
(In reply to Richard W.M. Jones from comment #13)
> msiexec is a piece of junk that causes these kinds of random problems (see
> also uninstall VMware Tools not working).  But also we need to rewrite
> firstboot so it can run scripts in sequence and with dependencies between
> scripts (bug 1788823) which would help a lot here.

Considering the efforts in getting v2v 2.0 on RHEL-9, I'd suggest close WONFIX this one and track the improvement of the use-case described here in Bug 1788823...
Comment 16 John Ferlan 2022-10-27 20:23:32 UTC 
Rich - the stale date is approaching (strange that the bot didn't fire). Do you believe we should extend the stale date or just decide this wont be fixed. I see the bug Klaus mentions above is in Verified, but I'm unclear how that affects what's described here though.
Comment 18 Richard W.M. Jones 2022-10-28 09:23:08 UTC 
Kill stale bug stuff.  We still have to work on scheduling firstboot
scripts on Windows which is a long term effort.
Comment 19 RHEL Program Management 2023-09-22 17:31:05 UTC 
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.
Comment 20 RHEL Program Management 2023-09-22 17:36:12 UTC 
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.
Note You need to log in before you can comment on or make changes to this bug.