Bug 1821168

Summary: [RFE][virtio-win-installer] Bundle virtio-win drivers, spice, qemu and rhv msis into one installer
Product: Red Hat Enterprise Linux 8 Reporter: Gal Zaidman <gzaidman>
Component: virtio-winAssignee: Gal Zaidman <gzaidman>
virtio-win sub component: others QA Contact: xiagao
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: ailan, jinzhao, juzhang, kanderso, lijin, mtessun, vrozenfe
Version: 8.2Keywords: FutureFeature
Target Milestone: rc   
Target Release: 8.2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1821173 (view as bug list) Environment:
Last Closed: 2020-07-21 15:32:52 UTC Type: Feature Request
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1821173    
Bug Blocks:    
Attachments:
Description Flags
log
none
the screenshot after uninstall drivers via tool
none
the screenshot after install driver
none
unkown publisher
none
uninstall failed on win10(2004)
none
insall log in win10
none
uninstall log in win10 none

Description Gal Zaidman 2020-04-06 08:39:02 UTC
Description of problem:
Current state:
- Virtio-win installer has custom logic for installing spice agent and driver, and rhv agent.
- Virtio-win installer doesn't install qemu-ga.
- Virtio-win installer is created for each arch x64,x86.

Problem with current state.
- Customer need to install qemu-ga separately.
- On 8.2.1 spice released there own msi installer for agent and drivers, we need to consume it instead of implementing our own logic

What should be:
One installer to rule them all, an exe bundle of the different msis (virtio-win drivers, spice driver, spice agent, qemu-ga, (optional and disabled by default) rhv agent).

Comment 3 xiagao 2020-06-03 07:28:16 UTC
Created attachment 1694728 [details]
log

Testing updating drivers,but failed, the failed UI and all log are in attachment.
[0970:1244][2020-06-03T06:48:46]e000: Error 0x80070643: Failed to install MSI package.
[0970:1244][2020-06-03T06:48:46]e000: Error 0x80070643: Failed to execute MSI package.
[05E4:0B48][2020-06-03T06:48:46]e000: Error 0x80070643: Failed to configure per-machine MSI package.
[05E4:0B48][2020-06-03T06:48:46]i319: Applied execute package: QxlWddmDod_x86.msi, result: 0x80070643, restart: None
[05E4:0B48][2020-06-03T06:48:46]e000: Error 0x80070643: Failed to execute MSI package.

Comment 4 Gal Zaidman 2020-06-03 07:43:22 UTC
(In reply to xiagao from comment #3)
> Created attachment 1694728 [details]
> log
> 
> Testing updating drivers,but failed, the failed UI and all log are in
> attachment.
> [0970:1244][2020-06-03T06:48:46]e000: Error 0x80070643: Failed to install
> MSI package.
> [0970:1244][2020-06-03T06:48:46]e000: Error 0x80070643: Failed to execute
> MSI package.
> [05E4:0B48][2020-06-03T06:48:46]e000: Error 0x80070643: Failed to configure
> per-machine MSI package.
> [05E4:0B48][2020-06-03T06:48:46]i319: Applied execute package:
> QxlWddmDod_x86.msi, result: 0x80070643, restart: None
> [05E4:0B48][2020-06-03T06:48:46]e000: Error 0x80070643: Failed to execute
> MSI package.


Sorry such a stupid mistake, fixed on:
https://github.com/virtio-win/virtio-win-guest-tools-installer/commit/6bdf73bc92eda3d723cad3703513503c688cbc8e
creating a new build for you now

Comment 7 xiagao 2020-06-04 06:12:43 UTC
Hi Gal,
Confirm 2 things with you.
1) Testing win8-32, pop up "QXL-WDDM-DOD is only supported on Windows 10 or higher systems", is it normal?
2) When do uninstall test via this tool, virtio-blk and virtio-scsi couldn't be uninstalled, right?

BR
xiagao

Comment 8 xiagao 2020-06-04 06:18:00 UTC
(In reply to xiagao from comment #7)
> Hi Gal,
> Confirm 2 things with you.
> 1) Testing win8-32, pop up "QXL-WDDM-DOD is only supported on Windows 10 or
> higher systems", is it normal?

When I click 'ok' button, the virtio-win-guest-tools setup failed.
[0E90:0E7C][2020-06-04T06:07:11]i325: Registering dependency: {7e64b8af-5529-4472-ac91-f48cdf714307} on package provider: {AF6058B7-4D4F-4B3D-B959-4C364B2388A4}, package: virtio_win_gt_x86.msi
[0E90:0E7C][2020-06-04T06:07:11]i323: Registering package dependency provider: {FE63DAB6-40FC-4BB1-8FF0-E3814CA88477}, version: 0.20.0.0, package:QxlWddmDod_x86.msi
[0E90:0E7C][2020-06-04T06:07:11]i301: Applying execute package: QxlWddmDod_x86.msi, action: Install, path: C:\ProgramData\Package Cache\{FE63DAB6-40FC-4BB1-8FF0-E3814CA88477}v0.20.0.0\QxlWddmDod-x86.msi, arguments: ' ARPSYSTEMCOMPONENT="1" MSIFASTINSTALL="7"'
[0E90:0E7C][2020-06-04T06:12:49]e000: Error 0x80070643: Failed to install MSI package.
[0E90:0E7C][2020-06-04T06:12:49]e000: Error 0x80070643: Failed to execute MSI package.
[03E0:0C78][2020-06-04T06:12:49]e000: Error 0x80070643: Failed to configure per-machine MSI package.
[03E0:0C78][2020-06-04T06:12:49]i319: Applied execute package: QxlWddmDod_x86.msi, result: 0x80070643, restart: None
[03E0:0C78][2020-06-04T06:12:49]e000: Error 0x80070643: Failed to execute MSI package.
[0E90:0E7C][2020-06-04T06:12:49]i318: Skipped rollback of package: QxlWddmDod_x86.msi, action: Uninstall, already: Absent
[03E0:0C78][2020-06-04T06:12:49]i319: Applied rollback package: QxlWddmDod_x86.msi, result: 0x0, restart: None
[0E90:0E7C][2020-06-04T06:12:49]i329: Removed package dependency provider: {FE63DAB6-40FC-4BB1-8FF0-E3814CA88477}, package: QxlWddmDod_x86.msi
[0E90:0E7C][2020-06-04T06:12:49]i351: Removing cached package: QxlWddmDod_x86.msi, from path: C:\ProgramData\Package Cache\{FE63DAB6-40FC-4BB1-8FF0-E3814CA88477}v0.20.0.0\
[0E90:0E7C][2020-06-04T06:12:49]i329: Removed package dependency provider: {6F1C8EEF-5815-4616-A352-CF2CA12B2AE1}, package: QxlWddmDod_x64.msi
[0E90:0E7C][2020-06-04T06:12:49]i326: Removed dependency: {7e64b8af-5529-4472-ac91-f48cdf714307} on package provider: {AF6058B7-4D4F-4B3D-B959-4C364B2388A4}, package virtio_win_gt_x86.msi


> 2) When do uninstall test via this tool, virtio-blk and virtio-scsi couldn't
> be uninstalled, right?
> 
> BR
> xiagao

Comment 9 Gal Zaidman 2020-06-04 07:01:10 UTC
(In reply to xiagao from comment #8)
> (In reply to xiagao from comment #7)
> > Hi Gal,
> > Confirm 2 things with you.
> > 1) Testing win8-32, pop up "QXL-WDDM-DOD is only supported on Windows 10 or
> > higher systems", is it normal?
> 
> When I click 'ok' button, the virtio-win-guest-tools setup failed.
> [0E90:0E7C][2020-06-04T06:07:11]i325: Registering dependency:
> {7e64b8af-5529-4472-ac91-f48cdf714307} on package provider:
> {AF6058B7-4D4F-4B3D-B959-4C364B2388A4}, package: virtio_win_gt_x86.msi
> [0E90:0E7C][2020-06-04T06:07:11]i323: Registering package dependency
> provider: {FE63DAB6-40FC-4BB1-8FF0-E3814CA88477}, version: 0.20.0.0,
> package:QxlWddmDod_x86.msi
> [0E90:0E7C][2020-06-04T06:07:11]i301: Applying execute package:
> QxlWddmDod_x86.msi, action: Install, path: C:\ProgramData\Package
> Cache\{FE63DAB6-40FC-4BB1-8FF0-E3814CA88477}v0.20.0.0\QxlWddmDod-x86.msi,
> arguments: ' ARPSYSTEMCOMPONENT="1" MSIFASTINSTALL="7"'
> [0E90:0E7C][2020-06-04T06:12:49]e000: Error 0x80070643: Failed to install
> MSI package.
> [0E90:0E7C][2020-06-04T06:12:49]e000: Error 0x80070643: Failed to execute
> MSI package.
> [03E0:0C78][2020-06-04T06:12:49]e000: Error 0x80070643: Failed to configure
> per-machine MSI package.
> [03E0:0C78][2020-06-04T06:12:49]i319: Applied execute package:
> QxlWddmDod_x86.msi, result: 0x80070643, restart: None
> [03E0:0C78][2020-06-04T06:12:49]e000: Error 0x80070643: Failed to execute
> MSI package.
> [0E90:0E7C][2020-06-04T06:12:49]i318: Skipped rollback of package:
> QxlWddmDod_x86.msi, action: Uninstall, already: Absent
> [03E0:0C78][2020-06-04T06:12:49]i319: Applied rollback package:
> QxlWddmDod_x86.msi, result: 0x0, restart: None
> [0E90:0E7C][2020-06-04T06:12:49]i329: Removed package dependency provider:
> {FE63DAB6-40FC-4BB1-8FF0-E3814CA88477}, package: QxlWddmDod_x86.msi
> [0E90:0E7C][2020-06-04T06:12:49]i351: Removing cached package:
> QxlWddmDod_x86.msi, from path: C:\ProgramData\Package
> Cache\{FE63DAB6-40FC-4BB1-8FF0-E3814CA88477}v0.20.0.0\
> [0E90:0E7C][2020-06-04T06:12:49]i329: Removed package dependency provider:
> {6F1C8EEF-5815-4616-A352-CF2CA12B2AE1}, package: QxlWddmDod_x64.msi
> [0E90:0E7C][2020-06-04T06:12:49]i326: Removed dependency:
> {7e64b8af-5529-4472-ac91-f48cdf714307} on package provider:
> {AF6058B7-4D4F-4B3D-B959-4C364B2388A4}, package virtio_win_gt_x86.msi
> 

That actually makes sense, and it is a real but I need to resolve (tested only on win10 on my side),
QxlWddmDod is a driver that officially supports only windows 10 and we try to install it on all platforms.
I will send a fix for that.

> > 2) When do uninstall test via this tool, virtio-blk and virtio-scsi couldn't
> > be uninstalled, right?

That is right, they are the only drivers that will not get uninstalled

> > 
> > BR
> > xiagao

Comment 12 xiagao 2020-06-05 09:44:21 UTC
Created attachment 1695396 [details]
the screenshot after uninstall drivers via tool

Comment 13 xiagao 2020-06-05 09:45:18 UTC
Created attachment 1695397 [details]
the screenshot after install driver

Comment 16 xiagao 2020-06-08 06:46:59 UTC
Another issue:  the tool's Publisher is unknown now.

Comment 17 Gal Zaidman 2020-06-08 07:01:38 UTC
(In reply to xiagao from comment #16)
> Another issue:  the tool's Publisher is unknown now.

Do you mean the EXE?
How are you checking that?
What is the effect it has?

Comment 19 xiagao 2020-06-08 07:09:24 UTC
(In reply to Gal Zaidman from comment #17)
> (In reply to xiagao from comment #16)
> > Another issue:  the tool's Publisher is unknown now.
> 
> Do you mean the EXE?
> How are you checking that?
> What is the effect it has?

yes, using a none administrator user to login OS, and try to install the tool.exe, will pop up the info(see attachment).

Comment 20 xiagao 2020-06-08 07:10:12 UTC
Created attachment 1696003 [details]
unkown publisher

Comment 21 Gal Zaidman 2020-06-08 07:23:56 UTC
(In reply to xiagao from comment #20)
> Created attachment 1696003 [details]
> unkown publisher

OK started a build with a fix, I will add a link here in another ~30m

Comment 30 xiagao 2020-06-09 02:44:02 UTC
Hi Gal,
I have another thing to confirm with you.
In my test env, only test virtio-win driver, spice, qxl and guest-agent feature. 
For ovirt-guest-agent, I don't know how to test it. 
Can you tell me how to do it or ask ovirt team to cover it?

BR,
xiagao

Comment 31 Gal Zaidman 2020-06-09 06:50:22 UTC
(In reply to xiagao from comment #30)
> Hi Gal,
> I have another thing to confirm with you.
> In my test env, only test virtio-win driver, spice, qxl and guest-agent
> feature. 
> For ovirt-guest-agent, I don't know how to test it. 
> Can you tell me how to do it or ask ovirt team to cover it?
> 
> BR,
> xiagao

No need to test it, that I RHV responsibility (since it is for RHV systems and so on), So you don't need to select it.
Plus installing it on a non-RHV env can cause unexpected behavior that is why it is disabled by default and has a warning.

Comment 32 xiagao 2020-06-09 09:35:53 UTC
Gal, hi
could you reply comment 28?

Comment 33 Gal Zaidman 2020-06-09 10:13:57 UTC
(In reply to xiagao from comment #32)
> Gal, hi
> could you reply comment 28?

So I looked at the log and I think that the system you were working with was not clean,
And I don't know what was done on if before you started the installation, on the logs I see:

1. Some of the packages where already installed:
[03FC:0640][2020-06-08T09:49:12]i100: Detect begin, 10 packages
[03FC:0640][2020-06-08T09:49:12]i102: Detected related bundle: {7b9dbcd6-7df5-4564-9fd6-b1b41e13cfdb}, type: Upgrade, scope: PerMachine, version: 1.9.11.0, operation: None
[03FC:0640][2020-06-08T09:49:12]i103: Detected related package: {F9943750-0EFF-44D1-A5A8-C797662965C0}, scope: PerMachine, version: 0.10.2.0, language: 0 operation: Downgrade
and:
[03FC:0640][2020-06-08T09:49:12]i101: Detected package: QxlWddmDod_x64.msi, state: Present, cached: Complete
[03FC:0640][2020-06-08T09:49:12]i101: Detected package: spice_vdagent_x64.msi, state: Present, cached: Complete
[03FC:0640][2020-06-08T09:49:12]i101: Detected package: qemu_ga_x86_64.msi, state: Present, cached: Complete

2. the error:
[03FC:18EC][2020-06-08T09:49:27]e000: Error 0x80070001: Failed to extract all files from container, erf: 1:2:0
[03FC:06A4][2020-06-08T09:49:27]e000: Error 0x80070001: Failed to wait for operation complete.
[03FC:06A4][2020-06-08T09:49:27]e000: Error 0x80070001: Failed to open container.
[03FC:06A4][2020-06-08T09:49:27]e000: Error 0x80070001: Failed to open container: WixAttachedContainer.
[03FC:06A4][2020-06-08T09:49:27]e312: Failed to extract payloads from container: WixAttachedContainer to working path: C:\Users\xxx\AppData\Local\Temp\{50E2D9CC-3F53-4629-A0E3-4A8F114F7B8B}\1A3D639DC2D82B7B829BFDFF2D0D1F0CEBD316FB, error: 0x80070001.
[03FC:0640][2020-06-08T09:49:27]e000: Error 0x80070001: Failed while caching, aborting execution.
[1050:1DBC][2020-06-08T09:49:27]i372: Session end, registration key: SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{2be1d2f6-0395-4cc4-a793

When I look at: https://wikifixes.com/errors/0x/0x80070001/#:~:text=The%20%220x80070001%22%20error%20is%20commonly,system%20settings%20to%20restore%20stability.
I see:
"The "0x80070001" error is commonly caused by incorrectly configured system settings or irregular entries in the Windows registry."

So I don't know how we got to that stage, I don't think that you just had started a new windows VM and installed the drivers, I assume that this could be caused by:
- Some unsuccessful installtion/uninstalltion on that machine that left something (which should be easy to reproduce if that is the case)
- Maybe you installed/uninstalled previous builds I gave you above on that machine, this can cause problems since the version wasn't changed  in between runs.
- The fact that spice and qemu are detected but no drivers are detected is a warning sign for me, since there are dependencies between them so it is definitely unsupported...


I will advise you to start with a clean env and try to run the tool again, if you remember what was done on that machine and you are able to reproduce than please open a bug on that

Comment 34 Gal Zaidman 2020-06-09 10:14:59 UTC
(In reply to xiagao from comment #32)
> Gal, hi
> could you reply comment 28?

So I looked at the log and I think that the system you were working with was not clean,
And I don't know what was done on if before you started the installation, on the logs I see:

1. Some of the packages where already installed:
[03FC:0640][2020-06-08T09:49:12]i100: Detect begin, 10 packages
[03FC:0640][2020-06-08T09:49:12]i102: Detected related bundle: {7b9dbcd6-7df5-4564-9fd6-b1b41e13cfdb}, type: Upgrade, scope: PerMachine, version: 1.9.11.0, operation: None
[03FC:0640][2020-06-08T09:49:12]i103: Detected related package: {F9943750-0EFF-44D1-A5A8-C797662965C0}, scope: PerMachine, version: 0.10.2.0, language: 0 operation: Downgrade
and:
[03FC:0640][2020-06-08T09:49:12]i101: Detected package: QxlWddmDod_x64.msi, state: Present, cached: Complete
[03FC:0640][2020-06-08T09:49:12]i101: Detected package: spice_vdagent_x64.msi, state: Present, cached: Complete
[03FC:0640][2020-06-08T09:49:12]i101: Detected package: qemu_ga_x86_64.msi, state: Present, cached: Complete

2. the error:
[03FC:18EC][2020-06-08T09:49:27]e000: Error 0x80070001: Failed to extract all files from container, erf: 1:2:0
[03FC:06A4][2020-06-08T09:49:27]e000: Error 0x80070001: Failed to wait for operation complete.
[03FC:06A4][2020-06-08T09:49:27]e000: Error 0x80070001: Failed to open container.
[03FC:06A4][2020-06-08T09:49:27]e000: Error 0x80070001: Failed to open container: WixAttachedContainer.
[03FC:06A4][2020-06-08T09:49:27]e312: Failed to extract payloads from container: WixAttachedContainer to working path: C:\Users\xxx\AppData\Local\Temp\{50E2D9CC-3F53-4629-A0E3-4A8F114F7B8B}\1A3D639DC2D82B7B829BFDFF2D0D1F0CEBD316FB, error: 0x80070001.
[03FC:0640][2020-06-08T09:49:27]e000: Error 0x80070001: Failed while caching, aborting execution.
[1050:1DBC][2020-06-08T09:49:27]i372: Session end, registration key: SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{2be1d2f6-0395-4cc4-a793

When I look at: https://wikifixes.com/errors/0x/0x80070001/#:~:text=The%20%220x80070001%22%20error%20is%20commonly,system%20settings%20to%20restore%20stability.
I see:
"The "0x80070001" error is commonly caused by incorrectly configured system settings or irregular entries in the Windows registry."

So I don't know how we got to that stage, I don't think that you just had started a new windows VM and installed the drivers, I assume that this could be caused by:
- Some unsuccessful installtion/uninstalltion on that machine that left something (which should be easy to reproduce if that is the case)
- Maybe you installed/uninstalled previous builds I gave you above on that machine, this can cause problems since the version wasn't changed  in between runs.
- The fact that spice and qemu are detected but no drivers are detected is a warning sign for me, since there are dependencies between them so it is definitely unsupported...


I will advise you to start with a clean env and try to run the tool again, if you remember what was done on that machine and you are able to reproduce than please open a bug on that

Comment 42 xiagao 2020-06-24 07:04:52 UTC
Created attachment 1698540 [details]
uninstall failed on win10(2004)

(In reply to Gal Zaidman from comment #36)
> New build available:
> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=29423573
> 
> The exe will remain unsigned until we figure out the signature problem.
> 
> OK so to sum up we decided to get this bug verified once the following
> criteria is met:
> 1. When using the installer, install installs all the chosen drivers and
> agents on all OS supported versions.
> 2. When using the installer, uninstall uninstalls all the chosen drivers and
> agents on all OS supported versions.
> 3. Unsupported OS versions fail and gives a clear error message.
> 
> And open the following bugs on the installer for 8.2.1:
> 1. Running the exe, pop up unknown publisher -> exe is not signed.
Filed a new bug: https://bugzilla.redhat.com/show_bug.cgi?id=1847849

> 2. Install drivers via virtio-win tool, then uninstall them via virtio-win
> tool, then install drivers again via tool, found keyboard and mouse didn't
> work.
This issue only happened in Win10(2004),try again with previous virtio-win-guest-tools version,
found the root cause is uninstallation failed(but tool didn't show any error.)
You can refer to the log in attachment.


> 3. Install all drivers via virtio-win tool, then uninstall input driver, try
> to use 'repair' button to install input driver again, it failed.
Filed a new bug: https://bugzilla.redhat.com/show_bug.cgi?id=1845368


> 4. Install/uninstall doesn't always prompt to reboot machine, we need to
> always ask for a reboot to be on the safe side.
Filed a new bug: https://bugzilla.redhat.com/show_bug.cgi?id=1847856


> 
> Please verify that the bugs are real (on the uploaded build) before opening
> them and add all the logs on %TEMP% and screenshots to the bug.

Comment 46 xiagao 2020-06-28 03:34:17 UTC
Created attachment 1699013 [details]
insall log in win10

Comment 47 xiagao 2020-06-28 03:35:30 UTC
Created attachment 1699014 [details]
uninstall log in win10

Comment 53 lijin 2020-06-29 07:25:13 UTC
Hi Martin, GaL

Could you provide devel_ack and pm_ack?

Thanks

Comment 59 xiagao 2020-06-29 08:08:22 UTC
Win2019 hits black screen after do uninstallation test.
Gal is debugging this issue now.

Comment 60 Gal Zaidman 2020-06-29 08:42:28 UTC
(In reply to xiagao from comment #59)
> Win2019 hits black screen after do an uninstallation test.
> Gal is debugging this issue now.

So I looked at the machine and there is a problem with spice qxldod msi.
We will open a bug on qxldod msi, basically what happens is that the virtio-win installer installs spice msis in the process.
When we run uninstall then each msi uninstall method will run, I can see that after a successful installation the display adopter is recognized as qxl and after we uninstall just the qxl msi (meaning not using the bundle uninstall method) then we get a black screen (I assume that no display adopter is recognized), after a reboot, I see that the display adopter is "Microsoft basic display adopter".
We will open bugs on virtio-win for tracking it and on spice for solving, documenting workaround for this issue

Comment 61 xiagao 2020-06-30 02:50:42 UTC
(In reply to Gal Zaidman from comment #60)
> (In reply to xiagao from comment #59)
> > Win2019 hits black screen after do an uninstallation test.
> > Gal is debugging this issue now.
> 
> So I looked at the machine and there is a problem with spice qxldod msi.
> We will open a bug on qxldod msi, basically what happens is that the
> virtio-win installer installs spice msis in the process.
> When we run uninstall then each msi uninstall method will run, I can see
> that after a successful installation the display adopter is recognized as
> qxl and after we uninstall just the qxl msi (meaning not using the bundle
> uninstall method) then we get a black screen (I assume that no display
> adopter is recognized), after a reboot, I see that the display adopter is
> "Microsoft basic display adopter".
> We will open bugs on virtio-win for tracking it and on spice for solving,
> documenting workaround for this issue

yes, it's spice's issue. File a new bug: https://bugzilla.redhat.com/show_bug.cgi?id=1851845

Comment 63 xiagao 2020-06-30 02:57:25 UTC
verified on virtio-win-1.9.12-1.el8.

Comment 65 errata-xmlrpc 2020-07-21 15:32:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:3055

Comment 66 Red Hat Bugzilla 2023-09-14 05:55:05 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days