Bug 1972056

Summary: [RFE] Add support for fwcfg64 driver - 64 bits
Product: Red Hat Enterprise Linux 9 Reporter: Meirav Dean <mdean>
Component: virtio-winAssignee: Vadim Rozenfeld <vrozenfe>
virtio-win sub component: virtio-win-prewhql QA Contact: leidwang <leidwang>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: unspecified    
Priority: unspecified CC: jinzhao, juzhang, lijin, phou, qizhu, vrozenfe, yvugenfi
Version: 9.0Keywords: FutureFeature, Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1983947 (view as bug list) Environment:
Last Closed: 2022-11-15 06:47:32 UTC Type: Feature Request
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1986665    
Bug Blocks: 1983947    

Description Meirav Dean 2021-06-15 07:49:44 UTC
Description of problem:

The fwcfg64 driver builds the crash dump header. It uses a qemu device that is in icpi space. 

This driver helps us to get the crash dump from the guest snapshot: In the guest we build the header of the crash dump in run time. The driver passes it from the guest to the host. The host saves the dump and when we do guest mem dump we can add the physical raw mem to the header and a small stat.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Peixiu Hou 2021-06-16 13:41:51 UTC
Hi Meirav,

Thanks you file issue out, I have a few basic questions about this new feature:

1) How can I connect the fwcfg64 device to a vm through qemu command line(I tried this command "-fw_cfg name=opt/com.name.domain.your.example,string=1", but booted up vm failed, it seems it's wrong)? 
2) And in the vm, how to check the device's function?
3) If this device support hotpug/unplug?
4) When we do vm migration test, if will touch this driver's codes?

Thanks a lot~
Peixiu

Comment 3 Vadim Rozenfeld 2021-06-22 13:39:05 UTC
(In reply to Peixiu Hou from comment #2)
> Hi Meirav,
> 
> Thanks you file issue out, I have a few basic questions about this new
> feature:
> 
> 1) How can I connect the fwcfg64 device to a vm through qemu command line(I
> tried this command "-fw_cfg name=opt/com.name.domain.your.example,string=1",
> but booted up vm failed, it seems it's wrong)? 
> 2) And in the vm, how to check the device's function?
> 3) If this device support hotpug/unplug?
> 4) When we do vm migration test, if will touch this driver's codes?
> 
> Thanks a lot~
> Peixiu

Hi Peixiu,
fwcfg64 is a quite new feature. 
Give me some time to find the answers.

Best regards,
Vadim.

Comment 4 Vadim Rozenfeld 2021-06-23 06:25:42 UTC
(In reply to Vadim Rozenfeld from comment #3)
> (In reply to Peixiu Hou from comment #2)
> > Hi Meirav,
> > 
> > Thanks you file issue out, I have a few basic questions about this new
> > feature:
> > 
> > 1) How can I connect the fwcfg64 device to a vm through qemu command line(I
> > tried this command "-fw_cfg name=opt/com.name.domain.your.example,string=1",
> > but booted up vm failed, it seems it's wrong)? 

Run VM with "-device=vmcoreinfo" option (currently only x64 version seems to be working 
properly even thoug that x86 version is also included ito the build)


> > 2) And in the vm, how to check the device's function?
Inside of VM install fwcfgdmp.sys driver ( I use devcon for this) 
After installation you should see QEMU FwCfg Device" under "System Devices" category in
Device Manager Dialog.
Next, from the qemu monitor run "dump-guest-memory -w memory.dmp". Upon completion this 
command you should be able to open memory.dmp file with WinDbg as a normal crash dump file,
generated during BSOD processing.

> > 3) If this device support hotpug/unplug?
I'm quite sure that it doesn't. This device is in ACPI space. 

> > 4) When we do vm migration test, if will touch this driver's codes?

Migration should work AFAIU, hoverer it is not possible to create a memory dump during migration. 

> > 
> > Thanks a lot~
> > Peixiu
> 
> Hi Peixiu,
> fwcfg64 is a quite new feature. 
> Give me some time to find the answers.
> 
> Best regards,
> Vadim.

Comment 5 Peixiu Hou 2021-06-27 03:52:47 UTC
(In reply to Vadim Rozenfeld from comment #4)
> (In reply to Vadim Rozenfeld from comment #3)
> > (In reply to Peixiu Hou from comment #2)
> > > Hi Meirav,
> > > 
> > > Thanks you file issue out, I have a few basic questions about this new
> > > feature:
> > > 
> > > 1) How can I connect the fwcfg64 device to a vm through qemu command line(I
> > > tried this command "-fw_cfg name=opt/com.name.domain.your.example,string=1",
> > > but booted up vm failed, it seems it's wrong)? 
> 
> Run VM with "-device=vmcoreinfo" option (currently only x64 version seems to
> be working 
> properly even thoug that x86 version is also included ito the build)
> 
> 
> > > 2) And in the vm, how to check the device's function?
> Inside of VM install fwcfgdmp.sys driver ( I use devcon for this) 
> After installation you should see QEMU FwCfg Device" under "System Devices"
> category in
> Device Manager Dialog.
> Next, from the qemu monitor run "dump-guest-memory -w memory.dmp". Upon
> completion this 
> command you should be able to open memory.dmp file with WinDbg as a normal
> crash dump file,
> generated during BSOD processing.
> 

Hi Vadim,

Thanks a lot for your answer, I tested on win10-64 and win2019 guest vm, the device works well.

Used version:
RHEL9:
kernel-5.13.0-0.rc7.51.el9.x86_64
qemu-kvm-6.0.0-2.el9.x86_64
seabios-bin-1.14.0-4.el9.noarch
virtio-win-prewhql-202

RHEL8.5.0:
kernel-4.18.0-316.el8.x86_64
qemu-kvm-6.0.50-20.scrmod+el8.5.0+11566+2ca95551.wrb210623.x86_64
seabios-1.14.0-1.scrmod+el8.5.0+11566+2ca95551.x86_64
virtio-win-prewhql-202

And I also have some another questions want to confirm:
Q1: It seems this feature only support on windows guest? could you help to confirm it? thanks~

Q2: Yes, I tried test on win8.1-32 guest, the fwcfg driver can be installed successfully, but command dump-guest-memory hit error message as follows:
 (qemu) dump-guest-memory -w mem-8132.dmp
 Error: win-dump: invalid header, expected 'DU64', got 'DUMP'
If this feature will support on both windows x86_64 and windows i386 guest os in the future? 

Q3: I plan follows tests for this feature, could you help to review it?
1) Whql tests with "QEMU FwCfg Device".

2) Function test with fwcfg64 driver:
1. Do basic function test with fwcfg64 driver.
   step1: Boot a guest with "-device=vmcoreinfo" option
   step2: Install fwcfg64 driver in guest vm
   step3: Trigger a BSOD through "nmi"
   step4: Run "dump-guest-memory -w memory.dmp" in qemu monitor
   step5: Check the memory.dmp can be open with windb tools
2. Do system reboot/ system shutdown test with fwcfg device.
3. Do fwcfg driver install/uninstall test.
4. Use signtool.exe to check whether the driver is digital signed 

Thanks a lot~
Peixiu

Comment 6 Vadim Rozenfeld 2021-06-27 05:13:41 UTC
(In reply to Peixiu Hou from comment #5)
> (In reply to Vadim Rozenfeld from comment #4)
> > (In reply to Vadim Rozenfeld from comment #3)
> > > (In reply to Peixiu Hou from comment #2)
> > > > Hi Meirav,
> > > > 
> > > > Thanks you file issue out, I have a few basic questions about this new
> > > > feature:
> > > > 
> > > > 1) How can I connect the fwcfg64 device to a vm through qemu command line(I
> > > > tried this command "-fw_cfg name=opt/com.name.domain.your.example,string=1",
> > > > but booted up vm failed, it seems it's wrong)? 
> > 
> > Run VM with "-device=vmcoreinfo" option (currently only x64 version seems to
> > be working 
> > properly even thoug that x86 version is also included ito the build)
> > 
> > 
> > > > 2) And in the vm, how to check the device's function?
> > Inside of VM install fwcfgdmp.sys driver ( I use devcon for this) 
> > After installation you should see QEMU FwCfg Device" under "System Devices"
> > category in
> > Device Manager Dialog.
> > Next, from the qemu monitor run "dump-guest-memory -w memory.dmp". Upon
> > completion this 
> > command you should be able to open memory.dmp file with WinDbg as a normal
> > crash dump file,
> > generated during BSOD processing.
> > 
> 
> Hi Vadim,
> 
> Thanks a lot for your answer, I tested on win10-64 and win2019 guest vm, the
> device works well.
> 
> Used version:
> RHEL9:
> kernel-5.13.0-0.rc7.51.el9.x86_64
> qemu-kvm-6.0.0-2.el9.x86_64
> seabios-bin-1.14.0-4.el9.noarch
> virtio-win-prewhql-202
> 
> RHEL8.5.0:
> kernel-4.18.0-316.el8.x86_64
> qemu-kvm-6.0.50-20.scrmod+el8.5.0+11566+2ca95551.wrb210623.x86_64
> seabios-1.14.0-1.scrmod+el8.5.0+11566+2ca95551.x86_64
> virtio-win-prewhql-202
> 
> And I also have some another questions want to confirm:
> Q1: It seems this feature only support on windows guest? could you help to
> confirm it? thanks~
> 
> Q2: Yes, I tried test on win8.1-32 guest, the fwcfg driver can be installed
> successfully, but command dump-guest-memory hit error message as follows:
>  (qemu) dump-guest-memory -w mem-8132.dmp

"-w" is for windows windbg 
dump-guest-memory command should work for linux guests but it is out the scope of
this bug.

>  Error: win-dump: invalid header, expected 'DU64', got 'DUMP'
> If this feature will support on both windows x86_64 and windows i386 guest
> os in the future? 
> 
That's true. 32-bit quests are not supported yet, and we need to clarify 
if they need to be supported, because adding support for 32-bit quest will
require some work from the quest driver side and may require some changes at 
qemu side.


> Q3: I plan follows tests for this feature, could you help to review it?
> 1) Whql tests with "QEMU FwCfg Device".
> 
> 2) Function test with fwcfg64 driver:
> 1. Do basic function test with fwcfg64 driver.
>    step1: Boot a guest with "-device=vmcoreinfo" option
>    step2: Install fwcfg64 driver in guest vm
steps 1 and 2 age good
>    step3: Trigger a BSOD through "nmi"

Do you need "nmi" here to get a proper dump file?
I was able to generated a valid Windbg dump file even
without this step. This is the most interesting and useful
scenario since it allows to generate dump file on a working
system for the future forensic data analysis.  

>    step4: Run "dump-guest-memory -w memory.dmp" in qemu monitor
>    step5: Check the memory.dmp can be open with windb tools
> 2. Do system reboot/ system shutdown test with fwcfg device.
> 3. Do fwcfg driver install/uninstall test.
> 4. Use signtool.exe to check whether the driver is digital signed 

All the rest is good.

Best regards,
Vadim.

> 
> Thanks a lot~
> Peixiu

Comment 7 Peixiu Hou 2021-06-28 10:35:20 UTC
(In reply to Vadim Rozenfeld from comment #6)
> (In reply to Peixiu Hou from comment #5)
> > (In reply to Vadim Rozenfeld from comment #4)
> > > (In reply to Vadim Rozenfeld from comment #3)
> > > > (In reply to Peixiu Hou from comment #2)
> > > > > Hi Meirav,
> > > > > 
> > > > > Thanks you file issue out, I have a few basic questions about this new
> > > > > feature:
> > > > > 
> > > > > 1) How can I connect the fwcfg64 device to a vm through qemu command line(I
> > > > > tried this command "-fw_cfg name=opt/com.name.domain.your.example,string=1",
> > > > > but booted up vm failed, it seems it's wrong)? 
> > > 
> > > Run VM with "-device=vmcoreinfo" option (currently only x64 version seems to
> > > be working 
> > > properly even thoug that x86 version is also included ito the build)
> > > 
> > > 
> > > > > 2) And in the vm, how to check the device's function?
> > > Inside of VM install fwcfgdmp.sys driver ( I use devcon for this) 
> > > After installation you should see QEMU FwCfg Device" under "System Devices"
> > > category in
> > > Device Manager Dialog.
> > > Next, from the qemu monitor run "dump-guest-memory -w memory.dmp". Upon
> > > completion this 
> > > command you should be able to open memory.dmp file with WinDbg as a normal
> > > crash dump file,
> > > generated during BSOD processing.
> > > 
> > 
> > Hi Vadim,
> > 
> > Thanks a lot for your answer, I tested on win10-64 and win2019 guest vm, the
> > device works well.
> > 
> > Used version:
> > RHEL9:
> > kernel-5.13.0-0.rc7.51.el9.x86_64
> > qemu-kvm-6.0.0-2.el9.x86_64
> > seabios-bin-1.14.0-4.el9.noarch
> > virtio-win-prewhql-202
> > 
> > RHEL8.5.0:
> > kernel-4.18.0-316.el8.x86_64
> > qemu-kvm-6.0.50-20.scrmod+el8.5.0+11566+2ca95551.wrb210623.x86_64
> > seabios-1.14.0-1.scrmod+el8.5.0+11566+2ca95551.x86_64
> > virtio-win-prewhql-202
> > 
> > And I also have some another questions want to confirm:
> > Q1: It seems this feature only support on windows guest? could you help to
> > confirm it? thanks~
> > 
> > Q2: Yes, I tried test on win8.1-32 guest, the fwcfg driver can be installed
> > successfully, but command dump-guest-memory hit error message as follows:
> >  (qemu) dump-guest-memory -w mem-8132.dmp
> 
> "-w" is for windows windbg 
> dump-guest-memory command should work for linux guests but it is out the
> scope of
> this bug.
> 
> >  Error: win-dump: invalid header, expected 'DU64', got 'DUMP'
> > If this feature will support on both windows x86_64 and windows i386 guest
> > os in the future? 
> > 
> That's true. 32-bit quests are not supported yet, and we need to clarify 
> if they need to be supported, because adding support for 32-bit quest will
> require some work from the quest driver side and may require some changes at 
> qemu side.
> 
> 
> > Q3: I plan follows tests for this feature, could you help to review it?
> > 1) Whql tests with "QEMU FwCfg Device".
> > 
> > 2) Function test with fwcfg64 driver:
> > 1. Do basic function test with fwcfg64 driver.
> >    step1: Boot a guest with "-device=vmcoreinfo" option
> >    step2: Install fwcfg64 driver in guest vm
> steps 1 and 2 age good
> >    step3: Trigger a BSOD through "nmi"
> 
> Do you need "nmi" here to get a proper dump file?
> I was able to generated a valid Windbg dump file even
> without this step. This is the most interesting and useful
> scenario since it allows to generate dump file on a working
> system for the future forensic data analysis.  
> 

I got it, I'll delete this step for the case, thank you for your analysis~ 

And I drafted RHEL8 - Fwcfg64_Test_Plan as attachment, could you help to review it? and any else do you want to add or modify?

Thanks a lot~
peixiu

> >    step4: Run "dump-guest-memory -w memory.dmp" in qemu monitor
> >    step5: Check the memory.dmp can be open with windb tools
> > 2. Do system reboot/ system shutdown test with fwcfg device.
> > 3. Do fwcfg driver install/uninstall test.
> > 4. Use signtool.exe to check whether the driver is digital signed 
> 
> All the rest is good.
> 
> Best regards,
> Vadim.
> 
> > 
> > Thanks a lot~
> > Peixiu

Comment 16 leidwang@redhat.com 2022-08-11 01:43:28 UTC
Passed whql test on Win11,Win2022,Win2019,Win2016,Win10-64,Win2012,Win2012-R2.

Set the bug to verified.Thanks