Created attachment 1769901 [details]
MSFT doc on the various CPUID values

This describes pretty clearly what's expected to come back in the various CPUID values... including 0x40000002:

0x40000002 EAX Build Number
EBX Bits 31-16: Major Version
Bits 15-0: Minor Version
ECX Service Pack
EDX Bits 31-24: Service Branch
Bits 23-0: Service Number

Hm, CPUID 40000002.EAX should not be zero.

When any of Hyper-V emulation features (hv-*) were enabled on the QEMU command line,

CPUID 0x40000002.EAX is supposed to be '0x1bbc'. I just checked this:

# qemu-system-x86_64 -nodefaults -smp 1 -machine q35,accel=kvm -cpu host,hv-relaxed -drive file=/my/linux/vm/image.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -vnc :1 -vga std -m 2G -boot menu=on -serial stdio

....

# cpuid -r -l 0x40000002
CPU 0:
   0x40000002 0x00: eax=0x00001bbc ebx=0x00060001 ecx=0x00000000 edx=0x00000000

If you're seeing '0' in your configuration then something is not right.

Also, you can try 'hv-passthrough' feature in QEMU (instead of all 'hv-*' options). It will
take CPUID values from KVM and it'll be '0x3839' (which is greater than '0x23F0' you're asking
for):

# qemu-system-x86_64 -nodefaults -smp 1 -machine q35,accel=kvm -cpu host,hv-passthrough -drive file=/my/linux/vm/image.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -vnc :1 -vga std -m 2G -boot menu=on -serial stdio

....

# cpuid -r -l 0x40000002
CPU 0:
   0x40000002 0x00: eax=0x00003839 ebx=0x000a0000 ecx=0x00000000 edx=0x00000000

The returned number is more or less 'random', I'm absolutely fine with bumping them if
needed.

Thank you for such a fast and authoritative reply. We greatly appreciate it.

Hmmmm... yes.  Strange.  

We walked-into the KDNET application code yesterday afternoon, using the debugger, running on the Windows guest yesterday on a machine running qemu-kvm on Centos 8, after having this problem on two RHEL 8 systems and one Centos 8 system (all three very recently installed).  We were definitely getting back 0 in EAX in response to CPUID 0x40000002, thus causing the failure I mentioned to be displayed.

We'll do a bit more debugging/investigation.

In the meantime, we're building and running these virtual machines using virt-manager -- So I'm sorry to say it's not so easy (particularly for us, who are not either Linux nor qemu-kvm experts) to change the hv-* parameters and try hv-passthrough.... unless there's a way to do that using virt-manager or virsh that we don't know about (which is entirely possible -- if that's the case, would you please tell us what it is?).  The ability to use hv-passthrough would be really great for us as a work-around.

Thanks again for such a fast and helpful reply.

In case it's of any use, here is the -cpu parameter that's being passed to qemu-kvm, when we see the problem:

-cpu Skylake-Server-IBRS,ss=on,vmx=on,hypervisor=on,tsc-adjust=on,clflushopt=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaves=on,ibpb=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,mpx=off,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff

You should see 0x1bbc in CPUID.0x40000002.EAX with this configuration.

I don't think there's a convenient way to run QEMU with 'hv-passthrough'
CPU option through libvirt, the option is basically intended for debugging,
not for running in production (as it at least blocks migration).

What you can do is copy qemu command line from libvirt logs (or just do 'ps auxwww'
and copy from there) and run it manually. Unfortunately, this is also not very
straightforward as you'll have to drop libvirt-specific things (like fds it opens).

Here is a great example I've found:
https://developers.redhat.com/blog/2020/03/06/configure-and-run-a-qemu-based-vm-outside-of-libvirt/

Alternatively, I can provide you a patch for e.g. upstream QEMU bumping the number
in case you're able to build it. The patch is very simple:

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 6b3e9467f177..8f81964adbcc 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -6672,7 +6672,7 @@ static void x86_cpu_hyperv_realize(X86CPU *cpu)
     cpu->hyperv_interface_id[3] = 0;
 
     /* Hypervisor system identity */
-    cpu->hyperv_version_id[0] = 0x00001bbc;
+    cpu->hyperv_version_id[0] = 0x00003839;
     cpu->hyperv_version_id[1] = 0x00060001;
 
     /* Hypervisor implementation limits */

After you build upstream QEMU with this fix you can change the path to QEMU binary
in your libvirt config.

(An unrelated side-note: you're missing a lot of important Hyper-V enlightenments
like 'hv-synic','hv-stimer', 'hv-tlbflush', ... on the command line and these are
important for performance. Feel free to enable everything from hyperv.txt you've
 mentioned above).

Ah, AGAIN thank you for the quick and helpful reply.

We had racing updates to this bug. I was just writing:

---

Argh, OK, you're right... and I was wrong. The returned value is not zero... it’s just too small.  As you said, we are absolutely getting back 0x1bbc as you noted.  Sorry!

So, KDNET wants the host OS build number to be at least 9200 (0x23F0) but KVM is reporting 7100 (0x1BBC).

---

Yes... we had some "fun" trying to extract the qemu-kvm command line that virt-manager uses, as part of debugging this problem.  It wasn't fun.  Wish I had seen that blog post you pointed me to last week ;-)

We'd DEFINTELY like to try patching the upstream QEMU. Would the right repo to clone be this: https://github.com/qemu/qemu ??

I'll give some thought to filing an RFE for the virt-manager folks to allow people to tweak the hv-* parameters.  Or to shut them off entirely.

I'm sure the patch you provided will work just fine, but we'll make an attempt to build it (remember, we're Windows guys) and try it and report back here.

Can I assume you're going to change this in some official future release?

Many, many, thanks for being so helpful and responsive.  It's really helpful to us.

(In reply to Peter Viscarola from comment #6)
> Ah, AGAIN thank you for the quick and helpful reply.
> 
> We had racing updates to this bug. I was just writing:
> 
> ---
> 
> Argh, OK, you're right... and I was wrong. The returned value is not zero...
> it’s just too small.  As you said, we are absolutely getting back 0x1bbc as
> you noted.  Sorry!
> 
> So, KDNET wants the host OS build number to be at least 9200 (0x23F0) but
> KVM is reporting 7100 (0x1BBC).
> 

Ok, this matches my expectation.

> ---
> 
> Yes... we had some "fun" trying to extract the qemu-kvm command line that
> virt-manager uses, as part of debugging this problem.  It wasn't fun.  Wish
> I had seen that blog post you pointed me to last week ;-)
> 
> We'd DEFINTELY like to try patching the upstream QEMU. Would the right repo
> to clone be this: https://github.com/qemu/qemu ??

The main repo is on gitlab: https://gitlab.com/qemu-project/qemu
but github has the exact copy.

> 
> I'll give some thought to filing an RFE for the virt-manager folks to allow
> people to tweak the hv-* parameters.  Or to shut them off entirely.
> 
> I'm sure the patch you provided will work just fine, but we'll make an
> attempt to build it (remember, we're Windows guys) and try it and report
> back here.
> 
> Can I assume you're going to change this in some official future release?

Sure, we can try targeting 6.1 (too late for 6.0) but it would be great if I
get testing results from you first so we're 100% sure the fix works (as it may
take a while for it to get merged and then backported to RHEL).

I managed to build a new version of qemu, and test it.

I'm getting back 0x3839 in EAX in response to CPUID 0x40000002.

Yay!

The Windows kernel-mode debugger for network transport (KDNET, mentioned previously) now happily reports:

"Network debugging is supported by this Microsoft Hypervisor Virtual Machine."

BUT... we're still not (yet) able to get kernel debugging working via the network transport. This, despite the fact that we're passing in a dedicated NIC to the VM... and we know this NIC is supported (it works when we do not enable any enlightenments).

So, it seems the change you've made with the return value of EAX is necessary, but perhaps not sufficient, to allow us to use kernel debugging via the network, on a Windows guest.

We're going to continue to debug this, and we'll let you know what we find.

To make things even more complicated, I'll be on holiday next week.  So it might be a while before I can get you any more feedback.

Again, we are very grateful for your help on this problem.  I know small and esoteric one.  But we'll be very happy if we can eventually debug Windows drivers on guest systems where Enlightenments are enabled.

Thanks again.

I don't know much about KDNET myself but I recall a special "synthetic debugger
interface" was added to KVM some time ago:

https://lore.kernel.org/kvm/20200529134543.1127440-1-arilou@gmail.com/

Maybe it is related to what you're trying to achieve? Unfortunately, I don't see
any traces of this interface in upstream QEMU and it is certainly needed. I'd
suggest you reach out to the author of the patchset I mention above.

Hi Peter,

can you try the following cpu option hv-vendor-id=KVMKVMKVM ?

Thanks,
Vadim.

Once again, many thanks for your help and interest.  We really do appreciate it.

Vitaly: Thanks for the pointer to that patch set.  Interesting.  I sent the author an email.

Vadim:  I tried changing the hv-vendor-id as you suggested, as you can see:

C:\PGV>cpuid
0x40000000:
EAX: 0x40000005
EBX: 0x4b4d564b
ECX: 0x564b4d56
EDX: 0x0000004d

0x40000001:
EAX: 0x31237648
EBX: 0x00000000
ECX: 0x00000000
EDX: 0x00000000

0x40000002:
EAX: 0x00003839
EBX: 0x00060001
ECX: 0x00000000
EDX: 0x00000000

Unfortunately that also did not allow the Windows kernel debugger network transport to work.

I appreciate the suggestion.

I'm going to be on holiday for the next week so, while our interest in solving the larger issue remains high, I won't be able to DO anything much for the next week or so.

I *did* want to add that the change to CPUID leaf 0x40000002 is a good change, regardless of whether KDnet works or not... so I encourage you to make this change in any case.  The value that's being returned now doesn't correspond to any actual version of Windows.

Thanks again!

Peter

Is there any specific hardware requirement to reproduce this issue? 

We can enable kernel debug in win2019 successfully in our svvp testing.
Adding 'hv-vendor-id=KVMKVMKVM' flag works for us(https://bugzilla.redhat.com/show_bug.cgi?id=1766061#c48)

Hi,

We can pass the "Debug Capability Test" job on ws2019(boot sut with 'hv-vendor-id=KVMKVMKVM' and transport is net). 

I'm not sure if it is related to os version, could you update your guest os version, or try with ws2019 instead?

More info about "Debug Capability Test"
https://docs.microsoft.com/en-us/windows-hardware/test/hlk/testref/83974a17-ec9f-4996-a0d6-5873bf1eb3ac

Thanks
Yu Wang

Hi

Tried with 3 scenarios:

1 the same flag with #comment4

Failed at "Verify Debug Configuration"
details:
System.Exception: Kernel debugging over the network fails due to NIC hardware initialization failed. error code -1073741438.Please address the problem or run the test from another debug connection.
   at KernelDebugLogoTestSetup.DebugSetupVerifier.DebugSetupVerify()


2 flags with #comment4 + "hv-vendor-id=KVMKVMKVM"

Failed at "Run debug capability test"
Details
The test timed out waiting for the kernel debugger to break in.

3 flags with "-cpu Skylake-Server,hv_stimer,hv_synic,hv_time,hv_vpindex,hv_relaxed,hv_spinlocks=0xfff,hv_vapic,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,hv-vendor-id=KVMKVMKVM"

Pass this job(this is our svvp scenario)

Could you try with flags in scenario 3 instead?

Thanks
Yu Wang

Thanks so very much for your work on attention to this issue. 

I’m sorry that I’m out of the office (on Holiday) until mid next week, so I can’t try your suggestion. 

I will try it as soon as I’m back. And thanks again for your help on this issue.

Additional info:


1 boot with flags as below:
-cpu Skylake-Server-IBRS,ss=on,vmx=on,hypervisor=on,tsc-adjust=on,clflushopt=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaves=on,ibpb=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,mpx=off,hv_stimer,hv_synic,hv_time,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,hv-vendor-id=KVMKVMKVM

or
-cpu Skylake-Server-IBRS,ss=on,vmx=on,hypervisor=on,tsc-adjust=on,clflushopt=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaves=on,ibpb=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,mpx=off,hv_time,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv-vendor-id=KVMKVMKVM


Pass this job.

2 boot without flag "hv-vendor-id=KVMKVMKVM"

Failed this job as:
System.Exception: Kernel debugging over the network fails due to NIC hardware initialization failed. error code -1073741438.Please address the problem or run the test from another debug connection.
   at KernelDebugLogoTestSetup.DebugSetupVerifier.DebugSetupVerify()

Hi Folks,

I am *extremely* happy to report that following Mr. Wang's suggestion, using the following exact CPU flags:

-cpu Skylake-Server-IBRS,ss=on,vmx=on,hypervisor=on,tsc-adjust=on,clflushopt=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaves=on,ibpb=on,amd-ssbd=on, \
skip-l1dfl-vmentry=on,pschange-mc-no=on,mpx=off,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=KVMKVMKVM

Does make the Windows guest KDNet network transport WORK CORRECTLY!

YAY!

This is very exciting, and we are *very* grateful for the help you all have provided on this issue.

FYI: I have also contacted the Windows kernel-mode debugger team and asked them to provide an option to "force" the debugger to NOT use the Enlightenments and fall-back to its fully virtualized behavior. I think that THIS (MSFT fixing the debugger) is the proper solution.  It shouldn't be up to qemu to guess how to fool Windows guest systems into proper behavior.

Again, many thanks to you both.  My team and I are very grateful for your help.

(In reply to Peter Viscarola from comment #17)
> Hi Folks,
> 
> I am *extremely* happy to report that following Mr. Wang's suggestion, using
> the following exact CPU flags:
> 
> -cpu
> Skylake-Server-IBRS,ss=on,vmx=on,hypervisor=on,tsc-adjust=on,clflushopt=on,
> umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaves=on,
> ibpb=on,amd-ssbd=on, \
> skip-l1dfl-vmentry=on,pschange-mc-no=on,mpx=off,hv-time,hv-relaxed,hv-vapic,
> hv-spinlocks=0x1fff,hv-vendor-id=KVMKVMKVM
> 
> Does make the Windows guest KDNet network transport WORK CORRECTLY!
> 
> YAY!
> 
> This is very exciting, and we are *very* grateful for the help you all have
> provided on this issue.
> 
> FYI: I have also contacted the Windows kernel-mode debugger team and asked
> them to provide an option to "force" the debugger to NOT use the
> Enlightenments and fall-back to its fully virtualized behavior. I think that
> THIS (MSFT fixing the debugger) is the proper solution.  It shouldn't be up
> to qemu to guess how to fool Windows guest systems into proper behavior.
> 
> Again, many thanks to you both.  My team and I are very grateful for your
> help.

Glad to hear it works for you.
May I close it as Not-a-bug? We're happy to provide any support if you need.

Well... The value that's currently being returned in EAX for CPUID 0x40000002 ISN'T valid as released.  Changing it to 0x00003839 at least makes it valid (and changes the error reported by KDNet to something more sensible than "upgrade your hypervisor"), even if it doesn't solve the KDNet problem.

Given the fix is such a low impact, and has almost zero risk, I'd suggest this change be made.

Does that make sense?

And, again:  Thank you all for your help.  We really appreciate it.  I wish getting the Windows team to help was this easy ;-)

(In reply to Peter Viscarola from comment #19)
> Well... The value that's currently being returned in EAX for CPUID
> 0x40000002 ISN'T valid as released.  Changing it to 0x00003839 at least
> makes it valid (and changes the error reported by KDNet to something more
> sensible than "upgrade your hypervisor"), even if it doesn't solve the KDNet
> problem.
> 
> Given the fix is such a low impact, and has almost zero risk, I'd suggest
> this change be made.
> 
> Does that make sense?

Yes, it makes sense and the fix should be easy, so let's keep it.
Thanks.

> And, again:  Thank you all for your help.  We really appreciate it.  I wish
> getting the Windows team to help was this easy ;-)

I completely agree with Peter.

From the beginning we keep HYPERV_CPUID_VERSION value hard-coded as ox1bbc, which is corresponding to Windows Server 2008 R2 (build 7100)
https://patchwork.ozlabs.org/project/qemu-devel/patch/1318843022-20344-3-git-send-email-vrozenfe@redhat.com/

Introducing a new QEMU hyper-v related parameter, like "hv-version-id=ws2012|ws2016|ws2019|ws2022" ,
or even more flexible triple as "hv-version-id-build=0xXXXX,hv-version-id-major=0xYYY,hv-version-id-minor=0xZZZZ"  
can be a quite useful addition.  

Best,
Vadim.

Hi Vadim,

Is there any tool to check cpuid in windows?(Since I cannot find a tool to check this).

If we change the HYPERV_CPUID_VERSION value, how to verify this bug?

Thanks
Yu Wang

I’m probably “the Windows guy” for the purposes of this thread.  And, no, there IS no Windows utility that can be used to display CPUID.

There **is** a _cpuid “intrinsic” in C/C++, so it’s super easy to write a program to display the results of any CPUID leaf...  I’ll be happy to provide an example, if that would be helpful.

Peter

(In reply to Peter Viscarola from comment #23)
> I’m probably “the Windows guy” for the purposes of this thread.  And, no,
> there IS no Windows utility that can be used to display CPUID.
> 
> There **is** a _cpuid “intrinsic” in C/C++, so it’s super easy to write a
> program to display the results of any CPUID leaf...  I’ll be happy to
> provide an example, if that would be helpful.

Many thanks! I programmed a tool for checking CPUID just now. I will attach
it for reference. Thanks for your positive response!

Yu Wang

> 
> Peter

Created attachment 1775386 [details]
cpuid_check_tool

Hi,

I can reproduce this bug with different flags(with the attached tool):

1 boot with no flags:
    EAX: 0x00000000
 
2 boot with "hv-passthrough"
    EAX:0x00003839

3 boot with "hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi"
    EAX:0x00001bbc


Thanks
Yu Wang

It is a HYPERV_CPUID_VERSION value problem.
So we will not add to our test plan, verify this bug is enough.
change qe-test-coverage to "-"

Thanks
Yu Wang

Has anyone gotten a Windows10 (Version 1709) kernel debugger attached when running the Windows10 debugger VM and the Windows10 debugee VM on Xen 4.13.0 hypervisor? I am getting the same "NIC hardware initialization failed". Xen also uses QEMU internally, so I tried the arguments that worked for you all, but they did not work for me:
-cpu Skylake-Server-IBRS,ss=on,vmx=on,hypervisor=on,tsc-adjust=on,clflushopt=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaves=on,ibpb=on,amd-ssbd=on, \
skip-l1dfl-vmentry=on,mpx=off,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vendor-id=KVMKVMKVM

note: i had to remove the following 2 arguments due to errors from QEMU:
pschange-mc-no=on
hv_vpindex

Here was the error:
C:\Users\user\Desktop\oldDebuggers\x64>kdnet.exe

Network debugging is supported on the following NICs:
busparams=0.4.0, Intel(R) PRO/1000 MT Network Connection, Plugged in.
The Microsoft hypervisor running this VM does not support KDNET.
Please upgrade to the hypervisor shipped in Windows 8 or WS2012 or later.

KDNET initialization failed.  Status = 0xC0000182.
NIC hardware initialization failed.

I am using an Intel e1000 NIC emulated through QEMU because its supposedly a supported NIC for Windows kernel NET debugging.

Thanks in Advance!

I figured it out. Microsoft did not document that testsigning needs to be enabled for kdnet to work.

FWIW, I've sent a patch upstream which makes it possible to set any desired Hyper-V version:
https://lists.nongnu.org/archive/html/qemu-devel/2021-09/msg00453.html

I'm also suggesting to change the default to WS2016:
https://lists.nongnu.org/archive/html/qemu-devel/2021-09/msg00454.html

Bulk update: Move RHEL8 bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.

Patches were merged upstream, QEMU-6.2 will bring the following changes:

1) 6.2 machine types will default to WS2016 identification (hv-version-id-build=0x3839, hv-version-id-major=0x000A,hv-version-id-minor=0x0000)

2) It becomes possible to specify any HV identification:
 -cpu ...,hv-version-id-build=0xXXXX,
          hv-version-id-major=0xXXXX,
          hv-version-id-minor=0xXXXX,
          hv-version-id-spack=0xXXXX,
          hv-version-id-sbranch=0xXXXX,
          hv-version-id-snumber=0xXXX

https://bugzilla.redhat.com/show_bug.cgi?id=2027697 is ON_QA, so is
https://bugzilla.redhat.com/show_bug.cgi?id=2022604. Moving this too.

Base on  bz1945666 ON_QA status now, test 2 following scenarios:

package:
qemu-kvm-6.2.0-5.el9.x86_64
kernel-5.14.0-47.el9.x86_64

scenario 1:
  1) start a linux guest with following command:

  -cpu 'Skylake-Server',hv_frequencies \

  2) run command # cpuid -r -1 -l 0x40000002

result:  0x40000002 0x00: eax=0x00003839 ebx=0x000a0000 ecx=0x00000000 edx=0x00000000 (expected result)


scenario 2:

  1) start a linux guest with following command:

  -cpu 'Skylake-Server',hv_frequencies,hv-version-id-build=0x8181,hv-version-id-major=0x000A,hv-version-id-minor=0x00,hv-version-id-spack=0x550,hv-version-id-sbranch=0x0000,hv-version-id-snumber=0x00 \

  2) run command  # cpuid -r -1 -l 0x40000002

[root@vm-73-180 ~]# cpuid -r -1 -l 0x40000002
result:
CPU:
   0x40000002 0x00: eax=0x00008181 ebx=0x000a0000 ecx=0x00000550 edx=0x00000000 (expected result)


hi Vitaly ,

Base on above results, could we move this bz to verify status, or any other suggestions (mainly to confirm my testing is valid and enough to verify this bz)?




Thanks
Menghuan

(In reply to menli from comment #41)
> 
> Base on above results, could we move this bz to verify status, or any other
> suggestions (mainly to confirm my testing is valid and enough to verify this
> bz)?
> 

Both cases look good to me so I think we can go to VERIFIED.

Thanks!

Thanks Vitaly,
Base on comment 41 and coment 42, change status to verified.


Thanks
Menghuan