(In reply to YongkuiGuo from comment #0)

> Steps:
> 
> 1. Create a VM on OpenStack env with the latest rhel9.3 nightly compose
> 2. Run libguestfs-test-tool
> 
> [...]
> 
> Expected results:
> libguestfs-test-tool works.

No, libguestfs has never been expected to work in L2, in a nested virtualization setup (when L1 is a KVM guest and L2, i.e. the libguestfs appliance, is also a KVM guest). Such attempts have always been defeated by non-deterministic nested virtualization bugs.

If you insist you can reassign to kernel/KVM. It's a problem that's potentially triggered by specifying "-cpu max" on the qemu cmdline that is invoked in L1, while the L0 CPU is Icelake.

Interestingly this is 16/32 bit code:

0:  04 66                   add    al,0x66
2:  41                      inc    ecx
3:  eb f1                   jmp    0xfffffff6
5:  66 83 c9 ff             or     cx,0xffff
9:  66 89 c8                mov    ax,cx
c:  66 5b                   pop    bx
e:  66 5e                   pop    si
10: 66 5f                   pop    di
12: 66 c3                   retw
14: ea 5b e0 00 f0 30 36    jmp    0x3630:0xf000e05b
1b: 2f                      das
1c: 32 33                   xor    dh,BYTE PTR [ebx]
1e: 2f                      das
1f: 39 39                   cmp    DWORD PTR [ecx],edi
21: 00 fc                   add    ah,bh

> No, libguestfs has never been expected to work in L2

My reading is that this would be an L1 guest, with the libguestfs appliance running as L2,
which is something we do try to support (albeit, as you say, subject to various and multiple
nested virtualization bugs).

(In reply to Richard W.M. Jones from comment #2)
> Interestingly this is 16/32 bit code:
> 
> 14: ea 5b e0 00 f0 30 36    jmp    0x3630:0xf000e05b

Hm. This raises vague memories.

According to the Intel SDM, this seems like "JMP ptr16:32", "Jump far, absolute, address given in operand"; valid in compat/legacy mode, invalid in 64-bit mode.

This reminds me of early Linux boot code that performs the CPU mode switches, for entering long mode.

Something something about "unrestricted guest" support. This kind of jump had been very problematic in OVMF on KVM years ago (OVMF switches CPU modes), kept triggering KVM emulation failures. I don't remember the details.

Check "/sys/module/kvm_intel/parameters/unrestricted_guest" perhaps? If the processor lacks unrestricted guest support, I expect it will just not work; otherwise it should work fine. Seems like a pretty modern PCPU, so I'm unsure.

> Check "/sys/module/kvm_intel/parameters/unrestricted_guest" perhaps?

We don't have access to the L0 host (and unlikely to be able to get access).  However
in L1 where we run libguestfs that file is:

$ cat /sys/module/kvm_intel/parameters/unrestricted_guest
Y

Created attachment 1966861 [details]
qemu command

Further information:

TCG works (not surprising).  Another indication that it's a problem with nested virt.

The qemu command is attached.

Confusing:

(In reply to YongkuiGuo from comment #0)

> EIP=0000fff0 [...]
> [...]
> CS =f000 ffff0000 0000ffff 00009b00

this points to the reset vector where the BIOS is supposed to start... I wouldn't expect a far jump there (to a different code segment), before setting up segment descriptors. I'm unsure how reliable this register dump from QEMU is.

Also same error when running qemu directly:

/usr/libexec/qemu-kvm \
    -global virtio-blk-pci.scsi=off \
    -no-user-config \
    -nodefaults \
    -display none \
    -machine q35,accel=kvm:tcg,graphics=off \
    -cpu max,la57=off \
    -m 1280 \
    -no-reboot \
    -rtc driftfix=slew \
    -no-hpet \
    -global kvm-pit.lost_tick_policy=discard \
    -kernel /var/tmp/.guestfs-1000/appliance.d/kernel \
    -initrd /var/tmp/.guestfs-1000/appliance.d/initrd \
    -object rng-random,filename=/dev/urandom,id=rng0 \
    -device virtio-rng-pci,rng=rng0 \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/tmp/libguestfsfGeYfb/scratch1.img,cache=unsafe,format=raw,id=hd0,if=none \
    -device scsi-hd,drive=hd0 \
    -drive file=/var/tmp/.guestfs-1000/appliance.d/root,snapshot=on,id=appliance,cache=unsafe,if=none \
    -device scsi-hd,drive=appliance \
    -device virtio-serial-pci \
    -serial stdio \
    -chardev socket,path=/run/user/1000/libguestfsHuYYGU/guestfsd.sock,id=channel0 \
    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
    -append "panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=UUID=23e8de0a-4a72-4ffc-b711-b66d736f097d selinux=0 guestfs_verbose=1 TERM=xterm-256color"
qemu-kvm: -no-hpet: warning: -no-hpet is deprecated, use '-machine hpet=off' instead
KVM: entry failed, hardware error 0x8
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00080660
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=04 66 41 eb f1 66 83 c9 ff 66 89 c8 66 5b 66 5e 66 5f 66 c3 <ea> 5b e0 00 f0 30 36 2f 32 33 2f 39 39 00 fc 00 ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??

(In reply to Richard W.M. Jones from comment #2)

> 14: ea 5b e0 00 f0 30 36    jmp    0x3630:0xf000e05b
> 1b: 2f                      das
> 1c: 32 33                   xor    dh,BYTE PTR [ebx]

Wait, this disassembly is incorrect. The debugger / disassembler assumed an incorrect CPU mode, or it's a plain disassembler bug. The JMP instruction is actually

EA5BE000F0        jmp 0xf000:0xe05b

(i.e., ptr16:16, not ptr16:32), meaning a jump to 0xfe05b in real mode. And such a jump instruction is indeed valid as the first (and only) instruction that the reset vector points at.

The constant 0xe05b is present in SeaBIOS's "src/romlayout.S":

        ORG 0xe05b
entry_post:
        cmpl $0, %cs:HaveRunPost                // Check for resume/reboot
        jnz entry_resume
        ENTRY_INTO32 _cfunc32flat_handle_post   // Normal entry point

so that's the code we're trying to jump *to*, but the L2 domain crashes in that very jump first instruction, at the reset vector. IOW, the CS:EIP info in the QEMU register dump is actually correct.

hardware error 0x8 seems to be EXIT_REASON_NMI_WINDOW.

Setting kvm_intel.dump_invalid_vmcs=1 (in L1):

[18499.856522] VMCS 00000000b780e336, last attempted VM-entry on CPU 0
[18499.856526] *** Guest State ***
[18499.856527] CR0: actual=0x0000000000000030, shadow=0x0000000060000010, gh_mask=fffffffffffffff7
[18499.856530] CR4: actual=0x0000000000002040, shadow=0x0000000000000000, gh_mask=fffffffffffef871
[18499.856531] CR3 = 0x0000000000000000
[18499.856534] PDPTR0 = 0x0000000000000000  PDPTR1 = 0x0000000000000000
[18499.856538] PDPTR2 = 0x0000000000000000  PDPTR3 = 0x0000000000000000
[18499.856539] RSP = 0x0000000000000000  RIP = 0x000000000000fff0
[18499.856540] RFLAGS=0x00000002         DR7 = 0x0000000000000400
[18499.856545] Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
[18499.856550] CS:   sel=0xf000, attr=0x0009b, limit=0x0000ffff, base=0x00000000ffff0000
[18499.856557] DS:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
[18499.856562] SS:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
[18499.856568] ES:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
[18499.856574] FS:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
[18499.856579] GS:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
[18499.856582] GDTR:                           limit=0x0000ffff, base=0x0000000000000000
[18499.856594] LDTR: sel=0x0000, attr=0x00082, limit=0x0000ffff, base=0x0000000000000000
[18499.856598] IDTR:                           limit=0x0000ffff, base=0x0000000000000000
[18499.856603] TR:   sel=0x0000, attr=0x0008b, limit=0x0000ffff, base=0x0000000000000000
[18499.856605] EFER= 0x0000000000000000
[18499.856607] PAT = 0x0007040600070406
[18499.856611] DebugCtl = 0x0000000000000000  DebugExceptions = 0x0000000000000000
[18499.856613] Interruptibility = 00000000  ActivityState = 00000000
[18499.856614] InterruptStatus = 0000
[18499.856617] *** Host State ***
[18499.856620] RIP = 0xffffffffc0a0e863  RSP = 0xff2b812dc0947cb0
[18499.856627] CS=0010 SS=0018 DS=0000 ES=0000 FS=0000 GS=0000 TR=0040
[18499.856629] FSBase=00007fe67bdff640 GSBase=ff1ce2653ba00000 TRBase=fffffe0000003000
[18499.856633] GDTBase=fffffe0000001000 IDTBase=fffffe0000000000
[18499.856637] CR0=0000000080050033 CR3=0000000103f2c003 CR4=0000000000773ef0
[18499.856642] Sysenter RSP=fffffe0000003000 CS:RIP=0010:ffffffffaa0015f0
[18499.856644] EFER= 0x0000000000000d01
[18499.856646] PAT = 0x0407050600070106
[18499.856647] *** Control State ***
[18499.856648] CPUBased=0xb5a06dfa SecondaryExec=0x000033eb TertiaryExec=0x0000000000000000
[18499.856649] PinBased=0x000000ff EntryControls=0000d1ff ExitControls=002befff
[18499.856652] ExceptionBitmap=00060042 PFECmask=00000000 PFECmatch=00000000
[18499.856653] VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000
[18499.856654] VMExit: intr_info=00000000 errcode=00000000 ilen=00000000
[18499.856654]         reason=00000000 qualification=0000000000000000
[18499.856655] IDTVectoring: info=00000000 errcode=00000000
[18499.856657] TSC Offset = 0xffffd96342ad3040
[18499.856657] SVI|RVI = 00|00 TPR Threshold = 0x00
[18499.856660] APIC-access addr = 0x000000010776d000 virt-APIC addr = 0x0000000108c43000
[18499.856664] PostedIntrVec = 0xf2
[18499.856665] EPT pointer = 0x000000000122305e
[18499.856667] Virtual processor ID = 0x0001
[18503.601139] VMCS 00000000cd5f9037, last attempted VM-entry on CPU 0
[18503.601144] *** Guest State ***
[18503.601145] CR0: actual=0x0000000000000030, shadow=0x0000000060000010, gh_mask=fffffffffffffff7
[18503.601148] CR4: actual=0x0000000000002040, shadow=0x0000000000000000, gh_mask=fffffffffffef871
[18503.601149] CR3 = 0x0000000000000000
[18503.601152] PDPTR0 = 0x0000000000000000  PDPTR1 = 0x0000000000000000
[18503.601155] PDPTR2 = 0x0000000000000000  PDPTR3 = 0x0000000000000000
[18503.601155] RSP = 0x0000000000000000  RIP = 0x000000000000fff0
[18503.601157] RFLAGS=0x00000002         DR7 = 0x0000000000000400
[18503.601162] Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000
[18503.601166] CS:   sel=0xf000, attr=0x0009b, limit=0x0000ffff, base=0x00000000ffff0000
[18503.601173] DS:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
[18503.601178] SS:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
[18503.601184] ES:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
[18503.601189] FS:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
[18503.601195] GS:   sel=0x0000, attr=0x00093, limit=0x0000ffff, base=0x0000000000000000
[18503.601198] GDTR:                           limit=0x0000ffff, base=0x0000000000000000
[18503.601204] LDTR: sel=0x0000, attr=0x00082, limit=0x0000ffff, base=0x0000000000000000
[18503.601207] IDTR:                           limit=0x0000ffff, base=0x0000000000000000
[18503.601213] TR:   sel=0x0000, attr=0x0008b, limit=0x0000ffff, base=0x0000000000000000
[18503.601215] EFER= 0x0000000000000000
[18503.601216] PAT = 0x0007040600070406
[18503.601219] DebugCtl = 0x0000000000000000  DebugExceptions = 0x0000000000000000
[18503.601221] Interruptibility = 00000000  ActivityState = 00000000
[18503.601222] InterruptStatus = 0000
[18503.601225] *** Host State ***
[18503.601228] RIP = 0xffffffffc0a0e863  RSP = 0xff2b812dc0c17ca0
[18503.601235] CS=0010 SS=0018 DS=0000 ES=0000 FS=0000 GS=0000 TR=0040
[18503.601237] FSBase=00007f93a6930640 GSBase=ff1ce2653ba00000 TRBase=fffffe0000003000
[18503.601240] GDTBase=fffffe0000001000 IDTBase=fffffe0000000000
[18503.601245] CR0=0000000080050033 CR3=0000000001316006 CR4=0000000000773ef0
[18503.601249] Sysenter RSP=fffffe0000003000 CS:RIP=0010:ffffffffaa0015f0
[18503.601251] EFER= 0x0000000000000d01
[18503.601253] PAT = 0x0407050600070106
[18503.601255] *** Control State ***
[18503.601255] CPUBased=0xb5a06dfa SecondaryExec=0x000033eb TertiaryExec=0x0000000000000000
[18503.601256] PinBased=0x000000ff EntryControls=0000d1ff ExitControls=002befff
[18503.601260] ExceptionBitmap=00060042 PFECmask=00000000 PFECmatch=00000000
[18503.601260] VMEntry: intr_info=00000000 errcode=00000000 ilen=00000000
[18503.601261] VMExit: intr_info=00000000 errcode=00000000 ilen=00000000
[18503.601262]         reason=00000000 qualification=0000000000000000
[18503.601262] IDTVectoring: info=00000000 errcode=00000000
[18503.601264] TSC Offset = 0xffffd9612628d62e
[18503.601265] SVI|RVI = 00|00 TPR Threshold = 0x00
[18503.601267] APIC-access addr = 0x000000000771c000 virt-APIC addr = 0x000000000114f000
[18503.601271] PostedIntrVec = 0xf2
[18503.601273] EPT pointer = 0x000000000117d05e
[18503.601275] Virtual processor ID = 0x0002

The Intel SDM writes in appendix "VMX BASIC EXIT REASONS" (emphasis
mine):

> Every VM exit writes a 32-bit exit reason to the VMCS (see Section
> 21.9.1). *Certain VM-entry failures also do this* (see Section 23.7)
> [...]
>
> 8 -- NMI window. At the beginning of an instruction, there was no
>      virtual-NMI blocking; events were not blocked by MOV SS; and the
>      "NMI-window exiting" VM-execution control was 1.

Can we perhaps disable the "vnmi" kvm_intel module parameter in L1? Or
else (I assume: equivalently) remove the vmx-vnmi CPU model feature on
the QEMU command line?

... I've tried the former, it doesn't help.

Emanuele, any guesses on this one ? A long shot but could this be related to bug 2127128 even
though the error there was related to invalid control fields ?

(In reply to Bandan Das from comment #14)
> Emanuele, any guesses on this one ? A long shot but could this be related to
> bug 2127128 even
> though the error there was related to invalid control fields ?

That bug leads to some other bugs that appear more fitting: bug 2103118, bug 2105408, bug 2099216.

What I find relatively annoying is that bug 2103118 -- which was originally encountered via virt-customize, and which produced identical symptoms, i.e., crashing at the SeaBIOS reset vector: <ea> 5b e0 00 f0 -- had never been *root-caused*. We only said "broken in 8.2, functional in 8.6, let's go shopping". We never located the actual fix! In such cases a reverse bisection is recommended, to see what precisely fixed the problem. If we had done that there, for RHEL-8, now we wouldn't be standing here, with our trousers around our ankles.

I agree, and added a comment to the other bug to alert people that the bug
has probably not been fixed.

Latest rhel9.3 nightly compose is the L1 guest vm ? How about the host version? 

Dose the workaround listed in the comment https://bugzilla.redhat.com/show_bug.cgi?id=2103118#c50 work?

In L1:
host kernel = 5.14.0-316.el9.x86_64
qemu = 8.0.0-3.el9
seabios = 1.16.1-1.el9

I don't think we have any information about the L0 host, but maybe Yongkui has access.

The workaround probably only applies to OpenStack, but we're not using OpenStack to
launch the L2 VM.  The L2 VM is started with -cpu max so it'll have all features
potentially enabled.  If there is a particular CPU feature implicated then we could
try modifying the -cpu parameter if you can give us guidance.

(In reply to Qinghua Cheng from comment #17)
> Latest rhel9.3 nightly compose is the L1 guest VM ? 

Yes.

> How about the host version? 

I created this VM on our OpenStack env, and I have no permission to access the LO host.

OpenStack      flavor                     L1 guest CPU model                       L1 guest           libguestfs-test-tool      
--------------------------------------------------------------------------------------------------------------------------  
rhos-d         ci.standard.medium         Intel Xeon Processor (Icelake)           RHEL9.3 nightly      failed
rhos-d         ci.nested.virt.m1.medium   Intel Xeon Processor (Skylake, IBRS)     RHEL9.3 nightly      passed
rhos-01        ci.standard.medium         AMD EPYC-Rome Processor                  RHEL9.3 nightly      passed
--------------------------------------------------------------------------------------------------------------------------

1. In short, libguestfs-test-tool fails when the ci.xxx (excluding ci.nested.xxx) flavor is used and the L1 guest CPU model is Icelake in rhos-d OpenStack env. 
2. According to these docs[1][2], nested Virtualization is enabled on ci.nested.xxx flavor but disabled on ci.xxx flavor in rhos-d.
 
[1] https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=KB&title=PSI+OpenStack+Onboarding#PSIOpenStackOnboarding-NestedVirtualizationinRHOS-D
[2] https://docs.engineering.redhat.com/display/KB/PSI+OpenStack+Use+Cases+-+Aggregates

With my pretty limited knowledge on cpu models... here's what I see as a summary so far.

Hardware error seems to be caused by vmenter failure, which means 0x8 (VM_INSTRUCTION_ERROR) -> VM entry with invalid host-state field(s) (according to SDM 30.4).  It got trapped in L0 then delivered to L1 kvm with the same error.

So some host state seems wrong in L2's VMCS when hardware checks, probably something falls into SDM "26.2 CHECKS ON VMX CONTROLS AND HOST-STATE AREA".

I can try to read more in the vmsd dump in comment 12 in the latter days this week (which seems to be quite useful),  Before that I am curious on two things:

1. It seems that we're not be able to access the host (even so far I don't think anything mentioned on the host kernel version), then does it mean that even if we know it's a host kvm bug and we know a fix, it won't be fixed (because it'll need an upgrade of host kvm)?

2. Can we try a similar workaround as mentioned in comment 17 by Qinghua?  Is "-cpu max" required?  According to comment 21 where we do have a PASS use case, I'd give it a shot starting with "-cpu Skylake-Server-IBRS".

Also adding Vitaly (who sometimes has dealt with nested issues) to CC.

Getting information about L0 is crucial: most nested bugs end up being L0 KVM bugs, L1 hypervisor is usually innocent...

(In reply to Vitaly Kuznetsov from comment #25)
> Getting information about L0 is crucial: most nested bugs end up being L0
> KVM bugs, L1 hypervisor is usually innocent...

I am assuming this is something that reporter can provide. Hence, setting the needinfo.

(In reply to Nitesh Narayan Lal from comment #26)
> (In reply to Vitaly Kuznetsov from comment #25)
> > Getting information about L0 is crucial: most nested bugs end up being L0
> > KVM bugs, L1 hypervisor is usually innocent...
> 
> I am assuming this is something that reporter can provide. Hence, setting
> the needinfo.

I cannot get any info about L0 host(see comment 19). Currently, I use ci.nested.xxx flavor or create VM on rhos-01 as a workaround(see comment 21).

(In reply to Nitesh Narayan Lal from comment #29)
> (In reply to yduan from comment #28)
> > (In reply to Peter Xu from comment #23)
> > > With my pretty limited knowledge on cpu models... here's what I see as a
> > > summary so far.
> > > 
> > > Hardware error seems to be caused by vmenter failure, which means 0x8
> > > (VM_INSTRUCTION_ERROR) -> VM entry with invalid host-state field(s)
> > > (according to SDM 30.4).  It got trapped in L0 then delivered to L1 kvm with
> > > the same error.
> > > 
> > > So some host state seems wrong in L2's VMCS when hardware checks, probably
> > > something falls into SDM "26.2 CHECKS ON VMX CONTROLS AND HOST-STATE AREA".
> > > 
> > > I can try to read more in the vmsd dump in comment 12 in the latter days
> > > this week (which seems to be quite useful),  Before that I am curious on two
> > > things:
> > > 
> > > 1. It seems that we're not be able to access the host (even so far I don't
> > > think anything mentioned on the host kernel version), then does it mean that
> > > even if we know it's a host kvm bug and we know a fix, it won't be fixed
> > > (because it'll need an upgrade of host kvm)?
> > > 
> > 
> > IIUC, QE has no permission to touch the underlying host in PSI(PnT Shared
> > Infrastructure) OpenStack environment.
> > 
> > Additional info:
> > The test environment, Production Cloud-D - RHOS 16.1[1], is running on RHEL
> > 8.2 [2] which is same as Red
> > Hathttps://bugzilla.redhat.com/show_bug.cgi?id=2103118#c30.
> 
> Hi, Since the 8.2 kernel is pretty old and we need L0 information for
> further investigation.
> Can you please reproduce this issue on the latest rhel9/rhel8 host/guest
> combination?
>  
The problem is, as mentioned above, is that this is RHOS 16.1 and it's tied to the 
8.2 kernel and to fix this for RHOS, we have to fix the 8.2 kernel (which brings us 
to your comment above that 8.2 is pretty old and we probably don't want to be pushing 
nested virtualization fixes to it.)

Coming back to his bug, I was able to reproduce it fairly easily with a Icelake system
running 8.2 for L0 and a RHEL 9 for L1.

Unlike bug 2127128 that I linked above, it's not the hardware that's complaining. Rather, L0 KVM 
is crafting a message similar to what the host would do in such cases.

In my test, the error comes from L0 finding a mismatch between L1's CR4 (vmcs12->cr4) and the
acceptable values that it has kept for cr4_fixed1, specifically CR4.LA57[12]. I am pretty sure 
this could happen for any other cpuid bits that the 8.2 kernel doesn't know about.

to_vmx(vcpu)->nested.msrs.cr4_fixed1 is filled up in nested_vmx_cr_fixed1_bits_update() by checking
if the guest cpuid has the bit set and if the host too supports it and all would have been fine if we had this 
one line  in the 8.2 kernel: 
cr4_fixed1_update(X86_CR4_LA57, ecx, bit(X86_FEATURE_LA57)); What this would have done is set LA57 in 
nested.msrs.cr4_fixed1 if the host supported 5 level page tables and it's also set in the guest's cpu
model. (CR4_FIXED1 basically says that that bit in CR4 can be set to 1 is it's set in CR4_FIXED1
but if it's 0, setting that bit in CR4 would cause an exception)

Anyway, without that oneliner, to_vmx(vcpu)->nested.msrs.cr4_fixed1[12] = 0. When L1 tries to read MSR_IA32_VMX_CR0_FIXED1,
vmx_get_vmx_msr() returns the real hardware value of the msr(Another issue in the 8.2 kernel! The 8.2 kernel returns
vmx_get_vmx_msr(&vmcs_config.nested, msr->index, &msr->data) where as newer kernels return 
vmx_get_vmx_msr(&vmx->nested.msrs...)) and assuming the host has 5 level page tables, L1 would want to 
set that bit in CR4. 

So, in the end, we would hit (on L0):
if (CC(!nested_host_cr4_valid(vcpu, vmcs12->host_cr4)))
return -EINVAL
  |
  |
 \ /
if (nested_vmx_check_host_state(vcpu, vmcs12))
                return nested_vmx_fail(vcpu, VMXERR_ENTRY_INVALID_HOST_STATE_FIELD);  


Regarding a workaround, the only one I can think of is booting L1 with la57=off but given that this is
a managed instance, I don't know how much of it would be possible.

Regarding a fix, I think the change is trivial. I am not too keen on it though. This is 
nvmx after all and I can't guarantee that the la57 feature that's affecting my setup applies
to the reporter's environment. We would have to know what the host hardware is. Honestly,
I am inclined to close this as WONTFIX.

Thanks - I very much agree that we should close this WONTFIX.  I didn't
initially realise that the L0 kernel was so ancient.

By the way would you mind making your analysis in comment 30 public?
That way if someone hits this error message (which to be honest is not
a very good one) then they will find this bug and have an explanation
of what is going on.

Thanks Bandan for the anaylsis, closing this bug.