Xen guests need to use different processor instructions to make explicit calls into the Xen hypervisor depending on guest type and/or CPU vendor. In order to hide those differences, the hypervisor can fill a hypercall page with the needed instruction sequences, allowing the guest operating system to call into the hypercall page instead of having to choose the correct instructions. The hypercall page contains whole functions, which are written by the hypervisor and executed by the guest. With the lack of an interface between the guest OS and the hypervisor specifying how a potential modification of those functions should look like, the Xen hypervisor has no knowledge how any potential mitigation should look like or which hardening features should be put into place. This results in potential vulnerabilities if the guest OS is using any speculative mitigation that performs a compiler transform on "ret" instructions in order to work (e.g. the Linux kernel rethunk or safe-ret mitigations). Furthermore, the hypercall page has no provision for Control-flow Integrity schemes (e.g. kCFI/CET-IBT/FineIBT), and will simply malfunction in such configurations.
This issue has been addressed in the following products: Red Hat Enterprise Linux 8 Via RHSA-2025:3894 https://access.redhat.com/errata/RHSA-2025:3894
This issue has been addressed in the following products: Red Hat Enterprise Linux 8 Via RHSA-2025:3893 https://access.redhat.com/errata/RHSA-2025:3893
After upgrading the kernel on a RHEL 8.10 VM to 4.18.0-553.50.1.el8_10, the vm will stop with a kernel panic while running on a XenServer 8.4 host. Booting into 4.18.0-553.47.1.el8_10 works fine. [ 0.152052] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.158004] invalid opcode: 0000 [#1] SMP NOPTI [ 0.159000] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.18.0-553.50.1.el8_10.x86_64 #1 [ 0.159000] Hardware name: Xen HVM domU, BIOS 4.17 02/28/2025 [ 0.159000] RIP: 0010:xen_time_init+0x66/0x1f2 [ 0.159000] Code: c7 c0 20 c1 01 00 48 8b 14 d5 80 68 7c a7 8a 0d 7d 5b bc ff 48 63 34 02 31 d2 b8 18 00 00 00 48 83 f9 00 75 05 0f 01 c1 eb 03 <0f> 01 d9 85 c0 75 17 48 c7 c7 88 ac 6d a7 e8 47 18 d7 fd 48 c7 05 [ 0.159000] RSP: 0000:ffffa87240643e48 EFLAGS: 00010286 [ 0.159000] RAX: 0000000000000018 RBX: ffffffffa8a1cc90 RCX: ffffffffa83c1600 [ 0.159000] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000007 [ 0.159000] RBP: 0000000000000000 R08: ffffffffa7e38380 R09: c0000000ffff7fff [ 0.159000] R10: 0000000000000001 R11: ffffa87240643c38 R12: 0000000000000246 [ 0.159000] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 0.159000] FS: 0000000000000000(0000) GS:ffff99afcb400000(0000) knlGS:0000000000000000 [ 0.159000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.159000] CR2: ffff99af73c01000 CR3: 00000000b2210001 CR4: 00000000007706f0 [ 0.159000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 0.159000] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 0.159000] PKRU: 55555554 [ 0.159000] Call Trace: [ 0.159000] ? __die_body+0x1a/0x60 [ 0.159000] ? die+0x2a/0x50 [ 0.159000] ? do_trap+0xe7/0x110 [ 0.159000] ? xen_time_init+0x66/0x1f2 [ 0.159000] ? do_invalid_op+0x36/0x40 [ 0.159000] ? xen_time_init+0x66/0x1f2 [ 0.159000] ? invalid_op+0x14/0x20 [ 0.159000] ? xen_time_init+0x66/0x1f2 [ 0.159000] ? xen_time_init+0x33/0x1f2 [ 0.159000] native_smp_prepare_cpus+0xc4/0x166 [ 0.159000] xen_hvm_smp_prepare_cpus+0xc/0x65 [ 0.159000] kernel_init_freeable+0xd3/0x23a [ 0.159000] ? rest_init+0xaa/0xaa [ 0.159000] kernel_init+0xa/0x10a [ 0.159000] ret_from_fork+0x1f/0x40 [ 0.159000] Modules linked in: [ 0.492001] ---[ end trace 706188a2ec27f037 ]--- [ 0.498001] RIP: 0010:xen_time_init+0x66/0x1f2 [ 0.503000] Code: c7 c0 20 c1 01 00 48 8b 14 d5 80 68 7c a7 8a 0d 7d 5b bc ff 48 63 34 02 31 d2 b8 18 00 00 00 48 83 f9 00 75 05 0f 01 c1 eb 03 <0f> 01 d9 85 c0 75 17 48 c7 c7 88 ac 6d a7 e8 47 18 d7 fd 48 c7 05 [ 0.525000] RSP: 0000:ffffa87240643e48 EFLAGS: 00010286 [ 0.532003] RAX: 0000000000000018 RBX: ffffffffa8a1cc90 RCX: ffffffffa83c1600 [ 0.541001] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000007 [ 0.550000] RBP: 0000000000000000 R08: ffffffffa7e38380 R09: c0000000ffff7fff [ 0.559000] R10: 0000000000000001 R11: ffffa87240643c38 R12: 0000000000000246 [ 0.568000] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 0.573000] FS: 0000000000000000(0000) GS:ffff99afcb400000(0000) knlGS:0000000000000000 [ 0.579000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.584000] CR2: ffff99af73c01000 CR3: 00000000b2210001 CR4: 00000000007706f0 [ 0.590000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 0.595000] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 0.600000] PKRU: 55555554 [ 0.603000] Kernel panic - not syncing: Fatal exception
Hello Gerald, I confirm. We ran into the same issue today, while updating a RHEL8 VM on IBM Cloud (a "Classic VSI" instance, which is essentially a Xen based environment) to "4.18.0-553.50.1.el8_1". The system panic'ed at the very beginning. We downgraded the VM's kernel image to a older kernel and instance runs well again.
(gdb) list *xen_time_init+0x66 That looks like a BUG_ON() or something, but there isn't one in xen_time_init(). I strongly suspect this happens because Xen still thinks the guest is in 32-bit mode, because it didn't populate the hypercall page. See discussion at https://lore.kernel.org/all/54c892eded2b4ebdda8ee1085c383178f44414ad.camel@infradead.org/ As I said there, it would have been much better to just populate the hypercall page during early boot as we always did, and then just stop *using* it if we have CET or something else enabled.
https://lore.kernel.org/xen-devel/f7b820af-b548-4dc3-9af7-9d2214a232b2@amd.com/T/#u
Looks like it's the CPU detection to choose between vmcall/vmmcall. A whole bunch of pointless complexity which also shouldn't exist, because we could have just used the standard Xen hypercall page during early boot, even if we end up freeing that page later and not using it any more.
Seeing the exact same issue as Gerald on AWS with RHEL 8.10 on the t2.xlarge instance class with the 'legacy-bios' boot mode. Updated from kernel 4.18.0-553.47.1.el8_10 to 4.18.0-553.50.1.el8_10 and got this same kernel panic: [ 0.000000] Linux version 4.18.0-553.50.1.el8_10.x86_64 (mockbuild.eng.rdu2.redhat.com) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #1 SMP Thu Apr 10 16:09:41 EDT 2025 < snip > [ 0.000000] DMI: Xen HVM domU, BIOS 4.11.amazon 08/24/2006 [ 0.000000] Hypervisor detected: Xen HVM [ 0.000000] Xen version 4.11. < snip > [ 0.197132] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.207006] invalid opcode: 0000 [#1] SMP PTI [ 0.208000] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.18.0-553.50.1.el8_10.x86_64 #1 [ 0.208000] Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006 [ 0.208000] RIP: 0010:xen_time_init+0x66/0x1f2 [ 0.208000] Code: c7 c0 20 c1 01 00 48 8b 14 d5 80 68 9c bc 8a 0d 7d 5b bc ff 48 63 34 02 31 d2 b8 18 00 00 00 48 83 f9 00 75 05 0f 01 c1 eb 03 <0f> 01 d9 85 c0 75 17 48 c7 c7 88 ac 8d bc e8 47 18 d7 fd 48 c7 05 < snip > [ 0.208000] Call Trace: [ 0.208000] ? __die_body+0x1a/0x60 [ 0.208000] ? die+0x2a/0x50 [ 0.208000] ? do_trap+0xe7/0x110 [ 0.208000] ? xen_time_init+0x66/0x1f2 [ 0.208000] ? do_invalid_op+0x36/0x40 [ 0.208000] ? xen_time_init+0x66/0x1f2 [ 0.208000] ? invalid_op+0x14/0x20 [ 0.208000] ? xen_time_init+0x66/0x1f2 [ 0.208000] ? xen_time_init+0x33/0x1f2 [ 0.208000] native_smp_prepare_cpus+0xc4/0x166 [ 0.208000] xen_hvm_smp_prepare_cpus+0xc/0x65 [ 0.208000] kernel_init_freeable+0xd3/0x23a [ 0.208000] ? rest_init+0xaa/0xaa [ 0.208000] kernel_init+0xa/0x10a [ 0.208000] ret_from_fork+0x35/0x40 [ 0.208000] Modules linked in: [ 0.534002] ---[ end trace 5f5048ac9458349f ]---
There is a new kernel version 4.18.0-553.51.1.el8_10.x86_64 available. We can boot our xen vms into 4.18.0-553.51.1.el8_10.x86_64. That solves this issue for us. See also RHEL KB https://access.redhat.com/solutions/7116307