Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2121795

Summary: Starting VMs on a KVM-host with EL8.6-kernel sometimes produces timejumps into the future for other already running guest-VMs
Product: Red Hat Enterprise Linux 8 Reporter: Matthias Hensler <mails.bugzilla.redhat.com>
Component: kernelAssignee: Maxim Levitsky <mlevitsk>
kernel sub component: KVM QA Contact: Yanhui Ma <yama>
Status: CLOSED DUPLICATE Docs Contact:
Severity: high    
Priority: high CC: bdas, coli, ctatman, dgilbert, jinzhao, juzhang, mlevitsk, nilal, virt-maint, vkuznets
Version: 8.6Keywords: Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 8.7   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-07 15:48:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2096991    

Description Matthias Hensler 2022-08-26 16:30:42 UTC
Description of problem:
After upgrading from Rocky 8.5 to 8.6, we often encounter (huge) timejumps on already running guest-systems under KVM/Qemu, when a new, additional VM is started on the host (either freshly or because of an incoming migration).

The kind of operating system a guest is running, doesn't seem to matter. So far we saw the issues with EL6, EL7 and EL8 VMs, as well as different kinds of Debian and Ubuntu and even Microsoft Windows.

Whenever the problem is encountered for a guest-VM, it usually stops working for things like inbound SSH-connections and console logins, while basic functions like ICMP-network pings are still working. On the VGA console you can see things like CPU-cores getting stuck. On the journal sometimes messages from chronyd like "forward time jump detected" are visible.

The amount of forward time jumps differs from case to case. Sometimes the jump consists only of several hours, while other times the time jump is like several years, even up to hundreds of years into the future (like year 2252).

Starting a new VM on a host does not always trigger the problem, and there is no safe way to give an estimate on how often the problem is triggered. There seem to be several factors in play, like the load in the running VMs, etc. I would guess that the trigger-rate is somewhere in the range of around 5-10%, with the problem occuring sometimes quite often and other times with no trouble in a lot of VM starts.

Also, whenever the problem is triggered, only a couple of guest VMs are affected, but without any indication, that the affected guests did anything special in that timeframe (like high I/O or cpu-usage). The number of affected guests is usually something between 1 to 5 guests on a host running around 30-50 guests. The actual guest systems that are hit seem to be pretty random.

Either the used hardware or the actual kind of processes on the host-system seems to play a role there. So far we were only able to reproduce the issue on AMD EPYC 7502P and 7351P-systems, but not on Intel E3-1240v5-systems.

All of those systems are configured the same (by using an identical puppet-role-manifest). Disk images are provided via RDB-protocol on Ceph. Still, we had no luck on reproducing the issue on one of our Intel-systems, while pretty easily on our AMD-systems. However, the Intel-systems are more like a staging environment with significantly less load and fewer running VMs (only a couple on Intel, while >50 on AMD) in total.


It took quite some time to actually find the component which is responsible. While we first suspected the new Qemu 6 version (which came as an update from Qemu 4), it doesn't seem Qemu related. The culprit seems to be the used kernel of the KVM-Host, with the problem present on both 4.18.0-372.16.1, as well the lastest 4.18.0-372.19.1 from EL8.6. Downgrading to any kernel from EL8.5 (4.18.0-348.X) makes the problem go away.

The kernel running in the guest seems to be irrelevant, as the issue occured on all kind of different operating systems, including even different Microsoft Windows systems.


Version-Release number of selected component (if applicable):
kernel-4.18.0-372.16.1.el8_6.x86_64
kernel-4.18.0-372.19.1.el8_6.x86_64


How reproducible:
The rate of how often the problem is triggered, seems a bit random to me and is most likely related to a specific factor like load or maybe the moment of a context switch of a guest VM. However, while testing with a host running around 40-50 VMs we could trigger the issue quite often within a couple minutes to an hour by migrating VMs in and out.

And it is not the migration that triggers the issue, as even a fresh start of a VM via "virsh start ..." actually already affected other guests with that timejump problem. So more like a fresh qemu/kvm-process coming up will sometimes affect other running qemu-processes. Stopping (or migrating out) a VM seems to be safe.


Steps to Reproduce:
1. use fully updated EL8.6 (with qemu-kvm-6.2.0-11.module+el8.6.0+998+252a5635.2.x86_64)
2. boot into kernel-4.18.0-372.19.1.el8_6.x86_64 (or kernel-4.18.0-372.16.1.el8_6.x86_64)
3. start/migrate in a lot of guests (or start guests fresh)
4. maybe migrate out and back in one or more guest (or restart guests)
5. watch other guests for hints of timejumps (VGA-console or journal, etc)

Actual results:
Some guest VMs might experience timejumps into the future. (Please note that this affects
guests who are already running on the host, and not the guest that is currently started or migrated in)

journal might show a message like
Jul 28 21:06:42 news chronyd[15546]: Forward time jump detected!

After downgrading to kernel-4.18.0-348.23.1.el8_5.x86_64 (on the host-system) no further problem occurs.

Expected results:
No timejumps of course. Same setup was running for over a year with thousands of migrations and no issues. Problem started only after updating to the new EL8.6 kernel (and qemu in the process).

Additional info:
Unfortunately, this might be a difficult to reproduce bug, since we only see that on our AMD EPYC-plattform, and no other systems.
Also, reading back and forth other bugs in the kernel (or qemu) category brought nothing up. So I try to be as specific as possible.

After lots of testings several things, I am pretty sure that the kernel itself is the culprit here, but I have no clue on how to debug that further.

Commandline of a guest qemu-process (stripped of password for our RBD/Ceph):
/usr/libexec/qemu-kvm -name guest=rx_news,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1998-rx_news/master-key.aes"} -machine pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off,memory-backend=pc.ram -accel kvm -cpu EPYC-IBPB,ht=off,xsaves=off,cmp-legacy=on,extapic=off,skinit=off,wdt=off,tce=off,topoext=on,perfctr-core=on,perfctr-nb=off,monitor=off,x2apic=on,hypervisor=on,svm=off,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-synic=on,hv-stimer=on -m 1024 -object {"qom-type":"memory-backend-ram","id":"pc.ram","size":1073741824} -overcommit mem-lock=off -smp 1,sockets=1,dies=1,cores=1,threads=1 -uuid 8fa0d3f6-53e2-11e6-9782-005056bc1525 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=43,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -object {"qom-type":"secret","id":"libvirt-1-storage-auth-secret0","data":"XXX/XXX=","keyid":"masterKey0","iv":"XXX==","format":"base64"} -blockdev {"driver":"rbd","pool":"pureflash","image":"rx_news","server":[{"host":"10.128.0.17","port":"0"},{"host":"10.128.0.18","port":"0"},{"host":"10.128.0.19","port":"0"}],"user":"kvm","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-auth-secret0","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"} -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=drive-scsi0-0-0-0,drive=libvirt-1-format,id=scsi0-0-0-0,bootindex=1,write-cache=on -netdev tap,fd=48,id=hostnet0,vhost=on,vhostfd=73 -device virtio-net-pci,netdev=hostnet0,id=ne
t0,mac=52:54:00:00:XX:XX,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -audiodev {"id":"audio1","driver":"none"} -vnc [::]:4,password=on,audiodev=audio1 -k de -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -object {"qom-type":"rng-random","id":"objrng0","filename":"/dev/random"} -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

Comment 1 Matthias Hensler 2022-08-31 12:11:23 UTC
Some additional information: I could find similar reports for Proxmox after they updated from Linux 5.13 to 5.15. There are a couple of forum threads about that, and a global tracking bug at https://bugzilla.proxmox.com/show_bug.cgi?id=4073.

People there had success with a patched kernel for some FPU-saving bits, which seem to result out of this patchset: https://lore.kernel.org/lkml/CAJ6HWG66HZ7raAa+YK0UOGLF+4O3JnzbZ+a-0j8GNixOhLk9dA@mail.gmail.com/T/

Please let me know if there is anything I can provide to track that issue down.

Comment 2 Matthias Hensler 2022-08-31 13:46:27 UTC
Please also see this post with a couple of patches concerning some TSC-multiplier bits: https://lists.proxmox.com/pipermail/pve-devel/2022-August/053814.html
At least that is tightly similar to the issues we see with RHEL 8.6 and the Redhat-kernel.

Comment 3 Vitaly Kuznetsov 2022-08-31 13:54:33 UTC
Assuming this all happens on AMD (as EPYC-IBPB CPU model is being used), my
first guess would be that this commit:

commit 11d39e8cc43e1c6737af19ca9372e590061b5ad2
Author: Maxim Levitsky <mlevitsk>
Date:   Mon Jun 6 21:11:49 2022 +0300

    KVM: SVM: fix tsc scaling cache logic
    
resolves the issue. This was already picked up for the upcoming 8.7 but
8.6.z (where the offending commit 1ab9287add5e is present) is not yet
fixed.

Comment 4 Matthias Hensler 2022-08-31 13:58:18 UTC
Fixes for the TSC would at least relate to the Proxmox-patch from comment #2.

And yes, I can confirm that we only see that problem on AMD-cpus so far (EPYC 7351P and 7502P). The same environment on our Intel-cpus is fine.

Comment 6 Yanhui Ma 2022-09-01 01:52:26 UTC
QE just borrowed one AMD host (EPYC 7351P) and is trying to reproduce the bug.

Comment 9 Vitaly Kuznetsov 2022-09-01 13:36:23 UTC
Matthias,

we've backported the commit I've mentioned at https://bugzilla.redhat.com/show_bug.cgi?id=2121795#c3
to 8.6, could you please give this test kernel a spin so we know whether our guess about the root
cause was correct?

RPMs are available here:
https://people.redhat.com/~vkuznets/bug2121795/

Thanks!

Comment 10 Matthias Hensler 2022-09-01 13:46:02 UTC
Hi Vitaley,

many thanks. I have a maintenance-window on saturday and will give that test-kernel a go. Looking at the patch itself, I am at least pretty hopeful, since that code seems to address the root-cause pretty perfectly. I will come back with an update hopefully mid next week by the latest.

Comment 12 Dr. David Alan Gilbert 2022-09-05 08:32:18 UTC
I bet this could also be the cause of upstream gitlab issue:
https://gitlab.com/qemu-project/qemu/-/issues/1140

they reported a slightly different symptom, where the other guests would start eating lots of vcpu, but the backtrace was around tsc
and it was AMD only.

Comment 13 Matthias Hensler 2022-09-05 15:09:22 UTC
(In reply to Matthias Hensler from comment #10)
> I will come back with an update hopefully mid next week by the latest.

First (optimistic) update: we are running the test-kernel on one of our machines and have done around 1700 VMs starts without issues. I will continue testing tomorrow and if that runs well, install that testkernel on a second machine and do ping-pong-migrations between both machines to be extra sure that the issue is fixed.

Please expect my next feedback within the next 72 hours.

Comment 15 Matthias Hensler 2022-09-07 14:40:49 UTC
After doing tests with several thousand VM starts and migrations in our previous affected cluster, I am pretty confident now, that the kernel from comment #9 fixes the issue for us. So far I was not able to reproduce the issue on either EPYC 7351P or 7502P.

I will continue testing over the course of the next days, but at least from my side I would give a +1 to see that patch in the next upcoming kernel release for EL8.6.

Comment 17 Vitaly Kuznetsov 2022-09-07 15:48:53 UTC
(In reply to Matthias Hensler from comment #15)
> After doing tests with several thousand VM starts and migrations in our
> previous affected cluster, I am pretty confident now, that the kernel from
> comment #9 fixes the issue for us. So far I was not able to reproduce the
> issue on either EPYC 7351P or 7502P.
> 
> I will continue testing over the course of the next days, but at least from
> my side I would give a +1 to see that patch in the next upcoming kernel
> release for EL8.6.

Thank you Matthias,

I'm going to close this bug as DUPLICATE of BZ#2115240 as that's where the fix
was backported to 8.7. I'm going to request 8.6.z for this particular fix too.

Thanks for testing!

*** This bug has been marked as a duplicate of bug 2115240 ***

Comment 18 Maxim Levitsky 2022-10-31 15:02:19 UTC
@Yanhui Ma sorry for not noticing the NEEDINFO.

It is possible that the windows guest doens't crash because it doesn't use TSC, or something.

Please try with multiple cpus (-smp 16 or something) and/or with  with -cpu invtsc=on )


Best regards,
     Maxim Levitsky