Bug 2121795
| Summary: | Starting VMs on a KVM-host with EL8.6-kernel sometimes produces timejumps into the future for other already running guest-VMs | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Matthias Hensler <mails.bugzilla.redhat.com> |
| Component: | kernel | Assignee: | Maxim Levitsky <mlevitsk> |
| kernel sub component: | KVM | QA Contact: | Yanhui Ma <yama> |
| Status: | CLOSED DUPLICATE | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | bdas, coli, ctatman, dgilbert, jinzhao, juzhang, mlevitsk, nilal, virt-maint, vkuznets |
| Version: | 8.6 | Keywords: | Triaged |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | 8.7 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-09-07 15:48:53 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2096991 | ||
Some additional information: I could find similar reports for Proxmox after they updated from Linux 5.13 to 5.15. There are a couple of forum threads about that, and a global tracking bug at https://bugzilla.proxmox.com/show_bug.cgi?id=4073. People there had success with a patched kernel for some FPU-saving bits, which seem to result out of this patchset: https://lore.kernel.org/lkml/CAJ6HWG66HZ7raAa+YK0UOGLF+4O3JnzbZ+a-0j8GNixOhLk9dA@mail.gmail.com/T/ Please let me know if there is anything I can provide to track that issue down. Please also see this post with a couple of patches concerning some TSC-multiplier bits: https://lists.proxmox.com/pipermail/pve-devel/2022-August/053814.html At least that is tightly similar to the issues we see with RHEL 8.6 and the Redhat-kernel. Assuming this all happens on AMD (as EPYC-IBPB CPU model is being used), my
first guess would be that this commit:
commit 11d39e8cc43e1c6737af19ca9372e590061b5ad2
Author: Maxim Levitsky <mlevitsk>
Date: Mon Jun 6 21:11:49 2022 +0300
KVM: SVM: fix tsc scaling cache logic
resolves the issue. This was already picked up for the upcoming 8.7 but
8.6.z (where the offending commit 1ab9287add5e is present) is not yet
fixed.
Fixes for the TSC would at least relate to the Proxmox-patch from comment #2. And yes, I can confirm that we only see that problem on AMD-cpus so far (EPYC 7351P and 7502P). The same environment on our Intel-cpus is fine. QE just borrowed one AMD host (EPYC 7351P) and is trying to reproduce the bug. Matthias, we've backported the commit I've mentioned at https://bugzilla.redhat.com/show_bug.cgi?id=2121795#c3 to 8.6, could you please give this test kernel a spin so we know whether our guess about the root cause was correct? RPMs are available here: https://people.redhat.com/~vkuznets/bug2121795/ Thanks! Hi Vitaley, many thanks. I have a maintenance-window on saturday and will give that test-kernel a go. Looking at the patch itself, I am at least pretty hopeful, since that code seems to address the root-cause pretty perfectly. I will come back with an update hopefully mid next week by the latest. I bet this could also be the cause of upstream gitlab issue: https://gitlab.com/qemu-project/qemu/-/issues/1140 they reported a slightly different symptom, where the other guests would start eating lots of vcpu, but the backtrace was around tsc and it was AMD only. (In reply to Matthias Hensler from comment #10) > I will come back with an update hopefully mid next week by the latest. First (optimistic) update: we are running the test-kernel on one of our machines and have done around 1700 VMs starts without issues. I will continue testing tomorrow and if that runs well, install that testkernel on a second machine and do ping-pong-migrations between both machines to be extra sure that the issue is fixed. Please expect my next feedback within the next 72 hours. After doing tests with several thousand VM starts and migrations in our previous affected cluster, I am pretty confident now, that the kernel from comment #9 fixes the issue for us. So far I was not able to reproduce the issue on either EPYC 7351P or 7502P. I will continue testing over the course of the next days, but at least from my side I would give a +1 to see that patch in the next upcoming kernel release for EL8.6. (In reply to Matthias Hensler from comment #15) > After doing tests with several thousand VM starts and migrations in our > previous affected cluster, I am pretty confident now, that the kernel from > comment #9 fixes the issue for us. So far I was not able to reproduce the > issue on either EPYC 7351P or 7502P. > > I will continue testing over the course of the next days, but at least from > my side I would give a +1 to see that patch in the next upcoming kernel > release for EL8.6. Thank you Matthias, I'm going to close this bug as DUPLICATE of BZ#2115240 as that's where the fix was backported to 8.7. I'm going to request 8.6.z for this particular fix too. Thanks for testing! *** This bug has been marked as a duplicate of bug 2115240 *** @Yanhui Ma sorry for not noticing the NEEDINFO.
It is possible that the windows guest doens't crash because it doesn't use TSC, or something.
Please try with multiple cpus (-smp 16 or something) and/or with with -cpu invtsc=on )
Best regards,
Maxim Levitsky
|
Description of problem: After upgrading from Rocky 8.5 to 8.6, we often encounter (huge) timejumps on already running guest-systems under KVM/Qemu, when a new, additional VM is started on the host (either freshly or because of an incoming migration). The kind of operating system a guest is running, doesn't seem to matter. So far we saw the issues with EL6, EL7 and EL8 VMs, as well as different kinds of Debian and Ubuntu and even Microsoft Windows. Whenever the problem is encountered for a guest-VM, it usually stops working for things like inbound SSH-connections and console logins, while basic functions like ICMP-network pings are still working. On the VGA console you can see things like CPU-cores getting stuck. On the journal sometimes messages from chronyd like "forward time jump detected" are visible. The amount of forward time jumps differs from case to case. Sometimes the jump consists only of several hours, while other times the time jump is like several years, even up to hundreds of years into the future (like year 2252). Starting a new VM on a host does not always trigger the problem, and there is no safe way to give an estimate on how often the problem is triggered. There seem to be several factors in play, like the load in the running VMs, etc. I would guess that the trigger-rate is somewhere in the range of around 5-10%, with the problem occuring sometimes quite often and other times with no trouble in a lot of VM starts. Also, whenever the problem is triggered, only a couple of guest VMs are affected, but without any indication, that the affected guests did anything special in that timeframe (like high I/O or cpu-usage). The number of affected guests is usually something between 1 to 5 guests on a host running around 30-50 guests. The actual guest systems that are hit seem to be pretty random. Either the used hardware or the actual kind of processes on the host-system seems to play a role there. So far we were only able to reproduce the issue on AMD EPYC 7502P and 7351P-systems, but not on Intel E3-1240v5-systems. All of those systems are configured the same (by using an identical puppet-role-manifest). Disk images are provided via RDB-protocol on Ceph. Still, we had no luck on reproducing the issue on one of our Intel-systems, while pretty easily on our AMD-systems. However, the Intel-systems are more like a staging environment with significantly less load and fewer running VMs (only a couple on Intel, while >50 on AMD) in total. It took quite some time to actually find the component which is responsible. While we first suspected the new Qemu 6 version (which came as an update from Qemu 4), it doesn't seem Qemu related. The culprit seems to be the used kernel of the KVM-Host, with the problem present on both 4.18.0-372.16.1, as well the lastest 4.18.0-372.19.1 from EL8.6. Downgrading to any kernel from EL8.5 (4.18.0-348.X) makes the problem go away. The kernel running in the guest seems to be irrelevant, as the issue occured on all kind of different operating systems, including even different Microsoft Windows systems. Version-Release number of selected component (if applicable): kernel-4.18.0-372.16.1.el8_6.x86_64 kernel-4.18.0-372.19.1.el8_6.x86_64 How reproducible: The rate of how often the problem is triggered, seems a bit random to me and is most likely related to a specific factor like load or maybe the moment of a context switch of a guest VM. However, while testing with a host running around 40-50 VMs we could trigger the issue quite often within a couple minutes to an hour by migrating VMs in and out. And it is not the migration that triggers the issue, as even a fresh start of a VM via "virsh start ..." actually already affected other guests with that timejump problem. So more like a fresh qemu/kvm-process coming up will sometimes affect other running qemu-processes. Stopping (or migrating out) a VM seems to be safe. Steps to Reproduce: 1. use fully updated EL8.6 (with qemu-kvm-6.2.0-11.module+el8.6.0+998+252a5635.2.x86_64) 2. boot into kernel-4.18.0-372.19.1.el8_6.x86_64 (or kernel-4.18.0-372.16.1.el8_6.x86_64) 3. start/migrate in a lot of guests (or start guests fresh) 4. maybe migrate out and back in one or more guest (or restart guests) 5. watch other guests for hints of timejumps (VGA-console or journal, etc) Actual results: Some guest VMs might experience timejumps into the future. (Please note that this affects guests who are already running on the host, and not the guest that is currently started or migrated in) journal might show a message like Jul 28 21:06:42 news chronyd[15546]: Forward time jump detected! After downgrading to kernel-4.18.0-348.23.1.el8_5.x86_64 (on the host-system) no further problem occurs. Expected results: No timejumps of course. Same setup was running for over a year with thousands of migrations and no issues. Problem started only after updating to the new EL8.6 kernel (and qemu in the process). Additional info: Unfortunately, this might be a difficult to reproduce bug, since we only see that on our AMD EPYC-plattform, and no other systems. Also, reading back and forth other bugs in the kernel (or qemu) category brought nothing up. So I try to be as specific as possible. After lots of testings several things, I am pretty sure that the kernel itself is the culprit here, but I have no clue on how to debug that further. Commandline of a guest qemu-process (stripped of password for our RBD/Ceph): /usr/libexec/qemu-kvm -name guest=rx_news,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1998-rx_news/master-key.aes"} -machine pc-i440fx-rhel7.6.0,usb=off,dump-guest-core=off,memory-backend=pc.ram -accel kvm -cpu EPYC-IBPB,ht=off,xsaves=off,cmp-legacy=on,extapic=off,skinit=off,wdt=off,tce=off,topoext=on,perfctr-core=on,perfctr-nb=off,monitor=off,x2apic=on,hypervisor=on,svm=off,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-synic=on,hv-stimer=on -m 1024 -object {"qom-type":"memory-backend-ram","id":"pc.ram","size":1073741824} -overcommit mem-lock=off -smp 1,sockets=1,dies=1,cores=1,threads=1 -uuid 8fa0d3f6-53e2-11e6-9782-005056bc1525 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=43,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,clock=vm,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -object {"qom-type":"secret","id":"libvirt-1-storage-auth-secret0","data":"XXX/XXX=","keyid":"masterKey0","iv":"XXX==","format":"base64"} -blockdev {"driver":"rbd","pool":"pureflash","image":"rx_news","server":[{"host":"10.128.0.17","port":"0"},{"host":"10.128.0.18","port":"0"},{"host":"10.128.0.19","port":"0"}],"user":"kvm","auth-client-required":["cephx","none"],"key-secret":"libvirt-1-storage-auth-secret0","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"} -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=drive-scsi0-0-0-0,drive=libvirt-1-format,id=scsi0-0-0-0,bootindex=1,write-cache=on -netdev tap,fd=48,id=hostnet0,vhost=on,vhostfd=73 -device virtio-net-pci,netdev=hostnet0,id=ne t0,mac=52:54:00:00:XX:XX,bus=pci.0,addr=0x5 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -audiodev {"id":"audio1","driver":"none"} -vnc [::]:4,password=on,audiodev=audio1 -k de -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -object {"qom-type":"rng-random","id":"objrng0","filename":"/dev/random"} -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on