Bug 513138
| Summary: | Time of smp guest warps after migration | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | jason wang <jasowang> | ||||
| Component: | kvm | Assignee: | Zachary Amsden <zamsden> | ||||
| Status: | CLOSED WONTFIX | QA Contact: | Lawrence Lim <llim> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | low | ||||||
| Version: | 5.4 | CC: | azarembo, fschwarz, herrold, rlerch, tburke, tools-bugs, virt-maint, wnefal+redhatbugzilla, ykaul, zamsden | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | All | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: |
When the tsc is not stable on the host, including cpufreq changes and deep C state, or even migration into faster tsc host. To stop deep C states in which the TSC can stop, add
"processor.max_cstate=1" as a host kernel boot option.
To disable cpufreq (only necessary on hosts which lack constant_tsc flag in "flags" field of /proc/cpuinfo), edit /etc/sysconfig/cpuspeed
MIN_SPEED and MAX_SPEED variables to the highest frequency available.
|
Story Points: | --- | ||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2009-10-29 22:05:28 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 481013 | ||||||
| Bug Blocks: | 495630, 513501 | ||||||
| Attachments: |
|
||||||
|
Description
jason wang
2009-07-22 07:51:50 UTC
Interesting problem! Questions 1: Without migration, but with adding load on the host, can you see it happening? Question 2: Can you re-do it with offline migration (stop the vm before the migration command, do the migration, 'cont' the vm on the destination. Maybe we should cancel cpu scaling (not sure) - /sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq. Marcelo? (In reply to comment #1) > Interesting problem! > > Questions 1: > Without migration, but with adding load on the host, can you see it happening? > running unixbench and monotonic AMD platform: AMD Phenom(tm) 8750 Triple-Core Processor -smp 3 gettimeofday() PASS clock() PASS TSC() FAIL with a big warp value Could also notice obvious timedrift ( could be easily by watch -n 0 date ) Intel platform: Intel(R) Xeon(R) CPU E5310 @ 1.60GHz TSC() FAIL with a really small wrap ( something like -2010 ) Timedrift could also be noticed ( not as obvious as AMD processors but still could be see in just one minutes ) > Question 2: > Can you re-do it with offline migration (stop the vm before the migration > command, do the migration, 'cont' the vm on the destination. Still could reproduce the problem: Intel(R) Xeon(R) CPU E5310 @ 1.60GHz -smp 8 All three Test Failed. > Maybe we should cancel cpu scaling (not sure) - > /sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq. Marcelo? The problem is the TSC for different vcpus is not saved at exactly the same (real) time point, so the TSC's go out of sync between the vcpus on the destination. This is similar to what happens during guest initialization, which was "fixed" by: http://mirror.celinuxforum.org/gitstat//commit-detail.php?commit=53f658b3c33616a4997ee254311b335e59063289 It can probably be fixed in a similar way (looking into that). However note that handling of TSC during migration suffers from other issues (even after this bug is fixed). Dor, regarding cpufreq, if the hosts TSC does not tick at a constant rate, or if the host TSC stops for some reason (say ACPI deep sleep), its likely that Linux guests will encounter problems. Marcelo, can you add release note for it? How do you disable the cpufreq changes and the deep sleep states? Release note added. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: When the tsc is not stable on the host, including cpufreq changes and deep C state, or even migration into faster tsc host. To stop deep C states in which the TSC can stop, add "processor.max_cstate=1" as a host kernel boot option. To disable cpufreq (only necessary on hosts which lack constant_tsc flag in "flags" field of /proc/cpuinfo), edit /etc/sysconfig/cpuspeed MIN_SPEED and MAX_SPEED variables to the highest frequency available. Created attachment 355338 [details]
kvm-userspace-rhel5-savevm-tsc-synchronization.patch
jason, The failure with big tsc warp, is it during migration of SMP guest? how many vcpus? Ah. Do you mean that the processor.max_cstate=1 boot option eliminates the big tsc warp problem? (In reply to comment #10) > jason, > > The failure with big tsc warp, is it during migration of SMP guest? how many > vcpus? Forget to tell, 4 vcpus are used in all test. (In reply to comment #11) > Ah. Do you mean that the processor.max_cstate=1 boot option eliminates the big > tsc warp problem? Only up is in the test of processor.mac_cstate=1. So I would retest the smp guests with processor.mac_cstate=1 (In reply to comment #11) > Ah. Do you mean that the processor.max_cstate=1 boot option eliminates the big > tsc warp problem? Re-test in smp=4 wich processor.max_cstate=1 for five times. ALL gettimeofday() PASS ALL clock_gettime() PASS ALL TSC FAIL with big wraps. (In reply to comment #14) > (In reply to comment #11) > > Ah. Do you mean that the processor.max_cstate=1 boot option eliminates the big > > tsc warp problem? > > Re-test in smp=4 wich processor.max_cstate=1 for five times. > ALL gettimeofday() PASS > ALL clock_gettime() PASS > ALL TSC FAIL with big wraps. I don't see the big TSC warps here, with Intel host. Can you try that? At least the warp on system clock (which is what applications should be using) is reduced with the patch (it will increase as the number of vcpus increases). (In reply to comment #17) > (In reply to comment #14) > > (In reply to comment #11) > > > Ah. Do you mean that the processor.max_cstate=1 boot option eliminates the big > > > tsc warp problem? > > > > Re-test in smp=4 wich processor.max_cstate=1 for five times. > > ALL gettimeofday() PASS > > ALL clock_gettime() PASS > > ALL TSC FAIL with big wraps. > > I don't see the big TSC warps here, with Intel host. Can you try that? > > At least the warp on system clock (which is what applications should be using) > is reduced with the patch (it will increase as the number of vcpus increases). Marcelo: Test it with two Intel Xeon E5310s with eight physical cpu on each host. Use kvm-autoest to do the ping-pong migration (about 10 rounds in this case) until the monotonic_time finished. Test both with and without processor.max_cstate=1. Test the guest with 2 vcpus and 8 vcpus. All test FAIL with big warps (gettimeofday, clock_gettime() TSC). The warp value of TSC is relative small (for example -465834 or -612732) compared to gtod/clock, the warp value of gtod/clock is very big. Additional Information: 1 cat /proc/cpuinfo ... processor : 7 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU E5310 @ 1.60GHz stepping : 11 cpu MHz : 1595.927 cache size : 4096 KB physical id : 1 siblings : 4 core id : 3 cpu cores : 4 apicid : 7 fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx tm2 cx16 xtpr lahf_lm bogomips : 3191.89 clflush size : 64 cache_alignment : 64 address sizes : 38 bits physical, 48 bits virtual power management: 2. invariant TSC is zero in the host ( Xeon 5310) but the invariant TSC is set in the previous AMD host ( Quad-Core AMD Opteron(tm) Processor 1352) jason, Can you please test migration with the "clocksource=acpi_pm notsc" option passed to the guest kernel? TSC is not recommended in rhel5 guest. Either use the kvm pv clock or the PIT. |