Bug 629237 - time drift with VXTIME_PMTMR mode in case of early / short real ticks [rhel-4.8.z]
Summary: time drift with VXTIME_PMTMR mode in case of early / short real ticks [rhel-4...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel
Version: 4.8
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Don Howard
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On: 586307
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-09-01 12:02 UTC by RHEL Program Management
Modified: 2018-10-27 13:12 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
This change fixes forward time drift which has been observed with 64-bit Red Hat Enterprise Linux 4 virtual machines using PM timer based kernel tick accounting while running on KVM or HYPER-V hypervisor. Virtual machines that are booted with the "divider=x" kernel parameter set to a value greater than 1 and that show the following line of text in the kernel boot messages are prone to the problem: time.c: Using PM based timekeeping However, this change also uncovered a bug in the Xen hypervisor, possibly causing backward time drift when this change is applied. If such time drift is observed with fully-virtualized Xen guests satisfying the aforementioned conditions, the kernel parameter "pmtimer_fine_grained=0" can be used as a temporary workaround until the hypervisor is updated. Fixing the hypervisor for Red Hat Enterprise Linux 5 hosts is being tracked by BZ #633028.
Clone Of:
Environment:
Last Closed: 2010-12-01 22:25:12 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2010:0936 0 normal SHIPPED_LIVE Important: kernel security and bug fix update 2010-12-06 14:19:04 UTC

Description RHEL Program Management 2010-09-01 12:02:11 UTC 
This bug has been copied from bug #586307 and has been proposed
to be backported to 4.8 z-stream (EUS).
Comment 7 Paolo Bonzini 2010-09-28 11:20:37 UTC 
Regarding bug 633028, on the mailing list we decided that enabling by default the new option is _not_ a problem as far as Xen is concerned.

We should still make sure that the possible time drift is documented in the knowledge base.  Also, we may want to backport the change for bug 633028 to RHEL5.5 z-stream.
Comment 8 Glauber Costa 2010-09-28 12:51:55 UTC 
Paolo,

mind including references to the upstream discussions?

Thanks!
Comment 11 Don Howard 2010-11-05 18:43:23 UTC 
A patch addressing this issue has been included in kernel-2.6.9-89.32.1.
Comment 13 dawu 2010-11-27 14:18:24 UTC 
Test Status for BZ586307.
Now we are conducting a long term test. following is the 1st report of timing behavior of the 1st hour.
we have observed time drift on i686 guest on Intel platform.



=====================
A Test on Intel Host
=====================

model name	: Intel(R) Core(TM)2 Quad CPU    Q9400  @ 2.66GHz
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr sse4_1 lahf_lm
bogomips	: 5320.00

RPMS:
[root@dhcp-91-63 ~]# uname -r
2.6.18-229.el5
[root@dhcp-91-63 ~]# rpm -q kvm
kvm-83-207.el5


1).   x86_64 guest .
root     13083  4453  6 06:32 pts/0    00:04:54 /usr/libexec/qemu-kvm -m 2G -smp 2 -cpu qemu64,+sse2 -drive file=/var/lib/libvirt/images/rhel4u64.img,media=disk,if=virtio,boot=on,format=raw,cache=none,werror=stop -net nic,model=rtl8139,vlan=0,macaddr=00:10:16:12:10:11 -net tap,vlan=0,script=/etc/qemu-ifup -uuid 697fe4cc-49a3-41fb-a388-9275803eeefb -no-kvm-pit-reinjection -rtc-td-hack -name 64bit-63-intel -monitor stdio -vnc :1 -boot c -drive file=/var/lib/libvirt/images/rhel48-64-data.img,if=virtio,cache=none,werror=stop

[root@dhcp-91-57 ~]# uname -r
2.6.9-89.33.1.ELsmp
[root@dhcp-91-57 ~]# cat /proc/cmdline 
ro root=/dev/VolGroup00/LogVol00 rhgb quiet notsc divider=10 console=tty0

Result : no obvious time drift was found :
server 66.187.233.4, stratum 1, offset -0.011548, delay 0.27994
27 Nov 07:13:40 ntpdate[6538]: adjust time server 66.187.233.4 offset -0.011548 sec
server 66.187.233.4, stratum 1, offset -0.044367, delay 0.27991
27 Nov 07:28:42 ntpdate[6573]: adjust time server 66.187.233.4 offset -0.044367 sec
server 66.187.233.4, stratum 1, offset -0.077021, delay 0.28009
27 Nov 07:43:43 ntpdate[6575]: adjust time server 66.187.233.4 offset -0.077021 sec

2).   i686 guest.
root     13106  4489  4 06:32 pts/1    00:03:20 /usr/libexec/qemu-kvm -m 2G -smp 2 -cpu qemu64,+sse2 -drive file=/var/lib/libvirt/images/rhel4u32.img,media=disk,if=virtio,boot=on,format=raw,cache=none,werror=stop -net nic,model=rtl8139,vlan=0,macaddr=00:10:16:22:10:10 -net tap,vlan=0,script=/etc/qemu-ifup -uuid 5666eae0-4949-4f98-b296-ce15869ceee3 -no-kvm-pit-reinjection -rtc-td-hack -name 32bit-63-intel -monitor stdio -vnc :2 -boot c -drive file=/var/lib/libvirt/images/second.img,if=virtio,cache=none,werror=stop

[root@dhcp-91-60 ~]# uname -r
2.6.9-89.33.1.ELsmp
[root@dhcp-91-60 ~]# cat /proc/cmdline 
ro root=/dev/VolGroup00/LogVol00 rhgb quiet clock=pmtmr divider=10 

Result : Failed. big drift was found :
server 66.187.233.4, stratum 1, offset 0.238071, delay 0.27583
27 Nov 05:13:23 ntpdate[6524]: adjust time server 66.187.233.4 offset 0.238071 sec
server 66.187.233.4, stratum 1, offset 532.584184, delay 0.27531
27 Nov 05:28:25 ntpdate[6534]: step time server 66.187.233.4 offset 532.584184 sec
server 66.187.233.4, stratum 1, offset 533.277745, delay 0.27565
27 Nov 05:43:27 ntpdate[6536]: step time server 66.187.233.4 offset 533.277745 sec
server 66.187.233.4, stratum 1, offset 533.852720, delay 0.27513
27 Nov 05:58:29 ntpdate[18187]: step time server 66.187.233.4 offset 533.852720 sec


=====================
A Test on AMD Host
=====================

model name	: AMD Phenom(tm) 9600B Quad-Core Processor
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc nonstop_tsc pni cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy altmovcr8 abm sse4a misalignsse 3dnowprefetch osvw

[root@dhcp-91-65 images]# uname -r
2.6.18-229.el5
[root@dhcp-91-65 images]# rpm -qa kvm
kvm-83-206.el5

1). x86_64 guest
[root@dhcp-91-177 ~]# uname -r
2.6.9-89.33.1.ELsmp
[root@dhcp-91-177 ~]# cat /proc/cmdline 
ro root=/dev/VolGroup00/LogVol00 rhgb quiet notsc divider=10  console=tty0

Result : no drift was in the first hour
server 66.187.233.4, stratum 1, offset -0.007430, delay 0.28003
27 Nov 06:57:34 ntpdate[6505]: adjust time server 66.187.233.4 offset -0.007430 sec
server 66.187.233.4, stratum 1, offset -0.022844, delay 0.28004
27 Nov 07:12:37 ntpdate[6566]: adjust time server 66.187.233.4 offset -0.022844 sec
server 66.187.233.4, stratum 1, offset -0.037987, delay 0.27989
27 Nov 07:27:41 ntpdate[6585]: adjust time server 66.187.233.4 offset -0.037987 sec
server 66.187.233.4, stratum 1, offset -0.053340, delay 0.28008
27 Nov 07:42:43 ntpdate[6607]: adjust time server 66.187.233.4 offset -0.053340 sec

2). i686 guest
[root@dhcp-91-64 ~]# uname -r
2.6.9-89.33.1.ELsmp
[root@dhcp-91-64 ~]# cat /proc/cmdline 
ro root=/dev/VolGroup00/LogVol00 rhgb quiet clock=pmtmr divider=10 

Result : no drift was found in the first 30 mins .
server 66.187.233.4, stratum 1, offset 0.028464, delay 0.28011
27 Nov 07:03:19 ntpdate[6441]: adjust time server 66.187.233.4 offset 0.028464 sec
server 66.187.233.4, stratum 1, offset 0.164549, delay 0.28009
27 Nov 07:18:22 ntpdate[6461]: adjust time server 66.187.233.4 offset 0.164549 sec


I will go on doing bonnie++ testing on guest for longer time such as 1 or 2 days, and checking the time offset every 15 minutes, and give another report next monday.
Comment 14 dawu 2010-11-29 04:59:05 UTC 
Test Status for BZ586307 for 1.5 day
Following is the 2st report of timing behaviour of the 1.5 day
we have observed about 17 seconds time drift on i686 guest on AMD and Intel platform after more than 1.5 day,and 2-5 seconds time drift on x86_64 guest on AMD and Intel platform.



=====================
A Test on Intel Host
=====================

model name	: Intel(R) Core(TM)2 Quad CPU    Q9400  @ 2.66GHz
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr sse4_1 lahf_lm
bogomips	: 5320.00

RPMS:
[root@dhcp-91-63 ~]# uname -r
2.6.18-229.el5
[root@dhcp-91-63 ~]# rpm -q kvm
kvm-83-207.el5


1).   x86_64 guest .
root     13083  4453  6 06:32 pts/0    00:04:54 /usr/libexec/qemu-kvm -m 2G -smp 2 -cpu qemu64,+sse2 -drive file=/var/lib/libvirt/images/rhel4u64.img,media=disk,if=virtio,boot=on,format=raw,cache=none,werror=stop -net nic,model=rtl8139,vlan=0,macaddr=00:10:16:12:10:11 -net tap,vlan=0,script=/etc/qemu-ifup -uuid 697fe4cc-49a3-41fb-a388-9275803eeefb -no-kvm-pit-reinjection -rtc-td-hack -name 64bit-63-intel -monitor stdio -vnc :1 -boot c -drive file=/var/lib/libvirt/images/rhel48-64-data.img,if=virtio,cache=none,werror=stop

[root@dhcp-91-57 ~]# uname -r
2.6.9-89.33.1.ELsmp
[root@dhcp-91-57 ~]# cat /proc/cmdline 
ro root=/dev/VolGroup00/LogVol00 rhgb quiet notsc divider=10 console=tty0

Result : there is litter time drift after 32 hours, almost 1 seconds increment for every 8 hours,please refer to the following details:

27 Nov 08:13:46 ntpdate[6593]: adjust time server 66.187.233.4 offset -0.142620 sec
server 66.187.233.4, stratum 1, offset -0.175610, delay 0.27966
27 Nov 16:37:50 ntpdate[6956]: step time server 66.187.233.4 offset -1.007589 sec
server 66.187.233.4, stratum 1, offset -1.038314, delay 0.28003
28 Nov 00:08:57 ntpdate[7280]: step time server 66.187.233.4 offset -2.005197 sec
server 66.187.233.4, stratum 1, offset -2.021388, delay 0.28030
28 Nov 08:02:04 ntpdate[23859]: step time server 66.187.233.4 offset -3.033745 sec
server 66.187.233.4, stratum 1, offset -3.051843, delay 0.27966
28 Nov 16:02:47 ntpdate[24197]: step time server 66.187.233.4 offset -4.064722 sec
server 66.187.233.4, stratum 1, offset -4.078792, delay 0.32129


2).   i686 guest.
root     13106  4489  4 06:32 pts/1    00:03:20 /usr/libexec/qemu-kvm -m 2G -smp 2 -cpu qemu64,+sse2 -drive file=/var/lib/libvirt/images/rhel4u32.img,media=disk,if=virtio,boot=on,format=raw,cache=none,werror=stop -net nic,model=rtl8139,vlan=0,macaddr=00:10:16:22:10:10 -net tap,vlan=0,script=/etc/qemu-ifup -uuid 5666eae0-4949-4f98-b296-ce15869ceee3 -no-kvm-pit-reinjection -rtc-td-hack -name 32bit-63-intel -monitor stdio -vnc :2 -boot c -drive file=/var/lib/libvirt/images/second.img,if=virtio,cache=none,werror=stop

[root@dhcp-91-60 ~]# uname -r
2.6.9-89.33.1.ELsmp
[root@dhcp-91-60 ~]# cat /proc/cmdline 
ro root=/dev/VolGroup00/LogVol00 rhgb quiet clock=pmtmr divider=10 

Result : Failed. big drift was found after 32 hours, almost 1 seconds increment for every 2 hours,please refer to the following details:
27 Nov 08:48:27 ntpdate[6617]: adjust time server 66.187.233.4 offset 0.096371 sec
server 66.187.233.4, stratum 1, offset 0.215407, delay 0.28000
27 Nov 10:49:36 ntpdate[6810]: step time server 66.187.233.4 offset 1.052544 sec
server 66.187.233.4, stratum 1, offset 1.172695, delay 0.27998
27 Nov 12:50:20 ntpdate[6972]: step time server 66.187.233.4 offset 2.007129 sec
server 66.187.233.4, stratum 1, offset 2.126100, delay 0.28033
27 Nov 15:06:39 ntpdate[7142]: step time server 66.187.233.4 offset 3.098766 sec
server 66.187.233.4, stratum 1, offset 3.217681, delay 0.30984
27 Nov 17:07:31 ntpdate[7304]: step time server 66.187.233.4 offset 4.069576 sec
server 66.187.233.4, stratum 1, offset 4.192323, delay 0.31709
....
28 Nov 16:20:16 ntpdate[25479]: step time server 66.187.233.4 offset 15.069279 sec
server 66.187.233.4, stratum 1, offset 15.174743, delay 0.28004
28 Nov 18:21:15 ntpdate[25639]: step time server 66.187.233.4 offset 16.015031 sec
server 66.187.233.4, stratum 1, offset 16.130571, delay 0.28009



=====================
A Test on AMD Host
=====================

model name	: AMD Phenom(tm) 9600B Quad-Core Processor
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc nonstop_tsc pni cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy altmovcr8 abm sse4a misalignsse 3dnowprefetch osvw

[root@dhcp-91-65 images]# uname -r
2.6.18-229.el5
[root@dhcp-91-65 images]# rpm -qa kvm
kvm-83-206.el5

1). x86_64 guest
[root@dhcp-91-177 ~]# uname -r
2.6.9-89.33.1.ELsmp
[root@dhcp-91-177 ~]# cat /proc/cmdline 
ro root=/dev/VolGroup00/LogVol00 rhgb quiet notsc divider=10  console=tty0

Result : no obvious drift was found in the 32 hours,almost 1 seconds increment for every 17 hours,please refer to the following details:
27 Nov 08:27:47 ntpdate[6656]: adjust time server 66.187.233.4 offset -0.002572 sec
server 66.187.233.4, stratum 1, offset -0.003646, delay 0.27985
28 Nov 01:15:41 ntpdate[8529]: step time server 66.187.233.4 offset -1.008918 sec
server 66.187.233.4, stratum 1, offset -1.009079, delay 0.27985
28 Nov 18:17:19 ntpdate[21629]: step time server 66.187.233.4 offset -2.003760 sec
server 66.187.233.4, stratum 1, offset -2.004133, delay 0.31490


2). i686 guest
[root@dhcp-91-64 ~]# uname -r
2.6.9-89.33.1.ELsmp
[root@dhcp-91-64 ~]# cat /proc/cmdline 
ro root=/dev/VolGroup00/LogVol00 rhgb quiet clock=pmtmr divider=10 

Result : Failed. big drift was found after 32 hours, almost 1 seconds increment for every 2 hours,please refer to the following details:
7 Nov 08:02:50 ntpdate[6522]: adjust time server 66.187.233.4 offset 0.003507 sec
server 66.187.233.4, stratum 1, offset 0.147514, delay 0.28105
27 Nov 10:03:09 ntpdate[6947]: step time server 66.187.233.4 offset 1.122628 sec
server 66.187.233.4, stratum 1, offset 1.259722, delay 0.27994
27 Nov 11:48:30 ntpdate[7143]: step time server 66.187.233.4 offset 2.080055 sec
server 66.187.233.4, stratum 1, offset 2.223191, delay 0.29288
...
28 Nov 15:11:07 ntpdate[22199]: step time server 66.187.233.4 offset 17.116669 sec
server 66.187.233.4, stratum 1, offset 17.235999, delay 0.28018
28 Nov 16:56:24 ntpdate[22413]: step time server 66.187.233.4 offset 18.070935 sec
Comment 21 Igor Zhang 2010-11-30 15:04:09 UTC 
Thanks Glauber. Set it VERIFIED.
Comment 24 errata-xmlrpc 2010-12-01 22:25:12 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0936.html
Comment 25 ilya m. 2010-12-28 16:50:51 UTC 
Would this affect VmWare ESX(i) as well?
Comment 26 Paolo Bonzini 2011-01-03 14:33:09 UTC 
We couldn't reproduce the bug with ESX on normal workloads (only with a special kernel module that triggered the bug), but the patch would fix ESX as well.
Note You need to log in before you can comment on or make changes to this bug.