Description of problem: With the latest FC6 Xen kernel the system time is completely unreliable, jumping back and forth(!) even in dom0. [ask@app2 ~]$ uname -a Linux app2.la.sol 2.6.20-1.2933.fc6xen #1 SMP Mon Mar 19 11:17:46 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux [ask@app2 ~]$ while `/bin/true`; do date; sleep 1; done Thu Apr 12 16:33:16 PDT 2007 Thu Apr 12 16:33:17 PDT 2007 Thu Apr 12 16:33:18 PDT 2007 Thu Apr 12 16:33:18 PDT 2007 Thu Apr 12 16:33:20 PDT 2007 Thu Apr 12 16:33:21 PDT 2007 Thu Apr 12 16:33:22 PDT 2007 Thu Apr 12 16:33:23 PDT 2007 Thu Apr 12 16:33:24 PDT 2007 Thu Apr 12 16:33:24 PDT 2007 Thu Apr 12 16:33:26 PDT 2007 Thu Apr 12 16:33:27 PDT 2007 Thu Apr 12 16:33:28 PDT 2007 Thu Apr 12 16:33:28 PDT 2007 Thu Apr 12 16:33:30 PDT 2007 (the seconds that appears to be okay are only that because it seems to more or less average out to be correct over time - each line is very "jumpy"). On the non-xen kernel I get: [root@app3 ~]# uname -a Linux app3.la.sol 2.6.20-1.2933.fc6 #1 SMP Mon Mar 19 11:00:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux [root@app3 ~]# while `/bin/true`; do date; sleep 1; done Thu Apr 12 16:20:28 PDT 2007 Thu Apr 12 16:20:29 PDT 2007 Thu Apr 12 16:20:30 PDT 2007 Thu Apr 12 16:20:31 PDT 2007 Version-Release number of selected component (if applicable): How reproducible: Always.
The two boxes above are dual quad-core Intel Xeon 2.33Ghz; I see the same problem on a dual dual-core Intel Xeon 2.00Ghz. http://www.siliconmechanics.com/i5917/Xeon-Server.php dmesg of xen kernel at http://tmp.askask.com/2007/04/app2-xen.dmesg.txt dmesg of no-xen kernel at http://tmp.askask.com/2007/04/app3-noxen.dmesg.txt
FWIW, it seems to get better if I disable apic ...
A test using 'date --rfc-3339 ns' in the place of the 'date' command may make the problem more visible. Could you do this test?
very cool, didn't know about that option. Here's a non-xen kernel: [root@app3 ~]# while `/bin/true`; do date --rfc-3339 ns; sleep 1; done 2007-04-13 12:19:42.766020000-07:00 2007-04-13 12:19:43.770184000-07:00 2007-04-13 12:19:44.774319000-07:00 2007-04-13 12:19:45.777746000-07:00 2007-04-13 12:19:46.781122000-07:00 2007-04-13 12:19:47.790459000-07:00 2007-04-13 12:19:48.794074000-07:00 2007-04-13 12:19:49.797861000-07:00 2007-04-13 12:19:50.801986000-07:00 2007-04-13 12:19:51.805114000-07:00 Here's from the xen kernel: [root@app2 ~]# while /bin/true; do date --rfc-3339 ns; sleep 1; done 2007-04-13 12:20:15.672158000-07:00 2007-04-13 12:20:16.643917000-07:00 2007-04-13 12:20:17.672203000-07:00 2007-04-13 12:20:17.643942000-07:00 2007-04-13 12:20:19.006156000-07:00 2007-04-13 12:20:19.671671000-07:00 2007-04-13 12:20:20.672290000-07:00 2007-04-13 12:20:21.671717000-07:00 2007-04-13 12:20:21.656193000-07:00 2007-04-13 12:20:22.644064000-07:00 2007-04-13 12:20:23.672360000-07:00 2007-04-13 12:20:24.672386000-07:00 2007-04-13 12:20:25.582588000-07:00 2007-04-13 12:20:26.672925000-07:00 2007-04-13 12:20:27.672468000-07:00 2007-04-13 12:20:28.582667000-07:00 2007-04-13 12:20:28.672996000-07:00 2007-04-13 12:20:29.671917000-07:00 2007-04-13 12:20:29.656400000-07:00 2007-04-13 12:20:31.006121000-07:00 2007-04-13 12:20:31.672586000-07:00 2007-04-13 12:20:33.672630000-07:00 I booted the xen kernel with apic=off to see if it made a difference, but if it did it wasn't much: title Fedora Core (2.6.20-1.2933.fc6xen) root (hd0,0) kernel /xen.gz-2.6.20-1.2933.fc6 com1=9600,8n1 com2=115200,8n1 apic=off module /vmlinuz-2.6.20-1.2933.fc6xen ro root=/dev/vg0/root console=tty0 console=ttyS0 module /initrd-2.6.20-1.2933.fc6xen.img
Here's with the new kernel (notice how it starts out going backwards in time ...) [root@app2 ~]# while /bin/true; do date --rfc-3339 ns; sleep 1; done 2007-04-16 23:02:48.981407000-07:00 2007-04-16 23:02:48.930131000-07:00 2007-04-16 23:02:50.820784000-07:00 2007-04-16 23:02:50.930164000-07:00 2007-04-16 23:02:51.791174000-07:00 2007-04-16 23:02:51.953771000-07:00 2007-04-16 23:02:52.819448000-07:00 2007-04-16 23:02:53.019187000-07:00 2007-04-16 23:02:53.797581000-07:00 2007-04-16 23:02:54.981535000-07:00 2007-04-16 23:02:54.953835000-07:00 2007-04-16 23:02:55.930285000-07:00 2007-04-16 23:02:56.791280000-07:00 2007-04-16 23:02:56.797647000-07:00 2007-04-16 23:02:57.981602000-07:00 2007-04-16 23:02:57.953897000-07:00 2007-04-16 23:02:58.930349000-07:00 2007-04-16 23:02:59.791346000-07:00 2007-04-16 23:02:59.904411000-07:00 2007-04-16 23:03:00.930392000-07:00 2007-04-16 23:03:01.791389000-07:00 2007-04-16 23:03:02.797772000-07:00 2007-04-16 23:03:02.821041000-07:00 2007-04-16 23:03:03.954014000-07:00 2007-04-16 23:03:04.904499000-07:00 2007-04-16 23:03:04.930476000-07:00 2007-04-16 23:03:05.791478000-07:00 2007-04-16 23:03:06.797861000-07:00 2007-04-16 23:03:06.821126000-07:00 2007-04-16 23:03:07.954106000-07:00
[root@app2 ~]# uname -a Linux app2.la.sol 2.6.20-1.2944.fc6xen #1 SMP Tue Apr 10 18:03:37 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux If it can help I think we can arrange for something from RedHat to get access to one of these systems. (The boxes have an IPMI KVM and a serial console and IPDU power, so they are easy to mess with remotely). - ask
Hi, Sorry for taking so long to take a look on this bug. Could you check the results of the same command (while /bin/true; do date --rfc-3339 ns; sleep 1; done), using the kernel command line option "independent_wallclock"? Also, could you attach dmesg output when using the following command line options: - Using both: independent_wallclock permitted_clock_jitter=0 - Using only: permitted_clock_jitter=0 Probably it will flood your console with messages, but the information from them will be useful.
change QA contact
Hi Eduardo, Darn - I missed your comment earlier. FWIW changing independent_wallclock in the dom0 (with sysctl) doesn't seem to make a difference. I'll try with permitted_clock_jitter=0 later. - ask Linux app3.la.sol 2.6.20-2925.13.fc7xen #1 SMP Tue Jul 17 10:38:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux [ask@app3 ~]$ while /bin/true; do date --rfc-3339 ns; sleep 1; done 2007-08-07 18:31:35.155215000-07:00 2007-08-07 18:31:37.070061000-07:00 2007-08-07 18:31:38.081759000-07:00 2007-08-07 18:31:39.317794000-07:00 2007-08-07 18:31:41.120665000-07:00 2007-08-07 18:31:42.932862000-07:00 2007-08-07 18:31:44.005449000-07:00 2007-08-07 18:31:45.029116000-07:00 2007-08-07 18:31:46.037569000-07:00
The FC-6 updates system haven't added an automatic note on this bug and I haven't noticed. A patch was included since 2.6.20-1.3001.fc6 that should help fix this issue. Could you check what are the results when using 2.6.20-1.3001.fc6 or newer? If you are using Fedora 7, the patch was added on 2.6.20-2930.fc7.
This report targets FC6, which is now end-of-life. Please re-test against Fedora 7 or later, and if the issue persists, open a new bug. Thanks