Bug 703335 - KVM guest clocks jump forward one hour on reboot
KVM guest clocks jump forward one hour on reboot
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kvm (Show other bugs)
5.6
x86_64 Linux
high Severity high
: rc
: ---
Assigned To: Gleb Natapov
Virtualization Bugs
: Reopened
: 704949 (view as bug list)
Depends On:
Blocks: Rhel5KvmTier1
  Show dependency treegraph
 
Reported: 2011-05-09 22:58 EDT by Mark Wu
Modified: 2013-12-08 19:54 EST (History)
14 users (show)

See Also:
Fixed In Version: kvm-83-245.el5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-02-20 22:14:24 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Customer has verified this patch can fix the problem (917 bytes, patch)
2011-05-09 23:02 EDT, Mark Wu
no flags Details | Diff

  None (edit)
Description Mark Wu 2011-05-09 22:58:07 EDT
Description of problem:
I have a RHEL 5.6 installation configured to be a KVM host. This host has been updated with the latest errata using yum.  I have installed a guest that is also running RHEL 5.6 (it has no errata installed).
When I first start up the guest, it has the correct time. When I reboot the guest, its time jumps forward one hour. If I reboot the guest again, the time does not change again, but it is one ahead of the host.

If I destroy the guest and restart it, it starts up with the same time as the host. The time will jump forward one hour if the guest is rebooted, however. 

Version-Release number of selected component (if applicable):
kvm-83-224.el5
kvm guest: rhel5.6
kvm host: rhel5.6

How reproducible:
100%

Steps to Reproduce:
1.Set the time zone of host to "America/New_York", which is important since this issue only happens in daylight saving time.
2.Install a guest on kvm host.
3.After installation, run the kvm guest. It should have correct time.
4.Reboot the guest.
  
Actual results:
guest clock jump forward one hour on reboot

Expected results:
no jump on reboot

Additional info:

INIT: version 2.86 booting
                Welcome to Red Hat Enterprise Linux Server
                Press 'I' to enter interactive startup.
Thu 05 May 2011 10:19:05 AM EDT -0.840993 seconds
Setting clock  (utc): Thu May  5 10:19:06 EDT 2011 [  OK  ]
Starting udev: [  OK  ]
Loading default keymap (us): [  OK  ]
Setting hostname localhost.localdomain:  [  OK  ]


On shutdown:

Starting killall:  [  OK  ]
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Saving random seed:
Thu May 5 10:20:51 EDT 2011
Syncing hardware clock to system time type=1111 audit(1304605251.986:16): user pid=3003 uid=0 auid=4294967295 subj=system_u:system_r:hwclock_t:s0 msg='changing system time: exe="/sbin/hwclock" (hostname=?, addr=?, terminal=console res=success)'

Turning off swap:
Turning off quotas:
Unmounting pipe file systems:

Second boot:

INIT: version 2.86 booting
                Welcome to Red Hat Enterprise Linux Server
                Press 'I' to enter interactive startup.
Thu 05 May 2011 11:21:41 AM EDT -0.789181 seconds
Setting clock  (utc): Thu May  5 11:21:42 EDT 2011 [  OK  ]
Starting udev: [  OK  ]
Loading default keymap (us): [  OK  ]
Setting hostname localhost.localdomain:  [  OK  ]
Comment 3 Mike Cao 2011-05-16 03:06:41 EDT
*** Bug 704949 has been marked as a duplicate of this bug. ***
Comment 4 Glauber Costa 2011-05-18 09:07:14 EDT
This seems to be a non-issue, guest-configuration related. a correctly configured ntp should take care of this.

This is corroborated by your statement:

"1.Set the time zone of host to "America/New_York", which is important since
this issue only happens in daylight saving time."

I will close this as notabug, since it is what it seems.
If you still think otherwise, please reopen
Comment 8 Ronen Hod 2011-08-08 10:57:50 EDT
Gleb, I believe that it is the same as
https://bugzilla.redhat.com/show_bug.cgi?id=632541
Comment 9 Gleb Natapov 2011-08-08 11:02:36 EDT
(In reply to comment #8)
> Gleb, I believe that it is the same as
> https://bugzilla.redhat.com/show_bug.cgi?id=632541

Does not look like it to me.
Comment 11 Rik van Riel 2011-10-24 10:25:49 EDT
It looks like this could be a user error after all. The details are somewhat subtle, so please bear with me:

1) the user creates a new virtual machine, installs it without clicking the "bios clock uses utc" option in the installer
2) the virtual machine runs fine the first time around
3) the virtual machine shuts down, saves its time of day to the qemu bios clock (with DST offset!)
4) the virtual machine restarts, loads the qemu bios time and re-applies the DST offset to it ... now the time is wrong

The same issue can happen on bare metal.

If the virtual machine is installed with the "bios clock uses utc" option, everything should work as expected. 

In an already installed guest, this can be enabled by appending "UTC=true" to /etc/sysconfig/clock
Comment 12 Ronen Hod 2011-10-24 11:16:34 EDT
David,

We encountered such issues only when the clock config was inconsistent, and other users did not complain about this issue.

We believe that if you
- Run qemu without '-rtc base=localtime' (Probably OK)
- Configure the guest to use utc as explained in the previous comment
- Make sure that the host's time (TZ, daylight) is correct (Probably is)
it should work.

Note that any guest save to "CMOS" is persistent only as long as QEMU is up, and will be reset when QEMU is restarted, so it cannot be trusted.

Can you try it?
Thanks, Ronen.
Comment 13 Roland Friedwagner 2011-10-27 06:37:27 EDT
I can confirm this bug and yet do not see
a guest configuration error.
But there may be a bug in qemu triggered by "/sbin/hwclock --systohc -u".

The jump is triggered when the init script /etc/init.d/halt is executed on
shutdown and does a "/sbin/hwclock --systohc -u".

You can reproduce this:

  $ echo $TZ

  $ echo $LANG
  C
  $ echo $LC_ALL

  $ date; /sbin/hwclock --utc; /sbin/hwclock --systohc -u; /sbin/hwclock --utc
  Thu Oct 27 12:22:27 CEST 2011
  Thu Oct 27 12:22:28 2011  -0.434616 seconds
  Thu Oct 27 13:22:30 2011  -0.987230 seconds
  $ date; /sbin/hwclock --utc; /sbin/hwclock --systohc -u; /sbin/hwclock --utc
  Thu Oct 27 12:22:34 CEST 2011
  Thu Oct 27 13:22:35 2011  -0.872532 seconds
  Thu Oct 27 12:22:37 2011  -0.986939 seconds
  $ date; /sbin/hwclock --utc; /sbin/hwclock --systohc -u; /sbin/hwclock --utc
  Thu Oct 27 12:22:41 CEST 2011
  Thu Oct 27 12:22:42 2011  -0.521600 seconds
  Thu Oct 27 12:22:44 2011  -0.986640 seconds
  $ date; /sbin/hwclock --utc; /sbin/hwclock --systohc -u; /sbin/hwclock --utc
  Thu Oct 27 12:22:44 CEST 2011
  Thu Oct 27 12:22:45 2011  -0.089753 seconds
  Thu Oct 27 12:22:47 2011  -0.987135 seconds
  $ date; /sbin/hwclock --utc; /sbin/hwclock --systohc -u; /sbin/hwclock --utc
  Thu Oct 27 12:22:48 CEST 2011
  Thu Oct 27 12:22:49 2011  -0.745615 seconds
  Thu Oct 27 12:22:51 2011  -0.987360 seconds

=> So only the first write of cmos clock does trigger the jump.
   Further writes are ok.

WORKAROUND:
--- /etc/init.d/halt.ori        2011-06-07 23:56:18.000000000 +0200
+++ /etc/init.d/halt    2011-10-27 12:30:02.000000000 +0200
@@ -129,7 +129,7 @@
        ;;
 esac

-[ -x /sbin/hwclock ] && action $"Syncing hardware clock to system time" /sbin/hwclock $CLOCKFLAGS
+[ -x /sbin/hwclock ] && action $"Syncing hardware clock to system time" /sbin/hwclock $CLOCKFLAGS; /sbin/hwclock $CLOCKFLAGS;

 # Try to unmount tmpfs filesystems to avoid swapping them in.  Ignore failures.
 tmpfs=$(awk '$2 ~ /^\/($|proc|dev)/ { next; }



qemu-kvm version:

  kvm-83-239.el5

qemu-kvm cmdline:

  LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin HOME=/ QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel5.6.0 -m 16384 -mem-prealloc -mem-path /hugepages/libvirt/qemu -smp 1,sockets=1,cores=1,threads=1 -name test -uuid 3975d5fb-7a1c-448d-90b7-38ab1c48774d -monitor unix:/var/lib/libvirt/qemu/test.monitor,server,nowait -no-kvm-pit-reinjection -boot c -drive file=/mnt/kvm_images/test.raw,if=virtio,boot=on,format=raw,cache=none -drive if=ide,media=cdrom,bus=1,unit=0,readonly=on,format=raw -net nic,macaddr=54:52:00:1e:6b:b0,vlan=0,model=virtio -net tap,fd=17,vlan=0 -serial pty -parallel none -usb -vnc 127.0.0.1:0 -k de -vga cirrus -balloon virtio

Guest clock settings:

  /etc/sysconfig/clock:
    ZONE="Europe/Vienna"
    UTC=true
    ARC=false
  /etc/adjtime:
    -0.001264 1319701867 0.000000
    1319701867
    UTC

Host clock settings:

  /etc/sysconfig/clock:
    ZONE="Europe/Vienna"
    UTC=true
    ARC=false
  /etc/adjtime:
    -1.562533 1319021384 0.000000
    1319021384
    UTC

Kind Regards,
Roland
Comment 14 Ronen Hod 2011-11-07 08:38:30 EST
Gleb,
You said in an email

First of all if attached patch is really fixing the problem (which I
didn't try) then bug was likely introduced by 014f9f7f01858 since this
code path shouldn't change stored time at all. I would try to revert it
and check. ...
Comment 16 Ronen Hod 2011-11-14 07:57:45 EST
The patch got 3 acks, and since it was reported by a customer, we would like to include it in 5.8.
Comment 20 Shaolong Hu 2011-12-12 00:01:34 EST
I try to reproduce and verify the bug, but kvm 239 and 246 gives the same result:

guest:
2.6.18-274.el5
#cat /etc/sysconfig/clock
ZONE="America/New_York"
UTC=false
ARC=false

host:
#cat /etc/sysconfig/clock
ZONE="America/New_York"
UTC=true
ARC=false

with kvm-83-246.el5:
date; /sbin/hwclock --utc; /sbin/hwclock --systohc -u; /sbin/hwclock --utc
Mon Dec 12 03:42:47 CST 2011
Mon 12 Dec 2011 11:42:48 AM CST  -0.218349 seconds
Mon 12 Dec 2011 03:42:50 AM CST  -0.987314 seconds
date; /sbin/hwclock --utc; /sbin/hwclock --systohc -u; /sbin/hwclock --utc
Mon Dec 12 03:42:54 CST 2011
Mon 12 Dec 2011 03:42:55 AM CST  -0.626617 seconds
Mon 12 Dec 2011 03:42:57 AM CST  -0.992545 seconds
date; /sbin/hwclock --utc; /sbin/hwclock --systohc -u; /sbin/hwclock --utc
Mon Dec 12 03:43:10 CST 2011
Mon 12 Dec 2011 03:43:11 AM CST  -0.560703 seconds
Mon 12 Dec 2011 03:43:13 AM CST  -0.991310 seconds

with kvm-83-239.el5
date; /sbin/hwclock --utc; /sbin/hwclock --systohc -u; /sbin/hwclock --utc
Mon Dec 12 04:48:51 CST 2011
Mon 12 Dec 2011 12:48:52 PM CST  -0.293388 seconds
Mon 12 Dec 2011 04:48:55 AM CST  -0.992667 seconds
date; /sbin/hwclock --utc; /sbin/hwclock --systohc -u; /sbin/hwclock --utc
Mon Dec 12 04:48:55 CST 2011
Mon 12 Dec 2011 04:48:56 AM CST  -0.063632 seconds
Mon 12 Dec 2011 04:48:58 AM CST  -0.991397 seconds
date; /sbin/hwclock --utc; /sbin/hwclock --systohc -u; /sbin/hwclock --utc
Mon Dec 12 04:48:59 CST 2011
Mon 12 Dec 2011 04:49:00 AM CST  -0.555515 seconds
Mon 12 Dec 2011 04:49:02 AM CST  -0.994068 seconds

Also i observe first time reboot, there is no time jump with both 239 and 246, please correct me if i am wrong, thanks.
Comment 21 Gleb Natapov 2011-12-12 03:56:34 EST
(In reply to comment #20)
> I try to reproduce and verify the bug, but kvm 239 and 246 gives the same

The bug is reproducible only when host is in the day light saving mode, so you either need to wait till next summer to verify it or configure host time to summer time manually.
Comment 22 Shaolong Hu 2011-12-13 01:51:29 EST
guest:
2.6.18-274.el5
#cat /etc/sysconfig/clock
ZONE="America/New_York"
UTC=true
ARC=false

host:
#cat /etc/sysconfig/clock
ZONE="America/New_York"
UTC=true
ARC=false

Reproduced on kvm-83-239.el5.x86_64:
Thu May  5 18:33:20 CST 2011
Thu 05 May 2011 06:33:21 PM CST  -0.236552 seconds
Thu 05 May 2011 07:33:23 PM CST  -0.984625 seconds
Thu May  5 18:33:23 CST 2011
Thu 05 May 2011 07:33:24 PM CST  -0.246500 seconds
Thu 05 May 2011 06:33:26 PM CST  -0.985252 seconds
Thu May  5 18:33:26 CST 2011
Thu 05 May 2011 06:33:27 PM CST  -0.352006 seconds
Thu 05 May 2011 06:33:29 PM CST  -0.985707 seconds

Verified on kvm-83-246.el5.x86_64:
Thu May  5 18:29:07 CST 2011
Thu 05 May 2011 06:29:08 PM CST  -0.153188 seconds
Thu 05 May 2011 06:29:10 PM CST  -0.983594 seconds
Thu May  5 18:29:10 CST 2011
Thu 05 May 2011 06:29:11 PM CST  -0.348462 seconds
Thu 05 May 2011 06:29:13 PM CST  -0.984948 seconds
Thu May  5 18:29:13 CST 2011
Thu 05 May 2011 06:29:14 PM CST  -0.358444 seconds
Thu 05 May 2011 06:29:16 PM CST  -0.985231 seconds

Fixed.
Comment 24 errata-xmlrpc 2012-02-20 22:14:24 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0149.html

Note You need to log in before you can comment on or make changes to this bug.