Bug 672194

Summary: "hwclock --systz" sets the wrong time when hw clock tracks localtime
Product: [Fedora] Fedora Reporter: Peter Hjalmarsson <kanelxake>
Component: util-linuxAssignee: Karel Zak <kzak>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: atu, harald, johannbg, jonathan, kzak, lpoetter, metherid, mschmidt, notting, plautrba, sangu.fedora, tmraz
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 819945 (view as bug list) Environment:
Last Closed: 2012-02-21 18:07:23 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 819945, 827141, 829314    
Attachments:
Description Flags
untested systemd patch
none
untested systemd patch none

Description Peter Hjalmarsson 2011-01-24 11:02:50 UTC
I hit this bug in rawhide that seems related to hwclock and its handling of --systz. I am not sure if I saw this with Fedora14 too, think I did but cannot really remember or test it currently.


In short my BIOS clock is adjusted to local time.
/etc/adjtime has LOCAL as its third line
My timezone is set to "Europe/Stockholm"
Currently no DST.

Every reboot "hwclock --systz" change my clock back one hour (i.e. if my bios clock is 10.15 Fedora reports 09.15).
This becomes quiet annoying if I have to do two reboots or more without network (ntp automatically set the correct time), as it seems ath5k and other stuff reports problems is the system time change too much or jumping back and forth.

If you run "hwclock --systz" on my system it changes my clock back one hour every time I execute it, also with TZ="Europe/Stockholm" or "CET" set, but with "GMT+1" hwclock sets the clock forward one hour.
I tried TZ="Isreael" and it changed the clock backwards two hours (Israel/Jerusalem is supposed to be GMT+2 IIRC).
If I change /etc/adjtime to UTC (a no-go long term as I dual-boot windows) "hwclock --systz" does not change the system time whatever I set TZ to.

Also: if I remove "hwclock --systz" from my bootup and then run date from a console then it both reports the correct time, date and timezone (as CET which is also valid for Europe/Stockholm).

So the question I ask myself is what is "hwclock --systz" trying to do? Is it trying to set the system time to UTC?
Also is "hwclock --systz" really supposed to change the system time if the hardware clock tracks localtime? 

I would expect the reaction of "hwclock --systz" should be to set the system time to local time. So for example if BIOS tracks UTC and the system should show CET then hwclock should change system time forward one hour, and if BIOS tracks localtime hwclock should leave the time alone.
Currently it does to totally opposite on my system.

Comment 1 Karel Zak 2011-02-23 10:08:31 UTC
Please, read man hwclock, section "Clocks in a Linux System".

The System Clock (a clock inside the kernel) is always in UTC. It means that hwclock changes the clock from LOCAL to UTC (it also set in-kernel timezone because it's used by some filesystems, but it's unimportant detail).

The correct System Clock interpretation is happen in userspace by TZ variable.

Comment 2 Peter Hjalmarsson 2011-02-23 10:59:36 UTC
(In reply to comment #1)
> Please, read man hwclock, section "Clocks in a Linux System".
>
> The System Clock (a clock inside the kernel) is always in UTC.

So let me see if I get this straight:
System-clock is supposed to be UTC.
And hwclock is supposed to set System-clock to UTC if Hardware-clock is running localtime.

That thinking currently seems to break nearly everything...

The short description of what went wrong and how to reproduce this is the following:

1. Install Fedora 15 (I think I had this problem already in 14)
2. Makes sure the hardware clock tracks localtime.
3. Make sure you have LOCAL in /etc/adjtime and you have CET as tz.
4. Remove all network connections.
5. Boot the computer.
6. login and watch how gnome-shell/gnome-panel shows your correct time -1h.
7. open a terminal, run "date" and see the time it reports.
8. connect to a network, run ntpdate and watch how gnome-shell, gnome-panel and date jumps one hour forward.


So when /etc/adjtime is set to LOCAL, then date and ntp is broken and hwclock is right in changing the system clock between LOCAL(kernel) -> UTC(hwclock) -> LOCAL(ntpdate)

And gnome-panel, gnome-shell, date and so on is wrong (remember the hwclock is localtime, and observe that date shows CET which is supposed to be my local timezone):
$ sudo hwclock && date
ons 23 feb 2011 11.42.39  -0.579638 sekunder
ons feb 23 10:42:39 CET 2011

Comment 3 Karel Zak 2011-02-23 12:30:53 UTC
(In reply to comment #2)
> And gnome-panel, gnome-shell, date and so on is wrong (remember the hwclock is
> localtime, and observe that date shows CET which is supposed to be my local
> timezone):
> $ sudo hwclock && date
> ons 23 feb 2011 11.42.39  -0.579638 sekunder
> ons feb 23 10:42:39 CET 2011

The hwclock --show (the default option) does not print raw hw clock, it uses localtime() function. Use hwclock --show --debug to see more details. 

Please, try (as root):

  # cat /sys/class/rtc/rtc0/hctosys
  # hwclock --show --debug
  # tail -1 /etc/adjtime
  # echo $TZ
  # zdump /etc/localtime
  # date

Comment 4 Peter Hjalmarsson 2011-02-24 05:39:14 UTC
# cat /sys/class/rtc/rtc0/hctosys
1
# hwclock --show --debug
hwclock from util-linux 2.19
Using /dev interface to clock.
Last drift adjustment done at 1298491573 seconds after 1969
Last calibration done at 1298491573 seconds after 1969
Hardware clock is on local time
Assuming hardware clock is kept in local time.
Waiting for clock tick...
...got clock tick
Time read from Hardware Clock: 2011/02/24 06:24:11
Hw clock time : 2011/02/24 06:24:11 = 1298525051 seconds since 1969
Thu Feb 24 06:24:11 2011  -0.187013 seconds
# tail -1 /etc/adjtime
LOCAL
# echo $TZ

# zdump /etc/localtime
/etc/localtime  Thu Feb 24 05:24:10 2011 CET
# date
Thu Feb 24 05:24:10 CET 2011


Since I saw TZ was empty (which does not stop date from picking up the correct timezeon) I created the following file:
$ cat /etc/profile.d/tz.sh 
export TZ="Europe/Stockholm"

After a reboot:
# cat /sys/class/rtc/rtc0/hctosys
1
# hwclock --show --debug
hwclock from util-linux 2.19
Using /dev interface to clock.
Last drift adjustment done at 1298491573 seconds after 1969
Last calibration done at 1298491573 seconds after 1969
Hardware clock is on local time
Assuming hardware clock is kept in local time.
Waiting for clock tick...
...got clock tick
Time read from Hardware Clock: 2011/02/24 05:30:05
Hw clock time : 2011/02/24 05:30:05 = 1298521805 seconds since 1969
Thu Feb 24 05:30:05 2011  -0.207492 seconds
# tail -1 /etc/adjtime
LOCAL
# echo $TZ
Europe/Stockholm
# zdump /etc/localtime
/etc/localtime  Thu Feb 24 04:30:04 2011 CET
# date
Thu Feb 24 04:30:04 CET 2011


As you can see, no change.
Also, if you notices that the second output is one hour earlier then the first, then the answer is simple: I did not have the computer connected to the net, and thus did not let ntpdate update the time before rebooting. So with other word hwclock does adjust system clock backwards one hour at boot, but then stores the new time to hardware clock during shutdown.
gnome-shell in gdm shows the same time as date in all cases.

Comment 5 Clyde E. Kunkel 2011-06-26 17:10:18 UTC
In rawhide current as of 20110626, gnome shell clock time does not match hardwae clock.  BIOS is correct.  Nothing in $TZ. Setting timezone in Date and Time does not hold on reboot :


# cat /sys/class/rtc/rtc0/hctosys
1
# hwclock --show --debug
hwclock from util-linux 2.19.1
Using /dev interface to clock.
Last drift adjustment done at 1309099283 seconds after 1969
Last calibration done at 1309099283 seconds after 1969
Hardware clock is on local time
Assuming hardware clock is kept in local time.
Waiting for clock tick...
...got clock tick
Time read from Hardware Clock: 2011/06/26 12:50:30
Hw clock time : 2011/06/26 12:50:30 = 1309107030 seconds since 1969
Sun 26 Jun 2011 12:50:30 PM EDT  -0.736332 seconds
# tail -1 /etc/adjtime
LOCAL
# echo TZ

# zdump /etc/localtime
/etc/localtime  Sun Jun 26 20:50:29 2011 EDT
# date
Sun Jun 26 20:50:29 EDT 2011

Comment 6 Karel Zak 2011-06-27 10:28:06 UTC
Small summary:

1) The system time is set by kernel (CONFIG_RTC_HCTOSYS), but kernel does not read anything from /etc/adjtime. Your kernel expects that hwclock (BIOS) is in UTC, not in LOCAL time.

Use

   $ dmesg | grep "system clock"

to see more details about initial system time setting during kernel boot.

2) so to fix the problem with LOCAL time we call hwclock --systz from init scripts or from udev rules. 


For Fedora-15 we use systemd. The basic systemd design feature it to bypass existing utils (probably to save fork+exec(), so systemd has own --systz implementation. This implementation seems broken.

systemd/src/utils.c: hwclock_apply_localtime_delta(void)

this code sets only system timezone, but does not modify system clock.

Comment 7 Karel Zak 2011-06-27 10:29:11 UTC
Created attachment 510061 [details]
untested systemd patch

Comment 8 Karel Zak 2011-06-27 10:40:05 UTC
Created attachment 510064 [details]
untested systemd patch

Sorry, the previous patch was for something completely different... :-)

Comment 9 Clyde E. Kunkel 2011-06-28 13:48:00 UTC
(In reply to comment #8)
> Created attachment 510064 [details]
> untested systemd patch
> 
> Sorry, the previous patch was for something completely different... :-)

I assume your patch is for rawhide.  Correct?  If so, could we get it applied soon?  If someone could do a scratch build, I am more than willing to test it. Thanks.

Comment 10 Lennart Poettering 2011-06-28 19:49:09 UTC
Karel, we apply the delta in systemd natively only in systemd git, which hasn't hit rawhide yet. This bug is unrelated hence.

In F15 and current rawhide we invoke "hwclock --hctosys" in the "hwclock-load.service".

So I am not sure what systemd might be doing wrong here...

Karel, the current code in git uses the fact that at the first invocation of settimeofday() with a non-NULL second parameter is special and sets only the tz shift in the kernel (read the kernel sources for details). A longer explanation about what and why systemd git does what it does, see http://lists.freedesktop.org/archives/systemd-devel/2011-May/002526.html -- but again, this is orthogonal to this bug report, since it only applies to systemd git, nothing we would currently have in fedora.

Comment 11 Lennart Poettering 2011-06-28 19:52:31 UTC
Peter, are you manually calling "hwclock --systz"? Why?

Note that hwclock --systz simply reads the system clock, adds something to it and writes it back, hence it is naturally not idempotent.

Comment 12 Karel Zak 2011-06-29 08:29:37 UTC
(In reply to comment #10)
> Karel, we apply the delta in systemd natively only in systemd git, which hasn't
> hit rawhide yet. This bug is unrelated hence.

Ah, I read the upstream git tree only ;-)

> In F15 and current rawhide we invoke "hwclock --hctosys" in the
> "hwclock-load.service".

I see "ExecStart=/sbin/hwclock --systz" in 

  /lib/systemd/system/hwclock-load.service

This is correct on systems with CONFIG_RTC_HCTOSYS kernel. (The --hctosys is horrible and expensive voodoo...)

> So I am not sure what systemd might be doing wrong here...

So sorry ;-)

> Karel, the current code in git uses the fact that at the first invocation of
> settimeofday() with a non-NULL second parameter is special and sets only the tz
> shift in the kernel (read the kernel sources for details). A longer explanation
> about what and why systemd git does what it does, see

I know about this crazy settimeofday() semantic, unfortunately it's very problematic. 

It seems that we *always* need to set the timezone in kernel, because for example FAT filesystem driver uses the timezone to convert FAT (non-UTC) timestamps to usable time. See Scott's u-l commit 2eefcaaace7e07eabf4dc054d33f9e0fb1a2d6d8.

It means that hwclock (--hctosys and --systz) always sets the timezone independently on CMOS clock state. For example you can use

 hwclock --systz --utc

and it will set the timezone, but systime will be unmodified (because the clock uses UTC).

If the CMOS clock is in LOCAL time than &tv is modified too. 

So, hwclock does not rely on the settimeofday() semantic.

> again, this is orthogonal to this bug report, since it only applies to systemd
> git, nothing we would currently have in fedora.

Probably yes, reassigning back to u-l.

Comment 13 Lennart Poettering 2011-06-29 16:56:08 UTC
(In reply to comment #12)
> (In reply to comment #10)
> > Karel, we apply the delta in systemd natively only in systemd git, which hasn't
> > hit rawhide yet. This bug is unrelated hence.
> 
> Ah, I read the upstream git tree only ;-)
> 
> > In F15 and current rawhide we invoke "hwclock --hctosys" in the
> > "hwclock-load.service".
> 
> I see "ExecStart=/sbin/hwclock --systz" in 
> 
>   /lib/systemd/system/hwclock-load.service
> 
> This is correct on systems with CONFIG_RTC_HCTOSYS kernel. (The --hctosys is
> horrible and expensive voodoo...)
> 
> > So I am not sure what systemd might be doing wrong here...
> 
> So sorry ;-)
> 
> > Karel, the current code in git uses the fact that at the first invocation of
> > settimeofday() with a non-NULL second parameter is special and sets only the tz
> > shift in the kernel (read the kernel sources for details). A longer explanation
> > about what and why systemd git does what it does, see
> 
> I know about this crazy settimeofday() semantic, unfortunately it's very
> problematic. 
> 
> It seems that we *always* need to set the timezone in kernel, because for
> example FAT filesystem driver uses the timezone to convert FAT (non-UTC)
> timestamps to usable time. See Scott's u-l commit
> 2eefcaaace7e07eabf4dc054d33f9e0fb1a2d6d8.

Hmm, I am not convinced this is the right thing to do.

To my knowledge the kernel uses the timezone shift in the 11min mode (i.e. when NTP is used) to sync the RTC automatically from the system time. 

If we pass the timezone to the kernel even if the RTC is in UTC then this would break. I am not sure how the supposed usage in FAT interacts with the one in the 11min mode.

Comment 14 Karel Zak 2011-06-29 18:09:10 UTC
(In reply to comment #13)
> If we pass the timezone to the kernel even if the RTC is in UTC then this would
> break. 

settimeofday() allows:

 1/ set timezone and time (tv and tz are non-NULL)
 2/ set timezone and move from LOCAL to UTC (tv is NULL, tz is non-NULL)
 3/ set only time (tv is non-NULL, tz is NULL)

hwclock uses 1/ for years, the proper tv (according to the /etc/adjtime setting) is set in userspace.

Comment 15 Peter Hjalmarsson 2011-06-30 08:15:11 UTC
(In reply to comment #11)
> Peter, are you manually calling "hwclock --systz"? Why?
> 
> Note that hwclock --systz simply reads the system clock, adds something to it
> and writes it back, hence it is naturally not idempotent.

Not really. I did that to debug the initial bug:

I followed anaconda from the LiveCD, choosed the correct timezone and that my clock is localtime, since I dualboot windows. Nothing else changed from the original configuration.
Every time I started my computer into Fedora the clock was wrong until NTP got a working network connection. This made a whole lot of stuff angry/confused (like for example fsck complaining about root having a modified time set into the future and so on).
This migrated into the hwclock on shutdown in such a fashion that every time I rebooted my system into linux without a network connection the clock was set back another x hours during the following startup depending on my timezone and DST.
With other words, three fast reboots without network connection would mean that my clock jumpen from ex. 10:00 to 7:00.

That was the original bug, nothing more, nothing less.
When I come home today I can see if this is still true for a updated Fedora15 install, I have not had the time to confirm this and have gotten pretty used to the messages form fsck during bootup and other, so I have not reflected upon if it is gone or not.

Comment 16 Clyde E. Kunkel 2011-06-30 14:23:50 UTC
Comment 5 refers to rawhide while this bz was originally for F15.

Should a separate bz be opened for rawhide?  If so, happy to do it.  Really need to get this fixed.  Thank you for all of your efforts.

Comment 17 Peter Hjalmarsson 2011-07-01 15:54:10 UTC
(In reply to comment #15)
> When I come home today I can see if this is still true for a updated Fedora15
> install.

Checked this today, and it seems like the issue is gone on my Fedora15 laptop. So this seems to be fixed.

Comment 18 Karel Zak 2012-07-04 12:13:16 UTC
(In reply to comment #14)
> settimeofday() allows:
> 
>  1/ set timezone and time (tv and tz are non-NULL)
>  2/ set timezone and move from LOCAL to UTC (tv is NULL, tz is non-NULL)
>  3/ set only time (tv is non-NULL, tz is NULL)
> 
> hwclock uses 1/ for years, the proper tv (according to the /etc/adjtime
> setting) is set in userspace.

Just for the record, now hwclock --systz uses 2/ to be more robust.