Bug 821711

(In reply to comment #0)
> Description of problem:
> XFCE datetime applet forgets the timezone settings after reboot especially if a
> network connection is unavailable

xfce4-datetime-plugin does not have any timezone settings.

> Version-Release number of selected component (if applicable):
> ???

Please run 'rpm -q xfce4-datetime-plugin' to find out the version.

> Steps to Reproduce:
> 1. During installation select "Computer uses UTC", select your TZ
> 2. Configure network to connect information from DHCP
> 3. Configure chrony

What exactly did you configure? You don't need to configure anything, you just need to enable network time servers in the firstboot wizard.

> 4. Wait till time is synchronized
> 5. Reboot in such a way that network connection is lost during reboot
> 6. Log in as any user
> 7. Restore network connection
> 
> Actual results:
> After login the applet shows time in UTC. When network connection is restored,
> the time is set to a correct value after a few minutes. I do not know whether
> the applet does it on its own or whether it waits for chrony.

The applet just relies on the system time. Please give me the content of /etc/sysconfig/clock and /etc/localtime.

Created attachment 584669 [details]
/etc/localtime

Requested /etc/localtime

(In reply to comment #1)
> (In reply to comment #0)
> > Description of problem:
> > XFCE datetime applet forgets the timezone settings after reboot especially if a
> > network connection is unavailable
> 
> xfce4-datetime-plugin does not have any timezone settings.
> 
It says that the package is not installed yet I see time in the upper right corner. I do not know what exactly Xfce uses as default.

> > Version-Release number of selected component (if applicable):
> > ???
> 
> Please run 'rpm -q xfce4-datetime-plugin' to find out the version.
> 
> > Steps to Reproduce:
> > 1. During installation select "Computer uses UTC", select your TZ
> > 2. Configure network to connect information from DHCP
> > 3. Configure chrony
> 
> What exactly did you configure? You don't need to configure anything, you just
> need to enable network time servers in the firstboot wizard.
> 
1. during installation I selected the timezone and clicked that the computer uses UTC

2. Later (probably during firsdtboot) I selected synchronization with NTP and added two additional NTP servers available on LAN

3. In /etc/chrony.conf I commented out the public NTP servers (I have 20 computers with differents linux distros on LAN and it would be silly to synchronize all of them with the public servers). My /etc/chrony.conf was included in the original post.

4. I verified that chrony works:
$ chronyc sources
210 Number of sources = 4
MS Name/IP address           Stratum Poll LastRx Last sample
============================================================================
^* hroch486.icpf.cas.cz          3    7     29  -1410ns[  -61us] +/-   29ms
^+ klastr1.icpf.cas.cz           3    6     26   -587us[ -587us] +/-   51ms
^+ dca.asuch.cas.cz              4    6     24    +13ms[  +13ms] +/-  334ms
^+ dcc.asuch.cas.cz              3    6     23    +16ms[  +16ms] +/-  308ms


> > 4. Wait till time is synchronized
> > 5. Reboot in such a way that network connection is lost during reboot
> > 6. Log in as any user
> > 7. Restore network connection
> > 
> > Actual results:
> > After login the applet shows time in UTC. When network connection is restored,
> > the time is set to a correct value after a few minutes. I do not know whether
> > the applet does it on its own or whether it waits for chrony.
> 
> The applet just relies on the system time. Please give me the content of
> /etc/sysconfig/clock and /etc/localtime.

$ cat /etc/sysconfig/clock
# The time zone of the system is defined by the contents of /etc/localtime.
# This file is only for evaluation by system-config-date, do not rely on its
# contents elsewhere.
ZONE="Europe/Prague"

/etc/localtime was added as an attachment

(In reply to comment #3)
> It says that the package is not installed yet I see time in the upper right
> corner. I do not know what exactly Xfce uses as default.

This is the clock of xfce4-panel then. Adjusting component.

Nevertheless I doubt that the problem is in xfce4-panel. Is the time correct when you run the 'date' command in the console?

(In reply to comment #4)
> (In reply to comment #3)
> > It says that the package is not installed yet I see time in the upper right
> > corner. I do not know what exactly Xfce uses as default.
> 
> This is the clock of xfce4-panel then. Adjusting component.
> 
> Nevertheless I doubt that the problem is in xfce4-panel. Is the time correct
> when you run the 'date' command in the console?

Thanks for pointing to it. The 'date' command returns equally wrong value, thus the problem is in the system time. Do I need some special setting in chrony or elsewhere to have the time correct if network is unavailable after boot?

I have another observation from a different computer running Fedora 16. It runs as a server without graphics and network is started at boot. Chrony sybchronizes time very quickly so that I cannot see the problem if I just log to it remotely. However, services started at boot may see wrong time. This is my log from APC:

$ cat /var/log/apcupsd.events
2012-06-20 11:02:32 +0200  apcupsd 3.14.10 (13 September 2011) redhat startup succeeded
2012-06-20 10:42:40 +0200  apcupsd exiting, signal 15
2012-06-20 10:42:40 +0200  apcupsd shutdown succeeded
2012-06-20 11:40:58 +0200  apcupsd 3.14.10 (13 September 2011) redhat startup succeeded

Explanation:
The computer was started at 10:02 but due to lost timezone information the time reported was 11:02. I changed something in the network configuration and to be sure I rebooted the computer (I was logged remotely via ssh and issued reboot). The shutdown time is correct but the startup time is wrong. ls -l /var/log shows among others:

-rw-r--r--. 1 root     root       21261 Jun 20 11:40 boot.log
-rw-r--r--. 1 root     root         284 Jun 20 11:40 apcupsd.events

This means that the file modifications times are wrong too.

This bug is a stopper for me. The computer will soon be used for processing real time data. If power fails for a longer period and is shut down by UPS after batteries are discharged and then starts with a wrong date/time, it may lead to a loss of important data from the measuring devices or may report nonsense data to the central European database.

I have other compuiters with CentOS 4.x, CentOS 5.x and Fedora 13 and they do not have such a problem.

(In reply to comment #5)

> Thanks for pointing to it. The 'date' command returns equally wrong value,
> thus the problem is in the system time. 

Please reassign the ticket to chrony then and give it a meaningful summary.

> Do I need some special setting in
> chrony or elsewhere to have the time correct if network is unavailable after
> boot?

I don't know, I am the maintainer of the xfce4-datetime-plugin and your question is outside of my realm, sorry.

Done. Can I somehow increase the severity? When I reported it initially I did not recognize the real impact of the problem.

The source of the problem seems to be somewhere deeper. Since I need the correct time, I tried to disable chrony and enable ntpd, waited until the time was synchronized, then rebooted and the same bug was observed. I enable "undisciplined local clock" in /etc/ntp.conf but it did not help. Afterwards I waited until clock was synchronized, then disabled ntpd so that no time sync was configured, rebooted and the time was wrong. Currently I see one possibility: whenever my script for processing real time data needs the correct time, it has to run ntpdate against one of my loical ntp servers running on CentOS 5.x.

Finally I have found the source of the problem. Before chronyd starts, /usr/libexec/ntpdate-wrapper is executed. It looks to /etc/ntp/step-tickers but this file is empty. In such a case the ntpdate-wrapper exits with retval=6. File /etc/ntp/step-tickers in CentOS contains:
0.rhel.pool.ntp.org
1.rhel.pool.ntp.org
2.rhel.pool.ntp.org
127.127.1.0

After adding these lines to /etc/ntp/step-tickers Fedora starts with the correct time.

I am not sure whether I should set SYNC_HWCLOCK to yes in /etc/sysconfig/ntpdate
(my computer is a single boot Linux machine).

PS: It was a tough exercise to understand how systemd works...

There is still a problem if the computer starts while network is unavailable (or network interface does not start due to any reason). If the network is started manually later, it takes some time to synchronize. After adding the step-tickers, the following command causes immediate synchronization:

systemctl restart chronyd.service

(because ntpdate-wrapper does it). Setting SYNC_HWCLOCK to yes in /etc/sysconfig/ntpdate did not help.

Can chronyd be configured to block boot until the time is fully synchronised with the NTP servers? In such a case chronyd should wait and other scripts running at boot must not be executed. The reason is that due to this bug time is wrong after boot until it gets synchronised. It may be fast but time will remain wrong if NTP servers are inaccessible. Wrong time setting will cause disaustrous damage of files acquired at real time. It is safer if the computer does not start.

(In reply to comment #12)
> Can chronyd be configured to block boot until the time is fully synchronised
> with the NTP servers? In such a case chronyd should wait and other scripts
> running at boot must not be executed. The reason is that due to this bug
> time is wrong after boot until it gets synchronised. It may be fast but time
> will remain wrong if NTP servers are inaccessible. Wrong time setting will
> cause disaustrous damage of files acquired at real time. It is safer if the
> computer does not start.

I hope I found an answer to this question. I set RETRIES=987654321 in /etc/sysconfig/ntpdate so that ntpdate will give up after more than 30 years. Inportant services (crond.service and httpd.service) are set as After=... ntpdate.service and thus these critical services will not start with wrong date/time.

Another observation (maybe it helps to find the source of the problem), chrony reports:

chronyc> rtcdata
513 RTC driver not running

I found some explanation here: http://chrony.tuxfamily.org/FAQ.html#section_9

I tried to add the rtcfile directive. Since the driftfile is in /var/lib/chrony/drift, I entered:

rtcfile /var/lib/chrony/rtc

However, if this is included in /etc/chrony.conf, chronyd does not start.

I tried to look whether I may have hwclock as mentioned in 9.2 (the link above). My search attempts return:

# find / -name '*hwclock*'
/usr/sbin/hwclock
/usr/share/doc/util-linux-2.20.1/README.hwclock
/usr/share/man/man8/hwclock.8.gz
/sbin/hwclock
# find /lib/systemd/ -type f -exec grep -nH hwclock {} \;
Binary file /lib/systemd/systemd-timedated matches
# find /etc/rc.d/ -type f -exec grep -nH hwclock {} \;
# find /lib/systemd/ -type f -exec grep -nH timedated {} \;
/lib/systemd/system/systemd-timedated.service:14:ExecStart=/lib/systemd/systemd-timedated
Binary file /lib/systemd/systemd-timedated matches
# systemctl status systemd-timedated.service
systemd-timedated.service - Time & Date Service
          Loaded: loaded (/lib/systemd/system/systemd-timedated.service; static)
          Active: inactive (dead)
          CGroup: name=systemd:/system/systemd-timedated.service

I do not know whether this may mean that I have the hwclock problem.

dmesg returns among others:
[    1.516454] Refined TSC clocksource calibration: 2993.200 MHz.
[    1.516460] Switching to clocksource tsc

I have found another information: 
http://chrony.tuxfamily.org/manual.html#rtconutc-directive

However, adding rtconutc to chronyd.conf did not help although my clock uses UTC.

No matter what combination of directives in /etc/chrony.conf (rtcsync, rtconutc) and /etc/sysconfig/ntpdate (SYNC_HWCLOCK=yes|no) I use, I always have the following typical messages in /var/log/messages:

Jul 24 15:28:36 uchp-antonpaar network[888]: Determining IP information for p4p1... done.
Jul 24 15:28:36 uchp-antonpaar network[888]: [  OK  ]
Jul 24 15:28:36 uchp-antonpaar rpc.statd[1393]: Version 1.2.5 starting
Jul 24 15:28:36 uchp-antonpaar sm-notify[1394]: Version 1.2.5 starting
Jul 24 15:28:36 uchp-antonpaar systemd[1]: Startup finished in 2s 116ms 334us (kernel) + 2s 267ms 341us (initrd) + 33s 701ms 385us (userspace) = 38s 85ms 60us.
Jul 24 14:28:44 uchp-antonpaar chronyd[882]: Selected source 147.231.137.254
Jul 24 14:28:44 uchp-antonpaar chronyd[882]: System clock wrong by -3598.194311 seconds, adjustment started
Jul 24 14:28:44 uchp-antonpaar chronyd[882]: System clock was stepped by -3598.194 seconds

While this is fine on computers with network access, I still do not know how to deal with time on computers (notebooks) that may be used without a network. Is console available during boot? Can I do something as

#!/bin/sh
read datetime
/bin/date $datetime

and plug this small script to the boot sequence and delay the boot until time is entered manually? It is acceptable if the time is a few minutes wrong but it is not acceptable if the difference is one hour or more.

I thought that RETRIES=987654321 in /etc/sysconfig/ntpdate will solve the problem but it does not help. chronyd.service contains:

[Unit]
Description=NTP client/server
After=syslog.target ntpdate.service
Conflicts=ntpd.service

I have modified crond.service to contain:

[Unit]
Description=Command Scheduler
After=syslog.target auditd.service sssd.service ypbind.service ntpdate.service

I have rebooted with the ethernet cable unplugged in order to emulate inaccessible NTP servers. I thought that ntpdate will wait and will not allow crond to start but crond started (I have a simple script in /etc/crontab that writes the output of /bin/date to a file, so that I can see when the script was run).

Is there a possibility to delay start of other services (especially crond and httpd) until time is synchronised or should I just stupidly add "chronyc tracking" to each and every cron script and abort it if the Reference ID is 0.0.0.0 ()?

(In reply to comment #17)
> I thought that RETRIES=987654321 in /etc/sysconfig/ntpdate will solve the
> problem but it does not help. chronyd.service contains:
> 
> [Unit]
> Description=NTP client/server
> After=syslog.target ntpdate.service
> Conflicts=ntpd.service
> 
> I have modified crond.service to contain:
> 
> [Unit]
> Description=Command Scheduler
> After=syslog.target auditd.service sssd.service ypbind.service
> ntpdate.service
> 
> I have rebooted with the ethernet cable unplugged in order to emulate
> inaccessible NTP servers. I thought that ntpdate will wait and will not
> allow crond to start but crond started (I have a simple script in
> /etc/crontab that writes the output of /bin/date to a file, so that I can
> see when the script was run).
> 
> Is there a possibility to delay start of other services (especially crond
> and httpd) until time is synchronised or should I just stupidly add "chronyc
> tracking" to each and every cron script and abort it if the Reference ID is
> 0.0.0.0 ()?

The command at the first line of each and every script would be:

/usr/bin/chronyc tracking | /bin/grep 'Reference ID' | /bin/grep '0\.0\.0\.0' || exit 1

(everything on a single line)

(In reply to comment #18)
> (In reply to comment #17)
> > ...
> > Is there a possibility to delay start of other services (especially crond
> > and httpd) until time is synchronised or should I just stupidly add "chronyc
> > tracking" to each and every cron script and abort it if the Reference ID is
> > 0.0.0.0 ()?
> 
> The command at the first line of each and every script would be:
> 
> /usr/bin/chronyc tracking | /bin/grep 'Reference ID' | /bin/grep
> '0\.0\.0\.0' || exit 1
> 
> (everything on a single line)

Sorry for my mistake, the correct command is

/usr/bin/chronyc tracking | /bin/grep 'Reference ID' | /bin/grep '0\.0\.0\.0' && exit 1

(&&, not ||)

In order to overcome the problem I have given up the dependence on ntpdate.service because it did not help. Instead I have developed a script for delayed start. My /lib/systemd/system/crond.service is now as follows:

[Unit]
Description=Command Scheduler
After=syslog.target auditd.service sssd.service ypbind.service

[Service]
EnvironmentFile=/etc/sysconfig/crond
ExecStart=/usr/local/bin/delayedstart.sh /usr/sbin/crond -n $CRONDARGS

[Install]
WantedBy=multi-user.target


After modifying the ExecStart (in case after any modification) it is necessary to use:
systemctl --system daemon-reload

delayedstart.sh is give below.


#!/bin/bash
## $Id: delayedstart.sh 1221 2012-07-27 11:12:23Z zw $
delay=60
while true
do
  a=`/usr/bin/chronyc tracking`
  [ $? -ne 0 ] && /bin/sleep $delay && continue
  [ "$a" = '' ] && /bin/sleep $delay && continue
  echo $a | /bin/egrep 'Reference ID\s+:\s+0\.0\.0\.0' >/dev/null && /bin/sleep $delay && continue
  echo $a | /bin/egrep 'System time\s+:\s+0\..*seconds' >/dev/null && break
done
$*

Explanation: "chronyc tracking" returns a few pieces of information. Reference ID contains the IP address of an NTP server to which chrony is synchronized. If it is unsynchronized, the IP address is 0.0.0.0. System time contains the departure of the system time from the correct time (I hope). If /etc/ntp/step-tickers contains IP addresses and/or names of accessible NTP servers, synchronization is very fast so that this verification probably is not needed.

Note: depending on the font size some lines may be broken in this window. The script will soon be available from my web page http://icebearsoft.euweb.cz/
The license of the script is GPL.

I suspect your RTC runs in local time and it wasn't reset after changing the timezone. I think system-config-date should do that. The problem is that RTC isn't set to system time on shutdown anymore, see bug #750883 for more information.

As for the services which need correct time on start, enable the chrony-wait.service and add After=time-sync.target to the service units.

(In reply to comment #21)
> I suspect your RTC runs in local time and it wasn't reset after changing the

It is configured to use UTC since it is a Linux only computer (no dual boot).

> timezone. I think system-config-date should do that. The problem is that RTC
> isn't set to system time on shutdown anymore, see bug #750883 for more
> information.
> 
No, system-config-date is a GUI. I have no chance to use it. Imagine if power fails for a longer time than UPS can stand, the computer is shut down. When it restarts, it must have a correct time. I may take holidays so the computer my start when I am away and it may take 2 weeks before I can do anything in GUI. This is not acceptable. I can afford to lose some real time data but I must not lose two weeks of data. And I cannot afford to damage real time data due to wrong time.

> As for the services which need correct time on start, enable the
> chrony-wait.service and add After=time-sync.target to the service units.

It still does not solve a problem of a notebook running without a network access :-( I am afraid to upgrade from Fedora 13 which works.

(In reply to comment #21)
> I suspect your RTC runs in local time and it wasn't reset after changing the
> timezone. I think system-config-date should do that. The problem is that RTC
> isn't set to system time on shutdown anymore, see bug #750883 for more
> information.
> 
> As for the services which need correct time on start, enable the
> chrony-wait.service and add After=time-sync.target to the service units.

If the comment is correct, chrony-wait.service + time-sync.target wait at most 10 minutes. However, if a computer is started after power was resumed, probably other computers and switches/routers on the LAN may be started at the same time (or even later). There is small but nonzero probability that the NTP servers will be inaccessible within 10 minutes' interval. My solution does not suffer from this time-limit problem. Data loss in unpleasant but it is not far that bad as destroying EU's central database with false data and risking that scientific papers will be based on nonsense data.

I'm not sure where it came from, but your RTC is off by one hour. As hwclock --systohc is no longer called on shutdown/reboot, it needs to be run manually when it's off by more than 15 minutes, because the kernel RTC sync doesn't know/care about timezones and will adjust the clock only within +/- 15 minutes. 

When the chrony RTC tracking feature (rtcdrift and rtconutc) is used instead of the kernel sync, the chronyc trimrtc command will fix any offse and it can be run periodically. 

To disable the waitsync limit, copy the chrony-wait.service file to /etc and change the command to "/usr/bin/chronyc waitsync 0 0.1".

(In reply to comment #24)
> I'm not sure where it came from, but your RTC is off by one hour. As hwclock

It came from the motherboard manufacturer. When I unpack a new computer from a box, the time is sometimes (almost) correct, sometimes a few hours wrong, sometimes even more than 10 years wrong.

> --systohc is no longer called on shutdown/reboot, it needs to be run

Thank you, this helped. Probably Anaconda should do it if it can connect to a network and find NTP servers.

> manually when it's off by more than 15 minutes, because the kernel RTC sync
> doesn't know/care about timezones and will adjust the clock only within +/-
> 15 minutes. 
> 
> When the chrony RTC tracking feature (rtcdrift and rtconutc) is used instead
> of the kernel sync, the chronyc trimrtc command will fix any offse and it
> can be run periodically. 
> 
chronyc trimrtc reported "501 Not authorised" although I was root. I am looking into the manual in order to find what to do.

> To disable the waitsync limit, copy the chrony-wait.service file to /etc and
> change the command to "/usr/bin/chronyc waitsync 0 0.1".