Bug 981617 - please take /etc/adjtime out of the initramfs so it doesn't override the existing file system version
please take /etc/adjtime out of the initramfs so it doesn't override the exis...
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: dracut (Show other bugs)
rawhide
Unspecified Linux
unspecified Severity unspecified
: ---
: ---
Assigned To: dracut-maint
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-05 05:48 EDT by Andre Robatino
Modified: 2013-08-21 06:20 EDT (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-20 08:52:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/var/log/messages (note the time change when chrony resets the time, near the end of the file) (760.33 KB, text/plain)
2013-07-05 05:48 EDT, Andre Robatino
no flags Details
dmesg (42.00 KB, text/plain)
2013-07-05 09:08 EDT, Andre Robatino
no flags Details

  None (edit)
Description Andre Robatino 2013-07-05 05:48:02 EDT
Created attachment 769171 [details]
/var/log/messages (note the time change when chrony resets the time, near the end of the file)

Description of problem:
I have a dual-boot (WinXP/F19) machine with a clean F19 install. I used a registry hack to make Windows assume that the hardware clock is set to UTC instead of local time, so I can set the system clock to UTC in Fedora. My local time is currently 4 hours behind UTC. When I first boot, local time and UTC are both 4 hours fast, even though the RTC is correct. After about 30 seconds, chrony resets them to the correct values. For example (note the correct time was 04:56 EDT == 08:56 UTC):

[root@dell-pc ~]# date
Fri Jul  5 08:56:12 EDT 2013
[root@dell-pc ~]# timedatectl
      Local time: Fri 2013-07-05 08:56:13 EDT
  Universal time: Fri 2013-07-05 12:56:13 UTC
        RTC time: Fri 2013-07-05 08:56:13
        Timezone: America/New_York (EDT, -0400)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2013-03-10 01:59:59 EST
                  Sun 2013-03-10 03:00:00 EDT
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2013-11-03 01:59:59 EDT
                  Sun 2013-11-03 01:00:00 EST
[root@dell-pc ~]# date ; timedatectl
Fri Jul  5 04:56:24 EDT 2013
      Local time: Fri 2013-07-05 04:56:24 EDT
  Universal time: Fri 2013-07-05 08:56:24 UTC
        RTC time: Fri 2013-07-05 08:56:23
        Timezone: America/New_York (EDT, -0400)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2013-03-10 01:59:59 EST
                  Sun 2013-03-10 03:00:00 EDT
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2013-11-03 01:59:59 EDT
                  Sun 2013-11-03 01:00:00 EST
[root@dell-pc ~]#

If I set the system clock to local time instead of UTC, it works properly (even though timedatectl complains). Windows is not involved, since I can reboot Fedora and see the problem without Windows ever running. In any case, the RTC is always correct.

I have another dual-boot (Windows Vista/F19) machine, again a clean F19 install, and that machine does NOT show this problem, even though it's configured exactly the same. I have no explanation.

Version-Release number of selected component (if applicable):
chrony-1.28-0.1.pre1.fc19

How reproducible:
always on the one machine, never on a different one
Comment 1 Andre Robatino 2013-07-05 05:49:40 EDT
I have no idea what the correct Component is, so please reassign as appropriate and sorry if this is the wrong one.
Comment 2 Andre Robatino 2013-07-05 09:08:46 EDT
Created attachment 769239 [details]
dmesg

Nothing is added to dmesg when the clock is set back 4 hours. Note the entries

[    0.907737] Write protecting the kernel read-only data: 2276k
[    0.914338] systemd[1]: RTC configured in localtime, applying delta of -240 minutes to system time.
[    0.917759] systemd[1]: systemd 204 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ)
[    0.918314] systemd[1]: Running in initial RAM disk.

This is NOT true, the RTC is configured in UTC, as noted above. I'll change the component to systemd (still not sure that's correct, though).
Comment 3 Andre Robatino 2013-07-05 09:22:18 EDT
I have a F19-only machine, again with system clock set to UTC. The time works fine on it. The dmesg output for it is

[    5.409921] Write protecting the kernel read-only data: 2248k
[    5.489993] systemd[1]: systemd 204 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ)
[    5.491618] systemd[1]: Running in initial RAM disk.

so on that machine, systemd correctly sees that the RTC is set to UTC. On both of my dual-boot machines, with system time set to UTC, on one of which the time is broken and one where it works correctly, systemd falsely claims "RTC configured in localtime".
Comment 4 Kay Sievers 2013-07-05 09:29:52 EDT
RTC in UTC vs. LOCAL is defined by:
  /etc/adjtime

What's in that file on both machines?
Comment 5 Andre Robatino 2013-07-05 09:34:46 EDT
The third line in /etc/adjtime on both machines is "UTC", as expected.
Comment 6 Kay Sievers 2013-07-05 09:37:35 EDT
Ok.

What does:
  "systemd falsely claims "RTC configured in localtime"
exactly mean than?

What is "systemd" in that context?
Comment 7 Jóhann B. Guðmundsson 2013-07-05 09:44:09 EDT
(In reply to Andre Robatino from comment #3)
 On
> both of my dual-boot machines, with system time set to UTC, on one of which
> the time is broken and one where it works correctly, systemd falsely claims
> "RTC configured in localtime".

Hmm are you dual booting with windows on those machines? 

If so you might need to fix it ( via regedit hack ) on the windows side since windows is known to mishandle UTC and cause issues in dual booting setups. 

You should be able to confirm if that's the issue by opening Regedit, drill down to...

HKLM\SYSTEM\CurrentControlSet\Control\TimeZoneInformation
and create a new DWORD entry named “RealTimeIsUniversal”. Set the value to 1.
Comment 8 Andre Robatino 2013-07-05 09:51:07 EDT
(In reply to Jóhann B. Guðmundsson from comment #7)

> Hmm are you dual booting with windows on those machines? 
> 
> If so you might need to fix it ( via regedit hack ) on the windows side
> since windows is known to mishandle UTC and cause issues in dual booting
> setups.

See comment 0. I already did that (and also told Windows not to set the time). In any case, it's irrelevant, since this problem happens on repeated reboots of Fedora, even when Windows never runs.
Comment 9 Andre Robatino 2013-07-05 09:56:07 EDT
(In reply to Kay Sievers from comment #6)

> What does:
>   "systemd falsely claims "RTC configured in localtime"
> exactly mean than?
> 
> What is "systemd" in that context?

I don't understand the question. My RTC is configured in UTC (as confirmed by both /etc/adjtime and the timedatectl output in comment 0). The dmesg output on both dual-boot Windows/F19 machines says

[    0.914338] systemd[1]: RTC configured in localtime, applying delta of -240 minutes to system time.
Comment 10 Harald Hoyer 2013-07-05 10:07:41 EDT
What is the output of:

$ sudo lsinitrd -f etc/adjtime
Comment 11 Andre Robatino 2013-07-05 10:16:09 EDT
(In reply to Harald Hoyer from comment #10)
> What is the output of:
> 
> $ sudo lsinitrd -f etc/adjtime

On both of my dual-boot Windows/F19 machines (both the one where time works properly, and the one where it is broken) the output is

0.0 0 0.0
0
LOCAL

On my F19-only machine (where time works properly) the output is

0.0 0 0.0
0
Comment 12 Kay Sievers 2013-07-05 10:47:38 EDT
Seems, the initramfs image still included the old config file.

Running:
  dracut -f
should update it.
Comment 13 Andre Robatino 2013-07-05 11:41:31 EDT
(In reply to Kay Sievers from comment #12)
> Seems, the initramfs image still included the old config file.
> 
> Running:
>   dracut -f
> should update it.

Thanks, that works! I rebuilt the initramfs for all kernels on all 3 machines, and all of them work normally now. So where was the bug? Is it already fixed?
Comment 14 Kay Sievers 2013-07-05 12:00:20 EDT
It's not really a "bug". There is currently no reasonable infrastructure
to trigger a rebuild of the initrd when core config things change. It's a
manual step.
Comment 15 Andre Robatino 2013-07-05 13:21:45 EDT
OK, so I guess it's a side effect of the fact that I can't specify UTC vs. local time in the new installer, together with probably having applied the kernel updates before changing the system time to UTC (though I can't remember whether I did that). I'll close this as NOTABUG and keep an eye on the next kernel update, then. Thanks.
Comment 16 Steve Tyler 2013-07-06 12:53:46 EDT
(In reply to Kay Sievers from comment #14)
> It's not really a "bug". There is currently no reasonable infrastructure
> to trigger a rebuild of the initrd when core config things change. It's a
> manual step.

Maybe there should be a bug for this problem. The initramfs contains executables, including systemd, as well as config files.[1]

For example, the fix for this systemd bug did not take effect until the initramfs was rebuilt:
Bug 870535 - Regression in handling of localtime RTC: timezone offset applied twice

[1] $ sudo zcat /boot/initramfs-3.9.8-300.fc19.x86_64.img | cpio -tv
Comment 17 Andre Robatino 2013-07-13 09:37:22 EDT
I'm not the only one seeing this - see http://forums.fedoraforum.org/showthread.php?p=1659967 . I hope the anaconda devs take this into account in deciding whether to put back the UTC/local time choice, or alternatively if there is another way to prevent this bug that would be okay as well. I don't mind having to use system-config-date to change the setting if it actually works.
Comment 18 Steve Tyler 2013-07-13 12:19:12 EDT
Have you tried booting a rescue image?

The rescue initramfs doesn't have /etc/adjtime or /etc/localtime:
$ sudo zcat /boot/initramfs-0-rescue-*.img | cpio -tv --quiet | egrep 'adjtime|localtime'
Comment 19 Andre Robatino 2013-07-26 16:14:05 EDT
Reopening, reassigning to dracut and changing the Summary in response to https://bugzilla.redhat.com/show_bug.cgi?id=981793#c5 .
Comment 20 Steve Tyler 2013-07-27 01:42:30 EDT
If initramfs is rebuilt without /etc/adjtime, the system clock is correct on rebooting:

1. # lsinitrd -f /etc/adjtime # Verify contents of adjtime in initramfs.
2. # mv -i /etc/adjtime /etc/adjtime.BAK1 # Temporarily rename adjtime.
3. # dracut -f
4. # mv -i /etc/adjtime.BAK1 /etc/adjtime
5. # lsinitrd -f /etc/adjtime # Verify there is no adjtime in initramfs.
6. # reboot
7. # date # Verify the system clock is correct.

Note that chronyd does not log any messages about stepping the system clock.

Tested with a fresh, minimal install:
$ qemu-img create f19-test-2.img 12G
$ qemu-kvm -m 4096 -hda f19-test-2.img -cdrom ~/xfr/fedora/F19/Fedora-19-x86_64-DVD.iso -vga std -boot menu=on
Comment 21 Steve Tyler 2013-07-27 02:03:27 EDT
This is also true if the RTC is on local time:

1. Change UTC to LOCAL in /etc/adjtime.
2. Reboot with the qemu option "-rtc base=localtime".
3. # date
4. # grep chronyd /var/log/messages

If chronyd is disabled, in both cases (UTC or LOCAL), the system clock is also correct on booting:
# systemctl disable chronyd

Tested with:
$ qemu-kvm -m 4096 -hda f19-test-2.img -cdrom ~/xfr/fedora/F19/Fedora-19-x86_64-DVD.iso -vga std -boot menu=on -rtc base=localtime
Comment 22 Steve Tyler 2013-08-20 17:37:14 EDT
Harald Hoyer 2013-08-20 12:52:42 UTC
Status: ASSIGNED → CLOSED
Resolution: --- → RAWHIDE
Last Closed: 2013-07-05 13:21:45 → 2013-08-20 08:52:42

I can't see a recent commit related to /etc/adjtime:
http://git.kernel.org/cgit/boot/dracut/dracut.git/log/

The only commit related to /etc/adjtime is the one dated 2013-06-06 that installed /etc/adjtime and /etc/localtime to initramfs:
http://git.kernel.org/cgit/boot/dracut/dracut.git/log/?qt=grep&q=%2Fetc%2Fadjtime
Comment 23 Harald Hoyer 2013-08-21 04:43:34 EDT
(In reply to Steve Tyler from comment #22)
> Harald Hoyer 2013-08-20 12:52:42 UTC
> Status: ASSIGNED → CLOSED
> Resolution: --- → RAWHIDE
> Last Closed: 2013-07-05 13:21:45 → 2013-08-20 08:52:42
> 
> I can't see a recent commit related to /etc/adjtime:
> http://git.kernel.org/cgit/boot/dracut/dracut.git/log/
> 
> The only commit related to /etc/adjtime is the one dated 2013-06-06 that
> installed /etc/adjtime and /etc/localtime to initramfs:
> http://git.kernel.org/cgit/boot/dracut/dracut.git/log/
> ?qt=grep&q=%2Fetc%2Fadjtime

commit d27cd4dfdd51c7f5178c5f4cb8f5bf4668228995

http://git.kernel.org/cgit/boot/dracut/dracut.git/commit/?id=d27cd4dfdd51c7f5178c5f4cb8f5bf4668228995
Comment 24 Steve Tyler 2013-08-21 05:17:03 EDT
Thanks, Harald. Reverting the earlier commit is even better, but see below.

I was searching commit messages for "/etc/adjtime", but a search for "adjtime" finds it:
http://git.kernel.org/cgit/boot/dracut/dracut.git/log/?qt=grep&q=adjtime

Andre: This removes /etc/adjtime _and_ /etc/localtime from initramfs.[1]

This part of the commit may have some side-effects that we need to check:

-# setup system time
-if [ -f /etc/adjtime ]; then
-    if strstr "$(cat /etc/adjtime)" LOCAL; then
-        hwclock --hctosys --localtime
-    else
-        hwclock --hctosys --utc
-    fi
-fi

[1] Revert "base: setup correct system time and time zone in initrd"
http://git.kernel.org/cgit/boot/dracut/dracut.git/commit/?id=d27cd4dfdd51c7f5178c5f4cb8f5bf4668228995
Comment 25 Harald Hoyer 2013-08-21 05:26:21 EDT
(In reply to Steve Tyler from comment #24)
> This part of the commit may have some side-effects that we need to check:
> 
> -# setup system time
> -if [ -f /etc/adjtime ]; then
> -    if strstr "$(cat /etc/adjtime)" LOCAL; then
> -        hwclock --hctosys --localtime
> -    else
> -        hwclock --hctosys --utc
> -    fi
> -fi

Well, if no adjtime is in the initramfs that code fragment does not have any
effect anyway.
Comment 26 Steve Tyler 2013-08-21 05:51:24 EDT
Thanks for pointing that out:
-if [ -f /etc/adjtime ]; then
      ^^

I saw "hwclock" and panicked ... :-)
Comment 27 Steve Tyler 2013-08-21 06:20:58 EDT
Andre: This change is in dracut-032:
http://git.kernel.org/cgit/boot/dracut/dracut.git/commit/?id=032

Note You need to log in before you can comment on or make changes to this bug.