Bug 1202024

Summary: fsck on systems using PC-BIOS localtime perform fsck at every boot
Product: [Fedora] Fedora Reporter: Jonathan S <bugzilla_acct_1959>
Component: e2fsprogsAssignee: Eric Sandeen <esandeen>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 21CC: belegdol, bugzilla_acct_1959, esandeen, fedora, hedayatv, jayabharat, jgotts, josef, kzak, lovetide, martinthain99, mrmazda, oliver
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: e2fsprogs-1.42.12-4.fc20 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-05-02 18:07:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Logs for comment #14 none

Description Jonathan S 2015-03-14 16:17:01 UTC
Description of problem:

Since update to e2fsprogs-1.42.12-3.fc21, all PCs that have PC-BIOS set to localtime and have a timezone ahead of UTC (UTC+1, UTC+2, etc) fail the second fsck on the root filesystem with "Superblock last write time is in the future" and run fsck at every boot. This slows boot-time considerably.

See further https://bugzilla.redhat.com/show_bug.cgi?id=963283
comment 18 onwards.

Problem seems to be caused by the first fsck running under the initramfs, which always uses UTC. On leaving the initramfs, the second fsck is running at an *earlier* time (one hour earlier is you're UTC+1) - fooling fsck into thinking that the root filesystem has had its superblock written in the future.

This is a serious problem (hence severity marked high)- if one assumes that half the Fedora users are ahead of UTC, and half of those are using localtime (probably dual-booting Windows), then a quarter of all users are getting a boot-slowing fsck at every boot.

Comment 1 Eric Sandeen 2015-03-18 15:18:49 UTC
Can you attach the output of dumpe2fs -h for one of the filesystems which gets fsck'd, and the contents of /etc/e2fsck.conf as well?  Oh, and full e2fsck output if that's logged somewhere.

Something seems odd, it's not supposed to trigger a check if it's within 24h of the correct time.

Comment 2 Eric Sandeen 2015-03-18 16:56:05 UTC
In my testing, with TZ set to Europe/Amsterdam, RTC set to localtime, a 10-day check period on the filesystem, and broken_system_clock = 0 in e2fsck.conf, I do not see a full fsck.

I patched e2fsck to clearly state "Skipping" when check_if_skip() exits, and it's clearly doing so.

I'm going to need more information about what you've seen to investigate further.

Something like:

# journalctl -b --no-pager | grep fsck

would help, too.

Thanks,
-Eric

Comment 3 Eric Sandeen 2015-03-18 17:03:00 UTC
Wait, I take that back.  I do see something, let me investigate further.

Comment 4 Eric Sandeen 2015-03-18 18:46:21 UTC
Ok, the problem here is that the date-wrong-by-24h triggers a check, which sets E2F_FLAG_PROBLEMS_FIXED, which means the check won't get skipped.

I'll keep working it upstream...

Comment 5 Jonathan S 2015-03-18 19:59:33 UTC
Thanks for the stream of consciousness, Eric :-)
Glad you've identified the problem.
I guess you don't need it any more but you can see an extract of my journal in comment_20 of:
https://bugzilla.redhat.com/show_bug.cgi?id=963283
You can see that fsck says that the clock is out by less than a day, says 'FIXED' - and then goes on to do a full check anyway.

Comment 6 Eric Sandeen 2015-03-18 20:06:59 UTC
> Thanks for the stream of consciousness, Eric :-)

Sometimes talking to the duck works.  ;)

I'm sending a patch upstream now, we'll see if Ted likes it.

-Eric

Comment 7 Felix Miata 2015-03-21 10:52:56 UTC
Is this somehow distinguishable from bug 1201978 ?

Comment 8 Jonathan S 2015-03-21 22:15:52 UTC
Hi Eric

I noticed on comment_5 of
https://bugzilla.redhat.com/show_bug.cgi?id=1201978
you said:
"(if fsck is really being run twice, though, I can't explain that one)"

Well, I looked through my logs. First, looking at Fedora 21:
1) Kernel 3.17.7 and earlier ran fsck ONCE. There's no 'Switching root' and no change to the system clock - the log shows the clock remains the same throughout.
2) Kernel 3.17.8 changed bootup fairly radically and runs fsck twice. Here we get the first fsck under the initramfs, then 'Switching root' to the the root-filesystem, at which point the system clock changes. After the clock changes (when we have the root-filesystem mounted - presumably ro), fsck gets run again.

Note this seems to apply to only the root-filesystem. My non-dual-booting machine with BIOS set to UTC has a /home too. That gets fsck'd only once (no matter what kernel) - it happens a bit later in the boot sequence some time after the *second* /root fsck.
Looks like the initramsfs (on 3.17.8 and later) is checking that the root-filesystem to make sure it's consistent before it loads it.

Looking right back to May 2014 with kernel 3.11.10 (Fedora 20), that did switch root (with the clock change) but the fsck ran only once *after* the switch root.

Long and short, all the kernel changes seem to playing havoc with the running of fsck.

Comment 9 deadrat 2015-04-07 16:35:02 UTC
Is this bug a duplicate of bug 1201978 https://bugzilla.redhat.com/show_bug.cgi?id=1201978

Comment 10 Felix Miata 2015-04-08 03:27:00 UTC
(In reply to rat from comment #9)
> Is this bug a duplicate of bug 1201978
> https://bugzilla.redhat.com/show_bug.cgi?id=1201978

By strict definition it cannot be. One cannot duplicate something that does not exist. This bug did not exist when that bug was created. The elder bug has a longer CC list than does this, while both appear to be describing an identical problem.

Comment 11 Eric Sandeen 2015-04-10 16:00:18 UTC
Ted wrote his own patch to do something similar instead of using mine, I guess I need to test it.

http://git.kernel.org/cgit/fs/ext2/e2fsprogs.git/commit/?id=f096708126412c0569e40cfbd5740729976bf12a

Comment 12 Eric Sandeen 2015-04-10 16:01:11 UTC
(In reply to Jonathan S from comment #8)

> Long and short, all the kernel changes seem to playing havoc with the
> running of fsck.

The kernel really doesn't, and can't, affect how fsck runs.  That's a userspace issue.  If there's a problem, it probably needs its own separate bug.

Thanks,
-Eric

Comment 13 Jonathan S 2015-04-12 19:34:45 UTC
(In reply to Eric Sandeen from comment #12)

I didn't express that very well - it's all the attendant changes going on that have caused the problems rather than specifically the kernel. Anyway, I think I got carried away having a generalized moan. Sorry :-(

(To everyone)

Looking at https://bugzilla.redhat.com/show_bug.cgi?id=1201979 comment 3 onwards, a good workround is to boot your root-filesystem rw rather than ro (think grub config).
I think Zbigniew is a Fedora developer so he knows what he's talking about.

Comment 14 Ricardo Garcia 2015-04-25 11:08:42 UTC
Thanks everyone here for reporting and working on this issue. I'm also suffering this problem as I'm East of GMT and dual-boot Windows 7. The Fedora 21 installer chose to set the system clock to local time when it detected Windows.

I've cloned the e2fsprogs repository and temporarily replaced e2fsck and fsck.ext{2,3,4,4dev} with the compiled version from it. It appears to fix the issue on my computer. If there's anything I can do to test solutions or speed-up the process, let me know.

I'm attaching the critical-chain output and systemd-fsck-root service log from before and after using e2fsprogs from master.

Comment 15 Ricardo Garcia 2015-04-25 11:11:02 UTC
Created attachment 1018713 [details]
Logs for comment #14

Comment 16 Eric Sandeen 2015-05-01 01:15:56 UTC
Ok, I'm finally getting this patch built into fedora.  Sorry for the delay.

Comment 17 Fedora Update System 2015-05-01 02:02:37 UTC
e2fsprogs-1.42.12-4.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/e2fsprogs-1.42.12-4.fc21

Comment 18 Fedora Update System 2015-05-01 02:02:47 UTC
e2fsprogs-1.42.12-4.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/e2fsprogs-1.42.12-4.fc22

Comment 19 Fedora Update System 2015-05-01 02:03:43 UTC
e2fsprogs-1.42.12-4.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/e2fsprogs-1.42.12-4.fc20

Comment 20 LiuYan 2015-05-01 17:03:56 UTC
Thanks for fixing. This fsck bug really drive me crazy.

Comment 21 Eric Sandeen 2015-05-01 17:04:46 UTC
Sure thing, I'm sorry it took so long to get it built.  :(

Comment 22 Zbigniew Jędrzejewski-Szmek 2015-05-02 18:01:37 UTC
*** Bug 1204330 has been marked as a duplicate of this bug. ***

Comment 23 Fedora Update System 2015-05-02 18:07:26 UTC
e2fsprogs-1.42.12-4.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 24 Fedora Update System 2015-05-04 15:23:47 UTC
e2fsprogs-1.42.12-4.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 25 Fedora Update System 2015-05-17 06:43:12 UTC
e2fsprogs-1.42.12-4.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.