Bug 748920

Created attachment 530115 [details]
first reboot

Created attachment 530116 [details]
second reboot

Due to its severity proposing as F16 Blocker. There are no criteria fitting this use case, probably closest could be:
"All elements of the default panel (or equivalent) configuration in all release-blocking desktops must function correctly in common use"
(setting system clock breaks system boot, that's not "functioning correctly")

But I think we really don't need criteria for this. This is so easy to trigger and it has so severe implications that I believe this automatically qualifies for blocker.

That criterion doesn't work at all. It's not a desktop issue.

But we could consider this under "The installer must be able to create and install to any workable partition layout using any file system offered in a default installer configuration, LVM, software, hardware or BIOS RAID, or combination of the above" and "In most cases (see Blocker_Bug_FAQ), a system installed according to any of the above criteria (or the appropriate Beta or Final criteria, when applying this criterion to those releases) must boot to the 'firstboot' utility on the first boot after installation, without unintended user intervention, unless the user explicitly chooses to boot in non-graphical mode. This includes correctly accessing any encrypted partitions when the correct passphrase is supplied. The firstboot utility must be able to create a working user account".

Does F15 behave the same?



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

CCing Eric for an e2fsprogs perspective.

This bug is present in F15 - I just set the clock back on a VM I had after updating it. I get 

_gfx_test_week_2: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
    (ie without -a or -p options)


Dropping to debug shell.

sh: can't access tty; job control turned off


In my case I ran

fsck /dev/sda

fixed the time and then exit, which allowed the boot to continue and booted into gdm properly. I have two partitions sda1 and sda2, sda1 => /boot and sda2 is an LVM. I didn't have to do anything with that partition however, just sda1.

sorry I meant to say I ran fsck /dev/sda1

Right, this is not a new e2fsprogs "feature" (and feature it is, if you ask upstream, though I don't know for sure why there is this fetish about time ...)

If you really want to be able to set your system clock massively behind your filesystem, you can set

broken_system_clock = 1

in /etc/e2fsck.conf and it should skip that check.

-Eric

given that f15 did the same, it's a 'feature' of e2fsck, and no-one's exploded yet, i'm probably -1 blocker on this. any other votes?

I got bitten by this in F15 as well.  Since it's not something new, and it really is considered a "feature" upstream, I guess I'll vote -1 blocker on this.

I'm -1 blocker on this, too. It's not new and there are workarounds.

That makes for -3 blocker, rejecting.

i have some systems that  often get the time set back to jan 1 1970. as long as i can fsck all filesystems thats ok. but if we are unable to get to a shell and run fsck then i think thats a blocker. as a freshly installed system may not be bootable.

so im +1 blocker

dennis: this isn't anything new in f16, though...

If we as a distro think the time check is nuts, we can ship an e2fsck.conf to turn it off.  It'd need to get pulled into initFOO stuff too.  I agree that it's not a blocker for F16 but it's something we could consider.

-Eric

I think the ideal thing would be for it to be *noted* but not considered some sort of screaming error, but unfortunately, it seems like this isn't easily possible (it'd require a patch).

To be a little more clear - fsck complaining about the clock is not new.  I don't know if the problem where no shell is presented after the error might be new; if it is, it's not e2fsprogs newness but something in the boot process, I think.

Even though I'm outvoted, my opinion is clear: +1 blocker, +1 nth.

We have +2 blocker, -3 blocker right now, how do we count that? And why we didn't vote at least about NTH?

(In reply to comment #11)
> I'm -1 blocker on this, too. It's not new and there are workarounds.

What do you mean by workarounds? Running fsck in dracut shell? It's not as easy if you are using more complex disk layout. And I didn't even receive that shell on bare metal. And it's game over for common users. Or do you mean setting e2fsck.conf? That prior-problem workaround, not post-problem workaround, and no one knows about that anyway.

(In reply to comment #13)
> dennis: this isn't anything new in f16, though...

I never understood this concept "once broken, always broken". Does that mean that we will never fix such bugs? I understand that sometimes it is useful to know whether this is an immediate regression or a long-standing problem and decide accordingly. But it's not matra. In this case F15 doesn't really matter, it's too severe.


Guys, try to look at it from the perspective of first-time-linux-user people (these people are what drives me to improve Linux, I don't know what's your driver). When they set "incorrect" time, they are screwed. If someone puts his experience to a blog post ("Just set your system time to the past and Fedora will never boot again, hah hah!"), people will die in laughter how easy is to break the system. Imagine that Microsoft would release their system behaving this way. It would hit the headlines all over the world. This problem is embarrassing from PR perspective and from common user experience.

I understand there are technical reasons why the current state is what it is. Maybe some operations in the disk journal may go wrong if you mount the system with past time. Maybe there are other reasons. Maybe it's just a whim of e2fsprogs people and it's not that important at all. But I would like to have some reasonable solution in F16 if possible (let's say there is no risk of losing data so we can just set the option in /etc/e2fsck.conf) and some more proper solution in the future (fsck giving special return code about this, or recognizable output, or running full fsck automatically just to be sure nothing's broken, or the tools for setting system time may add some one-time filesystem flag that time inconsistency shouldn't break next mount because it's expected, whatever).

Even if this is not accepted as a blocker, can we still pull off some easy patch to have this fixed in F16?

Adding CommonBugs keyword, this should be certainly documented if released this way.

Created attachment 530275 [details]
no prompt on bare metal

This is how it looks on my bare metal machine. No emergency prompt whatsoever. The only option for me is to burn a LiveCD on a different machine, boot from it and fix the problem manually.

This does potentially seem like a problem in the init environment which is
worse than before.

I always hated the dropping-to-a-shell-for-the-clock, but it was semi-common
and fixable.

Shouldn't a fsck failure drop to the shell here too?  That seems like the most
severe bug.  Unfixable fs corruption for any reason (not just the clock) would
land us in the same stuck state.

harald, can you please look at why dracut isn't showing a prompt here?



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

if fixing the lack of prompt in dracut isn't plausible, i wouldn't really hate just disabling the check in e2fsprogs, frankly. It does seem like a dumb thing to consider a critical failure.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Changing the behavior by adding /etc/e2fsck.conf will probably require that that file get sucked into the init environment, as well.

-Eric

I'm re-proposing this as our understanding has changed: the lack of a prompt is new, as far as we know, and makes the problem more serious, as you can't just manually run fsck to escape.

so, if we wanted to 'fix' the date being a killer, we'd have to adjust e2fsck.conf in e2fsprogs and then I think add:

dracut_install -o /etc/e2fsck.conf

to /usr/share/dracut/modules.d/99fs-lib/module-setup.sh in dracut, AIUI.

Still, not getting a recovery console when there's a critical error seems like a problem that might still be a problem if you hit some *other* critical error too, and we'd really need harald to know why that happened. I'll see if I can reproduce it on my laptop, just a sec...

So I just tested on my laptop, and I *do* get the fsck errors, but I did get a prompt for both / from dracut and /home from systemd.

the thing that's not giving kparal a prompt looks to be systemd, not dracut, so re-assigning there for now.

lennart or kay, can you look at this?

kamil, can you try a few times and see whether systemd never gives you a prompt, or sometimes does?

when systemd does its fsck, note, the system's /etc/e2fsck.conf would be in play, I believe.

Progress report: I found out that I can work around the missing prompt by adding rdbreak=pre-mount to kernel options. Then I can manually 'fsck /dev/sda2' and continue booting. After I have done that, I received one more error about /dev/mapper/VolGroup-lv_home, but I was given an emergency shell in this case. So currently it seems I get a dracut or emergency shell for logical volumes, but not for classic partitions.

Will investigate further.

I tested on a fully updated i686 machine:
systemd-37-2
dracut-013-16

The trick with rdbreak=pre-mount (or rdbreak=mount) is useful for correcting the issue, but system doesn't boot after that and is very reluctant to reboot, so you'll probably need to hard-reboot (sysrq is not enabled by default). Next boot is fine.

I have done several attempts with setting back time [1]. Sometimes I received an error about lv_root, sometimes I did not. But I always (in dozen reboots) received an error about /dev/sda2 (/boot partition) and I never received a recovery shell for that.

Please note that there is no file /etc/e2fsck.conf in my system.

=================

I re-installed the machine (i386 non-PAE, TC2 DVD, default package set), enabled updates-testing, fully updated, used default layout (same as before):
/dev/sda1  BIOS boot
/dev/sda2  /boot
/dev/sda3  VolGroup
VolGroup: lv_root, lv_home, lv_swap

After I changed time and reboot, I received dracut shell for lv_root, fixed the issue, continued booting, received emergency shell for sda2, fixed the issue, continued booting, and received error for lv_home but *not* received any shell. Therefore the problem is a little different than before. After reboot I received an emergency shell for lv_home, fix ed the problem, but system didn't continue booting (similar as when I used rdbreak), had to reset it. After reboot everything started fine.

After I changed time again, I received dracut shell for lv_root, fixed the issue, continued booting, received error for sda2 but *no* received any shell (the same behavior as in the original report).

On all of the subsequent attempts I never received a shell for sda2, only error.

It means that the behavior may differ for the first time you do time manipulation, but all the following attempts work the same (for me).


[1] Use gnome clock to set the time. I had some issues with "date" command. I will investigate further on it, just a heads-up, use gnome clock.

it's always systemd-fsck that gives you the error with no prompt, yes?



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

tested again on my laptop, and I still get a prompt for two partitions. dracut prompts me for / (which is identified by UUID), and systemd for /boot (/dev/sda2) . I can attach a picture of the prompt.

A thought occurs: I'm testing this not by setting the time in Fedora but by doing it in the BIOS. Why wouldn't that always be a relatively easy workaround for this? Just go into the BIOS and set the time forward?



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Since we have differing experiences in this one, can some other people test? To test, just reboot, go to BIOS, set the clock back a couple days or a month or whatever, and then boot to Fedora. You should get a prompt from dracut about the failure and it'll tell you to run fsck manually on some disk: do so, and then continue the boot. during boot, systemd should then have the same issue. it *should* ask for your root password then give you a prompt from which you can run the fsck manually and then continue boot. let us know if it does or doesn't. thanks!

once you've finished testing you can of course just boot back to BIOS and fix the date.

Just a note:

/etc/e2fsck.conf is not currently installed on fedora.  It'd be a change to e2fsprogs to do so.

-Eric

Can't we just flip the compiled-in default to e2fsprogs? (Yes, and likely annoy upstream.)

We can fork anything we like, I suppose ;)

Seems like it might be better to use the existing upstream methods, though, no?

Sure, just wondering whether it's easier to change the default in cases where we can logically state 'upstream default is bad'.

to be honest, I do have one patch I've carried for years :(

I was able to work around this pretty easily when I changed the system clock after install but before first reboot (which is my understanding of the blocker issue) but I want to reproduce that again before describing it here since my subsequent "reproductions" have been different.

To put my system into this situation after first reboot, I've been doing 'touch /forcefsck', rebooting and setting my system clock back >= 1 month before the system comes back up.

When I do this, dracut errors out about fsck errors on my lv mounted as / and gives me a repair shell. I am able to fsck the / lv and continue the boot process.

Later on, systemd errors out with fsck errors on my /boot partition but gives me no option to use a maintenance shell. I can reboot any number of times and end up in the same place with the same errors.

However, if I use the 'fastboot' kernel parameter, all of a sudden systemd gives me the option to get into the maintenance shell so I can fix the fsck errors. Once I do this and exit the maintenance shell, systemd shows fsck errors for my /home lv and does not give the option to use a maintenance shell.

If I reboot one more time (without fastboot), systemd fails with fsck errors on my /home lv again but gives me the option of using the maintenance shell. Once I run fsck on my /home lv, the boot process finishes without any more problems.

I've been able to do this twice on my system but this doesn't feel quite right. It seems odd to me that the 'fastboot' param would affect systemd in this way.

notting: we _can_, yes. but in that case we'd want to make sure dracut respects the change, so we'd have to pull the config file into the initramfs, I think. and then we haven't really solved the underlying bug, which appears to be systemd not always giving you a rescue environment when fsck fails: we've just removed one possible cause of fsck failing. there are others, of course...



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

I've been able to do this twice now, so it doesn't seem to be a fluke.

With a fresh minimal install of i686 F16 on bare metal, I moved the system clock back 2 months after the installation but before the first reboot.

When I reboot the system, I end up in the dracut shell with fsck errors on my / lv. When I fix those and exist the dracut shell, systemd halts booting with more fsck errors about my /home lv and gives me no option to use the maintenance shell to fix it.

If I add 'fastboot' as a kernel parameter after rebooting, I am able to boot with no problems. Since that mounts everything, the last mount time is changed and subsequent boots are fine.

After getting a working system, I tried setting the system clock ahead by 2 months to trigger the usual fsck runs at boot. There were no failures and the system is still booting normally.

Can someone else try this as a workaround? Note that I've only been able to get this exact method to work on first reboot. This doesn't work when I've forced fsck on boot (see comment 38).

this is now the last remaining blocker. i'm still yet to hit a situation with this bug which couldn't reasonably be worked around, so i'm kind of shading -1 on it.

here's a thought: for those who get stuck with 'no prompt' from systemd can you try just blind-typing your root password and see if it dumps you to a prompt?



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

fastboot does work as a workaround here, too.

I think making sure we have docs/workaround noted is good enough for now, so -1 blocker for me.

We could also fix this in a update once we have had time to come up with a clean and tested solution in rawhide.

yes, it is susceptible to fixing with an update, another consideration.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

well, the present vote total is clearly not enough to accept this and all other blockers are addressed, so for at least the purposes of getting RC1 spun, let's mark this as rejected. We can re-test in more depth with RC1 and re-consider it if our understanding changes further.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Hmm, so what are you expecting me to fix in systemd here?

Lennart, I believe the problem we currently see in systemd related to this bug is that if system-fsck fails we receive the emergency console just sometimes, but not every time.

Is this still an issue or can this bug be closed?

I'll confirm when some F17 compose is out.

The latest rawhide VM I have (updated today) won't boot to a console. I get into starting services, but then suddenly I have a systemd-fsck line and then nothing. It sits there for awhile. I'll re-post if it every completes. However it seems still present if the symptoms are slightly different.

> To sum this up. Setting back time in Fedora for more than one day is a
> showstopper for a common user. Dracut shell = end of game.
I would like to point out the issue with common user - I have a common user using Fedora after Windows virus incident, so I don't have to solve such emergency again. But it is troublesome to explain, what is going on with drop to shell described in this bug and why he has to input cryptic commands instead of "just using" the system.

Please, consider automating this. If you need a new bug for "make boot user friendly", I will post it.

Note: I have Fedora 16 and issue in this bug is very common. It happens almost every time when suspend/hibernate does not succeed.

I have tried in a VM. The issue is still present. Maybe even worse, because gnome-shell was oopsing on me since I set back time. However, I hit several issues with setting time with regards to KVM, so I'll re-test properly on bare metal soon.

I have done some research and here are the results:

1. This problem still exists in Fedora 17. Both of them: a) Fedora does not boot if you set back time, and b) you don't get maintenance shell in specific scenarios

Reproducer:
a) Use the default partitioning
/dev/sda1 1MB GPT
/dev/sda2 500MB /boot
/dev/sda3 rest lvm
lvm contains:
lv_root, lv_home, lv_swap

b) If you set back time more than 1 day and reboot, you'll encounter dracut shell for vg-lv_root (see f17_virt_problem1.png). Fixing the problem and continuing boot will print out errors about /dev/sda2 (see f17_virt_problem2.png). You can't fix it, no maintenance shell provided. After reboot you'll skip vg-lv_root problem (that is already fixed), but you'll again encounter /dev/sda2 problem. If you are using plymouth, the boot will appear as "stuck" indefinitely (see f17_virt_splash.png, but you can use Esc to switch to text mode). You can fix this only by reverting system time in BIOS, using a rescue CD or some higher magic (like forcing dracut shell from grub).

c) If you use a different partitioning setup without lv_home, you'll be given a maintenance shell for /dev/sda2 problem, so you can fix it. The boot will hang after that, but next reboot is fine.

2. I was able to reproduce this consistently with a bare metal machine and with a VM. But be aware that when working with VM it can be tricky to change the system time, read http://kparal.wordpress.com/2012/02/28/changing-system-time-in-a-virtual-machine/

3. This problem was already reported as bug 577126 and minor fixes were introduced (like ignoring +-24 hours time changes).

4. This problem is largely prevalent, even though it might not seem so:
* Dennis Gilmore confirmed he saw system time changes in VMs often.
* Comment 51 confirmed this might happen after failed suspend/hibernate, and also the current fix procedure is "cryptic commands magic" from a common user perspective.
* There are Fedora admins who disabled fsck on boot completely (which is madness) just because they need an environment where the system survives users playing with it (including playing with time):
http://unix.stackexchange.com/questions/8409/how-can-i-avoid-run-fsck-manually-messages-while-allowing-experimenting-with-s
* System time can be changed knowingly, by accident, or by bug. For every approach there are many test cases. E.g.: I have already tried to reproduce several bugs by changing system time. Or have your kid play with your computer and you'll end up with shifted time very soon.
* Probably the best evidence how this issue is wide-spread is this link:
https://www.google.com/search?q=Superblock%20last%20mount%20time%20is%20in%20the%20future
Just skim through it and you'll be amazed how often this can break.
* Fedora users are believed to be much more technical than e.g. Ubuntu users. Still I'd bet anything that more than 50% of all Fedora users never used fsck and the maintenance shell is a showstopper for them.

5. I tried to reproduce the same problem in Ubuntu. I have not used LVM (the default installer doesn't support it), just standard partitions swap and /. After setting back time and reboot, the graphical splash gives you some options what to do (see ubuntu-splash.png). Hitting F (fix) or I (ignore) will let you boot into your system without problems. If you boot without splash, you'll get a similar choice in the text mode (see ubuntu-problem.png). Hitting F will fix the problem and reboot the machine (see ubuntu-problem-fixed.png). It's not perfect, but it's far superior user experience.

6. I didn't find any information why e2fsck upstream is so obsessed with time consistency. Eric, you seem like you have some contacts there, could you ask them, please?

7. If there are no clear risks involved, I'd like to set broken_system_clock = 1 in /etc/e2fsck.conf. My biggest argument is: You have to fix the superblock last mount time anyway, so why won't we fix it automatically? Nagging the user to write the commands by hand doesn't help anything.
Another approach (with lowered user experience, but maybe safer) is to provide users with an easy way to trigger fsck (without issuing manual commands), similarly to what Ubuntu does. "Hit F to fix automatically, M for manual recovery, ...". It can be handled by dracut and plymouth.


As you see, there are multiple issues involved. Either e2fsck defaults have to be changed, or a new functionality in dracut and plymouth has to be implemented, or at least the recovery console has to be offered using systemd. I'd like this bug to be used for the first or second issue, and I can report a separate bug against systemd if this doesn't seem like to be resolved soon.

Reassigning to e2fsprogs. Also proposing as F17 Final blocker.
(It was rejected as a blocker for F16, but the situation was a bit different back there - it was close to Final release date and the lack of maintenance shell happened just occasionally).

Created attachment 566310 [details]
f17_virt_problem1.png

Created attachment 566311 [details]
f17_virt_problem2.png

Created attachment 566312 [details]
f17_virt_splash.png

Created attachment 566313 [details]
ubuntu-splash.png

Created attachment 566314 [details]
ubuntu_problem.png

Created attachment 566315 [details]
ubuntu_problem_fixed.png

dracut/systemd/whateveritis definitely needs to be fixed if under these circumstances no rescue shell is offered to correct the problem, right?  Please do file that bug.

As for e2fsck, I still say it is working as designed, even if we don't like the way it's designed.

Perhaps routing around the damage in e2fsck.conf is the way to go.

(In reply to comment #60)
> dracut/systemd/whateveritis definitely needs to be fixed if under these
> circumstances no rescue shell is offered to correct the problem, right?  Please
> do file that bug.

I filed bug 798328. I proposed it as F17 Final blocker, which means it can lower the probability of accepting this one as a blocker as well.

eric: I'd say this comes under the heading of 'reasonable distribution customization' - I don't think we should interpret Fedora 'stick close to upstream' policy *so* strictly that we refuse to change a default configuration option which certainly seems capable of causing considerable user pain, as comprehensively described by Kamil.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Adam, that's fine as long as there's not some dire consequence that I'm missing.  Just sanity checking upstream, I don't mind turning it off, barring unforeseen consequences.

dracut includes /etc/e2fsck.conf, if present since dracut-014, and I agree with makeing the failcase nicer.

I will do an Ubuntu like key select thing.

(In reply to comment #64)
> dracut includes /etc/e2fsck.conf, if present since dracut-014, and I agree with
> makeing the failcase nicer.
> 
> I will do an Ubuntu like key select thing.

That is great, thanks. IIUIC even if set broken_system_clock = 1, there are still use cases where "Ubuntu like key select thing" gets used, like fsck after hard-poweroff.

I'd even say that if we have this functionality, we don't need to have broken_system_clock = 1. But that depends on a) whether there are risks related to this setting b) how fast the new functionality be available. We can switch to broken_system_clock = 1 now and consider reverting it back once the new functionality is present and working.

I'll just chime in with a non-power-user opinion. System clock inconsistency at the boot fsck just killed my boot process. Yes, the system clock was set wrong, I have no idea why it was suddenly wrong. As this happened at reboot after a major system update, I assumed that something broke in the update, not that unusual, and spent hours googling and rolling back updates to revive the system. Setting the clock right in BIOS finally made it work.

Now to my opinion: it seems that an incorrect system clock that trips the boot is not that uncommon. Killing the boot process is pretty harsh. How about it just suggests that the clock might be wrong and offer, but maybe not recommend, to ignore this error once and try to resume the boot? Leaving a novice user with this problem will make them leave linux.

I've found that several of the changes mentioned in this thread are required to remove the sanity check for last mount time at boot.

Here's my process which worked, three easy steps ;)

1. Disable the initramfs in your /boot/grub.conf (I use grub instead of grub2) file.  This is required because the initramfs performs a disk check which includes this sanity check.  

WARNING: The initramfs is required if you are using softraid (md) devices or lv's.

2. Move /lib/systemd/systemd-fsck to /lib/systemd/systemd-stopsMyBootUp, then recreate it with:

# echo "exit 0" > /lib/systemd/systemd-fsck

WARNING: This stops systemd's fsck, so no disk check will be run at boot time.

3. Add "broken_system_clock = 1" to your "/etc/e2fsck.config", if you don't have this file create it with:

# echo "broken_system_clock = 1" > /etc/e2fsck.config

For a training class, we have nearly 200 laptops that must be rolled back in time every week to a fixed date for the class start (kinda like Ground Hog Day).  We are running a Fedora VM using VMWare Player on a Windows 7 laptop.

I need a configuration that will start successfully when the clock is setback one week without having to run fsck on 200 laptops to fix this system error. 

Comment 67 does not really tell me in enough detail the three easy steps explicitly enough.  I need to know exactly what I need to change so that these VM's will start every week when the clock is set back without having to run fsck.  For example, how do I disable initramfs?  Show me the lines in grub.conf before and after editing.

Also, do I really need to disable all file system integrity checks?  Seems like all I really want is disable fatal start errors when the system date is earlier than the last file system mount time or if I could prevent the system from writing and tracking the last file system mount time.  As was stated earlier in this bugzilla, if Windows did something like this where a cryptic message halted system start, it would not be tolerated.

Discussed at 2012-04-20 blocker review meeting - http://meetbot.fedoraproject.org/fedora-bugzappers/2012-04-20/fedora-bugzappers.2012-04-20-17.01.log.txt . We had no devel or releng or FPL representation at the meeting and the call is clearly controversial (we had no clear consensus of those who were present), so we agreed to punt the decision to a meeting that is better attended. This bug remains a proposed blocker.

I think that to really fix this, whatever it is these days, dracut I guess :) that makes initramfs will need to pick up /etc/e2fsck.conf, if it exists.

once it does, putting:

broken_system_clock = 1

into /etc/e2fsck.conf should suffice for now.

....

You can include it in the initramfs manually by i.e.:

# dracut -v -I /etc/e2fsck.conf -f /boot/initramfs-3.3.2-1.fc16.x86_64.img 3.3.2-1.fc16.x86_64

Presumably you could write a dracut module to do it too, or maybe a core dracut change could be made, I'm not certain.

So this probably has 2 pieces; add the prescribed e2fsck.conf to e2fsprogs, and teach dracut and/or mkinitrd to pick it up.

-Eric

Harald, I'm game to toss /etc/e2fsck.conf in if dracut can pick it up...

# date
Sun Apr 20 16:27:42 EST 1980

That's after a reboot with no trouble :)  I just did what I mentioned above: added e2fsck.conf and rebuilt the initramfs.

I'll stick e2fsck.conf in e2fsprogs now, sorry for the delay.  Dracut will need to pick it up too.

e2fsprogs-1.42-3.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/e2fsprogs-1.42-3.fc17

Ok, I don't know whether the problem of not getting a shell to be able to fix the error has been addressed elsewhere, but the config file is in the package now at least.  Guess I'll file a dracut bug to pick it up.

Bug 814874 - Please include /etc/e2fsck.conf in initramfs

Package e2fsprogs-1.42-3.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing e2fsprogs-1.42-3.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-6299/e2fsprogs-1.42-3.fc17
then log in and leave karma (feedback).

Hi All,

This issue is complicated.  The last boot time "sanity" check is in 2 places that will stop the boot requiring an fsck or a "ctrl + d" to ignore.

1. /lib/systemd/systemd-fsck (which does NOT respond to the /etc/e2fsck.conf config)

 - Run this to remove this tool thus disabling the check:

 #  echo "exit 0" > /lib/systemd/systemd-fsck

2. /lib/systemd/systemd-fsck exists in two places, in dracut (the initramfs) and in the main fs run by systemd during boot.

 - Run this to disable it in your initramfs (dracut)

 # echo "exit0" > /lib/systemd/systemd-fsck
 # dracut --install /lib/systemd/systemd-fsck

3. Also setting using the "system-config-date" and unchecking "system clock uses utc" will prevent the time change that occurs during the boot between "Other" OS's and Fedora.



(In reply to comment #77)
> Package e2fsprogs-1.42-3.fc17:
> * should fix your issue,
> * was pushed to the Fedora 17 testing repository,
> * should be available at your local mirror within two days.
> Update it with:
> # su -c 'yum update --enablerepo=updates-testing e2fsprogs-1.42-3.fc17'
> as soon as you are able to.
> Please go to the following url:
> https://admin.fedoraproject.org/updates/FEDORA-2012-6299/e2fsprogs-1.42-3.fc17
> then log in and leave karma (feedback).

Michael, I think you are making the issue too complicated.  There should be no need to disable systemd-fsck.

All that should be required is the presence of the new /etc/e2fsck.conf in the initramfs, something which should be taken care of automatically if/when dracut is updated per bug 814874 and the newer e2fsprogs is installed.  Until then, you can rebuild the initramfs and manually include /etc/e2fsck.conf from an updated e2fsprogs.

Once the config file is present in the initramfs, e2fsck won't exit with error due to large clock offsets, and boot will continue as normal even if the clock is off.

Can you show otherwise?

I believe /lib/systemd/systemd-fsck is the application stopping the bootup due to the large time gap.  It's not part of the e2fsprogs package (according to rpm -ql), and doesn't seem to respond to the /etc/e2fsck.conf file, at least in my testing on F15.

I found "disabling" /lib/systemd/systemd-fsck in both the initramfs and the root fs was the easiest way to avoid the stop during boot.

/lib/systemd/systemd-fsck is started by the fsck-root service.  I noted in the fsck.c file in the systemd source that systemd-fsck seems to call "/sbin/fsck" using the execv call with the fsck args -a -T -l (and others depending on state).

That being the case I'm not sure why /sbin/fsck in the initramfs did not respond to the /etc/e2fsck.conf file installed into the initramfs when executed by systemd-fsck.  

Eric in your testing did you find that installed the /etc/e2fsck.conf file prevented the stop at boot?

(In reply to comment #80)

> I found "disabling" /lib/systemd/systemd-fsck in both the initramfs and the
> root fs was the easiest way to avoid the stop during boot.

Of course, but that's a big hammer.  If that completely disables any boot-time fsck, then yes, I would expect it to avoid the problematic fsck return value, but it's hardly an optimal solution.

(In reply to comment #81)

> Eric in your testing did you find that installed the /etc/e2fsck.conf file
> prevented the stop at boot?

In my initial testing, yes.  Before, boot stopped, because e2fsck failed, because the time was wrong.  The conf file tells e2fsck not to fail due to time differences, so I expect this to be a complete solution once it's present in the boot environment.

Adding it to the initramfs manually, and then setting back the time to 1980, booted fine for me.

You found otherwise?

Just remembered that setting the clock to 1980 is a bad test :/  e2fsck decides on its own that that is "too far back" and automatically ignores the time difference :(  grumble grumble.

And, crud, the format of the conf file I pushed was wrong.  If you want to re-test, try this in e2fsck.conf:

[options]
broken_system_clock = 1

I had missed the [options] header, and without that it doesn't work.

I'll build a new e2fsprogs :/

e2fsprogs-1.42-4.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/e2fsprogs-1.42-4.fc17

Just CCing me as I hit this on a systems that do not have RTC or RTC+backup battery (and there is a plenty of them, usually the embedded devices).

Adam, feel free to test the latest e2fsprogs-2.42-4.fc17, make a new initramfs including the new e2fsck.conf file, and see if that resolves it for you.

-Eric

Ok, dracut-018-23.git20120419.fc17 has been built, and automatically includes e2fsck.conf.

So with both updates in place, re-running dracut (or installing a new kernel) will create a new initramfs with an e2fsck which should not stop on the clock problem.

-Eric

Package e2fsprogs-1.42-4.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing e2fsprogs-1.42-4.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-6496/e2fsprogs-1.42-4.fc17
then log in and leave karma (feedback).

e2fsprogs-1.42-4.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.