Bug 771638

Summary: emergency shell did not appear after local-fs.target failure
Product: [Fedora] Fedora Reporter: Wolfgang Denk <wd>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 16CC: johannbg, lpoetter, metherid, mschmidt, notting, plautrba, systemd-maint
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: systemd-37-20.fc16 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-07 15:49:33 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Wolfgang Denk 2012-01-04 12:35:39 UTC
Trivial errors like a missing / failed disk drive or even an
incorrect file system type in /etc/fstab wioll cause hard boot
failures.

Version-Release number of selected component (if applicable):

systemd-37-3.fc16

How reproducible:

100%

Steps to Reproduce:
1. Use a working system, then change in /etc/fstab the file system
   type for any unessential (i. e. not really needed for a successful
   boot) file system.
2. For example, change the /etc/fstab entry for /boot from ext4 into
   ext3
3. Reboot.

(4. Boot from DVD in recovery mode.  This will not work either.)

Actual results:

The system fails to boot.  systemd complains about missing
dependencies.

Expected results:

The system should just boot. The inconsistent file system should just
be omitted from being mounted, but such a trivial issue should
_never_ block the whole system from booting.

Additonal notes:

Similar problem also prevent recovery of the system when booting in
recovery mode from DVD, see bug # 771635.

Comment 1 Michal Schmidt 2012-01-04 15:44:51 UTC
(In reply to comment #0)
> The system fails to boot.  systemd complains about missing
> dependencies.

To be precise... Do you get dropped into "emergency mode" then?

> The system should just boot. The inconsistent file system should just
> be omitted from being mounted, but such a trivial issue should
> _never_ block the whole system from booting.

This seems to be a common complaint. The old rc.sysinit did not require all mounts from fstab to succeed.
So I'm inclined to change the fstab entries to be WantedBy=local-fs.target by default (instead of the current RequiredBy).
The 'nofail' option would still disable the ordering dependency with respect to the target.

Comment 2 Wolfgang Denk 2012-01-05 11:49:43 UTC
(In reply to comment #1)
>
> > The system fails to boot.  systemd complains about missing
> > dependencies.
> 
> To be precise... Do you get dropped into "emergency mode" then?

No.  The system comes to a halt without any way to input any commands
or do anything else but to reset the system.

> > The system should just boot. The inconsistent file system should just
> > be omitted from being mounted, but such a trivial issue should
> > _never_ block the whole system from booting.
> 
> This seems to be a common complaint. The old rc.sysinit did not require all
> mounts from fstab to succeed.

Right, and this makes perfect sense.  It is _much_ easier to recover a
partitally running system than a dead one.

> So I'm inclined to change the fstab entries to be WantedBy=local-fs.target by
> default (instead of the current RequiredBy).
> The 'nofail' option would still disable the ordering dependency with respect to
> the target.

It would be nice if above mentioned "emergency mode" would be used in
case of problems.

Thanks.

Comment 3 Michal Schmidt 2012-01-05 12:26:39 UTC
(In reply to comment #2)
> No.  The system comes to a halt without any way to input any commands
> or do anything else but to reset the system.

I can't reproduce this. I get the emergency shell.
Could you take a picture of the errors you get?

Comment 4 Wolfgang Denk 2012-01-05 14:01:44 UTC
(In reply to comment #3)
> (In reply to comment #2)
> > No.  The system comes to a halt without any way to input any commands
> > or do anything else but to reset the system.
> 
> I can't reproduce this. I get the emergency shell.
> Could you take a picture of the errors you get?

Hm... On the systems I can play around with now it works, too (i. e. I get the
"Welcome to emergency mode" message and can enter a shell.  The servers where
this happened are back in operation again, so I cannot re-test this soon (I was
happy to haven them for a full day during xmas break).

I will re-test this as soon as we have to reboot any of the servers.  Sorry.

Comment 5 Michal Schmidt 2012-01-12 16:02:50 UTC
(In reply to comment #1)
> So I'm inclined to change the fstab entries to be WantedBy=local-fs.target by
> default (instead of the current RequiredBy).

Lennart convinced me that papering over failing mounts by default would not be safe.
systemd cannot know if the mount that failed is trivial or essential for the integrity of the system. By default it has to err on the safe side. Non-essential mounts should be marked with the 'nofail' option.

Let's keep this BZ open for the issue with the missing emergency shell.

Comment 6 Michal Schmidt 2012-03-18 14:58:33 UTC
We need someone who can reproduce the problem of the missing emergency shell to tell us the error messages, or we'll close this due to INSUFFICIENT_DATA.

Comment 7 Wolfgang Denk 2012-03-18 16:04:35 UTC
(In reply to comment #6)
> We need someone who can reproduce the problem of the missing emergency shell to
> tell us the error messages, or we'll close this due to INSUFFICIENT_DATA.

I'm not sure what you mean by "error messages".  When this happens, the box is more or less dead. There are no error messages, it's just hanging, with no other way to recover than to reset it.

Comment 8 Michal Schmidt 2012-03-19 14:20:41 UTC
I was referring to "systemd complains about missing dependencies.". I thought it also applied to this failure mode. If there are no error messages, we need to make it more verbose.
One option is to boot with "systemd.log_level=debug systemd.log_target=console". Ideally with a serial console connected ("console=ttyS0" or similar) to capture the logs.

Comment 9 Michal Schmidt 2012-06-07 15:49:33 UTC
I believe the missing emergency shell got fixed in systemd-37-20.fc16. It was caused by bug 798328.