Bug 1164334 - filesystem log device on lvm prevents normal bootup
Summary: filesystem log device on lvm prevents normal bootup
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: systemd
Version: 7.0
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: systemd-maint
QA Contact: Frantisek Sumsal
URL:
Whiteboard:
: 1226375 (view as bug list)
Depends On:
Blocks: 1133060 1289654
TreeView+ depends on / blocked
 
Reported: 2014-11-14 17:09 UTC by Vimal Kumar
Modified: 2019-09-12 08:05 UTC (History)
12 users (show)

Fixed In Version: systemd-219-3.el7
Doc Type: Bug Fix
Doc Text:
Cause: Some mounts can have dependencies on other ones. Consequence: There was no way how to easily setup the dependencies Fix: add x-systemd.requires and x-systemd.requires-mounts-for for /etc/fstab Result:
Clone Of:
: 1289654 (view as bug list)
Environment:
Last Closed: 2015-11-19 15:02:07 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2092 0 normal SHIPPED_LIVE systemd bug fix and enhancement update 2015-11-19 12:13:57 UTC

Description Vimal Kumar 2014-11-14 17:09:03 UTC
1) Description of problem:

While using a logical volume as an XFS log device, the machine halts at emergency mode while booting. This is not for the root filesystem, but a simple data filesystem.

Pressing 'Ctrl + D' proceeds the booting successfully, and the filesystem can be seen as mounted. 

This is only seen when the log device is on a logical volume, and not on a normal disk.

Adding '_netdev' to the mount options solve this problem.

2) Version-Release number of selected component (if applicable):

xfsprogs-3.2.0-0.10.alpha2.el7.x86_64

3) How reproducible:

Always

4) Steps to Reproduce:

a) Create an XFS filesystem with a log device on a logical volume.

	# lvcreate -L 256M -n lv_journal01 rhel
	# mkfs.xfs -l logdev=/dev/rhel/lv_journal01 /dev/sdb

b) Add an entry in /etc/fstab so that it will be mounted at boot.
	/dev/sdb	/mnt/test	xfs	defaults,logdev=/dev/rhel/lv_journal01 1 2

c) Reboot the machine, it should halt at emergency mode asking for the root password or press 'Ctrl+D'.

5) Actual results:

The machine halts without booting up asking for the root password. 

6) Expected results:

The machine should boot up without any intervention. 

7) Additional info:

This happens only when the log device is on an lvm partition. The machine boots properly if the log device is on a normal partition.

Comment 2 Eric Sandeen 2014-11-14 17:36:36 UTC
This is not an xfs[progs] bug; the device it is looking for is not available at the time XFS needs it.

I'm not sure where it should go ... let's try dracut for now.

Comment 3 Harald Hoyer 2014-11-20 12:55:12 UTC
(In reply to Eric Sandeen from comment #2)
> This is not an xfs[progs] bug; the device it is looking for is not available
> at the time XFS needs it.
> 
> I'm not sure where it should go ... let's try dracut for now.

That's not a dracut bug. This partition is used on the real root, not in the initramfs.

The problem is, that "logdev=.." has to be expressed in a device dependency. So the /mnt/test mount unit not only depends on "/dev/sdb", but also on "/dev/rhel/lv_journal01".

This can be solved from the systemd side by teaching fstab-generator about the logdev keyword.

Comment 4 Lennart Poettering 2014-11-20 13:28:32 UTC
So, systemd's "systemd-fstab-generator" is responsible for creating the mount units from /etc/fstab, with the right dependencies. 

I really don't want to have hacks for specific file systems in that tool, though, after all the various file systems have vastly different options and semantics for referencing external devices log file system logs. ext4 seems to prefer major/minor numbers while xfs resolves paths (including symlinks) inside the kernel. The options are named really differently. ext4 calls it journal_dev=, while xfs calls it logdev= and rtdev=. 

if the kernel folks can't agree in common naming and common semantics then systemd really isn't the place to abstract this brokeness away.

Hence: if you want automatic generation of the right depencies, then please do that in xfsutils: add a small generator there that parses /etc/fstab looking for xfs partitions, and adds .d/ drop-ins that add the dependencies on the right devices with that. Similar, if ext4 wants the same level of support they should add a generator that solves that specifically for ext4's options from a generator shipped with e2fsprogs.

For details on generators see:

http://www.freedesktop.org/wiki/Software/systemd/Generators/

Comment 5 Eric Sandeen 2014-11-25 22:09:35 UTC
Lennart, while some of the more oddball filesystem options might requires some custom work, I would expect and hope that systemd could handle common device dependencies like options containing /dev/mapper/$FOO

Systemd doesn't need to care whether it's referenced as journal_dev, logdev, or rtdev, or quxdev but surely it can recognize the part after the "=" - i.e. /dev/mapper/my-pretty-name 

Systemd must already be able to handle dependencies for the main device (right?), so why shouldn't it handle an option device as well?

IOWS if we have:

/dev/mapper/maindev	/mnt/test	xfs	defaults,logdev=/dev/mapper/logdev 1 2

How is it that /dev/mapper/logdev is xfs's responsibility, but /dev/mapper/maindev is handled just fine by systemd?

Comment 6 Eric Sandeen 2014-11-25 22:34:27 UTC
Lennart, what do you think?  re-implementing basic device dependency handling in several different filesystem utility packages just doesn't seem optimal.

Comment 7 Eric Sandeen 2014-11-25 22:46:57 UTC
If you'd be willing to give an example of what the output should look like, I'd take a stab at adding basic device option parsing to the existing fstab generator.  As it is, I don't know where to start.

Comment 8 Lennart Poettering 2014-12-10 16:49:58 UTC
systemd doesn't try to "guess" what mount options mean. We won't have heuristics in systemd that look at the mount options and adding deps to anything that "looks" like a device. I mean, first of all this wouldn't work, since some file systems reference those devices via major/minor anyway. And even if everybody would reference them via /dev/ full names, it doesn't necessarily mean that systemd really always should wait for it...

The generators allow you to extend the systemd dependency tree. Look at systemd-fstab-generator for an example how they work, and see the docs I linked for more information. Most likely you just have to iterate through fstab and create a single .requires/ symlink for the right device.

Comment 9 Lennart Poettering 2014-12-10 16:59:43 UTC
Alternatively, I am open to adding support for a generic scheme how these dependency device are handled. For example, if the ext4 and xfs people can think of a good generic scheme how we can recognize references to other devices in the mount options, then we can certainly support that in systemd. 

Another option would be to introduce x-systemd.requires= or so as additional userspace mount option in systemd-fstab-generator, and simply ask users to explicitly configure the dependencies with that when they specify logdev=. Or in other words, instead of just putting "logdev=/dev/foo" in the options, you'd always have to specify "logdev=/dev/foo,x-systemd.requires=/dev/foo". The first option would be passed to the kernel and tells XFS what to make of the device. The second option tells systemd-fstab-generator to add an additional dependency. This wouldn't be fully automatic then, but one could argue that it is more powerful. And given that logdev= is probably not the most commonly used option maybe dumping this problem on the admins is not the worst of ideas...

Summarizing, three options:

a) Unify in the kernel how additional devices are referenced by file system mount options. Then add support for recognizing this in systemd-fstab-generator.
b) Solve this in xfsprogs via a new generator, and do xfs-specific parsing of the mount options.
c) Make the dependency declarations explicit via a new "x-systemd.requires=" mount option, independent of logdev=, and make sure admins always add both mount options.

What I am really not interested in though is option:

d) Add all kinds of magic to systemd-fstab-generator to recognize all kinds of file system specific mount options, just because the kernel folks can't agree on a common interface for this.

Pick one of a), b) or c), I am fine with either.

Comment 10 Eric Sandeen 2014-12-10 19:49:24 UTC
Honestly, I think a) is already done, if you accept "option=<devicepath>" as the existing kernel standard.  These are the auxiliary device options I find in the kernel:

btrfs
  device=<devicepath>
xfs
  logdev=<devicepath>
  rtdev=<devicepath>
ext3
  journal_path=<devicepath>
ext4
  journal_path=<devicepath>
reiserfs
  jdev=<devicepath>

In every one of these cases, the specified device is required to mount.

(Yes, ext3/4 has a dumb device number option too, but it's deprecated and has the above modern alternative. I am happy to ignore it.  I can find no other similar non-device-path device-specification mount options)

Recognizing this in systemd itself has the benefit of being a centralized fix, and will make every system start working again with systemd, no other package updates or admin actions are required.  I don't think it's a lot of magic, it's only recognizing the single standard pattern above, "option=<devicepath>"

If filesystems have other weird & random fs-specific needs, I can see where a fs-specific generator makes sense.  But really, the above is just standard fare.

If you're concerned that it's assuming too much, what if each filesystem can simply provide a template of options which contain dependencies, i.e. if xfs had a template file containing:

rtdev=%d
logdev=%d

that could be parsed by the fstab generator, and know that "%d" is a device dependency if it matches the pattern above.  That way systemd is not assuming or maintaining anything fs-specific, the filesystems can give clear specifications, and filesystem utility packages don't need to carry around systemd bits.

-Eric

Comment 11 Dave Chinner 2014-12-10 21:57:12 UTC
Lennart,

I just went to look at systemd-fstab-generator with an eye to modifying the script to parse the filesystem device parameters directly. Turns out that this is a binary and so i can't modify it with a text editor on my running system. I can't even understand what parsing is being done or the output format without having the systemd source code installed on my system. Ouch.

Look at it this way - I'm not anti-systemd at all. However,I'm not a systemd developer, nor have I any desire to do init system development. The proposed generator solution forces us to have to support systemd directly. That means we need conditional build infrastructure to support non-systemd package builders, not to mention the added complexity of supporting packages on systems where the init system is user selectable.\

I know, Lennart, that you don't care about non-systemd systems, but as the upstream pacakge maintainer I do - having to add package dependencies (both build- and installation- time) just to ensure a specific init system behaves correctly is fundamentally wrong. If the init system has internal dependcies it needs to generate to work correctly, then it's up to the init system developers to ensure those dependencies are corectly generated - it is not the responsibility of every 3rd party to ensure the init system works correctly.

The simple fact is that this is a problem introduced by systemd and is caused by a deficiency in it's fstab parser. The parser that can be easily extended to extract the small subset of device parameters and generate the dependencies systemd requires to boot. There's only a very small number of options you need to care about, as Eric documented above, and doing so keeps the init processing encapsulated within the init package.

IOWs, the most efficient, simple and effective solution for everyone is to add support to the systemd-fstab-generator to support these mount options, not demand everyone else write and maintain generators. Having everyone write their own generators is a sure road to inconsistency, bugs and madness...

-Dave.

Comment 12 Lennart Poettering 2014-12-16 13:28:32 UTC
(In reply to Eric Sandeen from comment #10)
> Honestly, I think a) is already done, if you accept "option=<devicepath>" as
> the existing kernel standard.  These are the auxiliary device options I find
> in the kernel:
> 
> btrfs
>   device=<devicepath>
> xfs
>   logdev=<devicepath>
>   rtdev=<devicepath>
> ext3
>   journal_path=<devicepath>
> ext4
>   journal_path=<devicepath>
> reiserfs
>   jdev=<devicepath>
> 
> In every one of these cases, the specified device is required to mount.

What kind of "standard" is this? The fact that you put together a table for this is indication enough that nothing is standard, certainly not the parameter name.

Is it that difficult to talk to your file system hacker friends and agree on a single name to name the option? A name that is unlikely to clash against similar parameters from any future file system? (i.e. don't pick "jdev" or so, since that is too likely to clash).

> Recognizing this in systemd itself has the benefit of being a centralized
> fix, and will make every system start working again with systemd, no other
> package updates or admin actions are required.  I don't think it's a lot of
> magic, it's only recognizing the single standard pattern above,
> "option=<devicepath>"

"single pattern"? It means adding either a table for the precise option name of the various file systems, or by adding heuristics, that guesses that something is a mount option of this type. Neither I want to see in my sources.

systemd is really not the place to add work-arounds for the fact that the kernel folks can't agree on a commons scheme to name their options.

> If filesystems have other weird & random fs-specific needs, I can see where
> a fs-specific generator makes sense.  But really, the above is just standard
> fare.
> 
> If you're concerned that it's assuming too much, what if each filesystem can
> simply provide a template of options which contain dependencies, i.e. if xfs
> had a template file containing:
> 
> rtdev=%d
> logdev=%d

I'd much rather prefer if the file systems would agree on a common name for the options, insteads of adding another indirection layer by defining a new attribute that exposes which option means what.

> that could be parsed by the fstab generator, and know that "%d" is a device
> dependency if it matches the pattern above.  That way systemd is not
> assuming or maintaining anything fs-specific, the filesystems can give clear
> specifications, and filesystem utility packages don't need to carry around
> systemd bits.

This sounds quite complex for a simple problem.

I mean, if you stick to a common name for the option, this not only makes the option easily udnerstandable to userspace, but also to admins.

Comment 13 Lennart Poettering 2014-12-16 13:36:45 UTC
(In reply to Dave Chinner from comment #11)
> Lennart,
> 
> I just went to look at systemd-fstab-generator with an eye to modifying the
> script to parse the filesystem device parameters directly. Turns out that
> this is a binary and so i can't modify it with a text editor on my running
> system. I can't even understand what parsing is being done or the output
> format without having the systemd source code installed on my system. Ouch.

you are funny. The file system code in the kernel isn#t shell either, how can you hack on that? 

The parsing is glibc setmntent(), it's very easy, very standardsy. 

But here you go, the sources are online:

http://cgit.freedesktop.org/systemd/systemd/tree/src/fstab-generator/fstab-generator.c

> Look at it this way - I'm not anti-systemd at all. However,I'm not a systemd
> developer, nor have I any desire to do init system development. The proposed
> generator solution forces us to have to support systemd directly. That means
> we need conditional build infrastructure to support non-systemd package
> builders, not to mention the added complexity of supporting packages on
> systems where the init system is user selectable.\

Hmm? Shipping a generator is just dropping in a file. That certainly doesn't break anybody's system even if he doesn't use system. You can also just add it to your RPM instead of upstream.

Anyway, if you want the options supported by systemd, there are three ways I can live with, see comment #9. Pick one.

Comment 14 Eric Sandeen 2014-12-16 15:58:59 UTC
Then we pick c)

Comment 15 Eric Sandeen 2014-12-16 16:47:34 UTC
Update summary; not unique to xfs.

Having the data device on a normal partition and any dependent device on lvm fails for both ext4 and xfs, and presumably any other fs.

Also, if the data device and log device are both on lvm, it seems to work in F20 at least.  So that may be a workaround: keep both pieces on LVM.

Comment 16 Eric Sandeen 2014-12-16 16:59:14 UTC
FWIW, the reason I push back on "common" naming for the journal device is that it is not sufficient to solve the problem of external device dependencies in general; other dependent devices may exist now or in the future - see for example rtdev in xfs.

There may well be more dependent device specifications for other filesystems, in or out of the tree, now or in the future.  (specification of a dedicated metadata device, or a snapshot device, or whatever may come).  Rearranging just the journal device mount option name is not a permanent solution to the problem of systemd recognizing device dependencies at mount time.

Comment 18 Lennart Poettering 2015-05-18 14:06:54 UTC
c) is implented now upstream

Comment 20 Lukáš Nykrýn 2015-06-04 10:24:50 UTC
*** Bug 1226375 has been marked as a duplicate of this bug. ***

Comment 22 errata-xmlrpc 2015-11-19 15:02:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2092.html


Note You need to log in before you can comment on or make changes to this bug.