Description of problem: systemd-remount-fs fails if data=writeback is specified in /etc/fstab which causes non-working system except in grub "ro" is replaced by "rw" Version-Release number of selected component (if applicable): $ rpm -qf /usr/lib/systemd/systemd-remount-fs systemd-208-9.fc20.i686 How reproducible: Always, already happen on F19 afair Steps to Reproduce: 1. Upgrade F19 to F20 on a dedicated one-and-only partion (means no extra partion for /boot 2. Boot with default kernel options from grub.cfg, means "ro" Actual results: System will not proper start, some processes claiming about read-only / media systemd-remount-fs fails Expected results: systemd-remount-fs fails Additional info: Login on "ro" boot is possible as root, manual execution shows then: /usr/lib/systemd/systemd-remount-fs .... Cannot change data mode on remount Workaround: remove data=writeback from /etc/fstab Problematic entry: LABEL="Fedora20" / ext4 discard,data=writeback,noatime,nodiratime,defaults 1 1 Working entry: LABEL="Fedora20" / ext4 discard,noatime,nodiratime,defaults 1 1 Any hint available what causes this problem?
Well, we just invoke /bin/mount here, and if that fails, we fail too. Reassigning to util-linux hence. Not sure we can do much about this though. The kernel doesn't allow changing data=writeback during runtime, that's why it refuses this. Karel, anything we can do about this? Would "-s" help here? Should we use that switch from systemd-remount-fs?
(In reply to Lennart Poettering from comment #1) > Karel, anything we can do about this? Would "-s" help here? Should we use > that switch from systemd-remount-fs? No, the option -s is userspace only (and only mount.nfs cares about it). We don't have a way how say "ignore wrong options" to filesystem kernel drivers. The question is why the filesystem is mounted without data=writeback first time although the option is in fstab. Peter, maybe you can use: # tune2fs <device> -o journal_data_writeback and then remove data=writeback from fstab.
Lennart, something like "-s" will not resolve the problem -- it will hide the problem only. The problem is that user wants the filesystem with data writeback, but it's mounted without this feature and it's impossible to change it later by remount.
Hi, yes, tune2fs and removing data=writeback option from fstab should work as expected, however this really is just a workaround. As Karel mentioned the real problem is that the file system got mounted without data=writeback in the first place. "-s" option will not work here as ext4 does not support sloppy option. -Lukas
rootflags=discard,data=writeback,noatime,nodiratime on the kernel command line might also work for "rw"
Re-assingning back to systemd. Not sure if the problem is already fixed, but I'm sure it's something we cannot fix by mount(8) ;-)
Created attachment 972056 [details] kickstart file illustrating data=writeback vs. failure to remount / read-write
Created attachment 972057 [details] output of "journalctl --system" after failure to remount / rw during first boot after kickstart install
Created attachment 972058 [details] output of dmesg after failure to remount / as rw post kickstart install
Bumping version to F21, as I'm running into (what I strongly believe to be) the same issue. I'm trying to do a kickstart install for a SSD based machine, hence the need for data=writeback. If I add --fsoptions="defaults,data=writeback" to my "logvol /" statement in kickstart, it fails to remount / as read-write until I manually run "tune2fs -o journal_data_writeback /dev/mapper/vg.0-root" Adding "tune2fs" in %post is a workaround, and the system comes up fine, and is able to remount / as read-write (but I feel I shouldn't have to do that). I've attached the simplest kickstart file I could come up with to reproduce the problem I'm seeing, alogn with "journalctl --system" and "dmesg" output. Hopefully this helps figure out where this bug really belongs. In the unlikely event this is a different issue to what's been originally reported, I'd be happy to open a separate one (I would have opened it against anaconda, but then I found this one and decided it *must* be the same problem :) Thanks!
Ok, so after some talking to people it seems to be a dracut problem. The thing is that the file system is initially mounted 'ro' by dracut, however it seems that for some reason it was not mounted with the options specified /etc/fstab. Of course the file system is not yet, mounted but I was told fstab should be part of initramfs and it should be the same as the system /etc/fstab. If the /etc/fstab was changed, then initramfs needs to be regenerated though (not sure that was the case here ?) Anyway assigning to dracut to move things forward. Thanks! -Lukas
This message is a reminder that Fedora 21 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 21. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '21'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 21 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
(In reply to Lukáš Czerner from comment #11) > Ok, > > so after some talking to people it seems to be a dracut problem. The thing > is that the file system is initially mounted 'ro' by dracut, however it > seems that for some reason it was not mounted with the options specified > /etc/fstab. Of course the file system is not yet, mounted but I was told > fstab should be part of initramfs and it should be the same as the system > /etc/fstab. > > If the /etc/fstab was changed, then initramfs needs to be regenerated though > (not sure that was the case here ?) > > Anyway assigning to dracut to move things forward. > > Thanks! > -Lukas Well, I'll reassign this to component kernel, because after initrd-root-fs.target is executed and root is mounted readonly (as specified on the kernel cmdline), initrd-fs.target wants initrd-parse-etc.service, which does a daemon-reload and thus regenerates the mount units according to the fstab on the real root. fstab of the real root is _not_ copied in the initramfs to be more flexible and does not need to. If the kernel command line options (in this case ro equivalent to rootflags=ro) clashes with a remount to the fstab options it is not the job of dracut or systemd to figure out, that "data=writeback" should imply rw.
Ah righ, I knew there was something I wanted to talk about at the systemd conference :). Anyway what is you suggestion kernel should do about this ? We've got a file system which is already mounted and the user space requested remount with the options that's it's not supported for remount. Whether the file system is mounted read only, or read write does not matter at all so data=writeback" does not need to imply rw. The crux of the problem is that the file system was mounted with different options than specified by user and this option can't be changed on the fly. The good news is that it's at least read-only otherwise we might have more problems. -Lukas
(In reply to Lukáš Czerner from comment #14) > Ah righ, I knew there was something I wanted to talk about at the systemd > conference :). Anyway what is you suggestion kernel should do about this ? > > We've got a file system which is already mounted and the user space > requested remount with the options that's it's not supported for remount. > > Whether the file system is mounted read only, or read write does not matter > at all so data=writeback" does not need to imply rw. > > The crux of the problem is that the file system was mounted with different > options than specified by user and this option can't be changed on the fly. > The good news is that it's at least read-only otherwise we might have more > problems. > > -Lukas So, do you suggest to umount first and then remount? Might be a solution, but also slows down the boot process.
From my point of view the best solution would be to mount it with the right options in the first place, however if you have a good reason not to do that, then I suppose the umount/mount is the next best thing. Also note that this is not only ext4 problem, other file system have options that can not be changed online as well. Most notably xfs where the problem is much bigger, because they would currently silently drop the options (intentionally) they can't change on remount (vast majority of the mount options btw) leaving the user believing that they are running with a different options than they really are. This should be fixed though. Moreover potentially there might be options where specifying/unspecifying them might alter the file system in some way which will make this much worse if the fs was not mounted ro initially. An example of such option might be init_itable/noinit_itable where not specifying noinit_itable will run the inode table initialization which might have consequences for users that do not want that, but it's still mounted with the defaults. But that would be only if the fs was mounted rw. -Lukas
Exactly. IMHO remount has been introduced to have way how to toggle between ro and rw. That's all, everything else (especially FS specific) supported by remount is just an extra bonus. It would be nice to support MS_FORCE|MS_REMOUNT by mount(2) syscall and implement it as an atomic in-kernel umount+mount, then it would be possible to remount from usersapce with arbitrary mount options.
Agreed, that would be nice feature to have, though it still wouldn't completely fix this problem. -Lukas
(In reply to Lukáš Czerner from comment #16) > From my point of view the best solution would be to mount it with the right > options in the first place, however if you have a good reason not to do > that, then I suppose the umount/mount is the next best thing. It's always a problem to keep /etc/fstab, potential copies in the initramfs and the kernel command line (rootflags=) in sync. So we mount with rootflags and ro/rw and remount with the /etc/fstab of the real root.
Fedora 21 changed to end-of-life (EOL) status on 2015-12-01. Fedora 21 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.