Created attachment 509792 [details] The bootup hang (screen shot) Description of problem: Many people want to bind-mount /var/tmp to /tmp. This saves space, adds security (arguable, as two world writable areas are now one). This might be related to the NSA Linux hardening guide which recommended the bind-mount of /var/tmp to /tmp. Fedora 15 has problems booting-up when the bind-mount configuration is put in /etc/fstab. Once the system is up and operational in run-level-3, you are then able to succeed with the bind-mount. Put the following Lines /etc/fstab: # Bind mount /var/tmp to /tmp /tmp /var/tmp none bind 0 0 Here is an idea of the filesystem layout: logvol swap logvol / logvol /tmp logvol /var logvol /var/log logvol /var/log/audit logvol /data As you can see, /var & /tmp are provided by logical volume. he problem is NOT reproducible when /var is part of the rootfs (/). This implies that /var and/or /var/tmp might be mounted BEFORE /tmp. For this to work, two this should happen: 1. bind mounts occure last in the mount sequence. or 2. /tmp mount BEFORE /var A third possibility is to add some kind of smart mount sequencing grammar to the boot process, to tell smartd exactly how to mount what, dependency graphs, yadda... yadda.... which might exist in smartd already (I din't know). By this point you are probably wanting to see detailed problem statement about the error msg. I have attached a screen shot of the actual error MSG from systemd. Will also attach a screen of the `systemctl status mount.tmp` command output. When I go into emergency mode and attempt to bring these up myself, but encounter challenges. Cannot mount /tmp because /var/tmp is not available. So it would seem /var is not mounted BEFORE /tmp, so my speculation might have wrong? Version-Release number of selected component (if applicable): How reproducible: The right condition have to be satisfied. / /var/ /var/tmp --> /tmp /tmp When attempting to isolate all world-writable areas into their own mount points (to enable the use of secure mount options) AND bind-mounting /var/tmp to /tmp, the system is unable to boot. Steps to Reproduce: 1. Use the provided kickstart file (pay attention to the %post section): http://dl.dropbox.com/u/41369/fedora/15/var-tmp_bug.ks [ks also attached to this BUG] 2. boot FC15 normally 3. Will not reach the supposed notion init level 2 (in the systemd world, do run-levels even make sense?) Actual results: system hangs because mount dependencies are circular? Expected results: As with previous Fedora and RHEL releases I expect bind-mount's in /var/fstab to work, be processed last after all the local logical volumes, dedicated disks, SAN mounts or whatever other netfs mounts. Additional info: Please let me know if bind-mounts of /var/tmp need a special systemd unit thingy? Thanks.
Created attachment 509793 [details] a screen capture of the `systemctl status mount.tmp` output
Created attachment 509794 [details] a kickstart file that reproduces the problem statement An interesting change to make the problem go away is to comment the logvol line for "/var", to make /var part of the rootfs. Otherwise the %post script section will add the lines to /etc/fstab that bind mount things.
Created attachment 510136 [details] dmesg output This has "systemd.log_level=debug systemd.log_target=kmsg" appended to the boot kernel line.
Created attachment 510137 [details] systemctl show tmp.mount
Created attachment 510138 [details] systemctl show var.mount
Created attachment 510139 [details] systemctl show var-tmp.mount
During the bootup phase, I was able to enter into "emergency mode" I was able to make it work by manually mounting filesystem in the correct sequence. When I attempt to mount /tmp, there is an error that /var/tmp does not exist. This implies that mounting /tmp, and /var/tmp (the bind-mount) occure simultaneously, which is not possible if /tmp goes *BEDORE* /var in the sequence. So I simply mounted /var, and I noticed the /var/tmp mount-point then existed, and then mounting /tmp prodeed to work correctly. touching /tmp/foo file made the file visible in /var/tmp also.
As I look at the output of systemd --test I now have an idea of the sequencing of events. I see that the unit tmp.mount is "RequiredBy var-tmp.mount", and "Before var-tmp.mount" but missing "After var.mount". So the questions is how to get "After var.mount" into tmp.mount unit, or possibly into the var-tmp.mount unit? I suppose either place would be the same. How to make the system do this automatically when /var is a mount-point that hosts a bind-mount to another location on the top-level of the rootfs (or elsewhere)?
Karle, could you please take a look at this?: From #systemd (edited): <mezcalero> masta: so: https://bugzilla.redhat.com/attachment.cgi?id=510137 looks right from the configuration <mezcalero> masta: however mounting /tmp apparently failed with error 32 <michich> mezcalero, systemd spawns "/bin/mount $MOUNTPOINT", but mount misunderstands this <michich> mezcalero, mount matches it with the wrong fstab line. It thinks the "/tmp" argument stands for the device, not the mountpoint. <michich> mezcalero, this bug must happen whenever there are two lines in fstab like this: <michich> mezcalero, 1st: /dev/something /mntpoint ... <michich> mezcalero, 2nd: /mntpoint /somewhere/else none bind ... <mezcalero> so "mount foo" first looks for a device /foo and mounts that, and only then looks fo a path /foo? <michich> mezcalero, exactly. <mezcalero> sounds wrong in mount <mezcalero> it should first look for a path, then for a device <michich> here's the code in mount where the argument is tried first as a device and only if that fails, it tries it as a directory: http://git.kernel.org/?p=utils/util-linux/util-linux.git;a=blob;f=mount/mount.c;h=3ba705f3ee292e0bf79295610536e66f99401b21;hb=HEAD#l2349
Yeah, this is stupid disadvantage. Fixed by upstream commit 067e9b4934372f72b89b2c0442a3d28290834537. Note that the next generation of mount(8) (completely based on libmount, probably util-linux 2.21) will support options --target and --source to avoid ambiguous command line interpretation ... then systemd should be updated to use exec( mount --target /mountpoint )
*** Bug 711403 has been marked as a duplicate of this bug. ***
util-linux-2.19.1-1.3.fc15 has been submitted as an update for Fedora 15. https://admin.fedoraproject.org/updates/util-linux-2.19.1-1.3.fc15
Thank Kzak & michich, solved. Updated util-linux to version 2.19.1-1.3.fc15 from KOJI. # rpm -qf /bin/mount util-linux-2.19.1-1.3.fc15.x86_64 All the bind mounts appear to be working properly with systemd at bootup. Here is a idea of the configuration lv_root / lv_var /var lv_tmp /tmp lv_tmp /var/tmp (bind mount) ...
Package util-linux-2.19.1-1.3.fc15: * should fix your issue, * was pushed to the Fedora 15 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing util-linux-2.19.1-1.3.fc15' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/util-linux-2.19.1-1.3.fc15 then log in and leave karma (feedback).
util-linux-2.19.1-1.3.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report.