Bug 982007 - system does not boot with separate /var partition
system does not boot with separate /var partition
Product: Fedora
Classification: Fedora
Component: systemd (Show other bugs)
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: systemd-maint
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2013-07-07 13:22 EDT by Frank Ansari
Modified: 2014-06-19 19:46 EDT (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2014-06-19 19:46:34 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Frank Ansari 2013-07-07 13:22:55 EDT
Description of problem:
When I put /var on a separate partition the system hangs at boot.

Version-Release number of selected component (if applicable):

How reproducible:
Create two volume groups: vgSystem and vgData.

/etc/fstab like this:

/dev/mapper/vgSystem-root /      ext4  defaults  1 1
/dev/mapper/vgData-var    /var   ext4  defaults  1 1
/dev/mapper/vgData-home   /home  ext4  defaults  1 1
/dev/mapper/vgSystem-swap swap   swap  defaults  0 0
UUID=a9c10b79-6d97-4bfb-9ffb-621cc05cc599 /boot ext4  defaults 1 2
UUID=925B-7095          /boot/efi  vfat umask=0077,shortname=winnt 0 0

In this scenario vgSystem partitions are on an SSD HDD and vgData partitions on a SATA HDD.

Steps to Reproduce:
1. Boot the PC

Actual results:
System hangs at message:
Binary verified by the vendor certificate

Expected results:
Normal boot ending up on Gnome Desktop

Additional info:
This even happens when only /var/log is put to the separate partition

Also I noticed that after several resets of the PC it sometimes works.

On the other hand: if /var is on the same partition with / the boot works most of the times but some times hangs (so in this case it is just the other way round).
Comment 1 Harald Hoyer 2013-07-10 09:51:13 EDT
What is your kernel command line?

Does it help, if you add "rd.lvm.vg=vgData" to the kernel command line?

Without the workaround, to debug the issue:
Can you wait 5 minutes and get a login shell?
Comment 2 Frank Ansari 2013-07-14 07:17:57 EDT
Here you can see what is in my grub.cfg:

	linuxefi /vmlinuz-3.9.9-301.fc19.x86_64 root=/dev/mapper/vgSystem-root ro rd.md=0 rd.dm=0 rd.lvm.lv=vgSystem/swap  rd.lvm.lv=vgSystem/root rd.lvm.vg=vgData rd.luks=0 vconsole.font=latarcyrheb-sun16 vconsole.keymap=de rhgb quiet LANG=en_US.utf8
	initrdefi /initramfs-3.9.9-301.fc19.x86_64.img

Waiting for 5 min does not bring anything up.

Putting the whole /var partition to another disk makes booting very unlikely. I have to reset my machine up to perhaps ten times in order to make it boot.

As you can see here I have added your suggestion for at test. But unfortunately it is not better than before.

If I put only /var/log to another disk booting becomes more likely. Perhaps ever second attempt fails.
Comment 3 Lennart Poettering 2013-09-12 23:12:14 EDT
Where exactly does it hang? YOu should see an animation in the textboot progress that tells you what is being waited for.
Comment 4 Frank Ansari 2013-09-14 12:34:06 EDT
No unfortuantely it hangs at this message:
"Binary verified by the vendor certificate". At this point there is no GUI.

There is always some probability that it might boot or hang.

What I did not test so far is to disable secure boot with /var on a separate disk. Maybe I will test this. After UEFI update from 1101 to 1204 (ASUS board P8B75-M LE) secure boot is not working anymore. I have opened bug 1008084 for this and also wrote a complaint to ASUS.

I know this is another story - but this gave me idea to test with secure boot disabled. But I still have to do this.

But to be honest I doubt that the issue with the /var partiton has something to do with secure boot because as far as I understood nothing what is needed for secure boot is in /var.
Comment 5 Frank Ansari 2013-12-05 09:58:19 EST
Fixed. I changed to Arch Linux. There it works as expected.
Comment 6 jason 2013-12-15 16:37:37 EST
systemd doesn't seem to be mounting additional mount points early enough.  I am effected by this as well, but my system usually boots although some processes do not start, or are started before /var has been mounted and therefore are writing to the masked directory.  

/dev/mapper/VolGrp01-root /                       ext4    defaults,_netdev 1 1
UUID=3041eab9-f5b9-4086-bd79-c9db4538481f /boot                   ext4    defaults,_netdev 1 2
/dev/mapper/VolGrp01-var /var                    ext4    defaults,_netdev 1 2
macross:/storage /storage    nfs4       rw   0 0
macross:/home /home    nfs4       rw   0 0

And the boot.log which I had to do a bind-mount in order to retrieve as it was written to the VolGrp01-root partition and not the VolGrp01-var.

[root@htpc log]# cat /mnt/tmp/var/log/boot.log 
[  OK  ] Started Show Plymouth Boot Screen.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Basic System.
[   14.198817] dracut-initqueue[188]: iscsistart: version
[   14.753990] dracut-initqueue[188]: iscsistart: Connection1:0 to [target: iqn.2008-06.localdomain.macross:htpc_fedora.root, portal:,3260] through [iface: default] is operational now
[  OK  ] Started dracut initqueue hook.
         Mounting /sysroot...
[  OK  ] Mounted /sysroot.
[  OK  ] Reached target Initrd Root File System.
         Starting Reload Configuration from the Real Root...
[  OK  ] Started Reload Configuration from the Real Root.
[  OK  ] Reached target Initrd File Systems.
[  OK  ] Reached target Initrd Default Target.

Welcome to Fedora 19 (Schrödinger’s Cat)!

[  OK  ] Stopped Switch Root.
[  OK  ] Stopped target Switch Root.
[  OK  ] Stopped target Initrd File Systems.
[  OK  ] Stopped target Initrd Root File System.
         Expecting device dev-disk-by\x2duuid-3041eab9\x2df5b9\x2d4086\x2dbd79\x2dc9db4538481f.device...
[  OK  ] Listening on Syslog Socket.
         Starting Collect Read-Ahead Data...
         Starting Replay Read-Ahead Data...
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
         Starting Create static device nodes in /dev...
[  OK  ] Reached target Paths.
         Mounting Debug File System...
[  OK  ] Set up automount Arbitrary Executable File Formats File System Automount Point.
         Mounting POSIX Message Queue File System...
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Listening on udev Control Socket.
         Starting udev Coldplug all Devices...
         Mounting Huge Pages File System...
         Expecting device dev-mapper-VolGrp01\x2dvar.device...
[  OK  ] Reached target Encrypted Volumes.
[  OK  ] Listening on LVM2 metadata daemon socket.
[  OK  ] Listening on Device-mapper event daemon FIFOs.
         Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[  OK  ] Reached target Swap.
         Mounting Temporary Directory...
[  OK  ] Stopped Trigger Flushing of Journal to Persistent Storage.
         Stopping Journal Service...
[  OK  ] Stopped Journal Service.
         Starting Journal Service...
[  OK  ] Started Journal Service.
[  OK  ] Started Collect Read-Ahead Data.
[  OK  ] Started Replay Read-Ahead Data.
         Starting File System Check on Root Device...
         Starting Apply Kernel Variables...
         Mounting Configuration File System...
[  OK  ] Mounted Temporary Directory.
[  OK  ] Mounted Debug File System.
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Started File System Check on Root Device.
         Starting Remount Root and Kernel File Systems...
         Starting LVM2 metadata daemon...
[  OK  ] Mounted Configuration File System.
[   16.308897] systemd-fsck[468]: /dev/mapper/VolGrp01-root: clean, 70046/640848 files, 449229/2560000 blocks
[  OK  ] Started udev Coldplug all Devices.
[  OK  ] Started Create static device nodes in /dev.
         Starting udev Kernel Device Manager...
[  OK  ] Started LVM2 metadata daemon.
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Started udev Kernel Device Manager.
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Configure read-only root support...
         Starting Import network configuration from initramfs...
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Started Import network configuration from initramfs.
[  OK  ] Started Configure read-only root support.
[  OK  ] Started Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
[  OK  ] Reached target Local File Systems.
         Starting Tell Plymouth To Write Out Runtime Data...
         Starting Security Auditing Service...
         Starting Recreate Volatile Files and Directories...
[  OK  ] Started Recreate Volatile Files and Directories.
[FAILED] Failed to start Security Auditing Service.
See 'systemctl status auditd.service' for details.
[  OK  ] Started Tell Plymouth To Write Out Runtime Data.
[  OK  ] Reached target System Initialization.
[  OK  ] Reached target Timers.
[  OK  ] Reached target Basic System.
         Starting irqbalance daemon...
[  OK  ] Started irqbalance daemon.
         Starting Self Monitoring and Reporting Technology (SMART) Daemon...
[  OK  ] Started Self Monitoring and Reporting Technology (SMART) Daemon.
         Starting Machine Check Exception Logging Daemon...
         Starting System Logging Service...
[  OK  ] Started System Logging Service.
         Starting Hardware RNG Entropy Gatherer Daemon...
[  OK  ] Started Hardware RNG Entropy Gatherer Daemon.
[ INFO ] Network Manager is not active.
[DEPEND] Dependency failed for Network Manager Wait Online.
[  OK  ] Reached target Network is Online.
[  OK  ] Reached target Network.
         Starting Sendmail Mail Transport Agent...
         Mounting /home...
         Mounting /storage...
         Starting OpenSSH server daemon...
[  OK  ] Listening on Open-iSCSI iscsiuio Socket.
[  OK  ] Listening on Open-iSCSI iscsid Socket.
[  OK  ] Started Machine Check Exception Logging Daemon.
[  OK  ] Started OpenSSH server daemon.
[  OK  ] Mounted /storage.
[FAILED] Failed to start Sendmail Mail Transport Agent.
See 'systemctl status sendmail.service' for details.
[DEPEND] Dependency failed for Sendmail Mail Transport Client.
[  OK  ] Mounted /home.
[  OK  ] Found device htpc_fedora.
         Starting File System Check on /dev/disk/by-uuid/3041eab9-f5b9-4086-bd79-c9db4538481f...
[   17.099506] systemd-fsck[594]: /dev/sde1: clean, 315/128016 files, 125372/512000 blocks
[  OK  ] Started File System Check on /dev/disk/by-uuid/3041eab9-f5b9-4086-bd79-c9db4538481f.
         Mounting /boot...
[  OK  ] Mounted /boot.
[  OK  ] Found device /dev/mapper/VolGrp01-var.
         Starting File System Check on /dev/mapper/VolGrp01-var...
[   17.255581] systemd-fsck[611]: /dev/mapper/VolGrp01-var: clean, 2253/320000 files, 124691/1280000 blocks
[  OK  ] Started File System Check on /dev/mapper/VolGrp01-var.
         Mounting /var...
[  OK  ] Mounted /var.
[  OK  ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
[  OK  ] Listening on RPCbind Server Activation Socket.
         Starting RPC bind service...
[  OK  ] Listening on PC/SC Smart Card Daemon Activation Socket.
[  OK  ] Reached target Remote File Systems.
         Starting Trigger Flushing of Journal to Persistent Storage...
[  OK  ] Started RPC bind service.
[  OK  ] Started Trigger Flushing of Journal to Persistent Storage.
         Starting Permit User Sessions...
[  OK  ] Started Permit User Sessions.
         Starting Job spooling tools...
[  OK  ] Started Job spooling tools.
         Starting Command Scheduler...
[  OK  ] Started Command Scheduler.
         Starting Terminate Plymouth Boot Screen...
         Starting Wait for Plymouth Boot Screen to Quit...
Comment 7 Frank Ansari 2013-12-24 12:40:23 EST
Fedora 20 has systemd version 208. This version seems to fix this issue. I just installed Fedora 20 and it works even if the /var partition is on a different disk.
Comment 8 Lennart Poettering 2014-06-19 19:46:34 EDT
OK, closing then.

Note You need to log in before you can comment on or make changes to this bug.