Bug 465725

The livecd can't use a unionfs unless the kernel includes one

*** Bug 460854 has been marked as a duplicate of this bug. ***

*** Bug 500288 has been marked as a duplicate of this bug. ***

*** Bug 524598 has been marked as a duplicate of this bug. ***

In my situation (live image on CompactFlash for embedded PC/104 systems), I don't really need or want the persistence, just a writable root fs.  The current live images generated by livecd-tools is not adequate for this as the system will become horribly unstable once the LVM snapshot overlay has become full.  

Typical symptoms include:
  * the inability to log in via SSH despite the port listening
  * if local console access is provisioned, the inability to run most commands due to random segfaults, i/o errors and "command not found" errors

My workaround was to modify the live initrd to mount /var and /tmp on tmpfs.  Certainly these too can become full, but at least the OS remains stable, accessible and manageable.

This was accomplished with the following patches.  First up, sbin/real-init from the live initrd:

--- /tmp/tmp1/syslinux/tmp/sbin/real-init	2009-10-07 08:16:31.000000000 -0400
+++ real-init	2009-10-07 08:20:28.000000000 -0400
@@ -451,6 +451,17 @@
     ln -s /dev/mapper/live-rw /dev/root
 
     mount -n  /dev/mapper/live-rw /sysroot
+
+    # Create tmpfs file systems for /tmp and /var so that the LVM snapshot COW
+    # doesn't exhaust all free memory over longer uptimes.
+    echo "Creating tmpfs /tmp and /var ..."
+    mkdir /sysroot/tmp/.{tmp,var}_mnt
+    mount -t tmpfs tmpfs  /sysroot/tmp/.tmp_mnt
+    mount -t tmpfs tmpfs  /sysroot/tmp/.var_mnt
+    cp -a /sysroot/var/* /sysroot/tmp/.var_mnt/
+    mount --move /sysroot/tmp/.var_mnt/ /sysroot/var
+    mount --move /sysroot/tmp/.tmp_mnt/ /sysroot/tmp
+
     # here you can modify the rw ext3 fs for testing if you don't want to
     # respin the entire rootfs (which takes ages). Example
     #


Also needed, to make things like mount and df to show things correctly, is a patch to the rc.sysinit from the live root fs:
--- rc.sysinit.orig	2009-10-07 08:22:21.000000000 -0400
+++ rc.sysinit	2009-10-07 08:22:39.000000000 -0400
@@ -663,6 +663,8 @@
 	mount -f /sys >/dev/null 2>&1
 	mount -f /dev/pts >/dev/null 2>&1
 	mount -f /proc/bus/usb >/dev/null 2>&1
+        mount -f -t tmpfs tmpfs /tmp >/dev/null 2>&1
+        mount -f -t tmpfs tmpfs /var >/dev/null 2>&1
 fi
 
 # Mount all other filesystems (except for NFS and /proc, which is already

I made an F12 "livecd" USB install, and this same problem still occurs.  It really limits the usefulness of a USB install, as it would be unwise to run an unpatched Fedora, yet patching the install destroys it!

This silent-but-deadly failure mode is going to eat someone's thesis one day.

I asked on #fedora, and someone told me the overlay stuff is done with LVM.  Is that right?  If so, is there an LVM command that shows how much of the overlay has been used?  I know next to nothing about LVM, but I think there's a "lvs" command that gives that info.  Can we have a panel app which displays this number, kind of like a fuel gauge?  Can we pop up a warning each time the usage goes past a 10% increment?

I'm curious why the Fedora livecd doesn't use unionfs.  From Wikipedia, "As of January 2007, Unionfs 2.0 has been included into Andrew Morton's Linux -mm tree, thus slated for eventual inclusion in the Linux kernel mainline source tree. Version 2.x is the smallest implementation of unioning for Linux, it is heavily tested and inspected by many kernel developers, and it is more efficient".  Tell me this would be worse than the ticking time-bomb people have now?

  http://www.filesystems.org/project-unionfs.html

Also:

  https://fedorahosted.org/liveusb-creator/ticket/28

(In reply to comment #6)
> I asked on #fedora, and someone told me the overlay stuff is done with LVM.
> Is that right?

yes

> If so, is there an LVM command that shows how much of the overlay
> has been used?

dmsetup status live-rw

(In reply to comment #5)
> In my situation (live image on CompactFlash for embedded PC/104 systems), I
> don't really need or want the persistence, just a writable root fs.

You can also try to reuse a part of stateless support in rc.sysinit:
build livecd with a customized kickstart where you set TEMPORARY_STATE=yes in
 /etc/sysconfig/readonly-root
This bind-mounts files and directories listed in /etc/rwtab to a tmpfs

This won't help with yum updates, but instead of doing that, you could rebuild livecd image against Fedora updates repository and then replace LiveOS folder on the USB.

But ideally, yes, when unionfs enters official upstream kernel, livecd will use it for persistence.

(In reply to comment #8)
> You can also try to reuse a part of stateless support in rc.sysinit:
> build livecd with a customized kickstart where you set TEMPORARY_STATE=yes in
>  /etc/sysconfig/readonly-root
> This bind-mounts files and directories listed in /etc/rwtab to a tmpfs

Oh how cool!  I sure wish I'd know about that at the time since it looks so much simpler.  Thank you very much for pointing this out as I may be able to leverage this for an upcoming spin of our AOS, if we happen to make that prior to having unionfs support.

As an alternative to unionfs, could btrfs snapshots be used?

For reference: http://article.gmane.org/gmane.linux.kernel/974518

(In reply to comment #7)
> > If so, is there an LVM command that shows how much of the overlay
> > has been used?
> 
> dmsetup status live-rw    

I prefer:
# dmsetup status live-rw|awk '{split($4,x,"/"); print x[1]*100/x[2]}

the output is the used space in % of the total space units...

Sorry by the poor spelling and concordance...

overlayfs (http://marc.info/?l=linux-kernel&m=134461525325660&w=2 ) may be a better option than unionfs.  The Ubuntu folks are including overlayfs in their kernels.  Would be nice to see it in the Fedora kernels so it could be tested as a Live persistence solution.

Just an update here: overlayfs has been merged into kernel 3.18 [1].

[1] http://kernelnewbies.org/Linux_3.18#head-f514a511bf32b818dbde50c24a51fb095e81cc8e

Yeah, we totally forgot this bug was even open.  Overlayfs is enabled in Fedora kernels now.

Yay!

I'm still hoping that we are going to see union mounts as well (bug 519827)