Bug 248082

I haven't quite got around to fixing this.  I got distracted and had to fix that
redundant rootfs format bug first.  Since I went to the trouble of collecting
this data, which is also relevant for that bug as well, I'll go ahead and post
it here.

I performed 5 installs onto my vaio vgn-n250 laptop (core duo, sata laptop drive
(hdparm -t == 40 MB/s).  The flash used was 8.5MB/s.  cdrom used is 1.6MB/s.

3 live isos were used.  stock f7, replicated f7 (with livecd tools), and
replicated f7 with a 2.5G container instead of the default 4.0G container. 
about 2075MB of data within the filesystem.

each test was also performed after doing livecd-iso-to-usb. (thus the 6(-1) tests).

I measured the time the user waits for the install, this being 3 phases-

1) format rootfs
2) copy live image to drive
3) postinst stuff

I did not do a stock-f7 usb test, as mainly I used the stock-f7 cd test to
verify that my respun numbers were not off base.  30G rootfs was used in all tests.

respin-flash-2.5G

format   41s
copy     99s
postinst 75s

respin-flash-4.0G

format   41s 
copy    131s
postinst 75s

respin-cdrom-2.5G

format   49s
copy    262s
postinst 89s

respin-cdrom-4.0g 

format   49s
copy    290s
postinst 89s

stockf7-cdrom-4.0G

format   47s
copy    291s
postinst 89s

conclusions
-----------

copy benefit interpolated up by ~25% to factor for optimal case, i.e.
4.0G->2.1G, rather than 4.0G->2.5G

for flash case, overall

89+75=164 vs 41+131+75=247, 
83/164= "F7 took 50% longer to install than it could have"
41s saved from format, 42s saved from copy

for cdrom

253+89= 342 vs 49+290+89=428, 
86/342= "F7 took 25% longer to install than it could have"
49s saved from format, 37s saved from copy

I mentioned on the fedora-livecd-list to markmc that my solution involving
on-the-fly resize2fs-ing down to minimal isn't going to work.  Not only is there
the issue of ram usage, the fatal flaw with that theory is that the
resize2fs-minimization process will likely(??) require lots of reads hitting the
install media(e.g. cdrom), which I suspect will be atrocious.

The final (I say a bit prematurely) solution then hit me-

During livecd-creator compose time

1) after done composing the 4.0G os.img as happens now, do a resize2fs down to
minimal, truncate the file to minimal, resparsify it back to 4.0G.

This step by itself, while time consuming (make it a flag to livecd-creator if
this is an issue), is useful in that it will remove any data taken up by files
that were written during the install and then deleted for some reason (I'm
guessing there are at least a few).

2) now, still during livecd-creator compose, snapshot the image, overlay file on
disk, and run a resize2fs minimize pass.  Then (here is the magic) include that
snapshot file (which by all rights should be very small, if not, a pretty bad
bug with resize2fs) on the cdrom alongside os.img.

3) now, during anaconda liveinst time, recreate the snapshot device, thus you
have access to a minimal os.img for use in copying efficiently to drive.  If for
some reason having that overlay file on cdrom slows things down, read it into
/dev/shm(tmpfs) first.

This design does hinge on the overlay file being small, and the snapshot device
not imposing a serious penalty.  But my fairly strong hunch is that that will be
the case.

oops, forgot in step 1, after resparsifying back to 4.0G, resize2fs back to 4.0G
as well.

I have to wonder a bit if this is getting to the point of over-optimization --
the cost of doing all of the resizing + snapshotting is going to be pretty
large.  And being able to build live CD fast is going to be increasingly
important.  For cases where you really care about making this level of
optimization, we already let you specify the size of the rootfs image and so
that's going to likely be the better approach.

I disagree.  I think the time measurements show the benefits.

All the resizing + snapshotting happens at livecd creator time, which can be a
flag, so that it needs only be done for mass shipping spins.

The snapshot does get _used_ at install time, but I suspect it won't be that bad.

Anyway, I may still produce a patch and do some tests, and post.

Just setting the size at creation time, isn't an answer I think.  I think you
need the +2G space, such that livecd users with 2G of ram are allowed to utilize
it.  Actually, the issues with that are also related to issues with the
currently undefined persistence livecd wishlist feature.  I.e. do we support
persistance to an 8GB usbstick?  If so, how does that get implemented?

(though that discussion is I think for fedora-livecd-list, and not this bug)

Douglas McClendon :) posted a patch that takes care of much of this, as well as
reclaiming 5.5% wasted space on the F7-i686-LiveCD.  Please review that patch
ASAP.  If it looks like it will get applied more or less as is, I'll go forward
implementing the rest of the above, hopefully in time for the F8T1 devel freeze.

I can implement in such a way that it is easily verifiable that by default,
nothing will happen differently, either with the boot sequence, or anaconda.  If
however a boot arg is given (e.g. by f8t1 testers actively testing this
feature/bugfix), then a dm-linear layer will be inserted on top of the os.img
loop device (so it can be simultaneously used as part of the live-rw-rootfs, and
as part of the minimized-installable-rootfs), and anaconda will take advantage
of it, and the <<1MB extra minimized-fs-overlay file provided by livecd-creator
will be utilized.

As always, a little more mind-bending thought with devicemapper, and I realize
that no bootarg or futzing with the existing boot sequence will be needed.

I had been happy to discover that dm allows you to use a single device to create
multiple mountable devices.  (i.e. that rebootless theoretical workaround #5...).

But what I just realized, is that because the loop device associated with
os.img, is _already_ a component of devicemapper in the livecd scenario (i.e. a
part of the snapshotted /dev/mapper/live-rw), that it can be reused to build the
minimized image at runtime (without any extra stuff happening during early boot)