Bug 722987

Sorry about that. Changing to the intended Component (deltarpm).

Made delta ISOs for the test images in http://dl.fedoraproject.org/pub/alt/stage/ for 20110714/-> 20110721/,  20110721/->20110721-1/, and 20110721-1/->20110721-3/. The size for 20110714/-> 20110721/ was approximately the same size as the corresponding netinst, as expected, but the sizes for the later jumps was much smaller. Any ideas why this would be the case? (I'm saving the deltas and the 20110714/ ISOs so these images can all be rebuilt to test when needed.)

Compression between successive pairs of netinsts (% of full size):

Fedora-16-test-20110714_20110721-i386-netinst.diso: 87.9%
Fedora-16-test-20110714_20110721-x86_64-netinst.diso: 89.5%

Fedora-16-test-20110721-0_1-i386-netinst.diso: 26.3%
Fedora-16-test-20110721-0_1-x86_64-netinst.diso: 9.5%

Fedora-16-test-20110721-1_3-i386-netinst.diso: 26.0%
Fedora-16-test-20110721-1_3-x86_64-netinst.diso: 9.8%

Looking at the RPMs in each of the i386 ISOs (the ones containing RPMs, not the netinsts), there are RPM changes between  20110714/ and 20110721/, but none between 20110721/, 20110721-1/, and 20110721-3/. Maybe a change in at least one RPM is associated with some change in the corresponding netinst which prevents delta compression from working well?

(In reply to comment #4)
> Looking at the RPMs in each of the i386 ISOs (the ones containing RPMs, not the
> netinsts), there are RPM changes between  20110714/ and 20110721/, but none
> between 20110721/, 20110721-1/, and 20110721-3/. Maybe a change in at least one
> RPM is associated with some change in the corresponding netinst which prevents
> delta compression from working well?

The initial ramdisk in the images does change a fair amount too, since it's generated at compose-time (and not packaged).  Perhaps this accounts for differences you are seeing, even when no packages are changed.

This issue appears to be with delta compression generally, not just with that done by makedeltaiso. Using xdelta on RHEL6, I get the following ratios for the netinsts which are similar to those in comment 3:

20110714->20110721 i386: 87.8%
20110714->20110721 x86_64: 89.1%

20110721->20110721-1 i386: 25.6%
20110721->20110721-1 x86_64: 9.1%

20110721-1->20110721-3 i386: 25.5%
20110721-1->20110721-3 x86_64: 9.1%

I also checked how well rsync did on the same images and never got a speedup of more than 1.00, so there are basically no matching blocks between different netinsts, even closely related ones. So lack of matching blocks is not the issue, since they didn't exist before either.

I'm going to change the Component to "distribution" and hope that someone who understands install images has an idea which Component to use (or if this is a bug at all, though it would be really unfortunate if install images couldn't be made so delta compression worked as well as it used to).

I'd suspect squashfs. In any case, CC'ing Will who has worked in this area.

The netinst images contain, basically:
- kernel & initrd (~25MB)
- anaconda runtime image (~100-125MB)
- [x86 only] EFI boot images (~25MB)
- assorted metadata, config files, READMEs, GPG keys, etc. (negligible)

In F14 and earlier, the runtime image (install.img) was a squashfs image. I'm guessing these must delta fairly well (although I don't see any actual stats here).

In F15, the runtime image contents were placed in the initrd, which is an xz-compressed cpio archive. And I guess that doesn't delta well.

In current rawhide, the initrd contains a LiveOS-style image (ext4 filesystem image inside xz-compressed squashfs). I'm further guessing that squashfs-with-one-big-file doesn't delta as well as a normal squashfs'd filesystem. 

So.. I don't think it's likely that rsync/deltaiso is going to be very good at generating small deltas for files compressed in this way.

We're working to get anaconda to run from a simple squashfs image as it did before, but this probably won't be finished for F16.

You can see the delta ISO size for successive TC/RC netinsts for F14 and F15 development as a percentage of full ISO size at

http://lists.fedoraproject.org/pipermail/test/2011-May/100449.html

Chances are the numbers using xdelta would be similar. Anyway, good to know there's a chance the deltas will become smaller again eventually. Thanks for the info.

FYI, delta compression of non-RPM content is doing much better in 16 so far (ignoring the one from 15 to 16-Alpha.TC1). Here are the latest numbers to compare with those from the previous link.

Fedora-15_16-Alpha.TC1-x86_64-netinst.diso: 265652346 (98.6%)
Fedora-16-Alpha.TC1_RC1-x86_64-netinst.diso: 161720186 (60.0%)
Fedora-16-Alpha.RC1_RC2-x86_64-netinst.diso: 125467358 (46.7%)
Fedora-16-Alpha.RC2_RC3-x86_64-netinst.diso: 54251914 (20.3%)

I have a complete list of all the netinst->netinst disos (both i386 and x86_64) from 14-Alpha.TC1 onwards at http://robatino.fedorapeople.org/diso_netinst.txt which I will keep updated.

This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

I doubt this issue is directly due to squashfs-tools. There might be something with supporting xz compression indirectly causing an issue.

I'm not even sure what the latest status of this is. If you look at the latest entries in the link in comment 11, they are much smaller, but that might just be that the changes were small - I don't know if it's a permanent improvement.

What is the status of this? The delta compression was better during the 16 cycle, but seems worse during 17 (see the updated list at http://robatino.fedorapeople.org/diso_netinst.txt ).

I doubt this is directly a squashfs issue. Getting the full iso sizes down is more important (IMO) than getting the delta sizes down. It might be the way the deltas are done for the ISOs could be adjusted to get better compression.

(In reply to comment #16)
> I doubt this is directly a squashfs issue. Getting the full iso sizes down
> is more important (IMO) than getting the delta sizes down.

Maybe, but there isn't necessarily a tradeoff. xz compression was introduced in F12, the delta compression got worse in F15, and AFAIK there was no significant reduction in ISO size at that time.

> It might be the
> way the deltas are done for the ISOs could be adjusted to get better
> compression.

Possibly. wwoods, can you provide an update to comment 8 regarding the current status of anaconda?

Squashfs did not have upstream support for lzma/xz until the F15 development time frame. There was a significant (around 10%) improvement in compression for F15 live images, though a good chunk of that was used to balance out increased bloat. At least one image would have needed to cut out packages to make size without this feature.
See: http://fedoraproject.org/wiki/Features/LZMA_for_Live_Images

OK, but this bug is specifically about install images. Although it does raise the question of delta compression for live images. It would be nice if something like rsync/zsync was useful on live images, so testers could download them faster. I know that currently that's not true. kparal had a blog post on the subject

https://kparal.wordpress.com/2009/09/01/zsync-transfer-large-files-efficiently/

where he compares Ubuntu and Fedora in this respect. Apparently Ubuntu image downloads do speed up well with rsync/zsync. I don't know if Fedora's lives are superior in some other way and delta compression is an unavoidable tradeoff of that.

Is this still an issue? I really doubt the issue is squashfs-tools itself. It may be how it is being used in the process of creating images, but then the bug we be in whatever is doing that.

I don't know if this still happens, I haven't made delta ISOs since the install DVDs went away. Feel free to close this if you want.

OK. I've closed this.