Bug 504425 - RFE to have mksquashfs use lzma compression
Summary: RFE to have mksquashfs use lzma compression
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: squashfs-tools
Version: rawhide
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Bruno Wolff III
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-06-06 21:08 UTC by Bruno Wolff III
Modified: 2010-06-11 16:52 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-06-11 16:52:04 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Bruno Wolff III 2009-06-06 21:08:49 UTC
Description of problem:
Now that lzma is in the kernel, I am hoping the mksquashfs in f12 will be able to use lzma compression. Better compression will be useful for livecd images. In some cases this will allow more material to be included. In others there will be bandwidth savings with the reduced image sizes.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Bruno Wolff III 2009-06-23 01:14:33 UTC
When I originally submitted this I was envisioning that it was a matter of just enabling a plugin for lzma instead of bzip, but it seems there was more to it than that. I did find the message I have included below, which suggests lzma enabled squashfs may still make it in to 2.6.31.


	
Re: squashfs 4.0 lzma support

by Phillip Lougher :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message
Christopher Rogers wrote:
> Since lzma is in kernel 2.6.30 now, is it going to be added into kernel
> 2.6.31 or 2.6.32 merger window?

I started working on LZMA support a couple of days ago.  I'm hoping to
get it finished for the 2.6.31 merge window, though I may miss this.
It depends on whether I hit any problems, and how soon the merge window
closes.

>
> I'm only asking cause I'm still using the older squashfs lzma 3.4 for
> the time be. At some point thats going to break and there will be no
> squashfs-lzma option.
>

Yes I know. I expect to get a lot of requests for LZMA support when that
happens, and I aim to have something finished before then.

Phillip

Comment 2 Bruno Wolff III 2009-07-15 01:47:23 UTC
As far as I can tell Phillip didn't get this into 2.6.31. I'll try to remember to look for indications that a patch exists (possibly in linux-next) from time to time.

Comment 3 Bruno Wolff III 2009-10-13 01:26:57 UTC
It looks like this isn't going to happen for 2.6.32. But F13 will probably start with 2.6.33, so there is another chance to make it into F13.

Comment 4 Bruno Wolff III 2009-11-04 13:36:15 UTC
There have been replies to the old thread at http://thread.gmane.org/gmane.comp.file-systems.squashfs.devel/79/focus=88 by Phillip in late October that suggest he is actively working on this again. He mentions trying to merge the patch in 2.6.33.

Comment 5 Bruno Wolff III 2009-12-07 19:27:24 UTC
Lougher posted the patch set this morning. It isn't in just yet, but it looks like it will really happen for 2.6.33.
http://www.acetylcholine.com/node/44548

Comment 6 Bruno Wolff III 2009-12-09 00:37:09 UTC
The message from Lougher at the link below has some information about using / building squashfs-tools to use lzma.
http://marc.info/?l=linux-embedded&m=126029628723357&w=2

Comment 7 Bruno Wolff III 2009-12-09 01:15:41 UTC
I am willing to take a stab of building the devel version of mksquashfs in rawhide if you want. I'd like to see lzma enabled mksquashfs and livecd-tools testable by feature freeze and am willing to do some of the work.

Comment 8 Bruno Wolff III 2009-12-11 19:55:01 UTC
This is now a proposed feature:
https://fedoraproject.org/wiki/Features/LZMA_for_Live_Images
I won't ask FESCO to vote until Lougher's patches are accepted. (He posted a revision yesterday to address a few comments.)

Comment 9 Bruno Wolff III 2010-02-12 15:39:54 UTC
It isn't here, but the feature page has some extra notes covering it, the patches ended up in linux-next and are targeted for 2.6.34 which F13 probably won't release with.
The version of the SDK used by the dev version squashfs-tools isn't currently included in Fedora. Writing new wrappers for the xz library is tricky. And the lzma library is based on an upstream project that is dead. (The project was mainly for the utility tools which have been superceded by xz.)

Comment 10 Bruno Wolff III 2010-02-13 19:40:43 UTC
I was checking on the status of squashfs upstream today and noticed that Lougher added support for using the xz library about a week ago. That will probably make it a lot easier to get things working.
I won't try to beat the alpha release on this (particularly since the games spin needs attention right now), but I'll see if I can get it into rawhide shortly after the branch and maybe into the F13 beta if it doesn't seem to cause problems.

Comment 11 Bruno Wolff III 2010-02-13 20:19:31 UTC
I have a scratch build that builds. I haven't tested functionallity yet, so you don't need to look at that. But if you want to review what I have changed in the spec file or other packaging related stuff, then this build would be useful for that.
https://koji.fedoraproject.org/koji/taskinfo?taskID=1984187

Comment 12 Bruno Wolff III 2010-02-15 05:52:31 UTC
I did some testing with the scratch build.

I didn't strip the CVS metadata out of the archive file and that should get done before submitting a real build.

I used mksquashfs and unsquashfs with lzma compression and got back an identical (according to diff -r) directory tree.

I created a live spin (Desktop) on a system with the test squashfs and with the test squashfs in the repo used to build the image. The image built and ran correctly. Note this is testing that mksquashfs doesn't break things with the default zlib compression. lzma won't work yet because there is no kernel support yet.

mksquashfs by default uses as many processors as you have to simultaneously work on the compression. It does this with both zlib and lzma compression. There is an option to limit the number of processors that it will use.

I tested a hacked version of livecd-creator to compare zlib and lzma compression. The output won't actually run, but this gives us a good idea of the benefit in what is probably our main use case:
Games spin image using zlib: 4390387712
Games spin image using lzma: 3984588800

Since things seem to be mostly working, I'd like to get the OK to get 4.1 in rawhide right after the branch with an eye on putting it in f13 after the alpha. I think it will get enough testing before beta to be a reasonable risk to take.

In rawhide I would be looking to make some proposal for changes to livecd-creator with a target of F14. The kernel patches for lzma will probably land during the merge window, which isn't too far away.

Comment 13 Bruno Wolff III 2010-02-15 21:25:23 UTC
Kyle, can you give me a yeah or nay on putting the prerelease squashfs-tools 4.1 into rawhide right after F13 is branched?

Comment 14 Bruno Wolff III 2010-02-17 07:04:17 UTC
I made a new scratch build against f14 with a couple of cleanups relative to my last scratch build. This is what I am planning on importing to dist-f14.
http://koji.fedoraproject.org/koji/taskinfo?taskID=1992766

Comment 15 Bruno Wolff III 2010-03-03 21:35:49 UTC
On the kernel side, Lougher made a pull request which has been bounced by Linus as needing cleanup.

Comment 16 Bruno Wolff III 2010-03-06 20:47:28 UTC
Linus just pulled most of this. I am not sure that lzma is completely supported by this pull as Lougher's comment in the updated pull request suggested it was clean up and that more would be following later. But Linus also pulled some of the earlier stuff as well, so I am not sure. I'll update after I figure out more.
It does make it seem very likely to be in 2.6.34 now.

Comment 17 Bruno Wolff III 2010-03-07 14:11:47 UTC
I tested building a custom kernel which I believe included the lzma/squashfs related stuff that Linus pulled yesterday and was unable to mount an lzma squashfs image, but was able to mount a zlib squashfs image. So it looks like we do need to wait a bit yet.

Comment 18 Bruno Wolff III 2010-04-29 15:56:07 UTC
We aren't going to get LZMA Squashfs in 2.6.34. Lougher has recently made some bug fixes for some of the infrastructure changes that had been accepted earlier. He also made the comment "It's not something that's going to be fixed
in a couple of evenings :-(". So even 2.6.35 is iffy.

Comment 19 Phillip Lougher 2010-05-05 02:08:34 UTC
It's something I'm going to try and get finished for 2.6.35, whether I will is another matter.  Luckily CELF has offered payment for the LZMA stuff and so I may be able to work full time on it.

Comment 20 Bruno Wolff III 2010-05-05 07:38:42 UTC
Thanks. Even 2.6.36 would probably be good enough to get this in F14.
I also appreciate you adding support for the xz library to mksquashfs in addition to the lzma SDK. That made geting it to work in Fedora a lot easier.

Comment 21 Bruno Wolff III 2010-06-01 07:49:54 UTC
Status update: LZMA didn't make the merge window (though squashfs did gain xattr support which might allow for some changes in live images) for 2.6.35.

Comment 22 Bruno Wolff III 2010-06-09 04:03:22 UTC
Is it OK if I build a 4.1 prerelease version of mksquashfs for rawhide?
Even if we don't get lzma, we'll probably want the xattr support (since that's been accepted for 2.6.35). The default will still be zlib, so there shouldn't be a big impact (where we'd need to revert) if we don't get lzma for 2.6.36.

Comment 23 Bruno Wolff III 2010-06-09 04:37:40 UTC
I have made new scratch builds.
f14: http://koji.fedoraproject.org/koji/taskinfo?taskID=2239959
f13: http://koji.fedoraproject.org/koji/taskinfo?taskID=2239962

Comment 24 Bruno Wolff III 2010-06-09 04:46:35 UTC
I actually had fixed the %{?dist} reference, but did the scratch builds with the previous src rpm.

Comment 25 Bruno Wolff III 2010-06-09 05:54:50 UTC
I found that Phillip does have a squashfs lzma git repo which had some changes in early May, but probably weren't in good enough shape to ask for a pull during the 2.6.35 merge window.
http://git.kernel.org/?p=linux/kernel/git/pkl/squashfs-lzma.git;a=summary

Comment 26 Bruno Wolff III 2010-06-09 19:26:11 UTC
I tested a live image made with the x86_64 dist-f13 scratch build of squashfs-tools and it seemed to work OK.

Comment 27 Bruno Wolff III 2010-06-11 16:52:04 UTC
Since the main part of this we control is for squashfs-tools, which I am now building the version with lzma support in rawhide, I think i can close this.
If patches for lzma in 2.6.36 land, but I need them back ported to 2.6.35 in order to have them in the F14 release, I'll start a separate conversation for that.


Note You need to log in before you can comment on or make changes to this bug.