Description of problem: Now that lzma is in the kernel, I am hoping the mksquashfs in f12 will be able to use lzma compression. Better compression will be useful for livecd images. In some cases this will allow more material to be included. In others there will be bandwidth savings with the reduced image sizes. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
When I originally submitted this I was envisioning that it was a matter of just enabling a plugin for lzma instead of bzip, but it seems there was more to it than that. I did find the message I have included below, which suggests lzma enabled squashfs may still make it in to 2.6.31. Re: squashfs 4.0 lzma support by Phillip Lougher :: Rate this Message: Reply to Author | View Threaded | Show Only this Message Christopher Rogers wrote: > Since lzma is in kernel 2.6.30 now, is it going to be added into kernel > 2.6.31 or 2.6.32 merger window? I started working on LZMA support a couple of days ago. I'm hoping to get it finished for the 2.6.31 merge window, though I may miss this. It depends on whether I hit any problems, and how soon the merge window closes. > > I'm only asking cause I'm still using the older squashfs lzma 3.4 for > the time be. At some point thats going to break and there will be no > squashfs-lzma option. > Yes I know. I expect to get a lot of requests for LZMA support when that happens, and I aim to have something finished before then. Phillip
As far as I can tell Phillip didn't get this into 2.6.31. I'll try to remember to look for indications that a patch exists (possibly in linux-next) from time to time.
It looks like this isn't going to happen for 2.6.32. But F13 will probably start with 2.6.33, so there is another chance to make it into F13.
There have been replies to the old thread at http://thread.gmane.org/gmane.comp.file-systems.squashfs.devel/79/focus=88 by Phillip in late October that suggest he is actively working on this again. He mentions trying to merge the patch in 2.6.33.
Lougher posted the patch set this morning. It isn't in just yet, but it looks like it will really happen for 2.6.33. http://www.acetylcholine.com/node/44548
The message from Lougher at the link below has some information about using / building squashfs-tools to use lzma. http://marc.info/?l=linux-embedded&m=126029628723357&w=2
I am willing to take a stab of building the devel version of mksquashfs in rawhide if you want. I'd like to see lzma enabled mksquashfs and livecd-tools testable by feature freeze and am willing to do some of the work.
This is now a proposed feature: https://fedoraproject.org/wiki/Features/LZMA_for_Live_Images I won't ask FESCO to vote until Lougher's patches are accepted. (He posted a revision yesterday to address a few comments.)
It isn't here, but the feature page has some extra notes covering it, the patches ended up in linux-next and are targeted for 2.6.34 which F13 probably won't release with. The version of the SDK used by the dev version squashfs-tools isn't currently included in Fedora. Writing new wrappers for the xz library is tricky. And the lzma library is based on an upstream project that is dead. (The project was mainly for the utility tools which have been superceded by xz.)
I was checking on the status of squashfs upstream today and noticed that Lougher added support for using the xz library about a week ago. That will probably make it a lot easier to get things working. I won't try to beat the alpha release on this (particularly since the games spin needs attention right now), but I'll see if I can get it into rawhide shortly after the branch and maybe into the F13 beta if it doesn't seem to cause problems.
I have a scratch build that builds. I haven't tested functionallity yet, so you don't need to look at that. But if you want to review what I have changed in the spec file or other packaging related stuff, then this build would be useful for that. https://koji.fedoraproject.org/koji/taskinfo?taskID=1984187
I did some testing with the scratch build. I didn't strip the CVS metadata out of the archive file and that should get done before submitting a real build. I used mksquashfs and unsquashfs with lzma compression and got back an identical (according to diff -r) directory tree. I created a live spin (Desktop) on a system with the test squashfs and with the test squashfs in the repo used to build the image. The image built and ran correctly. Note this is testing that mksquashfs doesn't break things with the default zlib compression. lzma won't work yet because there is no kernel support yet. mksquashfs by default uses as many processors as you have to simultaneously work on the compression. It does this with both zlib and lzma compression. There is an option to limit the number of processors that it will use. I tested a hacked version of livecd-creator to compare zlib and lzma compression. The output won't actually run, but this gives us a good idea of the benefit in what is probably our main use case: Games spin image using zlib: 4390387712 Games spin image using lzma: 3984588800 Since things seem to be mostly working, I'd like to get the OK to get 4.1 in rawhide right after the branch with an eye on putting it in f13 after the alpha. I think it will get enough testing before beta to be a reasonable risk to take. In rawhide I would be looking to make some proposal for changes to livecd-creator with a target of F14. The kernel patches for lzma will probably land during the merge window, which isn't too far away.
Kyle, can you give me a yeah or nay on putting the prerelease squashfs-tools 4.1 into rawhide right after F13 is branched?
I made a new scratch build against f14 with a couple of cleanups relative to my last scratch build. This is what I am planning on importing to dist-f14. http://koji.fedoraproject.org/koji/taskinfo?taskID=1992766
On the kernel side, Lougher made a pull request which has been bounced by Linus as needing cleanup.
Linus just pulled most of this. I am not sure that lzma is completely supported by this pull as Lougher's comment in the updated pull request suggested it was clean up and that more would be following later. But Linus also pulled some of the earlier stuff as well, so I am not sure. I'll update after I figure out more. It does make it seem very likely to be in 2.6.34 now.
I tested building a custom kernel which I believe included the lzma/squashfs related stuff that Linus pulled yesterday and was unable to mount an lzma squashfs image, but was able to mount a zlib squashfs image. So it looks like we do need to wait a bit yet.
We aren't going to get LZMA Squashfs in 2.6.34. Lougher has recently made some bug fixes for some of the infrastructure changes that had been accepted earlier. He also made the comment "It's not something that's going to be fixed in a couple of evenings :-(". So even 2.6.35 is iffy.
It's something I'm going to try and get finished for 2.6.35, whether I will is another matter. Luckily CELF has offered payment for the LZMA stuff and so I may be able to work full time on it.
Thanks. Even 2.6.36 would probably be good enough to get this in F14. I also appreciate you adding support for the xz library to mksquashfs in addition to the lzma SDK. That made geting it to work in Fedora a lot easier.
Status update: LZMA didn't make the merge window (though squashfs did gain xattr support which might allow for some changes in live images) for 2.6.35.
Is it OK if I build a 4.1 prerelease version of mksquashfs for rawhide? Even if we don't get lzma, we'll probably want the xattr support (since that's been accepted for 2.6.35). The default will still be zlib, so there shouldn't be a big impact (where we'd need to revert) if we don't get lzma for 2.6.36.
I have made new scratch builds. f14: http://koji.fedoraproject.org/koji/taskinfo?taskID=2239959 f13: http://koji.fedoraproject.org/koji/taskinfo?taskID=2239962
I actually had fixed the %{?dist} reference, but did the scratch builds with the previous src rpm.
I found that Phillip does have a squashfs lzma git repo which had some changes in early May, but probably weren't in good enough shape to ask for a pull during the 2.6.35 merge window. http://git.kernel.org/?p=linux/kernel/git/pkl/squashfs-lzma.git;a=summary
I tested a live image made with the x86_64 dist-f13 scratch build of squashfs-tools and it seemed to work OK.
Since the main part of this we control is for squashfs-tools, which I am now building the version with lzma support in rawhide, I think i can close this. If patches for lzma in 2.6.36 land, but I need them back ported to 2.6.35 in order to have them in the F14 release, I'll start a separate conversation for that.