Bug 1648490 - PATCH: add zstd support to squashfs-tools
Summary: PATCH: add zstd support to squashfs-tools
Alias: None
Product: Fedora
Classification: Fedora
Component: squashfs-tools
Version: rawhide
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
Assignee: Bruno Wolff III
QA Contact: Fedora Extras Quality Assurance
Depends On:
TreeView+ depends on / blocked
Reported: 2018-11-09 22:02 UTC by John Friend
Modified: 2019-07-11 09:53 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-07-11 09:53:16 UTC
Type: Bug

Attachments (Terms of Use)
Patch 0001: add missing include files (gcc 8.2.1 complains of major/minor() being undefined) (2.16 KB, patch)
2018-11-09 22:04 UTC, John Friend
no flags Details | Diff
Patch 0002: Add zstd compression (15.22 KB, patch)
2018-11-09 22:05 UTC, John Friend
no flags Details | Diff
updated man file (6.26 KB, text/plain)
2018-11-09 22:06 UTC, John Friend
no flags Details

Description John Friend 2018-11-09 22:02:36 UTC
zstd is a newish compression algorithm from facebook. It offers compression
ratios comparable to xz, with significantly better performance than zlib.
The kernel added squashfs zstd support a while back, and current fedora kernel
package enables it by default (currently at 4.18.17-300).

The patch is not my own work, but by a certain Sean Purcell. Original
was taken from https://github.com/plougher/squashfs-tools/commit/6113361316d5ce5bfdc118d188e5617a1fcd747c.

a fork with the same GPL v2.0 license as the original. The commit
contains signed-off by both Sean Purcell and Nick Terrell from fb.

The same people are on the linux kernel commit adding zstd support
to squashfs in the kernel:

So, this looks legit in terms of license and creators.

Tested this by:
1. building the dist-git package (+attached patches) in f29 and installed it
2. creating a zstd compressed image using "mksquashfs foo bar -comp zstd" 
3. Succesfully mounted it  using "mount -o loop bar /mnt/zstd".

Seems to work.

The patches are against b85fd95, the current master commist in dist-git.
I'm a little confused about mksquashfs.1, its in .gitignore, but I'm not
sure if its checked in or downloaded by fedpkg by some hidden automagic,
so, I just updated the markup and i'm attaching the whole file for you to
either check in or store somewhere.

If these changes are acceptable, please remember to add your name to the
changelog entry (left blank), and to push this to f29, not just rawhide.

Thank you.

Comment 1 John Friend 2018-11-09 22:04:17 UTC
Created attachment 1503859 [details]
Patch 0001: add missing include files (gcc 8.2.1 complains of major/minor() being undefined)

Comment 2 John Friend 2018-11-09 22:05:46 UTC
Created attachment 1503860 [details]
Patch 0002: Add zstd compression

Comment 3 John Friend 2018-11-09 22:06:23 UTC
Created attachment 1503861 [details]
updated man file

Comment 4 Nick Terrell 2018-11-12 22:33:29 UTC
We've tested the zstd patch heavily at Facebook, since we use mksquashfs to create XARs [1]. It would be awesome to have zstd support in the upstream packages.

IIUC https://github.com/plougher/squashfs-tools is now the official repository. The maintainer of squashfs is plougher.

[1] https://code.fb.com/data-infrastructure/xars-a-more-efficient-open-source-system-for-self-contained-executables/

Comment 5 Bruno Wolff III 2018-11-12 23:02:32 UTC
I noticed that zstd was in the kernel.

I saw that Phillip had a github repo. There have been some other forks of squashfs that are getting used by people that is confusing things. We had some security reports that seemed to be for them, but I haven't always seen a direct statement from Phillip that they don't apply to his version. So I'm not sure what I want to do with those patches (that are applied, but I so think about if updating to the latest version of Phillip's tree.)

I haven't done much development in Fedora recently. This might be a good way to ease back into maintenance, but if someone else wants to run with it, don't wait on me.

Once it is in squashfs-tools, we should look at adding it as an option for livecd-creator. But no rush there unless it is a big improvement over xz.

Comment 6 Nick Terrell 2018-11-12 23:10:06 UTC
xz will still compress better than zstd (by a few percent). However, zstd read speed is more than 5x faster, as fast as lz4. The absolute read speed will depend on the number of cores squashfs is allowed to use, but it zstd will still be 5x faster than xz.

See the table in the kernel commit message https://github.com/torvalds/linux/commit/87bf54bb43ddd385d2538b777324bf737f243042#diff-9551d0d75294e4c653db34d347bf61c9.

Comment 7 John Friend 2018-11-19 14:11:29 UTC

Comment 8 John Friend 2019-05-09 21:21:28 UTC
There's been no action taken since I submitted this patch 5 months ago.
Bruno, you seemed inclined to merge when I posted this, why has this stalled?

Comment 9 John Friend 2019-05-14 23:17:40 UTC

Comment 10 Bruno Wolff III 2019-05-21 05:59:49 UTC
I'm looking at getting squashfs-tools based on Philip's github repo instead of the previous sourceforge location. I need to move slowly in messing with things since I've been essentially out of action long enough that the way things are done has changed. Also there are a couple of security related patches that I think are obsoleted, but that I should check to see if that is the case a bit more. The new HEAD has the zstd stuff in it. I just started actively playing with this and expect to have a rawhide build this weekend. Right now I'm working on switching source repos and getting a local build to work.

Comment 11 John Friend 2019-05-21 07:38:40 UTC
Glad to hear it. Thanks.

Comment 12 Bruno Wolff III 2019-05-21 08:37:06 UTC
I did some minor updates for a couple of other bugs, so I've got that part of the process down. I was worried about the changes in process causing larger issues. So I just need to tackle get the repo switched over (which includes trying to verify that the security fixes really don't appear to be needed any more).

Comment 13 Chris Murphy 2019-06-06 02:54:43 UTC
See also

Comment 14 Bruno Wolff III 2019-06-06 03:22:20 UTC
My plan was to add options for zstd to the tool for building live images after I had it working in squashfs-tools. It should be pretty easy to get a version working. My memory is I fixed some of the compression method specific parameters to something reasonable. So input on what those would be for zstd would be useful. I saw some discussion about that on one of the Fedora lists. For live images, we have generally optimized for best compression, since images don't get built as much as they get used and being able to fit more stuff in a fixed space seems to be more important.

My plan is to sync up with Phillip's tree, since he knows what he is doing and I'm not so sure about some of the forks. The main issue is trying to figure out if some security patches are still desired. My memory is that they were reported as bugs against forks and may or may not have been needed to be fixed in Phillip's version. And Phillip has also made changes that may have obsoleted them. But since it would be bad form to add security bugs back into a new release, I want to be careful about this.

Unfortunately I didn't get much done memorial day weekend. I did have a good burst of activity a few weeks ago when I added the test case to the CI.

Comment 15 Chris Murphy 2019-06-06 17:10:13 UTC
Is this already in squashfs-tools?


Comment 16 Bruno Wolff III 2019-06-06 17:19:09 UTC
zstd is in Phillip's tree. He hasn't cut a release in a while.
Fedora is not using the tip of his tree yet and does not have zstd support yet.
I need to resolve what to do with some security patches before switching to using the tip of his tree.

Comment 17 John Friend 2019-06-08 16:40:32 UTC
Bruno, the patches I attached for zstd support were cherry picked from that tree,
the commit Chris pointed out in fact. Seems like the security patch issue is
delaying zstd support indefinitely, while being unrelated. 

Would you please consider cherry picking he zstd commit (my patch) now, and
switching to tip when you find the time to deal with that? this has been waiting
for 6 months, and it was as ready then as it is now.

Comment 18 Fedora Update System 2019-06-25 01:22:22 UTC
FEDORA-2019-05496c236e has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-05496c236e

Comment 19 Fedora Update System 2019-06-25 01:24:04 UTC
FEDORA-2019-0d990c9ab0 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0d990c9ab0

Comment 20 Bruno Wolff III 2019-06-25 01:29:28 UTC
To get things going I used the cherry picked patch rather than moving to Phillip's current tree. Paul Frields reviewed the cherry picked patch, so I didn't have to.
I have done builds for f29, f30 and f31 (rawhide).
The CI test is failing in pagure, but I don't think it is because of squashfs-tools. I ran the test on one of my rawhide systems and it ran OK.
Thanks to people who helped with this, so that I didn't need to do a lot to get this out.

Comment 21 Fedora Update System 2019-06-26 02:47:53 UTC
squashfs-tools-4.3-21.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0d990c9ab0

Comment 22 Fedora Update System 2019-06-26 03:27:50 UTC
squashfs-tools-4.3-21.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-05496c236e

Comment 23 John Friend 2019-06-26 15:22:18 UTC

Comment 24 Fedora Update System 2019-07-11 00:57:33 UTC
squashfs-tools-4.3-21.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 25 Fedora Update System 2019-07-11 04:16:59 UTC
squashfs-tools-4.3-21.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.