Bug 1648490
| Summary: | PATCH: add zstd support to squashfs-tools | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | John Friend <jfrie> | ||||||||
| Component: | squashfs-tools | Assignee: | Bruno Wolff III <bruno> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||
| Severity: | unspecified | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | rawhide | CC: | bgilbert, bruno, bugzilla, JohnFriend, katzj, kyle, terrelln | ||||||||
| Target Milestone: | --- | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2019-07-11 09:53:16 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
|
Description
John Friend
2018-11-09 22:02:36 UTC
Created attachment 1503859 [details]
Patch 0001: add missing include files (gcc 8.2.1 complains of major/minor() being undefined)
Created attachment 1503860 [details]
Patch 0002: Add zstd compression
Created attachment 1503861 [details]
updated man file
We've tested the zstd patch heavily at Facebook, since we use mksquashfs to create XARs [1]. It would be awesome to have zstd support in the upstream packages. IIUC https://github.com/plougher/squashfs-tools is now the official repository. The maintainer of squashfs is plougher. [1] https://code.fb.com/data-infrastructure/xars-a-more-efficient-open-source-system-for-self-contained-executables/ I noticed that zstd was in the kernel. I saw that Phillip had a github repo. There have been some other forks of squashfs that are getting used by people that is confusing things. We had some security reports that seemed to be for them, but I haven't always seen a direct statement from Phillip that they don't apply to his version. So I'm not sure what I want to do with those patches (that are applied, but I so think about if updating to the latest version of Phillip's tree.) I haven't done much development in Fedora recently. This might be a good way to ease back into maintenance, but if someone else wants to run with it, don't wait on me. Once it is in squashfs-tools, we should look at adding it as an option for livecd-creator. But no rush there unless it is a big improvement over xz. xz will still compress better than zstd (by a few percent). However, zstd read speed is more than 5x faster, as fast as lz4. The absolute read speed will depend on the number of cores squashfs is allowed to use, but it zstd will still be 5x faster than xz. See the table in the kernel commit message https://github.com/torvalds/linux/commit/87bf54bb43ddd385d2538b777324bf737f243042#diff-9551d0d75294e4c653db34d347bf61c9. bump. There's been no action taken since I submitted this patch 5 months ago. Bruno, you seemed inclined to merge when I posted this, why has this stalled? bump. I'm looking at getting squashfs-tools based on Philip's github repo instead of the previous sourceforge location. I need to move slowly in messing with things since I've been essentially out of action long enough that the way things are done has changed. Also there are a couple of security related patches that I think are obsoleted, but that I should check to see if that is the case a bit more. The new HEAD has the zstd stuff in it. I just started actively playing with this and expect to have a rawhide build this weekend. Right now I'm working on switching source repos and getting a local build to work. Glad to hear it. Thanks. I did some minor updates for a couple of other bugs, so I've got that part of the process down. I was worried about the changes in process causing larger issues. So I just need to tackle get the repo switched over (which includes trying to verify that the security fixes really don't appear to be needed any more). My plan was to add options for zstd to the tool for building live images after I had it working in squashfs-tools. It should be pretty easy to get a version working. My memory is I fixed some of the compression method specific parameters to something reasonable. So input on what those would be for zstd would be useful. I saw some discussion about that on one of the Fedora lists. For live images, we have generally optimized for best compression, since images don't get built as much as they get used and being able to fit more stuff in a fixed space seems to be more important. My plan is to sync up with Phillip's tree, since he knows what he is doing and I'm not so sure about some of the forks. The main issue is trying to figure out if some security patches are still desired. My memory is that they were reported as bugs against forks and may or may not have been needed to be fixed in Phillip's version. And Phillip has also made changes that may have obsoleted them. But since it would be bad form to add security bugs back into a new release, I want to be careful about this. Unfortunately I didn't get much done memorial day weekend. I did have a good burst of activity a few weeks ago when I added the test case to the CI. Is this already in squashfs-tools? https://github.com/plougher/squashfs-tools/commit/6113361316d5ce5bfdc118d188e5617a1fcd747c zstd is in Phillip's tree. He hasn't cut a release in a while. Fedora is not using the tip of his tree yet and does not have zstd support yet. I need to resolve what to do with some security patches before switching to using the tip of his tree. Bruno, the patches I attached for zstd support were cherry picked from that tree, the commit Chris pointed out in fact. Seems like the security patch issue is delaying zstd support indefinitely, while being unrelated. Would you please consider cherry picking he zstd commit (my patch) now, and switching to tip when you find the time to deal with that? this has been waiting for 6 months, and it was as ready then as it is now. FEDORA-2019-05496c236e has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-05496c236e FEDORA-2019-0d990c9ab0 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0d990c9ab0 To get things going I used the cherry picked patch rather than moving to Phillip's current tree. Paul Frields reviewed the cherry picked patch, so I didn't have to. I have done builds for f29, f30 and f31 (rawhide). The CI test is failing in pagure, but I don't think it is because of squashfs-tools. I ran the test on one of my rawhide systems and it ran OK. Thanks to people who helped with this, so that I didn't need to do a lot to get this out. squashfs-tools-4.3-21.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0d990c9ab0 squashfs-tools-4.3-21.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-05496c236e Thanks! squashfs-tools-4.3-21.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report. squashfs-tools-4.3-21.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report. |