Bug 1743029

Summary: zstd compression level 20+ appear to produce same results as level 19
Product: [Fedora] Fedora Reporter: Chris Murphy <bugzilla>
Component: squashfs-toolsAssignee: Bruno Wolff III <bruno>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 31CC: bruno, katzj, kyle, samuel-rhbugs
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-24 15:25:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chris Murphy 2019-08-18 18:20:52 UTC
Description of problem:


Version-Release number of selected component (if applicable):
squashfs-tools-4.3-21.fc30.x86_64

How reproducible:
Always

Steps to Reproduce:
# mksquashfs /mnt/2/ squashfs19.img -comp zstd -Xcompression-level 19 -b 256K
# mksquashfs /mnt/2/ squashfs22.img -comp zstd -Xcompression-level 22 -b 256K
# ls -l

Actual results:

-rw-r--r--. 1 root  root  2020720640 Aug 18 11:52  squashfs19.img              │Access: 2019-08-18 11:23:22.890523592 -0600
-rw-r--r--. 1 root  root  2020712448 Aug 18 01:27  squashfs22.img 


Expected results:

22 should be smaller than this, should take longer

Additional info:


https://src.fedoraproject.org/rpms/squashfs-tools/c/258bc8ea26b48e872f6d886bd830f774634e6fc9?branch=master

I can follow code like I can follow breadcrumbs while blindfolded, but I don't see where the zstd --ultra flag would be used to unlock compression levels 20-22. And at least if zstd were called directly to compress a file with a level above 19 but without --ultra, it produces the same results as 19. And that's what I'm seeing here, hence I suspect mksquashfs isn't passing --ultra to unlock higher compression levels.

Comment 1 Bruno Wolff III 2019-08-19 00:40:22 UTC
It might be a ggod idea to report this upstream. It looks like Phillip is doing a fair amount of work on squashfs-tools right now and there might be an official 4.4 release soon.
I'm not feeling too great about trying to find how and when to pass the equivalent of the --ultra option to libzstd and hack it into the zstd squashfs-tools wrapper.

Comment 2 Bruno Wolff III 2019-08-19 01:28:16 UTC
It looks like ZSTD_c_strategy isn't set (unless a non-standard symbol is used) and probably defaults to something less than ZSTD_btultra or ZSTD_btultra2. I don't know what the consequences to changing one of those to be the default is. It seems like a strange design to allow the higher compression level to be set but not have it really work. These seem to affect the meaning of other parameters, so they probably shouldn't just be thrown in. It would probably be good to have someone who knows zstd well, make a concrete suggestion for upstream.

Comment 3 Chris Murphy 2019-08-19 06:12:21 UTC
Upstream report.
https://github.com/plougher/squashfs-tools/issues/67

Comment 4 Bruno Wolff III 2019-08-29 05:19:22 UTC
4.4 got released but there is no comment about your issue yet. At least initially, I'm not going to want to try to do custom changes for zstd at levels 19+. I'm going to try to get an update into rawhide within a few weeks.

Comment 5 Bruno Wolff III 2020-05-13 02:07:30 UTC
4.4 is now in rawhide, though I don't expect it to affect this bug. There were a couple of follow up comments in the upstream issue, but they didn't resolve the issue.

Comment 6 Ben Cotton 2020-11-03 15:32:37 UTC
This message is a reminder that Fedora 31 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '31'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 31 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 7 Ben Cotton 2020-11-24 15:25:58 UTC
Fedora 31 changed to end-of-life (EOL) status on 2020-11-24. Fedora 31 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.