Bug 1985561
| Summary: | All zero files not a multiple of the blocksize seem to get truncated on restore | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Bruno Wolff III <bruno> | ||||
| Component: | squashfs-tools | Assignee: | Bruno Wolff III <bruno> | ||||
| Status: | CLOSED RAWHIDE | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | rawhide | CC: | bruno, katzj, kyle, phillip.lougher | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-07-27 11:59:33 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Looking more carefully at which files got flagged, it looks like all zero fragments trigger the problem. The whole file doesn't need to be all zeros. Apparently CI gating isn't working currently as the build wasn't held back from the latest rawhide compose. This bug will occur if you get multiple files that are identical, except for a different length of trailing ZERO filled data. For example two files. File 1 has 128K of data, and 10 bytes of trailing ZEROs. File 2 has 128K of identical data, and 4096 bytes of trailing ZEROs. In this case, the last block of the file will be stored as 0 (to indicate sparse), and you get identical block lists for both files. block[0] = compressed size of 128K of data block[1] = 0 The duplicate checking code didn't recognise this special case, and in the case of file 2 it would return file 1, effectively truncating it. This bug is a regression caused by rewriting the duplicate checking code to be more aggressive looking for duplicates when using tail-end packing. Unfortunately, it didn't show up in testing, due to a relatively low probability of occurring (identical files with different lengths of ZERO filled trailing bytes, and these different lengths must be less than the block size). The patch is on git hub https://github.com/plougher/squashfs-tools/commit/19b161c1cd3e31f7a396ea92dea4390ad43f27b9 Thanks for the explanation. It's a bit late here tonight and I might not last long enough to get a build done tonight, but I will for sure get one done tomorrow. 4.5-2.fc35 includes the above commit as a patch and a local build of that package passes the regression test that initially flagged the issue. The official build needs a glibc update, that won't show up until after tonight's rawhide compose, to be installed. So I won't close this until after I test the official build tomorrow. But I expect that test to go fine. Once there is a new point release out I'll do a new build with that as well. I tested the official build and it didn't have a problem. |
Created attachment 1805026 [details] CI test script Description of problem: All zero files not a multiple of the blocksize seem to get truncated on restore Version-Release number of selected component (if applicable): 4.5 How reproducible: Seems 100% Steps to Reproduce: 1.run the CI test Squashfs-compression-test.sh 2. 3. Actual results: For each sub test: Files /tmp/test-squashfs/sq/combined-urandom-1M-1-zero-2047 and /tmp/test-squashfs/data/combined-urandom-1M-1-zero-2047 differ Files /tmp/test-squashfs/sq/combined-urandom-1M-1-zero-4095 and /tmp/test-squashfs/data/combined-urandom-1M-1-zero-4095 differ Files /tmp/test-squashfs/sq/combined-urandom-1M-2-zero-2047 and /tmp/test-squashfs/data/combined-urandom-1M-2-zero-2047 differ Files /tmp/test-squashfs/sq/combined-urandom-1M-2-zero-4095 and /tmp/test-squashfs/data/combined-urandom-1M-2-zero-4095 differ Files /tmp/test-squashfs/sq/combined-urandom-1M-3-zero-2047 and /tmp/test-squashfs/data/combined-urandom-1M-3-zero-2047 differ Files /tmp/test-squashfs/sq/combined-urandom-1M-3-zero-4095 and /tmp/test-squashfs/data/combined-urandom-1M-3-zero-4095 differ Files /tmp/test-squashfs/sq/combined-urandom-1M-4-zero-2047 and /tmp/test-squashfs/data/combined-urandom-1M-4-zero-2047 differ Files /tmp/test-squashfs/sq/combined-urandom-1M-4-zero-4095 and /tmp/test-squashfs/data/combined-urandom-1M-4-zero-4095 differ Files /tmp/test-squashfs/sq/combined-urandom-1M-5-zero-2047 and /tmp/test-squashfs/data/combined-urandom-1M-5-zero-2047 differ Files /tmp/test-squashfs/sq/combined-urandom-1M-5-zero-4095 and /tmp/test-squashfs/data/combined-urandom-1M-5-zero-4095 differ Files /tmp/test-squashfs/sq/combined-zero-1M-1-zero-2047 and /tmp/test-squashfs/data/combined-zero-1M-1-zero-2047 differ Files /tmp/test-squashfs/sq/combined-zero-1M-1-zero-4095 and /tmp/test-squashfs/data/combined-zero-1M-1-zero-4095 differ Files /tmp/test-squashfs/sq/combined-zero-1M-2-zero-2047 and /tmp/test-squashfs/data/combined-zero-1M-2-zero-2047 differ Files /tmp/test-squashfs/sq/combined-zero-1M-2-zero-4095 and /tmp/test-squashfs/data/combined-zero-1M-2-zero-4095 differ Files /tmp/test-squashfs/sq/combined-zero-1M-3-zero-2047 and /tmp/test-squashfs/data/combined-zero-1M-3-zero-2047 differ Files /tmp/test-squashfs/sq/combined-zero-1M-3-zero-4095 and /tmp/test-squashfs/data/combined-zero-1M-3-zero-4095 differ Files /tmp/test-squashfs/sq/combined-zero-1M-4-zero-2047 and /tmp/test-squashfs/data/combined-zero-1M-4-zero-2047 differ Files /tmp/test-squashfs/sq/combined-zero-1M-4-zero-4095 and /tmp/test-squashfs/data/combined-zero-1M-4-zero-4095 differ Files /tmp/test-squashfs/sq/combined-zero-1M-5-zero-2047 and /tmp/test-squashfs/data/combined-zero-1M-5-zero-2047 differ Files /tmp/test-squashfs/sq/combined-zero-1M-5-zero-4095 and /tmp/test-squashfs/data/combined-zero-1M-5-zero-4095 differ Files /tmp/test-squashfs/sq/combined-zero-4K-1-zero-1 and /tmp/test-squashfs/data/combined-zero-4K-1-zero-1 differ Files /tmp/test-squashfs/sq/combined-zero-4K-1-zero-2047 and /tmp/test-squashfs/data/combined-zero-4K-1-zero-2047 differ Files /tmp/test-squashfs/sq/combined-zero-4K-1-zero-4095 and /tmp/test-squashfs/data/combined-zero-4K-1-zero-4095 differ Files /tmp/test-squashfs/sq/combined-zero-4K-2-urandom-0 and /tmp/test-squashfs/data/combined-zero-4K-2-urandom-0 differ Files /tmp/test-squashfs/sq/combined-zero-4K-2-zero-0 and /tmp/test-squashfs/data/combined-zero-4K-2-zero-0 differ Files /tmp/test-squashfs/sq/combined-zero-4K-2-zero-1 and /tmp/test-squashfs/data/combined-zero-4K-2-zero-1 differ Files /tmp/test-squashfs/sq/combined-zero-4K-2-zero-2047 and /tmp/test-squashfs/data/combined-zero-4K-2-zero-2047 differ Files /tmp/test-squashfs/sq/combined-zero-4K-2-zero-4095 and /tmp/test-squashfs/data/combined-zero-4K-2-zero-4095 differ Files /tmp/test-squashfs/sq/combined-zero-4K-3-urandom-0 and /tmp/test-squashfs/data/combined-zero-4K-3-urandom-0 differ Files /tmp/test-squashfs/sq/combined-zero-4K-3-zero-0 and /tmp/test-squashfs/data/combined-zero-4K-3-zero-0 differ Files /tmp/test-squashfs/sq/combined-zero-4K-3-zero-1 and /tmp/test-squashfs/data/combined-zero-4K-3-zero-1 differ Files /tmp/test-squashfs/sq/combined-zero-4K-3-zero-2047 and /tmp/test-squashfs/data/combined-zero-4K-3-zero-2047 differ Files /tmp/test-squashfs/sq/combined-zero-4K-3-zero-4095 and /tmp/test-squashfs/data/combined-zero-4K-3-zero-4095 differ Files /tmp/test-squashfs/sq/combined-zero-4K-4-urandom-0 and /tmp/test-squashfs/data/combined-zero-4K-4-urandom-0 differ Files /tmp/test-squashfs/sq/combined-zero-4K-4-zero-0 and /tmp/test-squashfs/data/combined-zero-4K-4-zero-0 differ Files /tmp/test-squashfs/sq/combined-zero-4K-4-zero-1 and /tmp/test-squashfs/data/combined-zero-4K-4-zero-1 differ Files /tmp/test-squashfs/sq/combined-zero-4K-4-zero-2047 and /tmp/test-squashfs/data/combined-zero-4K-4-zero-2047 differ Files /tmp/test-squashfs/sq/combined-zero-4K-4-zero-4095 and /tmp/test-squashfs/data/combined-zero-4K-4-zero-4095 differ Files /tmp/test-squashfs/sq/combined-zero-4K-5-urandom-0 and /tmp/test-squashfs/data/combined-zero-4K-5-urandom-0 differ Files /tmp/test-squashfs/sq/combined-zero-4K-5-zero-0 and /tmp/test-squashfs/data/combined-zero-4K-5-zero-0 differ Files /tmp/test-squashfs/sq/combined-zero-4K-5-zero-1 and /tmp/test-squashfs/data/combined-zero-4K-5-zero-1 differ Files /tmp/test-squashfs/sq/combined-zero-4K-5-zero-2047 and /tmp/test-squashfs/data/combined-zero-4K-5-zero-2047 differ Files /tmp/test-squashfs/sq/combined-zero-4K-5-zero-4095 and /tmp/test-squashfs/data/combined-zero-4K-5-zero-4095 differ Files /tmp/test-squashfs/sq/file-zero-4K-2 and /tmp/test-squashfs/data/file-zero-4K-2 differ Files /tmp/test-squashfs/sq/file-zero-4K-3 and /tmp/test-squashfs/data/file-zero-4K-3 differ Files /tmp/test-squashfs/sq/file-zero-4K-4 and /tmp/test-squashfs/data/file-zero-4K-4 differ Files /tmp/test-squashfs/sq/file-zero-4K-5 and /tmp/test-squashfs/data/file-zero-4K-5 differ Files /tmp/test-squashfs/sq/frag-zero-1 and /tmp/test-squashfs/data/frag-zero-1 differ Files /tmp/test-squashfs/sq/frag-zero-2047 and /tmp/test-squashfs/data/frag-zero-2047 differ Files /tmp/test-squashfs/sq/frag-zero-4095 and /tmp/test-squashfs/data/frag-zero-4095 differ Expected results: There should be no differences reported.