Bug 984704
| Summary: | RFE: appliance-tools should use --block-size to prepare seekable xz images | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Richard W.M. Jones <rjones> |
| Component: | appliance-tools | Assignee: | Neal Gompa <ngompa13> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | rawhide | CC: | apevec, dhuff, jgoulding, mattdm, virt-maint |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | appliance-tools-008.0-4.fc25 appliance-tools-008.0-4.fc26 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-11-23 12:32:41 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component. This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component. This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component. While I see the value in adding --block-size, I'm not sure what value I should set here. Do you have any particular recommendations? appliance-tools-008.0-4.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-fdc8ea1d5d appliance-tools-008.0-4.fc25 livecd-tools-24.2-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-7799a1cc7c appliance-tools-008.0-4.fc25 livecd-tools-24.2-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-7799a1cc7c appliance-tools-008.0-4.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-fdc8ea1d5d appliance-tools-008.0-4.fc25, livecd-tools-24.2-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-7799a1cc7c appliance-tools-008.0-4.fc25, livecd-tools-24.2-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report. appliance-tools-008.0-4.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report. Reopening this old bug. nbdkit upstream may get the ability to transparently stream and uncompress xz files hosted on web servers: https://www.redhat.com/archives/libguestfs/2018-November/thread.html#00210 eg: nbdkit --filter=xz curl url=https://download.fedoraproject.org/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.raw.xz However this doesn't work very well with the current block size which appears to be 192M since we have to read many megabytes from the remote end in order to satisfy a single read. I recompressed that cloud image with a 16M block size and it's possible to boot from it, at least when it's hosted on my own web server (which is much closer than the Fedora mirrors, so it's not a great test, but it's the best I can do for now). Here's another example using a libguestfs template which is already compressed with a 16M block size: $ nbdkit -f -v --filter=cache --filter=xz curl url=http://builder.libguestfs.org/fedora-29.xz $ qemu-system-x86_64 -machine accel=kvm:tcg -cpu host -m 2048 -drive file=nbd:localhost:10809,if=virtio Again this is bootable for me, albeit slowly. (In reply to Neal Gompa from comment #4) > While I see the value in adding --block-size, I'm not sure what value I > should set here. Do you have any particular recommendations? I didn't answer this before but a good choice would be 16M (16777216 bytes). I recompressed the Fedora cloud image: before 194278292 after 202874868 which is about a 1% increase in size. appliance-tools uses 16MiB block size: https://pagure.io/appliance-tools/c/fe2609302233148b8c670417748c0f036092c3d7?branch=master That said, x86 images aren't made with appliance-creator. They're made with lorax and oz. |
Description of problem: Recent versions of xz (note: not all versions) have a new --block-size parameter which allows xz files to be created with a predictable block size. The block size is specified in bytes, for example: $ xz --block-size=16777216 --best test1.img $ xz --list test1.img.xz Strms Blocks Compressed Uncompressed Ratio Check Filename 1 7 18.9 KiB 100.0 MiB 0.000 CRC64 test1.img.xz ^^^^ The advantage of using a large, finite block size is that these xz-images become seekable: https://rwmj.wordpress.com/2013/06/24/xz-plugin-for-nbdkit/#content nbdkit, an NBD server, has support for seeking in xz files, provided they have been prepared with --block-size: https://github.com/libguestfs/nbdkit/tree/master/plugins/xz There is some overhead to using a block size, but with a 16MB block size the overhead is only around 1%. Note there is a trade off between the effectiveness of compression (larger block sizes are better) and the amount of time & memory needed when seeking (larger block sizes cause more data to be uncompressed and a larger amount of memory to be allocated). 16 MB seems like a good compromise for current hardware. BTW if I'm reading the appliance-tools code correctly, I think you should also allow a compression level to be specified, and/or default to --best. Version-Release number of selected component (if applicable): appliance-tools 007 Additional info: https://lists.fedoraproject.org/pipermail/devel/2013-July/thread.html#185369