Bug 984704

Summary: RFE: appliance-tools should use --block-size to prepare seekable xz images
Product: [Fedora] Fedora Reporter: Richard W.M. Jones <rjones>
Component: appliance-toolsAssignee: Neal Gompa <ngompa13>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: apevec, dhuff, jgoulding, mattdm, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: appliance-tools-008.0-4.fc25 appliance-tools-008.0-4.fc26 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-28 21:29:50 EDT Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Richard W.M. Jones 2013-07-15 14:28:54 EDT
Description of problem:

Recent versions of xz (note: not all versions) have a new
--block-size parameter which allows xz files to be created
with a predictable block size.  The block size is specified
in bytes, for example:

 $ xz --block-size=16777216 --best test1.img
 $ xz --list test1.img.xz
  Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
      1       7     18.9 KiB    100.0 MiB  0.000  CRC64   test1.img.xz
           ^^^^

The advantage of using a large, finite block size is that
these xz-images become seekable:

 https://rwmj.wordpress.com/2013/06/24/xz-plugin-for-nbdkit/#content

nbdkit, an NBD server, has support for seeking in xz files,
provided they have been prepared with --block-size:

 https://github.com/libguestfs/nbdkit/tree/master/plugins/xz

There is some overhead to using a block size, but with a 16MB
block size the overhead is only around 1%.  Note there is a trade
off between the effectiveness of compression (larger block sizes
are better) and the amount of time & memory needed when seeking
(larger block sizes cause more data to be uncompressed and a
larger amount of memory to be allocated).  16 MB seems like a
good compromise for current hardware.

BTW if I'm reading the appliance-tools code correctly, I think
you should also allow a compression level to be specified, and/or
default to --best.

Version-Release number of selected component (if applicable):

appliance-tools 007

Additional info:

https://lists.fedoraproject.org/pipermail/devel/2013-July/thread.html#185369
Comment 1 Fedora Admin XMLRPC Client 2014-01-12 14:49:56 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 2 Fedora Admin XMLRPC Client 2016-12-21 16:14:37 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 3 Fedora Admin XMLRPC Client 2016-12-21 16:16:26 EST
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 4 Neal Gompa 2017-03-12 17:28:43 EDT
While I see the value in adding --block-size, I'm not sure what value I should set here. Do you have any particular recommendations?
Comment 5 Fedora Update System 2017-03-12 18:44:13 EDT
appliance-tools-008.0-4.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-fdc8ea1d5d
Comment 6 Fedora Update System 2017-03-12 18:46:21 EDT
appliance-tools-008.0-4.fc25 livecd-tools-24.2-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-7799a1cc7c
Comment 7 Fedora Update System 2017-03-12 18:46:29 EDT
appliance-tools-008.0-4.fc25 livecd-tools-24.2-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-7799a1cc7c
Comment 8 Fedora Update System 2017-03-12 21:51:52 EDT
appliance-tools-008.0-4.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-fdc8ea1d5d
Comment 9 Fedora Update System 2017-03-13 21:22:50 EDT
appliance-tools-008.0-4.fc25, livecd-tools-24.2-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-7799a1cc7c
Comment 10 Fedora Update System 2017-03-28 21:29:50 EDT
appliance-tools-008.0-4.fc25, livecd-tools-24.2-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.
Comment 11 Fedora Update System 2017-04-01 12:58:09 EDT
appliance-tools-008.0-4.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.