Bug 984704 - RFE: appliance-tools should use --block-size to prepare seekable xz images
Summary: RFE: appliance-tools should use --block-size to prepare seekable xz images
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: appliance-tools
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Neal Gompa
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords: Reopened
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-07-15 18:28 UTC by Richard W.M. Jones
Modified: 2018-11-23 12:32 UTC (History)
5 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2018-11-23 12:32:41 UTC


Attachments (Terms of Use)

Description Richard W.M. Jones 2013-07-15 18:28:54 UTC
Description of problem:

Recent versions of xz (note: not all versions) have a new
--block-size parameter which allows xz files to be created
with a predictable block size.  The block size is specified
in bytes, for example:

 $ xz --block-size=16777216 --best test1.img
 $ xz --list test1.img.xz
  Strms  Blocks   Compressed Uncompressed  Ratio  Check   Filename
      1       7     18.9 KiB    100.0 MiB  0.000  CRC64   test1.img.xz
           ^^^^

The advantage of using a large, finite block size is that
these xz-images become seekable:

 https://rwmj.wordpress.com/2013/06/24/xz-plugin-for-nbdkit/#content

nbdkit, an NBD server, has support for seeking in xz files,
provided they have been prepared with --block-size:

 https://github.com/libguestfs/nbdkit/tree/master/plugins/xz

There is some overhead to using a block size, but with a 16MB
block size the overhead is only around 1%.  Note there is a trade
off between the effectiveness of compression (larger block sizes
are better) and the amount of time & memory needed when seeking
(larger block sizes cause more data to be uncompressed and a
larger amount of memory to be allocated).  16 MB seems like a
good compromise for current hardware.

BTW if I'm reading the appliance-tools code correctly, I think
you should also allow a compression level to be specified, and/or
default to --best.

Version-Release number of selected component (if applicable):

appliance-tools 007

Additional info:

https://lists.fedoraproject.org/pipermail/devel/2013-July/thread.html#185369

Comment 1 Fedora Admin XMLRPC Client 2014-01-12 19:49:56 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 2 Fedora Admin XMLRPC Client 2016-12-21 21:14:37 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 3 Fedora Admin XMLRPC Client 2016-12-21 21:16:26 UTC
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.

Comment 4 Neal Gompa 2017-03-12 21:28:43 UTC
While I see the value in adding --block-size, I'm not sure what value I should set here. Do you have any particular recommendations?

Comment 5 Fedora Update System 2017-03-12 22:44:13 UTC
appliance-tools-008.0-4.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-fdc8ea1d5d

Comment 6 Fedora Update System 2017-03-12 22:46:21 UTC
appliance-tools-008.0-4.fc25 livecd-tools-24.2-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-7799a1cc7c

Comment 7 Fedora Update System 2017-03-12 22:46:29 UTC
appliance-tools-008.0-4.fc25 livecd-tools-24.2-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-7799a1cc7c

Comment 8 Fedora Update System 2017-03-13 01:51:52 UTC
appliance-tools-008.0-4.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-fdc8ea1d5d

Comment 9 Fedora Update System 2017-03-14 01:22:50 UTC
appliance-tools-008.0-4.fc25, livecd-tools-24.2-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-7799a1cc7c

Comment 10 Fedora Update System 2017-03-29 01:29:50 UTC
appliance-tools-008.0-4.fc25, livecd-tools-24.2-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 11 Fedora Update System 2017-04-01 16:58:09 UTC
appliance-tools-008.0-4.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 12 Richard W.M. Jones 2018-11-21 18:35:33 UTC
Reopening this old bug.

nbdkit upstream may get the ability to transparently stream and
uncompress xz files hosted on web servers:

  https://www.redhat.com/archives/libguestfs/2018-November/thread.html#00210
  eg:
  nbdkit --filter=xz curl url=https://download.fedoraproject.org/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.raw.xz

However this doesn't work very well with the current block size which appears to
be 192M since we have to read many megabytes from the remote end in order
to satisfy a single read.

I recompressed that cloud image with a 16M block size and it's possible to
boot from it, at least when it's hosted on my own web server (which
is much closer than the Fedora mirrors, so it's not a great test, but
it's the best I can do for now).

Here's another example using a libguestfs template which is already compressed
with a 16M block size:

  $ nbdkit -f -v --filter=cache --filter=xz curl url=http://builder.libguestfs.org/fedora-29.xz
  $ qemu-system-x86_64 -machine accel=kvm:tcg -cpu host -m 2048 -drive file=nbd:localhost:10809,if=virtio

Again this is bootable for me, albeit slowly.

(In reply to Neal Gompa from comment #4)
> While I see the value in adding --block-size, I'm not sure what value I
> should set here. Do you have any particular recommendations?

I didn't answer this before but a good choice would be 16M
(16777216 bytes).

I recompressed the Fedora cloud image:

before 194278292
after  202874868

which is about a 1% increase in size.

Comment 13 Neal Gompa 2018-11-23 12:32:41 UTC
appliance-tools uses 16MiB block size: https://pagure.io/appliance-tools/c/fe2609302233148b8c670417748c0f036092c3d7?branch=master

That said, x86 images aren't made with appliance-creator. They're made with lorax and oz.


Note You need to log in before you can comment on or make changes to this bug.