RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1448739 - RFE: Support multicore decompression for OVA files using pigz and pxz
Summary: RFE: Support multicore decompression for OVA files using pigz and pxz
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libguestfs
Version: 7.3
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: 7.5
Assignee: Richard W.M. Jones
QA Contact: Virtualization Bugs
Yehuda Zimmerman
URL:
Whiteboard: V2V
Depends On:
Blocks: 1420851 1469590
TreeView+ depends on / blocked
 
Reported: 2017-05-07 23:14 UTC by Robert Scheck
Modified: 2021-06-10 12:17 UTC (History)
14 users (show)

Fixed In Version: libguestfs-1.36.5-1.el7
Doc Type: Enhancement
Doc Text:
Parallel decompression of `OVA` files supported With this release, the *pigz* and *pxz* decompression utilities are supported by the *virt-v2v* utility. These utilities speed up extraction of `OVA` files compressed with the *gzip* and *xz* utilities on multi-processor machines. In addition, the command-line interfaces for *pigz* and *pxz* are fully compatible with the command-line interfaces for *gzip* and *xz*. If *pigz* and *pxz* are installed, they are used by default. If *pigz* and *pxz* are not installed, there is no change to the extraction behavior.
Clone Of:
Environment:
Last Closed: 2018-04-10 09:15:08 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0677 0 normal SHIPPED_LIVE libguestfs new packages, bug fix and enhancement update 2018-04-10 13:13:49 UTC

Description Robert Scheck 2017-05-07 23:14:15 UTC 
Description of problem:
Currently when passing a compressed *.ova file to virt-v2v, the decompression
happens only using single-core consuming gzip, xz or bzip2. Can you please add
optional support for pigz, pxz and lbzip2, which would be preferred in case of
their availability? These tools allow to use multiple cores.

I know, some of the RPMs are currently not part of RHEL 7, thus I just would
like to see virt-v2v able to prefer multi-core tools in case they are already
installed. Alternatively allow the decompression tool to be explicitly passed 
using a new command line option.

Version-Release number of selected component (if applicable):
libguestfs-1.32.7-3.el7

How reproducible:
Everytime, see above and below.

Actual results:
Single-core decompression, only gzip, xz and bzip2 being used.

Expected results:
Multi-core decompression, additionally using pigz, pxz and lbzip2 before the
already implemented single-core tools.
Comment 2 Robert Scheck 2017-05-07 23:30:42 UTC 
Hackish workaround for gzip(1):

mkdir -p /tmp/v2v
ln -s /usr/bin/pigz /tmp/v2v/gzip
export PATH=/tmp/v2v:$PATH"
virt-v2v (+ regular options)
Comment 3 Richard W.M. Jones 2017-05-08 07:53:56 UTC 
I should say that we don't support bzip2, but likely no one is really
using that for OVAs.

I agree it would be nice to prefer pigz, pxz if available at runtime.
Comment 4 Richard W.M. Jones 2017-05-08 09:07:31 UTC 
https://www.redhat.com/archives/libguestfs/2017-May/msg00045.html
Comment 5 Richard W.M. Jones 2017-05-08 13:09:47 UTC 
Upstream commits:

6c0d520842d92fcdc022e598025a4e2b4a693b82
d2f2c3f4b249accbd73868303870f2bc16e19b17
5e55a9cd4ebb85e1b6905d731f97d5f216d653f2
0e63e59bf5533b5bcc455b4786c012fc4eeeada8

The first 3 are just code refactorings, the functionality is added
in the 4th commit.
Comment 6 Robert Scheck 2017-05-08 17:15:36 UTC 
Wow, thank you very much for implementing this quickly, Richard.

Cross-filed case 01845310 on the Red Hat customer portal to get this
backported into a future update for RHEL 7.x.
Comment 7 Richard W.M. Jones 2017-05-08 18:46:40 UTC 
(In reply to Robert Scheck from comment #6)
> Wow, thank you very much for implementing this quickly, Richard.
> 
> Cross-filed case 01845310 on the Red Hat customer portal to get this
> backported into a future update for RHEL 7.x.

By all means, but it's ACKed for RHEL 7.5 anyway so no other
action should be necessary (provided it is needed only for >= 7.5).
Comment 8 Pino Toscano 2017-08-25 12:54:15 UTC 
Note to self: almost all the refactoring commits were backported in 1.36.x (for 1.36.4), so only the actual commit with the feature is left to pick:
0e63e59bf5533b5bcc455b4786c012fc4eeeada8
Comment 11 mxie@redhat.com 2017-09-22 14:36:16 UTC 
Verify the bug with below builds:
virt-v2v-1.36.6-1.el7.x86_64
libguestfs-1.36.6-1.el7.x86_64
libvirt-3.7.0-2.el7.x86_64
qemu-kvm-rhev-2.9.0-16.el7_4.8.x86_64



Steps:
Prepare environment
Install pigz and pxz on v2v conversion server
# rpm -q pigz pxz
pigz-2.3.4-1.el7.x86_64
pxz-4.999.9-10.beta.20120930git.el7.x86_64

For compressing OVA files using pigz 
Scenario1:using virt-v2v to convert esx-rhel6.8-ova.tar.gz
1.1.Prepare ova folder and pack the folder with tar,get file:esx-rhel6.8-ova.tar

1.2.Compress esx-rhel6.8-ova.tar using pigz, get file:esx-rhel6.8-ova.tar.gz 
#pigz -k esx-rhel6.8-ova.tar

1.3.Using virt-v2v convert esx-rhel6.8-ova.tar.gz, conversion could be finished without error
# virt-v2v -i ova esx-rhel6.8-ova.tar.gz -os default -of raw
[   0.0] Opening the source -i ova esx-rhel6.8-ova.tar.gz
virt-v2v: warning: making OVA directory public readable to work around 
libvirt bug https://bugzilla.redhat.com/1045069
[   5.2] Creating an overlay to protect the source from being modified
[   8.0] Initializing the target -o libvirt -os default
[   8.0] Opening the overlay
[  11.0] Inspecting the overlay
[  20.4] Checking for sufficient free disk space in the guest
[  20.4] Estimating space required on target for each disk
[  20.4] Converting Red Hat Enterprise Linux Server release 6.8 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  63.2] Mapping filesystem data to avoid copying unused and blank areas
[  63.4] Closing the overlay
[  63.6] Checking if the guest needs BIOS or UEFI to boot
[  63.6] Assigning disks to buses
[  63.6] Copying disk 1/1 to /var/lib/libvirt/images/esx5.5-rhel6.8-x86_64-sda (raw)
    (100.00/100%)
[  78.5] Creating output metadata
Pool default refreshed

Domain esx5.5-rhel6.8-x86_64 defined from /tmp/v2vlibvirtcb0bd1.xml

[ 101.2] Finishing off

4.Check guest after finishing conversion,checkpoints are passed


Scenario2:using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.gz
2.1 Prepare ova folder whose vmdk file is compressed as gz file and pack the folder with tar, get file:esx-rhel6.8-vmdkgz.tar
# tar -tf esx-rhel6.8-vmdkgz.tar
esx-rhel6.8-vmdkgz/
esx-rhel6.8-vmdkgz/esx5.5-rhel6.8-x86_64.ovf
esx-rhel6.8-vmdkgz/esx5.5-rhel6.8-x86_64-disk1.vmdk.gz

2.2 Using pigz to compress this tar file, get file: esx-rhel6.8-vmdkgz.tar.gz
#pigz -k esx-rhel6.8-vmdkgz.tar

2.3 using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.gz, conversion could be finished without error


Scenario3:using virt-v2v to convert esx-rhel6.8.ova.gz
3.1 Prepare ova file and use pigz to compress it, get file:esx-rhel6.8.ova.gz
#pigz -k esx-rhel6.8.ova

3.2 using virt-v2v to convert esx-rhel6.8.ova.gz, conversion could be finished without error




For compressing OVA files using pxz
Scenario1:using virt-v2v to convert esx-rhel6.8-ova.tar.xz
1.1.Prepare file:esx-rhel6.8-ova.tar,compress it with pxz, get file:esx-rhel6.8-ova.tar.xz
# pxz -k -T4 esx-rhel6.8-ova.tar

1.2.Using virt-v2v convert esx-rhel6.8-ova.tar.xz,conversion could be finished without error


Scenario2:using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.xz
2.1 Prepare file:esx-rhel6.8-vmdkgz.tar, then use pxg to compress this file, get file: esx-rhel6.8-vmdkgz.tar.xz
#pxz -T4 esx-rhel6.8-vmdkgz.tar

2.2 using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.gz, conversion could be finished without error


Scenario3:using virt-v2v to convert esx-rhel6.8.ova.xz
3.1 Prepare ova file and use pxz to compress it, get file:esx-rhel6.8.ova.xz
#pxz esx-rhel6.8.ova

3.2 using virt-v2v to convertesx-rhel6.8.ova.xz, conversion could be finished without error


Result:
    Virt-v2v support multicore decompression for OVA files using pigz and pxz, so move this bug from ON_QA to VERIFIED
Comment 12 Richard W.M. Jones 2017-09-22 16:40:34 UTC 
I don't agree with the verification steps in comment 11 (or I don't
understand them).

pigz and pxz create ordinary gzip and xz files, just faster.

The issue is whether virt-v2v uses pigz/pxz to uncompress the
files, and you can only tell that by looking at the virt-v2v -v -x output.
Comment 13 mxie@redhat.com 2017-09-25 04:09:26 UTC 
Hi rjones,

   Thanks for your reminding, please help to check below verification steps

   1.Use virt-v2v to convert a ova file which is compressed into gz format, could found below command in v2v details log:
  ......
  pigz --help >/dev/null 2>&1
  pigz -c -d 'esx-rhel6.8-ova.tar.gz' | tar -xf - -C '/var/tmp/ova.ZoXjNy'
  ......
   
   2.Use virt-v2v to convert a ova file which is compressed into xz format, could found below command in v2v details log:
  ......
  pxz --help >/dev/null 2>&1
  pxz -c -d 'esx-rhel6.8-ova.tar.xz' | tar -xf - -C '/var/tmp/ova.FS9LCW' 
  ......
Comment 14 Richard W.M. Jones 2017-09-25 08:57:53 UTC 
Yes, it looks as if pigz and pxz are being used by virt-v2v, so
that being found in the log file would be sufficient to verify this bug.
Comment 17 Yaniv Kaul 2018-01-20 12:15:54 UTC 
I'm not sure I see the real benefit, as they don't really decompress in parallel.
From pigz man page:
Decompression  can't  be parallelized, at least not without specially prepared deflate streams for that purpose.  As a result, pigz uses a single thread (the main thread) for decompression, but will create three other threads for reading, writing, and check calculation, which can speed up decompression under some circumstances.
Comment 18 Robert Scheck 2018-01-20 13:16:55 UTC 
I wouldn't have proposed this, if I wouldn't have seen a real benefit on my
slow (non high-end) hardware here. So the benefit indeed might not be given
on any hardware though.
Comment 21 errata-xmlrpc 2018-04-10 09:15:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0677
Note You need to log in before you can comment on or make changes to this bug.