Description of problem:
Currently when passing a compressed *.ova file to virt-v2v, the decompression
happens only using single-core consuming gzip, xz or bzip2. Can you please add
optional support for pigz, pxz and lbzip2, which would be preferred in case of
their availability? These tools allow to use multiple cores.
I know, some of the RPMs are currently not part of RHEL 7, thus I just would
like to see virt-v2v able to prefer multi-core tools in case they are already
installed. Alternatively allow the decompression tool to be explicitly passed
using a new command line option.
Version-Release number of selected component (if applicable):
Everytime, see above and below.
Single-core decompression, only gzip, xz and bzip2 being used.
Multi-core decompression, additionally using pigz, pxz and lbzip2 before the
already implemented single-core tools.
Hackish workaround for gzip(1):
mkdir -p /tmp/v2v
ln -s /usr/bin/pigz /tmp/v2v/gzip
virt-v2v (+ regular options)
I should say that we don't support bzip2, but likely no one is really
using that for OVAs.
I agree it would be nice to prefer pigz, pxz if available at runtime.
The first 3 are just code refactorings, the functionality is added
in the 4th commit.
Wow, thank you very much for implementing this quickly, Richard.
Cross-filed case 01845310 on the Red Hat customer portal to get this
backported into a future update for RHEL 7.x.
(In reply to Robert Scheck from comment #6)
> Wow, thank you very much for implementing this quickly, Richard.
> Cross-filed case 01845310 on the Red Hat customer portal to get this
> backported into a future update for RHEL 7.x.
By all means, but it's ACKed for RHEL 7.5 anyway so no other
action should be necessary (provided it is needed only for >= 7.5).
Note to self: almost all the refactoring commits were backported in 1.36.x (for 1.36.4), so only the actual commit with the feature is left to pick:
Verify the bug with below builds:
Install pigz and pxz on v2v conversion server
# rpm -q pigz pxz
For compressing OVA files using pigz
Scenario1:using virt-v2v to convert esx-rhel6.8-ova.tar.gz
1.1.Prepare ova folder and pack the folder with tar,get file:esx-rhel6.8-ova.tar
1.2.Compress esx-rhel6.8-ova.tar using pigz, get file:esx-rhel6.8-ova.tar.gz
#pigz -k esx-rhel6.8-ova.tar
1.3.Using virt-v2v convert esx-rhel6.8-ova.tar.gz, conversion could be finished without error
# virt-v2v -i ova esx-rhel6.8-ova.tar.gz -os default -of raw
[ 0.0] Opening the source -i ova esx-rhel6.8-ova.tar.gz
virt-v2v: warning: making OVA directory public readable to work around
libvirt bug https://bugzilla.redhat.com/1045069
[ 5.2] Creating an overlay to protect the source from being modified
[ 8.0] Initializing the target -o libvirt -os default
[ 8.0] Opening the overlay
[ 11.0] Inspecting the overlay
[ 20.4] Checking for sufficient free disk space in the guest
[ 20.4] Estimating space required on target for each disk
[ 20.4] Converting Red Hat Enterprise Linux Server release 6.8 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 63.2] Mapping filesystem data to avoid copying unused and blank areas
[ 63.4] Closing the overlay
[ 63.6] Checking if the guest needs BIOS or UEFI to boot
[ 63.6] Assigning disks to buses
[ 63.6] Copying disk 1/1 to /var/lib/libvirt/images/esx5.5-rhel6.8-x86_64-sda (raw)
[ 78.5] Creating output metadata
Pool default refreshed
Domain esx5.5-rhel6.8-x86_64 defined from /tmp/v2vlibvirtcb0bd1.xml
[ 101.2] Finishing off
4.Check guest after finishing conversion,checkpoints are passed
Scenario2:using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.gz
2.1 Prepare ova folder whose vmdk file is compressed as gz file and pack the folder with tar, get file:esx-rhel6.8-vmdkgz.tar
# tar -tf esx-rhel6.8-vmdkgz.tar
2.2 Using pigz to compress this tar file, get file: esx-rhel6.8-vmdkgz.tar.gz
#pigz -k esx-rhel6.8-vmdkgz.tar
2.3 using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.gz, conversion could be finished without error
Scenario3:using virt-v2v to convert esx-rhel6.8.ova.gz
3.1 Prepare ova file and use pigz to compress it, get file:esx-rhel6.8.ova.gz
#pigz -k esx-rhel6.8.ova
3.2 using virt-v2v to convert esx-rhel6.8.ova.gz, conversion could be finished without error
For compressing OVA files using pxz
Scenario1:using virt-v2v to convert esx-rhel6.8-ova.tar.xz
1.1.Prepare file:esx-rhel6.8-ova.tar,compress it with pxz, get file:esx-rhel6.8-ova.tar.xz
# pxz -k -T4 esx-rhel6.8-ova.tar
1.2.Using virt-v2v convert esx-rhel6.8-ova.tar.xz,conversion could be finished without error
Scenario2:using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.xz
2.1 Prepare file:esx-rhel6.8-vmdkgz.tar, then use pxg to compress this file, get file: esx-rhel6.8-vmdkgz.tar.xz
#pxz -T4 esx-rhel6.8-vmdkgz.tar
2.2 using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.gz, conversion could be finished without error
Scenario3:using virt-v2v to convert esx-rhel6.8.ova.xz
3.1 Prepare ova file and use pxz to compress it, get file:esx-rhel6.8.ova.xz
3.2 using virt-v2v to convertesx-rhel6.8.ova.xz, conversion could be finished without error
Virt-v2v support multicore decompression for OVA files using pigz and pxz, so move this bug from ON_QA to VERIFIED
I don't agree with the verification steps in comment 11 (or I don't
pigz and pxz create ordinary gzip and xz files, just faster.
The issue is whether virt-v2v uses pigz/pxz to uncompress the
files, and you can only tell that by looking at the virt-v2v -v -x output.
Thanks for your reminding, please help to check below verification steps
1.Use virt-v2v to convert a ova file which is compressed into gz format, could found below command in v2v details log:
pigz --help >/dev/null 2>&1
pigz -c -d 'esx-rhel6.8-ova.tar.gz' | tar -xf - -C '/var/tmp/ova.ZoXjNy'
2.Use virt-v2v to convert a ova file which is compressed into xz format, could found below command in v2v details log:
pxz --help >/dev/null 2>&1
pxz -c -d 'esx-rhel6.8-ova.tar.xz' | tar -xf - -C '/var/tmp/ova.FS9LCW'
Yes, it looks as if pigz and pxz are being used by virt-v2v, so
that being found in the log file would be sufficient to verify this bug.
I'm not sure I see the real benefit, as they don't really decompress in parallel.
From pigz man page:
Decompression can't be parallelized, at least not without specially prepared deflate streams for that purpose. As a result, pigz uses a single thread (the main thread) for decompression, but will create three other threads for reading, writing, and check calculation, which can speed up decompression under some circumstances.
I wouldn't have proposed this, if I wouldn't have seen a real benefit on my
slow (non high-end) hardware here. So the benefit indeed might not be given
on any hardware though.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.