Bug 1448739

Hackish workaround for gzip(1):

mkdir -p /tmp/v2v
ln -s /usr/bin/pigz /tmp/v2v/gzip
export PATH=/tmp/v2v:$PATH"
virt-v2v (+ regular options)

I should say that we don't support bzip2, but likely no one is really
using that for OVAs.

I agree it would be nice to prefer pigz, pxz if available at runtime.

https://www.redhat.com/archives/libguestfs/2017-May/msg00045.html

Upstream commits:

6c0d520842d92fcdc022e598025a4e2b4a693b82
d2f2c3f4b249accbd73868303870f2bc16e19b17
5e55a9cd4ebb85e1b6905d731f97d5f216d653f2
0e63e59bf5533b5bcc455b4786c012fc4eeeada8

The first 3 are just code refactorings, the functionality is added
in the 4th commit.

Wow, thank you very much for implementing this quickly, Richard.

Cross-filed case 01845310 on the Red Hat customer portal to get this
backported into a future update for RHEL 7.x.

(In reply to Robert Scheck from comment #6)
> Wow, thank you very much for implementing this quickly, Richard.
> 
> Cross-filed case 01845310 on the Red Hat customer portal to get this
> backported into a future update for RHEL 7.x.

By all means, but it's ACKed for RHEL 7.5 anyway so no other
action should be necessary (provided it is needed only for >= 7.5).

Note to self: almost all the refactoring commits were backported in 1.36.x (for 1.36.4), so only the actual commit with the feature is left to pick:
0e63e59bf5533b5bcc455b4786c012fc4eeeada8

Verify the bug with below builds:
virt-v2v-1.36.6-1.el7.x86_64
libguestfs-1.36.6-1.el7.x86_64
libvirt-3.7.0-2.el7.x86_64
qemu-kvm-rhev-2.9.0-16.el7_4.8.x86_64



Steps:
Prepare environment
Install pigz and pxz on v2v conversion server
# rpm -q pigz pxz
pigz-2.3.4-1.el7.x86_64
pxz-4.999.9-10.beta.20120930git.el7.x86_64

For compressing OVA files using pigz 
Scenario1:using virt-v2v to convert esx-rhel6.8-ova.tar.gz
1.1.Prepare ova folder and pack the folder with tar,get file:esx-rhel6.8-ova.tar

1.2.Compress esx-rhel6.8-ova.tar using pigz, get file:esx-rhel6.8-ova.tar.gz 
#pigz -k esx-rhel6.8-ova.tar

1.3.Using virt-v2v convert esx-rhel6.8-ova.tar.gz, conversion could be finished without error
# virt-v2v -i ova esx-rhel6.8-ova.tar.gz -os default -of raw
[   0.0] Opening the source -i ova esx-rhel6.8-ova.tar.gz
virt-v2v: warning: making OVA directory public readable to work around 
libvirt bug https://bugzilla.redhat.com/1045069
[   5.2] Creating an overlay to protect the source from being modified
[   8.0] Initializing the target -o libvirt -os default
[   8.0] Opening the overlay
[  11.0] Inspecting the overlay
[  20.4] Checking for sufficient free disk space in the guest
[  20.4] Estimating space required on target for each disk
[  20.4] Converting Red Hat Enterprise Linux Server release 6.8 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  63.2] Mapping filesystem data to avoid copying unused and blank areas
[  63.4] Closing the overlay
[  63.6] Checking if the guest needs BIOS or UEFI to boot
[  63.6] Assigning disks to buses
[  63.6] Copying disk 1/1 to /var/lib/libvirt/images/esx5.5-rhel6.8-x86_64-sda (raw)
    (100.00/100%)
[  78.5] Creating output metadata
Pool default refreshed

Domain esx5.5-rhel6.8-x86_64 defined from /tmp/v2vlibvirtcb0bd1.xml

[ 101.2] Finishing off

4.Check guest after finishing conversion,checkpoints are passed


Scenario2:using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.gz
2.1 Prepare ova folder whose vmdk file is compressed as gz file and pack the folder with tar, get file:esx-rhel6.8-vmdkgz.tar
# tar -tf esx-rhel6.8-vmdkgz.tar
esx-rhel6.8-vmdkgz/
esx-rhel6.8-vmdkgz/esx5.5-rhel6.8-x86_64.ovf
esx-rhel6.8-vmdkgz/esx5.5-rhel6.8-x86_64-disk1.vmdk.gz

2.2 Using pigz to compress this tar file, get file: esx-rhel6.8-vmdkgz.tar.gz
#pigz -k esx-rhel6.8-vmdkgz.tar

2.3 using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.gz, conversion could be finished without error


Scenario3:using virt-v2v to convert esx-rhel6.8.ova.gz
3.1 Prepare ova file and use pigz to compress it, get file:esx-rhel6.8.ova.gz
#pigz -k esx-rhel6.8.ova

3.2 using virt-v2v to convert esx-rhel6.8.ova.gz, conversion could be finished without error




For compressing OVA files using pxz
Scenario1:using virt-v2v to convert esx-rhel6.8-ova.tar.xz
1.1.Prepare file:esx-rhel6.8-ova.tar,compress it with pxz, get file:esx-rhel6.8-ova.tar.xz
# pxz -k -T4 esx-rhel6.8-ova.tar

1.2.Using virt-v2v convert esx-rhel6.8-ova.tar.xz,conversion could be finished without error


Scenario2:using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.xz
2.1 Prepare file:esx-rhel6.8-vmdkgz.tar, then use pxg to compress this file, get file: esx-rhel6.8-vmdkgz.tar.xz
#pxz -T4 esx-rhel6.8-vmdkgz.tar

2.2 using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.gz, conversion could be finished without error


Scenario3:using virt-v2v to convert esx-rhel6.8.ova.xz
3.1 Prepare ova file and use pxz to compress it, get file:esx-rhel6.8.ova.xz
#pxz esx-rhel6.8.ova

3.2 using virt-v2v to convertesx-rhel6.8.ova.xz, conversion could be finished without error


Result:
    Virt-v2v support multicore decompression for OVA files using pigz and pxz, so move this bug from ON_QA to VERIFIED

I don't agree with the verification steps in comment 11 (or I don't
understand them).

pigz and pxz create ordinary gzip and xz files, just faster.

The issue is whether virt-v2v uses pigz/pxz to uncompress the
files, and you can only tell that by looking at the virt-v2v -v -x output.

Hi rjones,

   Thanks for your reminding, please help to check below verification steps

   1.Use virt-v2v to convert a ova file which is compressed into gz format, could found below command in v2v details log:
  ......
  pigz --help >/dev/null 2>&1
  pigz -c -d 'esx-rhel6.8-ova.tar.gz' | tar -xf - -C '/var/tmp/ova.ZoXjNy'
  ......
   
   2.Use virt-v2v to convert a ova file which is compressed into xz format, could found below command in v2v details log:
  ......
  pxz --help >/dev/null 2>&1
  pxz -c -d 'esx-rhel6.8-ova.tar.xz' | tar -xf - -C '/var/tmp/ova.FS9LCW' 
  ......

Yes, it looks as if pigz and pxz are being used by virt-v2v, so
that being found in the log file would be sufficient to verify this bug.

I'm not sure I see the real benefit, as they don't really decompress in parallel.
From pigz man page:
Decompression  can't  be parallelized, at least not without specially prepared deflate streams for that purpose.  As a result, pigz uses a single thread (the main thread) for decompression, but will create three other threads for reading, writing, and check calculation, which can speed up decompression under some circumstances.

I wouldn't have proposed this, if I wouldn't have seen a real benefit on my
slow (non high-end) hardware here. So the benefit indeed might not be given
on any hardware though.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0677