Hide Forgot
Description of problem: Currently when passing a compressed *.ova file to virt-v2v, the decompression happens only using single-core consuming gzip, xz or bzip2. Can you please add optional support for pigz, pxz and lbzip2, which would be preferred in case of their availability? These tools allow to use multiple cores. I know, some of the RPMs are currently not part of RHEL 7, thus I just would like to see virt-v2v able to prefer multi-core tools in case they are already installed. Alternatively allow the decompression tool to be explicitly passed using a new command line option. Version-Release number of selected component (if applicable): libguestfs-1.32.7-3.el7 How reproducible: Everytime, see above and below. Actual results: Single-core decompression, only gzip, xz and bzip2 being used. Expected results: Multi-core decompression, additionally using pigz, pxz and lbzip2 before the already implemented single-core tools.
Hackish workaround for gzip(1): mkdir -p /tmp/v2v ln -s /usr/bin/pigz /tmp/v2v/gzip export PATH=/tmp/v2v:$PATH" virt-v2v (+ regular options)
I should say that we don't support bzip2, but likely no one is really using that for OVAs. I agree it would be nice to prefer pigz, pxz if available at runtime.
https://www.redhat.com/archives/libguestfs/2017-May/msg00045.html
Upstream commits: 6c0d520842d92fcdc022e598025a4e2b4a693b82 d2f2c3f4b249accbd73868303870f2bc16e19b17 5e55a9cd4ebb85e1b6905d731f97d5f216d653f2 0e63e59bf5533b5bcc455b4786c012fc4eeeada8 The first 3 are just code refactorings, the functionality is added in the 4th commit.
Wow, thank you very much for implementing this quickly, Richard. Cross-filed case 01845310 on the Red Hat customer portal to get this backported into a future update for RHEL 7.x.
(In reply to Robert Scheck from comment #6) > Wow, thank you very much for implementing this quickly, Richard. > > Cross-filed case 01845310 on the Red Hat customer portal to get this > backported into a future update for RHEL 7.x. By all means, but it's ACKed for RHEL 7.5 anyway so no other action should be necessary (provided it is needed only for >= 7.5).
Note to self: almost all the refactoring commits were backported in 1.36.x (for 1.36.4), so only the actual commit with the feature is left to pick: 0e63e59bf5533b5bcc455b4786c012fc4eeeada8
Verify the bug with below builds: virt-v2v-1.36.6-1.el7.x86_64 libguestfs-1.36.6-1.el7.x86_64 libvirt-3.7.0-2.el7.x86_64 qemu-kvm-rhev-2.9.0-16.el7_4.8.x86_64 Steps: Prepare environment Install pigz and pxz on v2v conversion server # rpm -q pigz pxz pigz-2.3.4-1.el7.x86_64 pxz-4.999.9-10.beta.20120930git.el7.x86_64 For compressing OVA files using pigz Scenario1:using virt-v2v to convert esx-rhel6.8-ova.tar.gz 1.1.Prepare ova folder and pack the folder with tar,get file:esx-rhel6.8-ova.tar 1.2.Compress esx-rhel6.8-ova.tar using pigz, get file:esx-rhel6.8-ova.tar.gz #pigz -k esx-rhel6.8-ova.tar 1.3.Using virt-v2v convert esx-rhel6.8-ova.tar.gz, conversion could be finished without error # virt-v2v -i ova esx-rhel6.8-ova.tar.gz -os default -of raw [ 0.0] Opening the source -i ova esx-rhel6.8-ova.tar.gz virt-v2v: warning: making OVA directory public readable to work around libvirt bug https://bugzilla.redhat.com/1045069 [ 5.2] Creating an overlay to protect the source from being modified [ 8.0] Initializing the target -o libvirt -os default [ 8.0] Opening the overlay [ 11.0] Inspecting the overlay [ 20.4] Checking for sufficient free disk space in the guest [ 20.4] Estimating space required on target for each disk [ 20.4] Converting Red Hat Enterprise Linux Server release 6.8 (Santiago) to run on KVM virt-v2v: This guest has virtio drivers installed. [ 63.2] Mapping filesystem data to avoid copying unused and blank areas [ 63.4] Closing the overlay [ 63.6] Checking if the guest needs BIOS or UEFI to boot [ 63.6] Assigning disks to buses [ 63.6] Copying disk 1/1 to /var/lib/libvirt/images/esx5.5-rhel6.8-x86_64-sda (raw) (100.00/100%) [ 78.5] Creating output metadata Pool default refreshed Domain esx5.5-rhel6.8-x86_64 defined from /tmp/v2vlibvirtcb0bd1.xml [ 101.2] Finishing off 4.Check guest after finishing conversion,checkpoints are passed Scenario2:using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.gz 2.1 Prepare ova folder whose vmdk file is compressed as gz file and pack the folder with tar, get file:esx-rhel6.8-vmdkgz.tar # tar -tf esx-rhel6.8-vmdkgz.tar esx-rhel6.8-vmdkgz/ esx-rhel6.8-vmdkgz/esx5.5-rhel6.8-x86_64.ovf esx-rhel6.8-vmdkgz/esx5.5-rhel6.8-x86_64-disk1.vmdk.gz 2.2 Using pigz to compress this tar file, get file: esx-rhel6.8-vmdkgz.tar.gz #pigz -k esx-rhel6.8-vmdkgz.tar 2.3 using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.gz, conversion could be finished without error Scenario3:using virt-v2v to convert esx-rhel6.8.ova.gz 3.1 Prepare ova file and use pigz to compress it, get file:esx-rhel6.8.ova.gz #pigz -k esx-rhel6.8.ova 3.2 using virt-v2v to convert esx-rhel6.8.ova.gz, conversion could be finished without error For compressing OVA files using pxz Scenario1:using virt-v2v to convert esx-rhel6.8-ova.tar.xz 1.1.Prepare file:esx-rhel6.8-ova.tar,compress it with pxz, get file:esx-rhel6.8-ova.tar.xz # pxz -k -T4 esx-rhel6.8-ova.tar 1.2.Using virt-v2v convert esx-rhel6.8-ova.tar.xz,conversion could be finished without error Scenario2:using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.xz 2.1 Prepare file:esx-rhel6.8-vmdkgz.tar, then use pxg to compress this file, get file: esx-rhel6.8-vmdkgz.tar.xz #pxz -T4 esx-rhel6.8-vmdkgz.tar 2.2 using virt-v2v to convert esx-rhel6.8-vmdkgz.tar.gz, conversion could be finished without error Scenario3:using virt-v2v to convert esx-rhel6.8.ova.xz 3.1 Prepare ova file and use pxz to compress it, get file:esx-rhel6.8.ova.xz #pxz esx-rhel6.8.ova 3.2 using virt-v2v to convertesx-rhel6.8.ova.xz, conversion could be finished without error Result: Virt-v2v support multicore decompression for OVA files using pigz and pxz, so move this bug from ON_QA to VERIFIED
I don't agree with the verification steps in comment 11 (or I don't understand them). pigz and pxz create ordinary gzip and xz files, just faster. The issue is whether virt-v2v uses pigz/pxz to uncompress the files, and you can only tell that by looking at the virt-v2v -v -x output.
Hi rjones, Thanks for your reminding, please help to check below verification steps 1.Use virt-v2v to convert a ova file which is compressed into gz format, could found below command in v2v details log: ...... pigz --help >/dev/null 2>&1 pigz -c -d 'esx-rhel6.8-ova.tar.gz' | tar -xf - -C '/var/tmp/ova.ZoXjNy' ...... 2.Use virt-v2v to convert a ova file which is compressed into xz format, could found below command in v2v details log: ...... pxz --help >/dev/null 2>&1 pxz -c -d 'esx-rhel6.8-ova.tar.xz' | tar -xf - -C '/var/tmp/ova.FS9LCW' ......
Yes, it looks as if pigz and pxz are being used by virt-v2v, so that being found in the log file would be sufficient to verify this bug.
I'm not sure I see the real benefit, as they don't really decompress in parallel. From pigz man page: Decompression can't be parallelized, at least not without specially prepared deflate streams for that purpose. As a result, pigz uses a single thread (the main thread) for decompression, but will create three other threads for reading, writing, and check calculation, which can speed up decompression under some circumstances.
I wouldn't have proposed this, if I wouldn't have seen a real benefit on my slow (non high-end) hardware here. So the benefit indeed might not be given on any hardware though.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0677