This is needed to allow reducing the size of virt-v2v (and virt-v2v-wrapper) logs.
The current version in RHEL AV 8.2 is 1.16.2. nbdkit 1.18 was released last week, here are the release notes: http://libguestfs.org/nbdkit-release-notes-1.18.1.html Some things that are interesting for RHEL: - The changes to the Python plugin would allow significant performance improvements in virt-v2v -o rhv-upload. - VDDK plugin is simpler to use. - Reduced verbosity would help to reduce the size of virt-v2v log files. - nbdkit-extentlist-filter could be useful (Martin K?) - Further fixes to improve stability of nbdkit-retry-filter.
Test bug with below builds: nbdkit-basic-filters-1.20.6-2.module+el8.3.0+7637+9ed826e0.x86_64 nbdkit-curl-plugin-1.20.6-2.module+el8.3.0+7637+9ed826e0.x86_64 nbdkit-ssh-plugin-1.20.6-2.module+el8.3.0+7637+9ed826e0.x86_64 nbdkit-1.20.6-2.module+el8.3.0+7637+9ed826e0.x86_64 nbdkit-server-1.20.6-2.module+el8.3.0+7637+9ed826e0.x86_64 nbdkit-basic-plugins-1.20.6-2.module+el8.3.0+7637+9ed826e0.x86_64 nbdkit-python-plugin-1.20.6-2.module+el8.3.0+7637+9ed826e0.x86_64 nbdkit-vddk-plugin-1.20.6-2.module+el8.3.0+7637+9ed826e0.x86_64 Steps: 1.Test if nbdkit is a vulnerable version, nbdkit package has no vulnerable # nbdkit -fv null --run 'sleep 1 >/dev/tcp/localhost/10809' 2>&1 | grep -q 'open readonly' && echo vulnerable || echo patched patched 2.Build nbdkit packages from src package, nbdkit rpm packages can be built from src package successfully 2.1 Download nbdkit src package and create .rpmmacros file $ cat .rpmmacros %_topdir %(echo $HOME)/rpmbuild %_smp_mflags -j5 2.2 Resolve the dependence problem before rebuilding,then build nbdkit rpm packages from src package # rpmbuild --rebuild nbdkit-1.20.6-2.module+el8.3.0+7637+9ed826e0.src.rpm PASS: test-connect PASS: test-newstyle PASS: test-oldstyle PASS: test-truncate4.sh PASS: test-rate.sh PASS: test-file PASS: test-null PASS: test-random PASS: test-split PASS: test-streaming PASS: test-tar.sh SKIP: test-old-plugins-i686-Linux-v1.0.0.sh SKIP: test-old-plugins-i686-Linux-v1.2.8-2-gd2d934df.sh SKIP: test-old-plugins-i686-Linux-v1.8.4-3-g11f5a90d.sh SKIP: test-old-plugins-i686-Linux-v1.12.8-2-g1e2ccc27.sh SKIP: test-old-plugins-i686-Linux-v1.18.4.sh PASS: test-retry-reopen-fail.sh PASS: test-rate-dynamic.sh PASS: test-curl-header-script PASS: test-curl-cookie-script PASS: test-old-plugins-x86_64-Linux-v1.0.0.sh PASS: test-old-plugins-x86_64-Linux-v1.2.8.sh PASS: test-old-plugins-x86_64-Linux-v1.8.4.sh PASS: test-old-plugins-x86_64-Linux-v1.12.8.sh PASS: test-old-plugins-x86_64-Linux-v1.18.2.sh PASS: test-delay PASS: test-readahead.sh PASS: test-cache-max-size.sh ============================================================================ Testsuite summary for nbdkit 1.20.6 ============================================================================ # TOTAL: 172 # PASS: 161 # SKIP: 11 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ ...... + exit 0 3.Test option '-D nbdkit.backend.controlpath=0 and -D nbdkit.backend.datapath=0, These two options can suppress some debug message successfully # nbdkit -f -v -U - ssh host=10.73.224.33 /var/lib/xen/images/xen-hvm-rhel7.7-x86_64.img --run 'qemu-img convert -p -f raw $nbd /tmp/out.img' --filter=retry -D nbdkit.backend.controlpath=0 -D nbdkit.backend.datapath=0 nbdkit: debug: TLS disabled: could not load TLS certificates nbdkit: debug: registering plugin /usr/lib64/nbdkit/plugins/nbdkit-ssh-plugin.so nbdkit: debug: registered plugin /usr/lib64/nbdkit/plugins/nbdkit-ssh-plugin.so (name ssh) nbdkit: debug: ssh: load nbdkit: debug: registering filter /usr/lib64/nbdkit/filters/nbdkit-retry-filter.so nbdkit: debug: registered filter /usr/lib64/nbdkit/filters/nbdkit-retry-filter.so (name retry) nbdkit: debug: retry: load nbdkit: debug: retry: config key=host, value=10.73.224.33 nbdkit: debug: ssh: config key=host, value=10.73.224.33 nbdkit: debug: retry: config key=path, value=/var/lib/xen/images/xen-hvm-rhel7.7-x86_64.img nbdkit: debug: ssh: config key=path, value=/var/lib/xen/images/xen-hvm-rhel7.7-x86_64.img nbdkit: debug: retry: config_complete nbdkit: debug: ssh: config_complete nbdkit: debug: using thread model: serialize_requests nbdkit: debug: retry: get_ready nbdkit: debug: ssh: get_ready nbdkit: debug: bound to unix socket /tmp/nbdkitfpH79v/socket nbdkit: debug: forked into background (new pid = 89225) nbdkit: debug: retry: after_fork nbdkit: debug: ssh: after_fork nbdkit: debug: accepted connection nbdkit: ssh[1]: debug: retry: preconnect nbdkit: ssh[1]: debug: ssh: preconnect nbdkit: ssh[1]: debug: newstyle negotiation: flags: global 0x3 nbdkit: ssh[1]: debug: newstyle negotiation: client flags: 0x3 nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_STRUCTURED_REPLY: client requested structured replies nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: client requested export '' nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: set count: 1 nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: set base:allocation nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: replying with base:allocation id 1 nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: reply complete nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_GO: client requested export '' nbdkit: ssh[1]: debug: authentication methods offered by the server: 0x26 password publickey gssapi-with-mic nbdkit: ssh[1]: debug: opened libssh handle nbdkit: ssh[1]: debug: newstyle negotiation: flags: export 0x8c9 nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_GO: ignoring NBD_INFO_* request 3 (NBD_INFO_BLOCK_SIZE) nbdkit: ssh[1]: debug: handshake complete, processing requests serially (100.00/100%) nbdkit: ssh[1]: debug: client sent NBD_CMD_DISC, closing connection nbdkit: ssh[1]: debug: reopens needed: 0 4.Testing nbdkit with annobin, there is no FAIL result in below output # annocheck -v --skip-cf-protection --skip-glibcxx-assertions --skip-glibcxx-assertions --skip-stack-realign --section-size=.gnu.build.attributes --ignore-gaps nbdkit-server-1.20.6-2.module+el8.3.0+7637+9ed826e0.x86_64.rpm --debug-rpm=nbdkit-server-debuginfo-1.20.6-2.module+el8.3.0+7637+9ed826e0.x86_64.rpm annocheck: Version 9.21. Hardened: ./usr/sbin/nbdkit: info: Set binary producer to gcc version 8. Section Size: ./usr/sbin/nbdkit: .gnu.build.attributes: 0x5e4 Hardened: ./usr/sbin/nbdkit: WARN: DW_FORM_GNU_strp_alt not yet handled. Hardened: ./usr/sbin/nbdkit: WARN: DW_FORM_GNU_strp_alt not yet handled. Hardened: ./usr/sbin/nbdkit: WARN: DW_FORM_GNU_strp_alt not yet handled. Hardened: ./usr/sbin/nbdkit: WARN: DW_FORM_GNU_strp_alt not yet handled. Hardened: ./usr/sbin/nbdkit: WARN: DW_FORM_GNU_strp_alt not yet handled. Hardened: ./usr/sbin/nbdkit: skip: Test for -Wl,-z-now. (Dynamic segment present, but no dynamic relocations found). Hardened: ./usr/sbin/nbdkit: skip: Branch protection. (Not an AArch64 binary). Hardened: ./usr/sbin/nbdkit: skip: Compiled with -fcf-protection=all (x86 only, gcc 8+ only). Hardened: ./usr/sbin/nbdkit: PASS: One dynamic section/segment found. Hardened: ./usr/sbin/nbdkit: PASS: Entry point instruction is ENDBR. Hardened: ./usr/sbin/nbdkit: PASS: Compiled with -D_FORTIFY_SOURCE=2. Hardened: ./usr/sbin/nbdkit: skip: Compiled with -D_GLIBCXX_ASSERTIONS. Hardened: ./usr/sbin/nbdkit: skip: Test for -Wl,-z,relro. (No dynamic relocations). Hardened: ./usr/sbin/nbdkit: PASS: Stack not executable. Hardened: ./usr/sbin/nbdkit: PASS: Compiled with sufficient optimization. Hardened: ./usr/sbin/nbdkit: PASS: Compiled with PIC/PIE. Hardened: ./usr/sbin/nbdkit: PASS: Compiled as a position independent binary. Hardened: ./usr/sbin/nbdkit: PASS: Good GNU Property note. Hardened: ./usr/sbin/nbdkit: PASS: DT_RPATH/DT_RUNPATH absent or rooted at /usr. Hardened: ./usr/sbin/nbdkit: PASS: No RWX segments found. Hardened: ./usr/sbin/nbdkit: PASS: Consistent use of the -fshort-enum option. Hardened: ./usr/sbin/nbdkit: PASS: Compiled with -fstack-clash-protection. Hardened: ./usr/sbin/nbdkit: PASS: Compiled with sufficient stack protection. Hardened: ./usr/sbin/nbdkit: skip: Compiled with -mstackrealign (i686 only). Hardened: ./usr/sbin/nbdkit: PASS: No text relocations found. Hardened: ./usr/sbin/nbdkit: PASS: No thread cancellation problems. Hardened: ./usr/sbin/nbdkit: PASS: Compiled with either -Wall and/or -Wformat-security. Hardened: ./usr/sbin/nbdkit: PASS: GOT/PLT relocations are read only. Hardened: ./usr/sbin/nbdkit: PASS. Section Size: Section '.gnu.build.attributes' found in 1 files, total size: 0x5e4
Verify bug with below builds: nbdkit-basic-plugins-1.22.0-1.module+el8.3.0+7898+13f907d5.x86_64 nbdkit-python-plugin-1.22.0-1.module+el8.3.0+7898+13f907d5.x86_64 nbdkit-curl-plugin-1.22.0-1.module+el8.3.0+7898+13f907d5.x86_64 nbdkit-vddk-plugin-1.22.0-1.module+el8.3.0+7898+13f907d5.x86_64 nbdkit-ssh-plugin-1.22.0-1.module+el8.3.0+7898+13f907d5.x86_64 nbdkit-basic-filters-1.22.0-1.module+el8.3.0+7898+13f907d5.x86_64 nbdkit-server-1.22.0-1.module+el8.3.0+7898+13f907d5.x86_64 nbdkit-1.22.0-1.module+el8.3.0+7898+13f907d5.x86_64 Steps: 1.Test if nbdkit is a vulnerable version, nbdkit package has no vulnerable # nbdkit -fv null --run 'sleep 1 >/dev/tcp/localhost/10809' 2>&1 | grep -q 'open readonly' && echo vulnerable || echo patched patched 2.Build nbdkit packages from src package, nbdkit rpm packages can be built from src package successfully 2.1 Download nbdkit src package and create .rpmmacros file $ cat .rpmmacros %_topdir %(echo $HOME)/rpmbuild %_smp_mflags -j5 2.2 Resolve the dependence problem before rebuilding,then build nbdkit rpm packages from src package # rpmbuild --rebuild nbdkit-1.22.0-1.module+el8.3.0+7898+13f907d5.src.rpm ..... ============================================================================ Testsuite summary for nbdkit 1.22.0 ============================================================================ # TOTAL: 202 # PASS: 192 # SKIP: 10 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ ...... + exit 0 3.Test option '-D nbdkit.backend.controlpath=0 and -D nbdkit.backend.datapath=0, these two options can suppress some debug message successfully # nbdkit -f -v -U - ssh host=10.73.224.33 /var/lib/xen/images/xen-hvm-rhel7.7-x86_64.img --run 'qemu-img convert -p -f raw $nbd /tmp/out.img' --filter=retry -D nbdkit.backend.controlpath=0 -D nbdkit.backend.datapath=0 nbdkit: debug: TLS disabled: could not load TLS certificates nbdkit: debug: registering plugin /usr/lib64/nbdkit/plugins/nbdkit-ssh-plugin.so nbdkit: debug: registered plugin /usr/lib64/nbdkit/plugins/nbdkit-ssh-plugin.so (name ssh) nbdkit: debug: ssh: load nbdkit: debug: registering filter /usr/lib64/nbdkit/filters/nbdkit-retry-filter.so nbdkit: debug: registered filter /usr/lib64/nbdkit/filters/nbdkit-retry-filter.so (name retry) nbdkit: debug: retry: load nbdkit: debug: retry: config key=host, value=10.73.224.33 nbdkit: debug: ssh: config key=host, value=10.73.224.33 nbdkit: debug: retry: config key=path, value=/var/lib/xen/images/xen-hvm-rhel7.7-x86_64.img nbdkit: debug: ssh: config key=path, value=/var/lib/xen/images/xen-hvm-rhel7.7-x86_64.img nbdkit: debug: retry: config_complete nbdkit: debug: ssh: config_complete nbdkit: debug: using thread model: serialize_requests nbdkit: debug: retry: get_ready nbdkit: debug: ssh: get_ready nbdkit: debug: bound to unix socket /tmp/nbdkitfpH79v/socket nbdkit: debug: forked into background (new pid = 89225) nbdkit: debug: retry: after_fork nbdkit: debug: ssh: after_fork nbdkit: debug: accepted connection nbdkit: ssh[1]: debug: retry: preconnect nbdkit: ssh[1]: debug: ssh: preconnect nbdkit: ssh[1]: debug: newstyle negotiation: flags: global 0x3 nbdkit: ssh[1]: debug: newstyle negotiation: client flags: 0x3 nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_STRUCTURED_REPLY: client requested structured replies nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: client requested export '' nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: set count: 1 nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: set base:allocation nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: replying with base:allocation id 1 nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_SET_META_CONTEXT: reply complete nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_GO: client requested export '' nbdkit: ssh[1]: debug: authentication methods offered by the server: 0x26 password publickey gssapi-with-mic nbdkit: ssh[1]: debug: opened libssh handle nbdkit: ssh[1]: debug: newstyle negotiation: flags: export 0x8c9 nbdkit: ssh[1]: debug: newstyle negotiation: NBD_OPT_GO: ignoring NBD_INFO_* request 3 (NBD_INFO_BLOCK_SIZE) nbdkit: ssh[1]: debug: handshake complete, processing requests serially (100.00/100%) nbdkit: ssh[1]: debug: client sent NBD_CMD_DISC, closing connection nbdkit: ssh[1]: debug: reopens needed: 0 4.Testing nbdkit with annobin, there is no FAIL result in below output # annocheck -v --skip-cf-protection --skip-glibcxx-assertions --skip-glibcxx-assertions --skip-stack-realign --section-size=.gnu.build.attributes --ignore-gaps nbdkit-server-1.22.0-1.module+el8.3.0+7898+13f907d5.x86_64.rpm --debug-rpm=nbdkit-server-debuginfo-1.22.0-1.module+el8.3.0+7898+13f907d5.x86_64.rpm annocheck: Version 9.21. Hardened: ./usr/sbin/nbdkit: info: Set binary producer to gcc version 8. Section Size: ./usr/sbin/nbdkit: .gnu.build.attributes: 0x5e4 Hardened: ./usr/sbin/nbdkit: WARN: DW_FORM_GNU_strp_alt not yet handled. Hardened: ./usr/sbin/nbdkit: WARN: DW_FORM_GNU_strp_alt not yet handled. Hardened: ./usr/sbin/nbdkit: WARN: DW_FORM_GNU_strp_alt not yet handled. Hardened: ./usr/sbin/nbdkit: WARN: DW_FORM_GNU_strp_alt not yet handled. Hardened: ./usr/sbin/nbdkit: WARN: DW_FORM_GNU_strp_alt not yet handled. Hardened: ./usr/sbin/nbdkit: skip: Test for -Wl,-z-now. (Dynamic segment present, but no dynamic relocations found). Hardened: ./usr/sbin/nbdkit: skip: Branch protection. (Not an AArch64 binary). Hardened: ./usr/sbin/nbdkit: skip: Compiled with -fcf-protection=all (x86 only, gcc 8+ only). Hardened: ./usr/sbin/nbdkit: PASS: One dynamic section/segment found. Hardened: ./usr/sbin/nbdkit: PASS: Entry point instruction is ENDBR. Hardened: ./usr/sbin/nbdkit: PASS: Compiled with -D_FORTIFY_SOURCE=2. Hardened: ./usr/sbin/nbdkit: skip: Compiled with -D_GLIBCXX_ASSERTIONS. Hardened: ./usr/sbin/nbdkit: skip: Test for -Wl,-z,relro. (No dynamic relocations). Hardened: ./usr/sbin/nbdkit: PASS: Stack not executable. Hardened: ./usr/sbin/nbdkit: PASS: Compiled with sufficient optimization. Hardened: ./usr/sbin/nbdkit: PASS: Compiled with PIC/PIE. Hardened: ./usr/sbin/nbdkit: PASS: Compiled as a position independent binary. Hardened: ./usr/sbin/nbdkit: PASS: Good GNU Property note. Hardened: ./usr/sbin/nbdkit: PASS: DT_RPATH/DT_RUNPATH absent or rooted at /usr. Hardened: ./usr/sbin/nbdkit: PASS: No RWX segments found. Hardened: ./usr/sbin/nbdkit: PASS: Consistent use of the -fshort-enum option. Hardened: ./usr/sbin/nbdkit: PASS: Compiled with -fstack-clash-protection. Hardened: ./usr/sbin/nbdkit: PASS: Compiled with sufficient stack protection. Hardened: ./usr/sbin/nbdkit: skip: Compiled with -mstackrealign (i686 only). Hardened: ./usr/sbin/nbdkit: PASS: No text relocations found. Hardened: ./usr/sbin/nbdkit: PASS: No thread cancellation problems. Hardened: ./usr/sbin/nbdkit: PASS: Compiled with either -Wall and/or -Wformat-security. Hardened: ./usr/sbin/nbdkit: PASS: GOT/PLT relocations are read only. Hardened: ./usr/sbin/nbdkit: PASS. Section Size: Section '.gnu.build.attributes' found in 1 files, total size: 0x5e4 5.Use nbdkit to access VMware to convert guest, the conversion can finish without error #nbdkit -rfv -U - --exportname --filter=cacheextents --filter=retry vddk server=10.73.198.169 user=root password=+/home/passwd vm=moref=vm-2408 file='[esx7.0-matrix] esx7.0-win2019-x86_64/esx7.0-win2019-x86_64.vmdk' libdir=/home/vmware-vix-disklib-distrib thumbprint='B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78' transports=nbd --run 'qemu-img convert $nbd /var/tmp/out ..... nbdkit: debug: VDDK call: VixDiskLib_Exit () nbdkit: debug: VDDK_PhoneHome: VDDK PhoneHome succeed to exit nbdkit: debug: VixDiskLib: VixDiskLib_Exit: Unmatched Init calls so far: 1. nbdkit: debug: OBJLIB-LIB: ObjLib cleanup done. nbdkit: debug: retry: unload filter 6. Convert guest from VMware to rhv by v2v, the conversion can finish without error # virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -ip /home/passwd -o rhv-upload -oo rhv-direct -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt esx7.0-rhel8.2-x86_64 [ 1.2] Opening the source -i libvirt -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 esx7.0-rhel8.2-x86_64 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 [ 2.9] Creating an overlay to protect the source from being modified [ 3.8] Opening the overlay [ 17.8] Inspecting the overlay [ 42.2] Checking for sufficient free disk space in the guest [ 42.2] Estimating space required on target for each disk [ 42.2] Converting Red Hat Enterprise Linux 8.2 (Ootpa) to run on KVM virt-v2v: This guest has virtio drivers installed. [ 190.7] Mapping filesystem data to avoid copying unused and blank areas [ 191.8] Closing the overlay [ 192.2] Assigning disks to buses [ 192.2] Checking if the guest needs BIOS or UEFI to boot [ 192.2] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data [ 193.7] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.gBOEUN/nbdkit4.sock", "file.export": "/" } (raw) (100.00/100%) [ 669.5] Creating output metadata [ 671.6] Finishing off Result: No problem was found，so move the bug from ON_QA to VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5137