Bug 1513884

Summary: [RFE]Should update some vddk info in v2v man page
Product: Red Hat Enterprise Linux 7 Reporter: mxie <mxie>
Component: libguestfsAssignee: Richard W.M. Jones <rjones>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.5CC: juzhou, kuwei, lmiksik, mtessun, mzhan, ptoscano, rjones, tzheng, xiaodwan
Target Milestone: rcKeywords: FutureFeature
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: V2V
Fixed In Version: libguestfs-1.36.10-5.el7 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 09:20:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
nbd1.1.25_vddk6.0.log none

Description mxie@redhat.com 2017-11-16 08:31:51 UTC 
Description of problem:
[RFE]Should update some vddk info in v2v man page

Version-Release number of selected component (if applicable):
virt-v2v-1.36.10-2.el7.x86_64
libguestfs-1.36.10-2.el7.x86_6

How reproducible:
100%

Steps to Reproduce:

1.The link of nbdkit-vddk-plugin(1) in v2v man page which was given is http://man.he.net/man1/nbdkit-vddk-plugin, the content of this man page is hard to understand for customers. I found anther man page about nbdkit-vddk-plugin is more clear (https://github.com/libguestfs/nbdkit/blob/master/plugins/vddk/nbdkit-vddk-plugin.pod),it will be better if using the latter one to link nbdkit-vddk-plugin(1)


2.It is better that changing "nbdkit ≥ 1.1.14 is required" to "nbdkit ≥ 1.1.16 is required"  in below part due to the problem https://bugzilla.redhat.com/show_bug.cgi?id=1477912#c9
          
           You must also compile nbdkit, enabling the VDDK plugin.  At least
           nbdkit ≥ 1.1.14 is required, but it is usually best to compile from
           the git tree.

3.Should change "export PATH=/path/to/nbdkit:$PATH" to "export PATH=/path/to/nbdkit-1.1.x:$PATH" in below part, This change has been done at http://libguestfs.org/virt-v2v.1.html#input-from-vddk, but the v2v man didn't update this

           You can run nbdkit from its source directory without needing to
           install it.  Set $PATH to include the nbdkit top build directory
           (the directory containing a shell script called nbdkit)

            export PATH=/path/to/nbdkit:$PATH

4.Can't complile nbdkit(both 1.1.14 and 1.1.16) with vddk 6.0(both -6.0.2-3566099 and 6.0.0-2498720 ), it will pop up below error:
      ......
      vddk.c: In function ‘vddk_open’:
      vddk.c:301:11: error: ‘VixDiskLibConnectParams’ has no member named ‘vimApiVer’
      params.vimApiVer = (char *) vim_api_ver;
           ^
      vddk.c:303:11: error: ‘VixDiskLibConnectParams’ has no member named ‘nfcHostPort’
      params.nfcHostPort = nfc_host_port;
           ^
      ......

      
    If the problem "nbdkit can't compile with vddk6.0" is a bug, it is necessary adding VDDK >= 6.5  version in v2v man page in case customer compile nbdkit with vddk 6.0

5.I think --vddk is a required option because v2v will pop up error if there is no -vddk option when convert a guest from vmware with vddk library , but v2v man said only --vddk-thumbprint is required, the others are optional.should correct this description in v2v man page

# virt-v2v -ic vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/?no_verify=1 --vddk-thumbprint F3:BD:CC:18:78:A8:DB:66:9F:0C:F0:BF:A6:1A:4D:A0:84:C7:86:26 esx6.5-rhel7.3-x86_64 --password-file /tmp/passwd -of rawvirt-v2v: error: ‘--vddk-*’ options should only be used when conversion 
via the nbdkit VDDK plugin has been enabled, ie. using ‘--vddk’.

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]



Actual results:
As above description

Expected results:
Should update some vddk info in v2v man page

Additional info
Comment 2 Pino Toscano 2017-12-04 10:12:41 UTC 
(In reply to mxie from comment #0)
> 1.The link of nbdkit-vddk-plugin(1) in v2v man page which was given is
> http://man.he.net/man1/nbdkit-vddk-plugin, the content of this man page is
> hard to understand for customers. I found anther man page about
> nbdkit-vddk-plugin is more clear
> (https://github.com/libguestfs/nbdkit/blob/master/plugins/vddk/nbdkit-vddk-
> plugin.pod),it will be better if using the latter one to link
> nbdkit-vddk-plugin(1)

This was fixed upstream with
https://github.com/libguestfs/libguestfs/commit/441fa82ee0d878ed63b1c213d8197c1f0ffde2a0
which is in libguestfs >= 1.37.32.

However, the nbdkit man pages are not present on libguestfs.org yet.

> 2.It is better that changing "nbdkit ≥ 1.1.14 is required" to "nbdkit ≥
> 1.1.16 is required"  in below part due to the problem
> https://bugzilla.redhat.com/show_bug.cgi?id=1477912#c9

This was fixed upstream with
https://github.com/libguestfs/libguestfs/commit/415e9d82e901647438f68e4748720d88153778aa
which is in libguestfs >= 1.37.32.

> 3.Should change "export PATH=/path/to/nbdkit:$PATH" to "export
> PATH=/path/to/nbdkit-1.1.x:$PATH" in below part, This change has been done
> at http://libguestfs.org/virt-v2v.1.html#input-from-vddk, but the v2v man
> didn't update this

This was fixed upstream with
https://github.com/libguestfs/libguestfs/commit/d81a2ee185599c23a128ab65be9d48b415abf461
which is in libguestfs >= 1.37.30.

> 4.Can't complile nbdkit(both 1.1.14 and 1.1.16) with vddk 6.0(both
> -6.0.2-3566099 and 6.0.0-2498720 ), it will pop up below error:

This was fixed upstream with
https://github.com/libguestfs/libguestfs/commit/12d87361a4623587c53196e70157448158c02c7f
which is in libguestfs >= 1.37.32.

Basically VDDK >= 6.5 is the required version.

> 5.I think --vddk is a required option because v2v will pop up error if there
> is no -vddk option when convert a guest from vmware with vddk library , but
> v2v man said only --vddk-thumbprint is required, the others are
> optional.should correct this description in v2v man page

Hm note that the text about optional parameters does not apply to --vddk too, since the explanation of --vddk and the --vddk-* parameters are in different paragraphs.

> # virt-v2v -ic
> vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/
> ?no_verify=1 --vddk-thumbprint
> F3:BD:CC:18:78:A8:DB:66:9F:0C:F0:BF:A6:1A:4D:A0:84:C7:86:26
> esx6.5-rhel7.3-x86_64 --password-file /tmp/passwd -of rawvirt-v2v: error:
> ‘--vddk-*’ options should only be used when conversion 
> via the nbdkit VDDK plugin has been enabled, ie. using ‘--vddk’.

This error message seems more oriented to users who do not want to use VDDK but specified --vddk-* parameters regardless, rather than users who want to use VDDK and forgot --vddk-* parameters.
Comment 3 Richard W.M. Jones 2017-12-06 15:58:55 UTC 
I've done a bunch of testing today and in fact you can use any version
of VDDK back to 5.1.1 (although using the latest version is probably
a good idea).  It needed a small fix to nbdkit to make that work:

https://github.com/libguestfs/nbdkit/commit/1535d1885b15c6de61b21531250bc723c2be6f72

I have adjusted the virt-v2v documentation.  These commits are
candidates for RHEL 7.5 if there is still time:

https://github.com/libguestfs/libguestfs/commit/5ac24b490ebcf2e16e4ee2284a7f59bcc47cc808
https://github.com/libguestfs/libguestfs/commit/4b43697f9bc9b0fa33e26fbc2fce136ebaa89005
Comment 5 mxie@redhat.com 2017-12-15 09:12:25 UTC 
Verify the bug with below builds:
virt-v2v-1.36.10-4.el7.x86_64
libguestfs-1.36.10-4.el7.x86_64

Steps:

For problem 1: the link of nbdkit-vddk-plugin(1) in v2v man page which was given is http://man.he.net/man1/nbdkit-vddk-plugin, but the content of this nbdkit man page is hard to understand for customers

1.1 Check the link of nbdkit on latest virt-v2v online man page（http://libguestfs.org/virt-v2v.1.html） is "http://libguestfs.org/nbdkit-vddk-plugin.1.html" 

Result 1.1:
       the link of nbdkit is correct but the link can't be found due to comment2 "the nbdkit man pages are not present on libguestfs.org yet"


Problem 2: it is better that changing "nbdkit ≥ 1.1.14 is required" to "nbdkit ≥ 1.1.16 is required"  in below part due to the problem https://bugzilla.redhat.com/show_bug.cgi?id=1477912#c9
          
2.1 check nbdkit version in latest v2v man page as below
#man virt-v2v

           You must also compile nbdkit, enabling the VDDK plugin.  nbdkit ≥
           1.1.25 is recommended, but it is usually best to compile from the
           git tree.
  
Result 2.1: 
       nbdkit version has been updated to latest one which is excepted result
      

Problem 3:should change "export PATH=/path/to/nbdkit:$PATH" to "export PATH=/path/to/nbdkit-1.1.x:$PATH" 

3.1 check this problem in latest v2v man page as below
#man virt-v2v
.....
      VDDK: PREREQUISITES
      ....
           You do not need to run "make install" because you can run nbdkit from its source directory.  The source directory has a shell
           script called nbdkit which runs the locally built copy of nbdkit and its plugins.  So set $PATH to point to the nbdkit top
           build directory (that is, the directory containing the shell script called nbdkit), eg:

            export PATH=/path/to/nbdkit-1.1.x:$PATH
      ....
      VDDK: IMPORTING A GUEST
          The -it vddk parameter selects VDDK as the input transport for disks.

          To import a particular guest from vCenter server or ESXi hypervisor, use a command like the following, substituting the URI, guest
           name and SSL thumbprint:

          $ export PATH=/path/to/nbdkit:$PATH
            ....
Result 3.1:
       In "VDDK: PREREQUISITES" part, the path has been corrected as expected but the path is not corrected in "VDDK: IMPORTING A GUEST "part


Problem 4:can't complile nbdkit(both 1.1.14 and 1.1.16) with vddk 6.0(both -6.0.2-3566099 and 6.0.0-2498720 ),should add vddk version in v2v man page

4.1 Check required VDDK version in v2v man page
#man virt-v2v 

Result 4.1:
       Can't find any word about VDDK version in v2v man page


Problem 5:v2v man said only --vddk-thumbprint is required, the others are optional.so should correct this description in v2v man page

5.1 Check related vddk options info in v2v man page
   #man virt-v2v
        ....
        -it vddk
           Use VMware VDDK as a transport to copy the input disks.  See "INPUT FROM VDDK" below.  If you use this parameter then you may
           need to use other --vddk* options to specify how to connect through VDDK.
        ....
       --vddk-libdir LIBDIR
           Set the VDDK library directory.  This directory should contain subdirectories called include, lib64 etc., but do not include
           lib64 actually in the parameter.
           In most cases this parameter is required when using the -it vddk (VDDK) transport.  See "INPUT FROM VDDK" below for details.
        ....
       --vddk-thumbprint xx:xx:xx:...
           Set the thumbprintw of the remote VMware server.
           This parameter is required when using the -it vddk (VDDK) transport.  See "INPUT FROM VDDK" below for details.


Result 5.1:
         As desription in v2v man page, when convert guest using the -it vddk (VDDK) transport, --vddk-thumbprint is required,  --vddk-libdir LIBDIR is required in most cases 



Hi rjones, 
   For result1.1,could I think this problem has been fixed even if the nbdkit page is not ready on libguestfs.org now?

   According to result3.1 and result4.1, problem3 and problem4 are not fixed fully

   For result 5.1, I have a question,v2v man page said"-vddk-libdir LIBDIR is required in most cases",could you please tell me when vddk-libdir LIBDIR is not required when using the -it vddk (VDDK) transport?
Comment 7 Richard W.M. Jones 2017-12-18 10:36:22 UTC 
(In reply to mxie from comment #5)
> Verify the bug with below builds:
> virt-v2v-1.36.10-4.el7.x86_64
> libguestfs-1.36.10-4.el7.x86_64
> 
> Steps:
> 
> For problem 1: the link of nbdkit-vddk-plugin(1) in v2v man page which was
> given is http://man.he.net/man1/nbdkit-vddk-plugin, but the content of this
> nbdkit man page is hard to understand for customers
> 
> 1.1 Check the link of nbdkit on latest virt-v2v online man
> page（http://libguestfs.org/virt-v2v.1.html） is
> "http://libguestfs.org/nbdkit-vddk-plugin.1.html" 
> 
> Result 1.1:
>        the link of nbdkit is correct but the link can't be found due to
> comment2 "the nbdkit man pages are not present on libguestfs.org yet"

This will be fixed when we finally move to the new server.

> Problem 3:should change "export PATH=/path/to/nbdkit:$PATH" to "export
> PATH=/path/to/nbdkit-1.1.x:$PATH" 

Fixed in
https://github.com/libguestfs/libguestfs/commit/dab065a8eed6c6d8d9c53956393566812cfe6a2e

> Problem 4:can't complile nbdkit(both 1.1.14 and 1.1.16) with vddk 6.0(both
> -6.0.2-3566099 and 6.0.0-2498720 ),should add vddk version in v2v man page
> 
> 4.1 Check required VDDK version in v2v man page
> #man virt-v2v 
> 
> Result 4.1:
>        Can't find any word about VDDK version in v2v man page

This is fixed in nbdkit 1.1.25.  The older versions (1.1.14 or 1.1.16) only
work with VDDK 6.5.

> Problem 5:v2v man said only --vddk-thumbprint is required, the others are
> optional.so should correct this description in v2v man page
> 
> 5.1 Check related vddk options info in v2v man page
>    #man virt-v2v
>         ....
>         -it vddk
>            Use VMware VDDK as a transport to copy the input disks.  See
> "INPUT FROM VDDK" below.  If you use this parameter then you may
>            need to use other --vddk* options to specify how to connect
> through VDDK.
>         ....
>        --vddk-libdir LIBDIR
>            Set the VDDK library directory.  This directory should contain
> subdirectories called include, lib64 etc., but do not include
>            lib64 actually in the parameter.
>            In most cases this parameter is required when using the -it vddk
> (VDDK) transport.  See "INPUT FROM VDDK" below for details.
>         ....
>        --vddk-thumbprint xx:xx:xx:...
>            Set the thumbprintw of the remote VMware server.
>            This parameter is required when using the -it vddk (VDDK)
> transport.  See "INPUT FROM VDDK" below for details.
> 
> 
> Result 5.1:
>          As desription in v2v man page, when convert guest using the -it
> vddk (VDDK) transport, --vddk-thumbprint is required,  --vddk-libdir LIBDIR
> is required in most cases 
[...]
>    For result 5.1, I have a question,v2v man page said"-vddk-libdir LIBDIR
> is required in most cases",could you please tell me when vddk-libdir LIBDIR
> is not required when using the -it vddk (VDDK) transport?

Difficult to say.  When you compile the nbdkit VDDK plugin, there is a compiled-in
library path:

$ nbdkit vddk --dump-plugin
...
vddk_default_libdir=/opt/vddk/vmware-vix-disklib-distrib
...

If that is correct, then you don't need to specify --vddk-libdir.  However most
people will want to override the compiled-in library path, and they will need to
use --vddk-libdir, and it's not a problem to use --vddk-libdir all the time.
Comment 8 Richard W.M. Jones 2017-12-18 10:37:01 UTC 
(In reply to mxie from comment #6)
> I found related --in-place info is not removed from
> http://libguestfs.org/virt-v2v.1.html, pls help to check

Can you be more specific?  Which part of the information is wrong?
Comment 9 mxie@redhat.com 2017-12-18 12:35:23 UTC 
(In reply to Richard W.M. Jones from comment #8)
> (In reply to mxie from comment #6)
> > I found related --in-place info is not removed from
> > http://libguestfs.org/virt-v2v.1.html, pls help to check
> 
> Can you be more specific?  Which part of the information is wrong?

Because --in-place cannot be used in RHEL 7 (Bug 1328766),  related --in-place info should be removed from http://libguestfs.org/virt-v2v.1.html
Comment 10 mxie@redhat.com 2017-12-19 03:02:45 UTC 
(In reply to mxie from comment #9)
> (In reply to Richard W.M. Jones from comment #8)
> > (In reply to mxie from comment #6)
> > > I found related --in-place info is not removed from
> > > http://libguestfs.org/virt-v2v.1.html, pls help to check
> > 
> > Can you be more specific?  Which part of the information is wrong?
> 
> Because --in-place cannot be used in RHEL 7 (Bug 1328766),  related
> --in-place info should be removed from http://libguestfs.org/virt-v2v.1.html

File bug 1527257 to track this problem.thanks for Richard's confirmation.
Comment 11 mxie@redhat.com 2017-12-19 04:32:43 UTC 
> > Problem 4:can't complile nbdkit(both 1.1.14 and 1.1.16) with vddk 6.0(both
> > -6.0.2-3566099 and 6.0.0-2498720 ),should add vddk version in v2v man page
> > 
> > 4.1 Check required VDDK version in v2v man page
> > #man virt-v2v 
> > 
> > Result 4.1:
> >        Can't find any word about VDDK version in v2v man page
> 
> This is fixed in nbdkit 1.1.25.  The older versions (1.1.14 or 1.1.16) only
> work with VDDK 6.5.

Yes, nbdkit1.1.25 could compile with VDDK6.0 (6.0.2-3566099) now. but there is nbdkit error during v2v conversion under the test environment which is configured with nbdkit1.1.25 and vddk6.0, details pls refer to log"nbd1.1.25_vddk6.0.log"

# virt-v2v -ic vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/?no_verify=1 -it vddk --vddk-libdir /root/vmware-vix-disklib-distrib --vddk-thumbprint F3:BD:CC:18:78:A8:DB:66:9F:0C:F0:BF:A6:1A:4D:A0:84:C7:86:26 esx6.5-rhel7.4-x86_64 --password-file /tmp/passwd -of qcow2 -o rhv -os 10.73.131.93:/home/nfs_export
[   0.0] Opening the source -i libvirt -ic vpx://vsphere.local%5cAdministrator.199.71/data/10.73.196.89/?no_verify=1 esx6.5-rhel7.4-x86_64 -it vddk  --vddk-libdir /root/vmware-vix-disklib-distrib --vddk-thumbprint F3:BD:CC:18:78:A8:DB:66:9F:0C:F0:BF:A6:1A:4D:A0:84:C7:86:26
nbdkit: error: VixDiskLib: Invalid configuration file parameter. Failed to read configuration file.
nbdkit: error: VixDiskLib: Invalid configuration file parameter. Failed to read configuration file.
[   1.8] Creating an overlay to protect the source from being modified
[   4.5] Initializing the target -o rhv -os 10.73.131.93:/home/nfs_export
[   4.8] Opening the overlay
[  19.7] Inspecting the overlay
[  41.6] Checking for sufficient free disk space in the guest
[  41.6] Estimating space required on target for each disk
[  41.6] Converting Red Hat Enterprise Linux Server 7.4 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 181.7] Mapping filesystem data to avoid copying unused and blank areas
[ 181.9] Closing the overlay
[ 182.1] Checking if the guest needs BIOS or UEFI to boot
[ 182.1] Assigning disks to buses
[ 182.1] Copying disk 1/1 to /tmp/v2v.v9Iu75/bdf9c90b-e6f0-439c-aa9f-6305fd5fad7e/images/d5292305-0f09-4ae5-9de9-0cfd7fdfe551/e3bd30a2-1107-4f87-832f-e6350aa44a7a (qcow2)
    (100.00/100%)
[1482.6] Creating output metadata
[1482.7] Finishing off
Comment 12 mxie@redhat.com 2017-12-19 04:33:28 UTC 
Created attachment 1369772 [details]
nbd1.1.25_vddk6.0.log
Comment 13 Richard W.M. Jones 2017-12-19 09:00:10 UTC 
Apparently "error: VixDiskLib: Invalid configuration file parameter.
Failed to read configuration file." is just a noisy warning generated
by the older VDDK, and can be ignored.  There's not much we can do
about the crappy warnings/messages produced by VDDK.
Comment 14 mxie@redhat.com 2018-01-03 06:00:46 UTC 
> > Result 3.1:
> > In "VDDK: PREREQUISITES" part, the path has been corrected as expected but 
> > the path is not corrected in "VDDK: IMPORTING A GUEST "part
> Fixed in
> https://github.com/libguestfs/libguestfs/commit/
> dab065a8eed6c6d8d9c53956393566812cfe6a2e

Move bug from ON_QA to ASSIGNED to note which v2v build will include this patch
Comment 17 mxie@redhat.com 2018-01-17 06:50:16 UTC 
Verify comment14's problem with below builds:
virt-v2v-1.36.10-5.el7.x86_64
libguestfs-1.36.10-5.el7.x86_64

Steps:
1.check nbdkit path info in v2v man page
#man virt-v2v
.....
      VDDK: PREREQUISITES
      ....
           You do not need to run "make install" because you can run nbdkit from its source directory.  The source directory has a shell script called nbdkit which runs the locally built copy of nbdkit and its plugins.  So set $PATH to point to the nbdkit top build directory (that is, the directory containing the shell script called nbdkit), eg:

            export PATH=/path/to/nbdkit-1.1.x:$PATH
      ....
      VDDK: IMPORTING A GUEST
          The -it vddk parameter selects VDDK as the input transport for disks.

          To import a particular guest from vCenter server or ESXi hypervisor, use a command like the following, substituting the URI, guest name and SSL thumbprint:

            $ export PATH=/path/to/nbdkit-1.1.x:$PATH
            ....

Result:
       All nbdkit paths are corrected,so move this bug from ON_QA to VERIFIED
Comment 20 errata-xmlrpc 2018-04-10 09:20:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0677