RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1848862 - There is nbdkit curl error info if convert a guest from VMware without vddk by administrator account
Summary: There is nbdkit curl error info if convert a guest from VMware without vddk b...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: virt-v2v
Version: unspecified
Hardware: x86_64
OS: Unspecified
medium
medium
Target Milestone: beta
: ---
Assignee: Richard W.M. Jones
QA Contact: mxie@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-19 05:49 UTC by mxie@redhat.com
Modified: 2022-11-15 10:23 UTC (History)
19 users (show)

Fixed In Version: virt-v2v-2.0.2-1.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-11-15 09:55:44 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
administrator-account-without-vddk.log (1.58 MB, text/plain)
2020-06-19 05:49 UTC, mxie@redhat.com
no flags Details
import_rhel_8_2_vm_from_vmware_6 (981.73 KB, text/plain)
2020-10-28 13:00 UTC, Beni Pelled
no flags Details
virt-v2v-1.42.0-11-vmware-adminstrator-without-vddk.log (974.55 KB, text/plain)
2021-04-30 07:45 UTC, mxie@redhat.com
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 6575861 0 None None None 2021-12-09 23:01:03 UTC
Red Hat Product Errata RHSA-2022:7968 0 None None None 2022-11-15 09:56:01 UTC

Description mxie@redhat.com 2020-06-19 05:49:14 UTC
Created attachment 1698046 [details]
administrator-account-without-vddk.log

Description of problem:
There is nbdkit curl error info if convert a guest from VMware without vddk by administrator account


Version-Release number of selected component (if applicable):
virt-v2v-1.42.0-4.module+el8.3.0+6798+ad6e66be.x86_64
libguestfs-1.42.0-2.module+el8.3.0+6798+ad6e66be.x86_64
libvirt-6.4.0-1.module+el8.3.0+6881+88468c00.x86_64
qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420.x86_64
nbdkit-1.20.3-1.module+el8.3.0+6969+c553f4a0.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Convert a guest from VMware without vddk by administrator account
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.73.141/data/10.73.75.219/?no_verify=1 -ip /home/passwd esx6.7-rhel8.2-x86_64
[   0.0] Opening the source -i libvirt -ic vpx://vsphere.local%5cAdministrator.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel8.2-x86_64
[   2.9] Creating an overlay to protect the source from being modified
[   3.6] Opening the overlay
[  48.9] Inspecting the overlay
[ 154.9] Checking for sufficient free disk space in the guest
[ 154.9] Estimating space required on target for each disk
[ 154.9] Converting Red Hat Enterprise Linux 8.2 (Ootpa) to run on KVM
nbdkit: curl[2]: error: pread: curl_easy_perform: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.73.141/folder/esx6.7-rhel8.2-x86%5f64/esx6.7-rhel8.2-x86%5f64-flat.vmdk?dcPath=data&dsName=esx6.7-matrix]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.73.141/folder/esx6.7-rhel8.2-x86%5f64/esx6.7-rhel8.2-x86%5f64-flat.vmdk?dcPath=data&dsName=esx6.7-matrix]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.73.141/folder/esx6.7-rhel8.2-x86%5f64/esx6.7-rhel8.2-x86%5f64-flat.vmdk?dcPath=data&dsName=esx6.7-matrix]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.73.141/folder/esx6.7-rhel8.2-x86%5f64/esx6.7-rhel8.2-x86%5f64-flat.vmdk?dcPath=data&dsName=esx6.7-matrix]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: pread: curl_easy_perform: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: pread: curl_easy_perform: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.73.141/folder/esx6.7-rhel8.2-x86%5f64/esx6.7-rhel8.2-x86%5f64-flat.vmdk?dcPath=data&dsName=esx6.7-matrix]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.73.141/folder/esx6.7-rhel8.2-x86%5f64/esx6.7-rhel8.2-x86%5f64-flat.vmdk?dcPath=data&dsName=esx6.7-matrix]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.73.141/folder/esx6.7-rhel8.2-x86%5f64/esx6.7-rhel8.2-x86%5f64-flat.vmdk?dcPath=data&dsName=esx6.7-matrix]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
^Cnbdkit: curl[2]: error: aborting sleep to shut down


2.Use virsh to connect vCenter server during v2v converting guest in step1, find vcenter server can't connected
]# virsh -c vpx://root.73.141/data/10.73.75.219/?no_verify=1
Enter root's password for 10.73.73.141: 
error: failed to connect to the hypervisor
error: internal error: HTTP response code 503 for call to 'RetrieveServiceContent'


3.Try to open the web of vSphere client during v2v converting guest in step1
, there is below error info in web interface

Could not connect to one or more vCenter Server systems:https://vm-73-141.lab.eng.pek2.redhat.com:443/sdk.


Actual results:
As above description

Expected results:
Can convert a guest from VMware without vddk by administrator account


Additional info:
1.Can't reproduce the problem if convert a guest from VMware without vddk by root account
# virt-v2v  -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -ip /home/passwd esx6.7-rhel8.2-x86_64
[   0.0] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel8.2-x86_64
[   3.0] Creating an overlay to protect the source from being modified
[   3.6] Opening the overlay
[  42.5] Inspecting the overlay
[ 135.1] Checking for sufficient free disk space in the guest
[ 135.1] Estimating space required on target for each disk
[ 135.1] Converting Red Hat Enterprise Linux 8.2 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[2172.8] Mapping filesystem data to avoid copying unused and blank areas
[2175.4] Closing the overlay
[2175.5] Assigning disks to buses
[2175.5] Checking if the guest needs BIOS or UEFI to boot
[2175.5] Initializing the target -o libvirt -os default
[2175.5] Copying disk 1/1 to /var/lib/libvirt/images/esx6.7-rhel8.2-x86_64-sda (raw)
^C  (49.31/100%)


2.Can't reproduce the problem if convert a guest from VMware with vddk by administrator account
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.73.141/data/10.73.75.219/?no_verify=1 -o rhv-upload -os nfs_data -of raw  -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -oo rhv-cafile=/home/ca.pem -oo rhv-cluster=NFS -oo rhv-direct -ip /home/passwd esx6.7-rhel7.8-x86_64
[   1.1] Opening the source -i libvirt -ic vpx://vsphere.local%5cAdministrator.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel7.8-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA
[   3.0] Creating an overlay to protect the source from being modified
[   4.0] Opening the overlay
[  10.2] Inspecting the overlay
[  47.5] Checking for sufficient free disk space in the guest
[  47.5] Estimating space required on target for each disk
[  47.5] Converting Red Hat Enterprise Linux Server 7.8 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 228.9] Mapping filesystem data to avoid copying unused and blank areas
[ 230.2] Closing the overlay
[ 230.5] Assigning disks to buses
[ 230.5] Checking if the guest needs BIOS or UEFI to boot
[ 230.5] Initializing the target -o rhv-upload -oc https://hp-dl360eg8-03.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[ 231.9] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.dfL23T/nbdkit4.sock", "file.export": "/" } (raw)
    (35.27/100%)

Comment 1 mxie@redhat.com 2020-06-19 08:45:56 UTC
Hi rjones, 

   The command line which is used to generate attached log file is 
"# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.73.141/folder-m/dc-m/cluster-m/10.73.75.219/?no_verify=1 -ip /home/passwd esx6.7-rhel7.8-x86_64 -v -x |& tee > administrator-account-without-vddk.log "

  Below is similar v2v conversion which uses same vSphere client structure with above command but uses root account

# virt-v2v -ic vpx://root.198.169/folder-m/dc-m/cluster-m/10.73.199.217/?no_verify=1  esx7.0-rhel6.10-x86_64 -ip /home/passwd
[   0.0] Opening the source -i libvirt -ic vpx://root.198.169/folder-m/dc-m/cluster-m/10.73.199.217/?no_verify=1 esx7.0-rhel6.10-x86_64
[   3.0] Creating an overlay to protect the source from being modified
[   3.6] Opening the overlay
[  35.9] Inspecting the overlay
[ 166.3] Checking for sufficient free disk space in the guest
[ 166.3] Estimating space required on target for each disk
[ 166.3] Converting Red Hat Enterprise Linux Server release 6.10 (Santiago) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[1438.5] Mapping filesystem data to avoid copying unused and blank areas
[1441.1] Closing the overlay
[1441.5] Assigning disks to buses
[1441.5] Checking if the guest needs BIOS or UEFI to boot
[1441.5] Initializing the target -o libvirt -os default
[1441.5] Copying disk 1/1 to /var/lib/libvirt/images/esx7.0-rhel6.10-x86_64-sda (raw)
^C  (0.00/100%)

Comment 2 Richard W.M. Jones 2020-06-19 12:50:31 UTC
Preliminary patch posted:

https://www.redhat.com/archives/libguestfs/2020-June/thread.html#00086

This just disables readahead and I'm still testing it.

Comment 7 Richard W.M. Jones 2020-09-24 09:39:10 UTC
I posted the patch again as part of:
https://www.redhat.com/archives/libguestfs/2020-September/msg00184.html

Comment 9 Beni Pelled 2020-10-28 12:59:08 UTC
Having the same issue [1] on EL8.2 (RHV Host) with:

- libguestfs-1.42.0-2.module+el8.3.0+6798+ad6e66be.x86_64
- virt-v2v-1.42.0-6.module+el8.3.0+7898+13f907d5.x86_64
- nbdkit-1.22.0-2.module+el8.3.0+8203+18ecf00e.x86_64
- libvirt-6.6.0-6.module+el8.3.0+8125+aefcf088.x86_64
- vdsm-4.40.35-1.el8ev.x86_64

PS> Importing the same VM using the RHV-UI works as expected even though
the same url error appears in the import log - import log attached.


# virt-v2v -ic vpx://vsphere.local%5cadministrator.lab.tlv.redhat.com/TLV/cougar02.scl.lab.tlv.redhat.com/?no_verify=1 -ip /tmp/passwd automation_v2v_rhel_8_2_vmware
[   0.0] Opening the source -i libvirt -ic vpx://vsphere.local%5cadministrator.lab.tlv.redhat.com/TLV/cougar02.scl.lab.tlv.redhat.com/?no_verify=1 automation_v2v_rhel_8_2_vmware
[   3.1] Creating an overlay to protect the source from being modified
[   3.7] Opening the overlay
libvirt needs authentication to connect to libvirt URI qemu:///system
(see also: http://libvirt.org/auth.html http://libvirt.org/uri.html)
Please enter your authentication name: <QEMU_RHV_USER>
Please enter your password: <QEMU_RHV_PASS>
[ 406.6] Inspecting the overlay
[ 505.2] Checking for sufficient free disk space in the guest
[ 505.2] Estimating space required on target for each disk
[ 505.2] Converting Red Hat Enterprise Linux 8.2 (Ootpa) to run on KVM
nbdkit: curl[2]: error: pread: curl_easy_perform: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://compute-vmware-env.scl.lab.tlv.redhat.com/folder/automation%5fv2v%5frhel%5f8%5f2/automation%5fv2v%5frhel%5f8%5f2-flat.vmdk?dcPath=TLV&dsName=Datastore%5fHP3PAR]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://compute-vmware-env.scl.lab.tlv.redhat.com/folder/automation%5fv2v%5frhel%5f8%5f2/automation%5fv2v%5frhel%5f8%5f2-flat.vmdk?dcPath=TLV&dsName=Datastore%5fHP3PAR]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
.
.
.
.
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://compute-vmware-env.scl.lab.tlv.redhat.com/folder/automation%5fv2v%5frhel%5f8%5f2/automation%5fv2v%5frhel%5f8%5f2-flat.vmdk?dcPath=TLV&dsName=Datastore%5fHP3PAR]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://compute-vmware-env.scl.lab.tlv.redhat.com/folder/automation%5fv2v%5frhel%5f8%5f2/automation%5fv2v%5frhel%5f8%5f2-flat.vmdk?dcPath=TLV&dsName=Datastore%5fHP3PAR]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
virt-v2v: This guest has virtio drivers installed.
[2443.5] Mapping filesystem data to avoid copying unused and blank areas
[2451.4] Closing the overlay
[2452.3] Assigning disks to buses
[2452.3] Checking if the guest needs BIOS or UEFI to boot
[2452.3] Initializing the target -o libvirt -os default
virt-v2v: error: cannot get libvirt hypervisor capabilities: authentication 
failed: Failed to start SASL negotiation: -4 (SASL(-4): no mechanism 
available: No worthy mechs found)

If reporting bugs, run virt-v2v with debugging enabled and include the 
complete output:

  virt-v2v -v -x [...]

Comment 10 Beni Pelled 2020-10-28 13:00:00 UTC
Created attachment 1724812 [details]
import_rhel_8_2_vm_from_vmware_6

Comment 15 Pino Toscano 2020-11-19 17:46:17 UTC
I'm starting to think that the issue behind these failures is not due to the VMware user used for the conversion. I personally can perform conversions using a non-administrator user with the right roles enabled.

One common thing of all these failures is that the VMware instances are QE resources, and thus used for testing using automated and manual jobs on them. Because of them, there may be various sessions open with the vCenter instance at the same time.
According to some old sources [1], it looks like there is some kind of limit for SDK sessions, although I cannot find a way to either get this limit or fetch statistics on the active sessions.

[1] https://communities.vmware.com/t5/vSphere-Management-SDK/Max-SOAP-sessions/m-p/1534369

So: Ming, Beni, Tamir: please try to convert again, monitoring the logson the vCenter appliance for possible errors like explained in these two VMware KB articles:
- https://kb.vmware.com/s/article/50114010
- https://kb.vmware.com/s/article/1031039

Do you get any of the errors mentioned in the articles?

Comment 19 Richard W.M. Jones 2020-12-01 12:25:18 UTC
I've disabled readahead for VMware-over-https upstream:
https://github.com/libguestfs/virt-v2v/commit/9f4940068022d4e7abdfea6617b73a2b206f19aa

I've also enabled cookie-scripts support which should help with
authentication timeouts:
https://github.com/libguestfs/virt-v2v/commit/2b9a11743b74ef3716b66a7e395108a26382e331

Comment 21 Tamir 2020-12-03 08:25:30 UTC
I found this error on the VMware's VCenter 7 server, in the file /var/log/vmware/vpxd/vpxd.log.
It looks like virt-v2v creates too many HTTP sessions to the VCenter and it results in 503 error in VMware's VCenter services.

* Workaround for the HTTP method *
I found a workaround to allow VCenter to accept as many sessions as possible you can change the file /etc/vmware-vpx/vpxd.cfg on the VCenter's server
and add inside the <vmacore> XML tag the following XML:
"
    <soap>
      <maxSessionCount>0</maxSessionCount>
    </soap>
"

You shouldn't replace all the <vmacore> contents, just add the above lines inside.
It will look something like that:

"
  <vmacore>
    <cacheProperties>true</cacheProperties>
    <soap>
      <maxSessionCount>0</maxSessionCount>
    </soap>
    <ssl/>
    <threadPool>
      <TaskMax>90</TaskMax>
      <threadNamePrefix>vpxd</threadNamePrefix>
    </threadPool>
  </vmacore>

"

Comment 22 Richard W.M. Jones 2020-12-03 10:07:16 UTC
Thanks, I've added this to the virt-v2v documentation to help others
who have this problem in future:

https://github.com/libguestfs/virt-v2v/commit/00649e98be43b4b05a28c5c8c858a54ec4fe3e7a

Comment 23 Raghvendra Singh 2021-04-16 18:34:58 UTC
Hi Richard W.M. Jones: 

Description of problem:
There is nbdkit curl error info if convert a guest from VMware to oVIrt during import.

nbdkit: curl[3]: error: problem doing HEAD request to fetch size of URL [https://<vcenter>/folder/tdfdettsovr002s/tdfdettsovr002s-flat.vmdk?dcPath=TTCE-CC03-RESOURCE&dsName=vol%5fvm%5fd2esr1%5fcdc03%5f14%5fkbe08]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable

It does retry for 5 times and eventually it fails, I see that some of the patches being worked by Redhat team and you. When the virt v2-v patch would be released. Appreciate your help.

I see the latest version of  virt-v2v is 1.42.0, when the remediation of above problem is expected.

It has become blocker for us to move to oVIrt. Please let me know if you need any more details.

Comment 24 Richard W.M. Jones 2021-04-16 21:21:49 UTC
That patches are all open source.  See comment 22 for the one which documents
how to fix this bug.

Comment 27 Richard W.M. Jones 2021-04-27 15:40:47 UTC
So I've partially resolved this bug in AV 8.5.0 by cherry picking:

9f4940068022d4e7abdfea6617b73a2b206f19aa
v2v: Disable readahead for VMware curl sources too (RHBZ#1848862).

00649e98be43b4b05a28c5c8c858a54ec4fe3e7a
docs: Document how to remove "Out of HTTP sessions" limit.

However the main fix here -- using cookie-scripts -- is not backportable
to RHEL 8 because related changes would require us to remove the
virt-v2v-copy-to-local tool.  This tool has been removed in RHEL 9
but I don't want to remove the tool from RHEL 8.

For the complete fix you're going to have to move to RHEL 9 or
the upstream package.

Comment 28 mxie@redhat.com 2021-04-30 07:43:42 UTC
Test the bug with below builds:
virt-v2v-1.42.0-11.module+el8.5.0+10793+d881d728.x86_64
libguestfs-1.44.0-3.module+el8.5.0+10681+17a9b157.x86_64
libvirt-client-7.0.0-13.module+el8.4.0+10604+5608c2b4.x86_64
qemu-kvm-5.2.0-15.module+el8.4.0+10650+50781ca0.x86_64
nbdkit-1.24.0-1.module+el8.4.0+9341+96cf2672.x86_64

Steps:
1. Convert a guest from VMware without vddk by administrator account

# # virt-v2v  -ic vpx://vsphere.local%5cAdministrator.198.169/data/10.73.199.217/?no_verify=1 -ip /home/passwd Auto-esx7.0-rhel8.3-pci-passthrough -o local -os /home
[   0.0] Opening the source -i libvirt -ic vpx://vsphere.local%5cAdministrator.198.169/data/10.73.199.217/?no_verify=1 Auto-esx7.0-rhel8.3-pci-passthrough
[   2.8] Creating an overlay to protect the source from being modified
[   3.5] Opening the overlay
[  43.5] Inspecting the overlay
[ 319.3] Checking for sufficient free disk space in the guest
[ 319.3] Estimating space required on target for each disk
[ 319.3] Converting Red Hat Enterprise Linux 8.3 (Ootpa) to run on KVM
nbdkit: curl[2]: error: pread: curl_easy_perform: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.198.169/folder/Auto-esx7.0-rhel8.3-pci-passthrough/Auto-esx7.0-rhel8.3-pci-passthrough-flat.vmdk?dcPath=data&dsName=esx7.0-function]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.198.169/folder/Auto-esx7.0-rhel8.3-pci-passthrough/Auto-esx7.0-rhel8.3-pci-passthrough-flat.vmdk?dcPath=data&dsName=esx7.0-function]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.198.169/folder/Auto-esx7.0-rhel8.3-pci-passthrough/Auto-esx7.0-rhel8.3-pci-passthrough-flat.vmdk?dcPath=data&dsName=esx7.0-function]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.198.169/folder/Auto-esx7.0-rhel8.3-pci-passthrough/Auto-esx7.0-rhel8.3-pci-passthrough-flat.vmdk?dcPath=data&dsName=esx7.0-function]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.198.169/folder/Auto-esx7.0-rhel8.3-pci-passthrough/Auto-esx7.0-rhel8.3-pci-passthrough-flat.vmdk?dcPath=data&dsName=esx7.0-function]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.198.169/folder/Auto-esx7.0-rhel8.3-pci-passthrough/Auto-esx7.0-rhel8.3-pci-passthrough-flat.vmdk?dcPath=data&dsName=esx7.0-function]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.198.169/folder/Auto-esx7.0-rhel8.3-pci-passthrough/Auto-esx7.0-rhel8.3-pci-passthrough-flat.vmdk?dcPath=data&dsName=esx7.0-function]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.198.169/folder/Auto-esx7.0-rhel8.3-pci-passthrough/Auto-esx7.0-rhel8.3-pci-passthrough-flat.vmdk?dcPath=data&dsName=esx7.0-function]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.198.169/folder/Auto-esx7.0-rhel8.3-pci-passthrough/Auto-esx7.0-rhel8.3-pci-passthrough-flat.vmdk?dcPath=data&dsName=esx7.0-function]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
.......


2.Convert a guest from VMware with vddk by administrator account
#  virt-v2v  -ic vpx://vsphere.local%5cAdministrator.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vddk7.0 -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78 -ip /home/passwd Auto-esx7.0-rhel8.3-pci-passthrough -o local -os /home
[   0.0] Opening the source -i libvirt -ic vpx://vsphere.local%5cAdministrator.198.169/data/10.73.199.217/?no_verify=1 Auto-esx7.0-rhel8.3-pci-passthrough -it vddk  -io vddk-libdir=/home/vddk7.0 -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78
[   1.7] Creating an overlay to protect the source from being modified
[   2.7] Opening the overlay
[   7.5] Inspecting the overlay
[  19.9] Checking for sufficient free disk space in the guest
[  19.9] Estimating space required on target for each disk
[  19.9] Converting Red Hat Enterprise Linux 8.3 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[  70.7] Mapping filesystem data to avoid copying unused and blank areas
[  71.8] Closing the overlay
[  72.0] Assigning disks to buses
[  72.0] Checking if the guest needs BIOS or UEFI to boot
[  72.0] Initializing the target -o local -os /home
[  72.0] Copying disk 1/1 to /home/Auto-esx7.0-rhel8.3-pci-passthrough-sda (raw)
    (100.00/100%)
[ 388.1] Creating output metadata
[ 388.1] Finishing off


Hi Richard,

   The bug is not fixed with virt-v2v-1.42.0-11, please check the debug log, thanks!

Comment 29 mxie@redhat.com 2021-04-30 07:45:57 UTC
Created attachment 1777620 [details]
virt-v2v-1.42.0-11-vmware-adminstrator-without-vddk.log

Comment 30 Richard W.M. Jones 2021-04-30 08:11:25 UTC
I know!  The main fix can't be done til RHEL 9 (see comment 27).  I only
backported a few patches that are possible in RHEL AV.

If you want we can move this bug to RHEL 9 where it's already been fixed
(comment 25).

Comment 31 mxie@redhat.com 2021-04-30 09:40:47 UTC
(In reply to Richard W.M. Jones from comment #30)
> I know!  The main fix can't be done til RHEL 9 (see comment 27).  I only
> backported a few patches that are possible in RHEL AV.
> 
> If you want we can move this bug to RHEL 9 where it's already been fixed
> (comment 25).

Oh sorry, I didn't read the comment27 carefully, but seems the patches "v2v: Disable readahead for VMware curl sources too" and "docs: Document how to remove "Out of HTTP sessions" limit" have little effect for the bug, did I miss anything for the bug?


1.Check man page "virt-v2v-input-vmware" and found below info:
# man virt-v2v-input-vmware |grep "Out of HTTP sessions: Limited to ..." -A 7
   vCenter: "Out of HTTP sessions: Limited to ..."
       VMware vCenter appears to limit HTTP sessions and in some circumstances virt-v2v may exceed this number.  You can adjust or remove
       the limit by editing /etc/vmware-vpx/vpxd.cfg on the vCenter server.  Increase the "<maxSessionCount>" field, or set it to 0 which
       makes it unlimited:

        <soap>
          <maxSessionCount>0</maxSessionCount>
        </soap>
2. Log into vCenter Server to modify /etc/vmware-vpx/vpxd.cfg but can't found info about <maxSessionCount>, 
root@vm-198-169 [ ~ ]# cat /etc/vmware-vpx/vpxd.cfg |grep "<maxSessionCount>"
root@vm-198-169 [ ~ ]# cat /etc/vmware-vpx/vpxd.cfg |grep "soap"
    <soapPort>8085</soapPort>

3.So add related info to the file manually on vCenter Server
root@vm-198-169 [ ~ ]#  cat /etc/vmware-vpx/vpxd.cfg |grep "<maxSessionCount>" -A 1 -B 1
    <soap>
       <maxSessionCount>0</maxSessionCount>
    </soap>

4.Convert a guest from VMware without vddk by administrator account
# virt-v2v  -ic vpx://vsphere.local%5cAdministrator.198.169/data/10.73.199.217/?no_verify=1 -ip /home/passwd Auto-esx7.0-rhel8.3-pci-passthrough -o local -os /home
[   0.0] Opening the source -i libvirt -ic vpx://vsphere.local%5cAdministrator.198.169/data/10.73.199.217/?no_verify=1 Auto-esx7.0-rhel8.3-pci-passthrough
[   2.8] Creating an overlay to protect the source from being modified
[   3.4] Opening the overlay
[  44.2] Inspecting the overlay
[ 328.5] Checking for sufficient free disk space in the guest
[ 328.5] Estimating space required on target for each disk
[ 328.5] Converting Red Hat Enterprise Linux 8.3 (Ootpa) to run on KVM
nbdkit: curl[2]: error: pread: curl_easy_perform: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.198.169/folder/Auto-esx7.0-rhel8.3-pci-passthrough/Auto-esx7.0-rhel8.3-pci-passthrough-flat.vmdk?dcPath=data&dsName=esx7.0-function]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.198.169/folder/Auto-esx7.0-rhel8.3-pci-passthrough/Auto-esx7.0-rhel8.3-pci-passthrough-flat.vmdk?dcPath=data&dsName=esx7.0-function]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: pread: curl_easy_perform: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.198.169/folder/Auto-esx7.0-rhel8.3-pci-passthrough/Auto-esx7.0-rhel8.3-pci-passthrough-flat.vmdk?dcPath=data&dsName=esx7.0-function]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: pread: curl_easy_perform: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
nbdkit: curl[2]: error: problem doing HEAD request to fetch size of URL [https://10.73.198.169/folder/Auto-esx7.0-rhel8.3-pci-passthrough/Auto-esx7.0-rhel8.3-pci-passthrough-flat.vmdk?dcPath=data&dsName=esx7.0-function]: HTTP response code said error: The requested URL returned error: 503 Service Unavailable
.....

Comment 32 Richard W.M. Jones 2021-04-30 11:22:23 UTC
Since we haven't really fixed this, and backporting the patches to fix it
will require removing virt-v2v-copy-to-local, let's move this to RHEL 9.

Comment 38 Peter Lauterbach 2022-01-05 20:47:19 UTC
Currently imports via the RHV UI are broken.  RHV will be based on RHEL8 for some time, and this functionality will no longer work.

Comment 39 mxie@redhat.com 2022-01-06 01:48:07 UTC
(In reply to Peter Lauterbach from comment #38)
> Currently imports via the RHV UI are broken.  RHV will be based on RHEL8 for
> some time, and this functionality will no longer work.

The bug has been fixed on rhel8.6, details please refer to bug2018173

Comment 40 Richard W.M. Jones 2022-01-06 08:27:03 UTC
(In reply to Peter Lauterbach from comment #38)
> Currently imports via the RHV UI are broken.  RHV will be based on RHEL8 for
> some time, and this functionality will no longer work.

I'm also confused about what you mean by this statement.  Imports
via RHV UI should work as far as I know.

Comment 43 Richard W.M. Jones 2022-01-13 08:05:47 UTC
(In reply to Abhishekh Patil from comment #42)
> Hi Team,
> 
> Customer #02962210 would like to know what version of RHVH this issue will
> be fixed ?

It's fixed in RHEL 8.6 which is out in May.  You'll have to ask the RHV team
if/when it'll appear in a RHV node.  They can use RHEL as a node.  For early
access see the hotfix process.

Comment 44 Klaus Heinrich Kiwi 2022-01-13 14:04:50 UTC
(In reply to Richard W.M. Jones from comment #43)
> (In reply to Abhishekh Patil from comment #42)
> > Hi Team,
> > 
> > Customer #02962210 would like to know what version of RHVH this issue will
> > be fixed ?
> 
> It's fixed in RHEL 8.6 which is out in May.  You'll have to ask the RHV team
> if/when it'll appear in a RHV node.  They can use RHEL as a node.  For early
> access see the hotfix process.

Rich, can we adjust the BZ to reflect that this was fixed in RHEL 8.6?

Abhishekh - you may need to request a z-stream if you need this fix in a released version before RHEL 8.6 (or the customer is not able/willing to upgrade to RHEL 8.6).

 -Klaus

Comment 45 mxie@redhat.com 2022-01-13 14:40:16 UTC
(In reply to Richard W.M. Jones from comment #43)
> (In reply to Abhishekh Patil from comment #42)
> > Hi Team,
> > 
> > Customer #02962210 would like to know what version of RHVH this issue will
> > be fixed ?
> 
> It's fixed in RHEL 8.6 which is out in May.  You'll have to ask the RHV team
> if/when it'll appear in a RHV node.  They can use RHEL as a node.  For early
> access see the hotfix process.

Hi Chen Shao, could you help to check which rhvh version contains virt-v2v version >= 1.42.0-16.el8

Comment 46 Richard W.M. Jones 2022-01-13 14:45:50 UTC
(In reply to Klaus Heinrich Kiwi from comment #44)
> (In reply to Richard W.M. Jones from comment #43)
> > (In reply to Abhishekh Patil from comment #42)
> > > Hi Team,
> > > 
> > > Customer #02962210 would like to know what version of RHVH this issue will
> > > be fixed ?
> > 
> > It's fixed in RHEL 8.6 which is out in May.  You'll have to ask the RHV team
> > if/when it'll appear in a RHV node.  They can use RHEL as a node.  For early
> > access see the hotfix process.
> 
> Rich, can we adjust the BZ to reflect that this was fixed in RHEL 8.6?
> 
> Abhishekh - you may need to request a z-stream if you need this fix in a
> released version before RHEL 8.6 (or the customer is not able/willing to
> upgrade to RHEL 8.6).

Sorry I was confused myself.  The RHEL 8.6 bug exists already:
https://bugzilla.redhat.com/show_bug.cgi?id=2018173

Comment 47 cshao 2022-01-14 01:51:28 UTC
(In reply to mxie from comment #45)
> (In reply to Richard W.M. Jones from comment #43)
> > (In reply to Abhishekh Patil from comment #42)
> > > Hi Team,
> > > 
> > > Customer #02962210 would like to know what version of RHVH this issue will
> > > be fixed ?
> > 
> > It's fixed in RHEL 8.6 which is out in May.  You'll have to ask the RHV team
> > if/when it'll appear in a RHV node.  They can use RHEL as a node.  For early
> > access see the hotfix process.
> 
> Hi Chen Shao, could you help to check which rhvh version contains virt-v2v
> version >= 1.42.0-16.el8

Hi Mxie,
The latest RHVH is rhvh-4.4.9.4-0.20220107.0+1, and virt-v2v version is virt-v2v-1.42.0-15.module+el8.5.0+12264+1ee0d523.x86_64.
So 1.42.0-16.el8 is not included by RHVH yet.

# imgbase w
You are on rhvh-4.4.9.4-0.20220107.0+1
# rpm -qa| grep virt-v2v
virt-v2v-1.42.0-15.module+el8.5.0+12264+1ee0d523.x86_64

Comment 48 Ulhas Surse 2022-04-05 11:11:35 UTC
The version virt-v2v-1.42.0-16 is in rhvh 4.4.10. Does that mean the fix is available in this RHVH? 

~~~
[root@dhcp210-238 ~]#  rpm -qa| grep virt-v2v
virt-v2v-1.42.0-16.module+el8.5.0+13900+a08c0464.x86_64

[root@dhcp210-238 ~]# imgbase w
You are on rhvh-4.4.10.2-0.20220310.0+1
~~~

Also, the Bug status is NEW, so just wanted to confirm, is it correct?

Comment 49 Richard W.M. Jones 2022-04-05 11:21:05 UTC
The bug has been fixed for a long time in RHEL 9, I think this bug
just fell through the cracks.

As you're using RHEL 8, you'll need to look at this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=2018173

Comment 50 Richard W.M. Jones 2022-04-05 11:22:23 UTC
And this one too (z-stream clone for 8.5-z):
https://bugzilla.redhat.com/show_bug.cgi?id=2040778

Comment 53 mxie@redhat.com 2022-04-20 09:44:45 UTC
Verify the bug with below builds:
virt-v2v-2.0.3-1.el9.x86_64
libguestfs-1.48.1-1.el9.x86_64
guestfs-tools-1.48.0-1.el9.x86_64
nbdkit-server-1.30.2-1.el9.x86_64
libnbd-1.12.2-1.el9.x86_64
libvirt-libs-8.2.0-1.el9.x86_64
qemu-img-6.2.0-13.el9.x86_64


Steps:
1.Convert a guest from ESXi7.0 without vddk by administrator account and regular account which has suffix "@vsphere.client" 

1.1 # virt-v2v  -ic vpx://vsphere.local%5cAdministrator.198.169/data/10.73.199.217/?no_verify=1 -ip /home/passwd esx7.0-rhel9.0-x86_64 -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data -b ovirtmgmt
[   0.1] Setting up the source: -i libvirt -ic vpx://vsphere.local%5cAdministrator.198.169/data/10.73.199.217/?no_verify=1 esx7.0-rhel9.0-x86_64
[   2.5] Opening the source
[  36.1] Inspecting the source
[ 589.5] Checking for sufficient free disk space in the guest
[ 589.5] Converting Red Hat Enterprise Linux 9.0 Beta (Plow) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[2830.2] Mapping filesystem data to avoid copying unused and blank areas
[2852.8] Closing the overlay
[2853.0] Assigning disks to buses
[2853.0] Checking if the guest needs BIOS or UEFI to boot
[2853.0] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[2873.3] Copying disk 1/1
█ 100% [****************************************]
[3169.7] Creating output metadata
[3179.3] Finishing off

1.2 # virt-v2v  -ic vpx://vsphere.local%5cmxie.198.169/data/10.73.199.217/?no_verify=1 -ip /home/passwd esx7.0-win11-x86_64 -o rhv-upload -of qcow2 -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data -b ovirtmgmt
[   0.1] Setting up the source: -i libvirt -ic vpx://vsphere.local%5cmxie.198.169/data/10.73.199.217/?no_verify=1 esx7.0-win11-x86_64
[   2.4] Opening the source
[  31.7] Inspecting the source
[ 605.6] Checking for sufficient free disk space in the guest
[ 605.6] Converting Windows 10 Enterprise to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 778.1] Mapping filesystem data to avoid copying unused and blank areas
[2806.2] Closing the overlay
[2806.5] Assigning disks to buses
[2806.5] Checking if the guest needs BIOS or UEFI to boot
[2806.5] Setting up the destination: -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -os nfs_data
[2819.5] Copying disk 1/1
█ 100% [****************************************]
[3766.4] Creating output metadata
[3778.1] Finishing off

1.3 Check guests after v2v conversion, checkpoints of guests are passed except bug1988279

2.Convert a guest from ESXi6.7 without vddk by administrator account and regular account which has suffix "@vsphere.client" 

2.1 # virt-v2v  -ic vpx://vsphere.local%5cAdministrator.73.141/data/10.73.75.219/?no_verify=1 -ip /home/passwd esx6.7-rhel8.5-x86_64
[   0.0] Setting up the source: -i libvirt -ic vpx://vsphere.local%5cAdministrator.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel8.5-x86_64
[   2.5] Opening the source
[  41.0] Inspecting the source
[ 956.6] Checking for sufficient free disk space in the guest
[ 956.6] Converting Red Hat Enterprise Linux 8.5 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[3965.7] Mapping filesystem data to avoid copying unused and blank areas
[3990.0] Closing the overlay
[3990.3] Assigning disks to buses
[3990.3] Checking if the guest needs BIOS or UEFI to boot
[3990.3] Setting up the destination: -o libvirt
[3991.8] Copying disk 1/1
█ 100% [****************************************]
[7117.9] Creating output metadata
[7118.0] Finishing off

2.2 # virt-v2v  -ic vpx://vsphere.local%5cmxie.73.141/data/10.73.75.219/?no_verify=1 -ip /home/passwd esx6.7-win2022-x86_64 
[   0.1] Setting up the source: -i libvirt -ic vpx://vsphere.local%5cmxie.73.141/data/10.73.75.219/?no_verify=1 esx6.7-win2022-x86_64
[   2.5] Opening the source
[  38.0] Inspecting the source
[ 786.3] Checking for sufficient free disk space in the guest
[ 786.3] Converting Windows Server 2022 Standard to run on KVM
virt-v2v: This guest has virtio drivers installed.
[1033.4] Mapping filesystem data to avoid copying unused and blank areas
[1179.5] Closing the overlay
[1179.8] Assigning disks to buses
[1179.8] Checking if the guest needs BIOS or UEFI to boot
[1179.8] Setting up the destination: -o libvirt
[1181.5] Copying disk 1/1
█ 100% [****************************************]
[6817.9] Creating output metadata
[6817.9] Finishing off

2.3 Check guests after v2v conversion, checkpoints of guests are passed


Result:
    The bug has been fixed, move the bug from ON_QA to VERIFIED

Comment 56 errata-xmlrpc 2022-11-15 09:55:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: virt-v2v security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7968


Note You need to log in before you can comment on or make changes to this bug.