Bug 1430287
Summary: | kernel XFS driver hits 2000 sec timeout during XFS log recovery in virt-v2v conversion of RHEL 7.2 guest | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | kuwei <kuwei> | ||||||
Component: | libguestfs | Assignee: | Richard W.M. Jones <rjones> | ||||||
Status: | CLOSED DUPLICATE | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||
Severity: | high | Docs Contact: | |||||||
Priority: | urgent | ||||||||
Version: | 7.4 | CC: | juzhou, kuwei, mxie, mzhan, ptoscano, rjones, tzheng, xiaodwan | ||||||
Target Milestone: | rc | Keywords: | TestBlocker | ||||||
Target Release: | --- | ||||||||
Hardware: | x86_64 | ||||||||
OS: | Linux | ||||||||
Whiteboard: | V2V | ||||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2017-03-10 12:15:18 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Attachments: |
|
(In reply to kuwei from comment #0) > 1:Convert a rhel7.2 guest from vmware to kvm. > # # virt-v2v -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 > esx5.5-rhel7.2-x86_64 --password-file /tmp/passwd -on test1 > [ 0.0] Opening the source -i libvirt -ic > vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.2-x86_64 > [ 1.2] Creating an overlay to protect the source from being modified > [ 1.8] Initializing the target -o libvirt -os default > [ 1.8] Opening the overlay > [ 35.0] Inspecting the overlay > ^c In this case, there I/O errors when reading the disks, and most probably that causes the kernel to throw error traces. My guess is that it is an issue either in QEMU (the curl/https driver), or in the kernel. Can you please paste the result of `rpm -qa | grep kernel | sort -V`, on the host where virt-v2v was run? I see you are using qemu-kvm-rhev, could you please try with the plain qemu-kvm? > 2.Convert a windows guest from vmware to kvm. Can you please provide the logs for this case? (In reply to Pino Toscano from comment #2) > In this case, there I/O errors when reading the disks, and most probably > that causes the kernel to throw error traces. My guess is that it is an > issue either in QEMU (the curl/https driver), or in the kernel. > > Can you please paste the result of `rpm -qa | grep kernel | sort -V`, on the > host where virt-v2v was run? # rpm -qa | grep kernel | sort -V kernel-3.10.0-567.el7.x86_64 kernel-devel-3.10.0-567.el7.x86_64 kernel-headers-3.10.0-567.el7.x86_64 kernel-tools-3.10.0-567.el7.x86_64 kernel-tools-libs-3.10.0-567.el7.x86_64 > I see you are using qemu-kvm-rhev, could you please try with the plain > qemu-kvm? Using qemu-kvm can convert guest from vmware to kvm successfully,you can see my addition info comment 0. But we can't start the guest on qemu-kvm packages,it is a bug https://bugzilla.redhat.com/show_bug.cgi?id=1430214.I think the bug maybe some help. > > 2.Convert a windows guest from vmware to kvm. > > Can you please provide the logs for this case? Created attachment 1261197 [details]
v2v-windows
(In reply to kuwei from comment #3) > > I see you are using qemu-kvm-rhev, could you please try with the plain > > qemu-kvm? > > Using qemu-kvm can convert guest from vmware to kvm successfully,you can see > my addition info comment 0. Oh you are right, sorry. > But we can't start the guest on qemu-kvm > packages,it is a bug https://bugzilla.redhat.com/show_bug.cgi?id=1430214.I > think the bug maybe some help. This is a libvirt 3.1.0 regression, see bug 1430258. You should be able to work around it by explicitly using the direct backend: export LIBGUESTFS_BACKEND=direct (Remember to drop this workaround once the libvirt bug is fixed.) > > > 2.Convert a windows guest from vmware to kvm. > > > > Can you please provide the logs for this case? Hmm... does it work if by chance you install libguestfs-rescue together with virt-v2v? It looks like lsof might needed by the base library, because of a local workaround. The timeout happens after 2000 seconds, which happens to be the timeout that we set in the qemu curl driver: https://github.com/libguestfs/libguestfs/blob/0b793550d430cd8135c177f74dec13341c731b6d/v2v/vCenter.ml#L224 In qemu commit http://git.qemu-project.org/?p=qemu.git;a=commit;h=796a060bc0fab40953997976a2e30d9d6235bc7b I changed the curl driver so it's now supposed to print the underlying error if there is one. Since that didn't happen in this case, I conclude that the curl driver really is hitting the timeout, and there is not any other error. While we could increase the timeout again (it was previously increased from 600 - see bug 1146007), it's not clear to me that we should be hitting the timeout at all. That would indicate that the XFS driver in the kernel isn't reading or writing any data over the connection for 2000 seconds, but is instead either hung or doing some kind of processing. I obtained the source image from the source server. However I was able to open the disk locally just fine, both with Fedora 26 and RHEL 7.4-preview, so I don't really know what's going on here. On RHEL 7.4 I am using: kernel-3.10.0-567.el7.x86_64 10:qemu-kvm-rhev-2.8.0-5.el7.x86_64 I also tried qemu-kvm-1.5.3-126.el7.x86_64 Hi, rjones, i try some different qemu-kvm-rhev versions and virt-v2v to convert guest from vmware to kvm, below list is the result. Scenario 1: when i using qemu-kvm-rhev-2.8.0-5.el7.x86_64, virt-v2v-1.36.1-1.el7.x86_64, --- failed virt-v2v-1.36.2-1.el7.x86_64 , ---failed virt-v2v-1.32.7-3.el7.x86_64 --- failed. Scenario 2: Using qemu-kvm-rhev-2.8.0-6.el7.x86_64, virt-v2v-1.36.1-1.el7.x86_64,--- passed virt-v2v-1.36.2-1.el7.x86_64 --- passed. Scenario 3: Using qemu-kvm-rhev-2.6.0-28.el7_3.6.x86_64, virt-v2v-1.36.2-1.el7.x86_64 , ---passed So, I think qemu-kvm-rhev-2.8.0-5.el7.x86_64 maybe has some issues. qemu-kvm-rhev 2.8.0-6 fixes bug 1425700, something to do with the handling of virtio-scsi queues. It is plausible that this change may have fixed this bug. I am going to test the -6 package myself using the reproducer in comment 10. Let's keep this bug open to see if the bug appears again. Can confirm that this appears to be fixed by qemu-kvm-rhev-2.8.0-6.el7. On that basis I will mark it as a duplicate of bug 1425700. *** This bug has been marked as a duplicate of bug 1425700 *** |
Created attachment 1261163 [details] v2v.log Description of problem: Convert a guest from vmware to kvm by virt-v2v can't be finished in 2 hours Version-Release number of selected component (if applicable): virt-v2v-1.36.1-1.el7.x86_64 libguestfs-1.36.1-1.el7.x86_64 libvirt-3.1.0-1.el7.x86_64 qemu-kvm-rhev-2.8.0-5.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1:Convert a rhel7.2 guest from vmware to kvm. # # virt-v2v -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.2-x86_64 --password-file /tmp/passwd -on test1 [ 0.0] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.3.19/?no_verify=1 esx5.5-rhel7.2-x86_64 [ 1.2] Creating an overlay to protect the source from being modified [ 1.8] Initializing the target -o libvirt -os default [ 1.8] Opening the overlay [ 35.0] Inspecting the overlay ^c Result: the above conversion can't be finished in 2 hours. For more information you can see the attached v2v.log. 2.Convert a windows guest from vmware to kvm. # virt-v2v -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2008-x86_64 --password-file /tmp/passwd [ 0.0] Opening the source -i libvirt -ic vpx://root.75.182/data/10.73.72.61/?no_verify=1 esx6.0-win2008-x86_64 [ 1.4] Creating an overlay to protect the source from being modified [ 2.4] Initializing the target -o libvirt -os default [ 2.4] Opening the overlay [ 24.4] Inspecting the overlay [ 109.0] Checking for sufficient free disk space in the guest [ 109.0] Estimating space required on target for each disk [ 109.0] Converting Windows Server (R) 2008 Standard to run on KVM virt-v2v: warning: /usr/share/virt-tools/pnp_wait.exe is missing. Firstboot scripts may conflict with PnP. ^C Result: the above conversion can't be finished in 3 hours. Actual results: As above description Expected results: It should be convert a guest from vmware to kvm successfully in 1 hours. Addition info: Guest can be convert from vmware to kvm with below builds: virt-v2v-1.36.1-1.el7.x86_64 libguestfs-1.36.1-1.el7.x86_64 libvirt-3.1.0-1.el7.x86_64 qemu-kvm-1.5.3-132.el7.x86_64