Bug 1849850

Summary: KVM Importing fails due to missing readinto function on the VMAdapter
Product: [oVirt] vdsm Reporter: michtere
Component: CoreAssignee: Steven Rosenberg <srosenbe>
Status: CLOSED CURRENTRELEASE QA Contact: Nisim Simsolo <nsimsolo>
Severity: high Docs Contact:
Priority: high    
Version: 4.40.3CC: ahadas, bugs, dfodor, egupovav, jiri.slezka, mavital, mtessun, mzamazal, nsimsolo, srosenbe
Target Milestone: ovirt-4.4.2Keywords: Reopened
Target Release: 4.40.25Flags: mtessun: ovirt-4.4?
mtessun: planning_ack+
ahadas: devel_ack+
mavital: testing_ack+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: v4.40.25 Doc Type: Bug Fix
Doc Text:
Cause: The readinto function was missing from the VMAdapter class. Consequence: Importing a disk from a path would fail due to the missing function. Fix: Added the readinto function to the VMAdapter class. Result: KVM importing succeeds.
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-18 07:12:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description michtere 2020-06-23 01:29:15 UTC
Description of problem: KVM import fails due to VMAdapter not containing readinto method

This causes a failure in the kvm2ovirt log:
[    0.4] preparing for copy
[    0.4] Copying disk 1/1 to /rhev/data-center/mnt/blockSD/...../images/...../.....
    (0/100%)^M    (100/100%)^MTraceback (most recent call last):
  File "/usr/libexec/vdsm/kvm2ovirt", line 23, in <module>
    kvm2ovirt.main()
  File "/usr/lib/python3.6/site-packages/vdsm/kvm2ovirt.py", line 269, in main
    handle_path(con, diskno, src, dst, options)
  File "/usr/lib/python3.6/site-packages/vdsm/kvm2ovirt.py", line 233, in handle_path
    download_disk(vmAdapter, physical, physical, dst, options.bufsize)
  File "/usr/lib/python3.6/site-packages/vdsm/kvm2ovirt.py", line 163, in download_disk
    op.run()
  File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/ops.py", line 63, in run
    self._run()
  File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/ops.py", line 147, in _run
    self._receive_chunk(buf, count)
  File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/ops.py", line 162, in _receive_chunk
    n = self._src.readinto(v)
AttributeError: 'VMAdapter' object has no attribute 'readinto'

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Within the Virtual Machine View of the engine, choose the more button and import.
2. Choose a KVM external import and choose next for both screens
3.

Actual results:

The Importing fails in the engine with the following error:

2020-06-23 03:53:44,594+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-42) [50ac1647] EVENT_ID: IMPORTEXPORT_IMPORT_VM_FAILED(1,153), Failed to import Vm pdc to Data Center Default, Cluster Default

Expected results:
The Import should succeed.

Additional info:

Probably Related: Bug 1798175 (https://bugzilla.redhat.com/show_bug.cgi?id=1798175)

Comment 1 Steven Rosenberg 2020-07-15 12:34:55 UTC
This looks like the same issue as the referenced issue which was fixed in 4.40.7 [1]


Maybe it can be retested. The function "readinto" has been added and it is not related to the libvirt sparseness issue.


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1798175

Comment 2 Nisim Simsolo 2020-07-26 14:58:35 UTC
(In reply to Steven Rosenberg from comment #1)
> This looks like the same issue as the referenced issue which was fixed in
> 4.40.7 [1]
> 
> 
> Maybe it can be retested. The function "readinto" has been added and it is
> not related to the libvirt sparseness issue.
> 
> 
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1798175

Retested, issue was not reproduced:
ovirt-engine-4.4.1.8-0.7.el8ev
vdsm-4.40.22-1.el8ev.x86_64
libvirt-daemon-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
qemu-kvm-4.2.0-29.module+el8.2.1+7297+a825794d.x86_64

Comment 3 Jiří Sléžka 2020-07-27 14:59:29 UTC
Hello,

I still experience this bug in oVirt 4.4.1 with vdsm version vdsm-4.40.22-1.el8.x86_64

ovirt-engine-4.4.1.10-1.el8.noarch (hosted engine)

on host involved in import

vdsm-4.40.22-1.el8.x86_64
libvirt-daemon-6.0.0-17.el8.x86_64
qemu-kvm-4.2.0-19.el8.x86_64


[    0.5] preparing for copy
[    0.5] Copying disk 1/1 to /rhev/data-center/mnt/glusterSD/10.0.4.11:_engine/bd15bfa7-67f7-494a-a0b6-138187247541/images/dd414a78-a139-4ef9-9f5a-a785e06028bd/9197763a-9ccd-474d-acb9-8ad3c4afc4d9
Traceback (most recent call last):
  File "/usr/libexec/vdsm/kvm2ovirt", line 23, in <module>
    kvm2ovirt.main()
  File "/usr/lib/python3.6/site-packages/vdsm/kvm2ovirt.py", line 269, in main
    handle_path(con, diskno, src, dst, options)
  File "/usr/lib/python3.6/site-packages/vdsm/kvm2ovirt.py", line 233, in handle_path
    download_disk(vmAdapter, physical, physical, dst, options.bufsize)
  File "/usr/lib/python3.6/site-packages/vdsm/kvm2ovirt.py", line 163, in download_disk
    op.run()
  File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/ops.py", line 51, in run
    self._run()
  File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/ops.py", line 137, in _run
    self._receive_chunk(count)
  File "/usr/lib64/python3.6/site-packages/ovirt_imageio/_internal/ops.py", line 152, in _receive_chunk
    n = self._src.readinto(v)
AttributeError: 'VMAdapter' object has no attribute 'readinto'

Comment 4 Arik 2020-07-27 20:02:57 UTC
Thanks Jiří.
It indeed looks different than bz 1798175 (this one is about VMAdapter while the fix for bz 1798165 added 'readinto' to StreamAdapter).
To reproduce this we need the disk(s) of the source VM to either be placed in a block storage or on a local path on the host (that is not within libvirt's storage pool).

Comment 5 Jiří Sléžka 2020-07-28 16:07:30 UTC
Hello Arik, the source disk is of this vm resides on lvm (on remote kvm of course). What exactly should I do to help debug this? Should I dd this disk to some lvm volume or file on oVirt host? It is possible but what next?

Comment 6 Arik 2020-07-28 20:44:48 UTC
(In reply to Jiří Sléžka from comment #5)
> Hello Arik, the source disk is of this vm resides on lvm (on remote kvm of
> course). What exactly should I do to help debug this? Should I dd this disk
> to some lvm volume or file on oVirt host? It is possible but what next?

Thanks for the additional information.
I think it should be easy for us to reproduce now, no need for additional resources.

Comment 7 Jiří Sléžka 2020-07-29 07:43:43 UTC
just for record, source kvm is CentOS7, vm's definition is this

<domain type='kvm' id='5'>
  <name>dione</name>
  <uuid>518fc09f-ef0d-4f81-a217-b49dadb4c404</uuid>
  <description>dione.slu.cz</description>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Conroe</model>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/vmdata/dione'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:22:d1:84'/>
      <source bridge='br-vlan902'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-5-dione-new/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
      <image compression='off'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <stats period='5'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c73,c919</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c73,c919</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

Comment 8 Steven Rosenberg 2020-08-17 19:16:20 UTC
This bug is assigned to me, but I'm the wrong Steven Rosenberg.

Comment 9 Steven Rosenberg 2020-08-18 08:06:02 UTC
(In reply to Steven Rosenberg from comment #8)
> This bug is assigned to me, but I'm the wrong Steven Rosenberg.

It is assigned to me. Thank you.

Comment 10 Nisim Simsolo 2020-08-24 07:48:08 UTC
Verified:
ovirt-engine-4.4.2.3-0.6.el8ev
vdsm-4.40.26-1.el8ev.x86_64
libvirt-daemon-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
qemu-kvm-4.2.0-29.module+el8.2.1+7297+a825794d.x86_64
KVM server: 
Red Hat Enterprise Linux release 8.2 (Ootpa)
libvirt-daemon-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64
qemu-kvm-4.2.0-29.module+el8.2.1+7297+a825794d.x86_64

1. Import KVM VM with block storage.
2. Verify import succeeds, run VM and verify VM is running and OS booted.
3. Import the same VM again as clone.
4. Verify import succeeds, run VM and verify VM is running and OS booted.
5. Also verified the next table from:
https://bugzilla.redhat.com/show_bug.cgi?id=1663135#c72

Comment 11 Sandro Bonazzola 2020-09-18 07:12:04 UTC
This bugzilla is included in oVirt 4.4.2 release, published on September 17th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.