Bug 1590640

Summary: qemu-kvm: block/io.c:1098: bdrv_co_do_copy_on_readv: Assertion `skip_bytes < pnum' failed.
Product: Red Hat Enterprise Linux 7 Reporter: mxie <mxie>
Component: qemu-kvm-rhevAssignee: Kevin Wolf <kwolf>
Status: CLOSED ERRATA QA Contact: CongLi <coli>
Severity: high Docs Contact:
Priority: high    
Version: 7.6CC: chayang, coli, eblake, juzhang, juzhou, knoel, kwolf, michen, mxie, mzhan, ngu, ptoscano, rjones, tzheng, virt-maint, xiaodwan
Target Milestone: rcKeywords: Regression, TestBlocker
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: P2V
Fixed In Version: qemu-kvm-rhev-2.12.0-8.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-01 11:10:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
virt-p2v-mpt3sas-1.38.2-1
none
virt-p2v-mpt3sas-1.36.10-1
none
screenshot-mpt3sas-1.38.2-1.png
none
virt-p2v-mpt3sas-1.38.2-1-log
none
virt-p2v-mpt3sas-qemu-kvm-rhev-2.12.0-8-log none

Description mxie@redhat.com 2018-06-13 06:06:39 UTC
Created attachment 1450791 [details]
virt-p2v-mpt3sas-1.38.2-1

Description of problem:
Can't convert mpt3sas host by virt-p2v

Version-Release number of selected component (if applicable):
virt-p2v-1.38.2-1.el7.iso
virt-v2v-1.38.2-3.el7.x86_64
libguestfs-1.38.2-3.el7.x86_64
libvirt-4.3.0-1.el7.x86_64
qemu-kvm-rhev-2.12.0-3.el7.x86_64


How reproducible:
100%

Steps to Reproduce:
1.Install rhel7.5 on mpt3sas host
1.Boot mpt3sas host into virt-p2v client
2.Input conversion server info and pass the test connection
3.Input info and start convert but the conversion is failed during opening overlay,details pls refer to log "virt-p2v-mpt3sas-1.38.2-1"


Actual results:
As above description

Expected Results:
Can convert mpt3sas host by virt-p2v

Additional info:
1.Can't reproduce the problem with virt-p2v-1.36.10-1.el7,details pls refer to log "virt-p2v-mpt3sas-1.36.10-1"

Comment 3 mxie@redhat.com 2018-06-13 06:07:26 UTC
Created attachment 1450792 [details]
virt-p2v-mpt3sas-1.36.10-1

Comment 4 mxie@redhat.com 2018-06-13 06:09:11 UTC
Created attachment 1450793 [details]
screenshot-mpt3sas-1.38.2-1.png

Comment 5 Richard W.M. Jones 2018-06-15 12:50:06 UTC
According to the logs it seems to fail without error around 2% into
the disk copy:

qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'raw' '/var/tmp/v2vovl4d64d3.qcow2' '/var/tmp/mpt3sas-sda'
    (0.00/100%)
    (1.00/100%)
    (2.00/100%)

But the error report says ...

> 3.Input info and start convert but the conversion is failed during
> opening overlay,details pls refer to log "virt-p2v-mpt3sas-1.38.2-1"

I think you've got the wrong log files?

Comment 6 mxie@redhat.com 2018-06-15 14:43:37 UTC
I uploaded two logs, one is "virt-p2v-mpt3sas-1.38.2-1" which is the detailed log for the bug, the other one is "virt-p2v-mpt3sas-1.36.10-1" which is the log for additional info's scenario, I also upload a new log "virt-p2v-mpt3sas-1.38.2-1-log" for your reference, thanks

Comment 7 mxie@redhat.com 2018-06-15 14:43:52 UTC
Created attachment 1451948 [details]
virt-p2v-mpt3sas-1.38.2-1-log

Comment 8 Richard W.M. Jones 2018-06-15 14:52:57 UTC
Oh indeed you did, thanks.
The error is .. strange.  It fails at:

  [    1.994948] Error: Driver 'pcspkr' is already registered, aborting...
  libguestfs: child_cleanup: 0x28ba790: child process died

The error is not the pcspkr one, because it appears in other logs.
In normal output we might expect to see other drivers being
loaded such as this one:

  [    0.722479] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
  [    0.723225] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni)

This smells of a qemu crashing bug because of a problem with
emulating an AES-NI instruction.

Is it possible that you still have the file
  /var/log/libvirt/qemu//guestfs-rcm8meh2boll2jnc.log
still around from that run?  It may contain information from
why qemu crashed.

If that file doesn't have any useful information, then perhaps
there was a core dump which was collected by coredumpctl?

Comment 9 mxie@redhat.com 2018-06-15 15:01:50 UTC
Found the file:
 
# cat guestfs-rcm8meh2boll2jnc.log
2018-06-12 10:45:41.214+0000: starting up libvirt version: 4.3.0, package: 1.el7 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2018-05-21-07:46:32, x86-030.build.eng.bos.redhat.com), qemu version: 2.12.0qemu-kvm-rhev-2.12.0-3.el7, hostname: hp-dl380eg8-02.lab.eng.pek2.redhat.com
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none TMPDIR=/var/tmp /usr/libexec/qemu-kvm -name guest=guestfs-rcm8meh2boll2jnc,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-5-guestfs-rcm8meh2boll/master-key.aes -machine pc-i440fx-rhel7.5.0,accel=kvm,usb=off,dump-guest-core=off -cpu host -m 2000 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 0b4dd276-89ad-47d3-928c-718e20b45e4a -display none -no-user-config -nodefaults -device sga -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-5-guestfs-rcm8meh2boll/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -no-acpi -boot strict=on -kernel /var/tmp/.guestfs-0/appliance.d/kernel -initrd /var/tmp/.guestfs-0/appliance.d/initrd -append 'panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 guestfs_network=1 TERM=linux guestfs_identifier=v2v' -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/var/tmp/v2vovlc709b4.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=unsafe,copy-on-read=on,discard=unmap -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1,write-cache=on -drive file=/tmp/libguestfsJuuRdk/overlay1.qcow2,format=qcow2,if=none,id=drive-scsi0-0-1-0,cache=unsafe -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0,write-cache=on -chardev socket,id=charserial0,path=/tmp/libguestfsicarXS/console.sock -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/tmp/libguestfsicarXS/guestfsd.sock -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x4 -netdev user,id=usernet,net=169.254.0.0/16 -device virtio-net-pci,netdev=usernet -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
2018-06-12 10:45:41.215+0000: Domain id=5 is tainted: custom-argv
2018-06-12 10:45:41.215+0000: Domain id=5 is tainted: host-cpu
qemu-kvm: block/io.c:1098: bdrv_co_do_copy_on_readv: Assertion `skip_bytes < pnum' failed.
2018-06-12 10:45:47.513+0000: shutting down, reason=destroyed

Comment 10 Kevin Wolf 2018-07-06 16:27:15 UTC
Would it be possible to get a full backtrace from a core dump?

Also, can you please post the 'qemu-img info' output for /var/tmp/v2vovlc709b4.qcow2, i.e. the image with copy-on-read=on? I can trigger an assertion failure with an image whose size is not aligned to full qcow2 clusters, but I don't know if this is your case.

Comment 11 Richard W.M. Jones 2018-07-07 06:49:59 UTC
I don't know if mxie will still have that temporary file around.

However we did run qemu-img info on that file (although early on,
before writing data to it), and the results are in the log:

libguestfs: command: run: qemu-img
libguestfs: command: run: \ info
libguestfs: command: run: \ --output json
libguestfs: command: run: \ /var/tmp/v2vovlc709b4.qcow2
libguestfs: parse_json: qemu-img info JSON output:\n{\n    "backing-filename-format": "raw",\n    "virtual-size": 1000204886016,\n    "filename": "/var/tmp/v2vovlc709b4.qcow2",\n    "cluster-size": 65536,\n    "format": "qcow2",\n    "actual-size": 212992,\n    "format-specific": {\n        "type": "qcow2",\n        "data": {\n            "compat": "1.1",\n            "lazy-refcounts": false,\n            "refcount-bits": 16,\n            "corrupt": false\n        }\n    },\n    "full-backing-filename": "nbd:localhost:40326",\n    "backing-filename": "nbd:localhost:40326",\n    "dirty-flag": false\n}\n\n

In particular the virtual size will not have changed.

Comment 12 Richard W.M. Jones 2018-07-07 07:11:03 UTC
1000204886016/65536 = 15261915.375

Comment 13 mxie@redhat.com 2018-07-10 08:34:45 UTC
(In reply to Kevin Wolf from comment #10)
> Would it be possible to get a full backtrace from a core dump?
> 
> Also, can you please post the 'qemu-img info' output for
> /var/tmp/v2vovlc709b4.qcow2, i.e. the image with copy-on-read=on? I can
> trigger an assertion failure with an image whose size is not aligned to full
> qcow2 clusters, but I don't know if this is your case.


Check overlay image with below builds:
virt-v2v-1.38.2-6.el7.x86_64
libguestfs-1.38.2-6.el7.x86_64
libvirt-4.5.0-2.el7.x86_64
qemu-kvm-rhev-2.12.0-7.el7.x86_64


1.Check the images during p2v conversion, I have no idea about the image with copy-on-read=on, We usually check "copy-on-read=on" in virt-v2v conversion log
# qemu-img info v2vovl770e0a.qcow2 
image: v2vovl770e0a.qcow2
file format: qcow2
virtual size: 932G (1000204886016 bytes)
disk size: 400K
cluster_size: 65536
backing file: nbd:localhost:46493
backing file format: raw
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

Comment 15 mxie@redhat.com 2018-07-23 06:03:46 UTC
Also can reproduce the bug when convert MD host by virt-p2v

Packages:
virt-v2v-1.38.2-8.el7.x86_64
libguestfs-1.38.2-8.el7.x86_64
libvirt-4.5.0-3.el7.x86_64
qemu-kvm-rhev-2.12.0-7.el7.x86_64
virt-p2v-1.38.2-2.el7.iso


Steps:
1.Install 6.9 OS with software RAID1 on physical machine
2.Boot host into virt-p2v client after finishing OS installation
3.Input virt-v2v conversion server info and pass the test connection
4.Input conversion info, then start conversion, but the conversion is failed during opening the overlay,check the log as below

# cat virt-p2v-20180720-mywjy36r/virt-v2v-conversion-log.txt 
....
[    2.949746] input: PC Speaker as /devices/platform/pcspkr/input/input3
[    2.998333] sd 2:0:0:0: Attached scsi generic sg0 type 0
[    3.008415] cryptd: max_cpu_qlen set to 1000
[    3.012598] sd 2:0:1:0: Attached scsi generic sg1 type 0
[    3.023188] sd 2:0:2:0: Attached scsi generic sg2 type 0
[    3.033783] sd 2:0:3:0: Attached scsi generic sg3 type 0
[    3.129503] AVX version of gcm_enc/dec engaged.
[    3.131644] AES CTR mode by8 optimization enabled
libguestfs: child_cleanup: 0x1f6e990: child process died
libguestfs: trace: v2v: launch = -1 (error)
virt-v2v: error: libguestfs error: guestfs_launch failed, see earlier error 
messages
....

#cat /var/log/libvirt/qemu/guestfs-2e2g011qbc3owfsf.log
2018-07-20 11:24:51.901+0000: starting up libvirt version: 4.5.0, package: 3.el7 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2018-07-13-12:14:36, x86-019.build.eng.bos.redhat.com), qemu version: 2.12.0qemu-kvm-rhev-2.12.0-7.el7, kernel: 3.10.0-915.el7.x86_64, hostname: hp-dl380eg8-02.lab.eng.pek2.redhat.com
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none TMPDIR=/var/tmp /usr/libexec/qemu-kvm -name guest=guestfs-2e2g011qbc3owfsf,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-39-guestfs-2e2g011qbc3o/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu host -m 2000 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid f0916c1a-c6f6-4a97-b32d-06fcfb8a87dd -display none -no-user-config -nodefaults -device sga -chardev socket,id=charmonitor,fd=29,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -no-acpi -boot strict=on -kernel /var/tmp/.guestfs-0/appliance.d/kernel -initrd /var/tmp/.guestfs-0/appliance.d/initrd -append 'panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdd selinux=0 guestfs_verbose=1 guestfs_network=1 TERM=linux guestfs_identifier=v2v' -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/var/tmp/v2vovl639132.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=unsafe,copy-on-read=on,discard=unmap -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1,write-cache=on -drive file=/var/tmp/v2vovl4b6a69.qcow2,format=qcow2,if=none,id=drive-scsi0-0-1-0,cache=unsafe,copy-on-read=on,discard=unmap -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0,write-cache=on -drive file=/var/tmp/v2vovl710999.qcow2,format=qcow2,if=none,id=drive-scsi0-0-2-0,cache=unsafe,copy-on-read=on,discard=unmap -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=2,lun=0,drive=drive-scsi0-0-2-0,id=scsi0-0-2-0,write-cache=on -drive file=/tmp/libguestfs00EUUT/overlay1.qcow2,format=qcow2,if=none,id=drive-scsi0-0-3-0,cache=unsafe -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=3,lun=0,drive=drive-scsi0-0-3-0,id=scsi0-0-3-0,write-cache=on -chardev socket,id=charserial0,path=/tmp/libguestfsW0HvHm/console.sock -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/tmp/libguestfsW0HvHm/guestfsd.sock -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x4 -netdev user,id=usernet,net=169.254.0.0/16 -device virtio-net-pci,netdev=usernet -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
2018-07-20 11:24:51.901+0000: Domain id=39 is tainted: custom-argv
2018-07-20 11:24:51.901+0000: Domain id=39 is tainted: host-cpu
qemu-kvm: block/io.c:1098: bdrv_co_do_copy_on_readv: Assertion `skip_bytes < pnum' failed.
2018-07-20 11:25:05.254+0000: shutting down, reason=crashed



Hi Kevin,

   Could you please help to fix the bug asap because this bug has blocked many p2v tests? Thanks

Comment 16 Richard W.M. Jones 2018-07-23 08:09:59 UTC
Fix is upstream in qemu commit b0ddcbbb36a66a605eb232b905cb49b1cc72e74e.

I have ACKed the downstream patch and it should go into the next
version of qemu-kvm-rhev fairly soon.

Comment 17 Miroslav Rezanina 2018-07-24 14:27:02 UTC
Fix included in qemu-kvm-rhev-2.12.0-8.el7

Comment 19 mxie@redhat.com 2018-07-26 09:32:01 UTC
Verify the bug with builds:
virt-v2v-1.38.2-9.el7.x86_64
libguestfs-1.38.2-9.el7.x86_64
libvirt-4.5.0-4.el7.x86_64
qemu-kvm-rhev-2.12.0-8.el7.x86_64
virt-p2v-1.38.2-2.el7.iso


Steps:
1.Install rhel7.5 on mpt3sas host
2.Boot mpt3sas host into virt-p2v client
3.Input conversion server info and pass the test connection
4 Input conversion info,then start convert 
5.But the conversion is failed during copying disk with below error,details pls refer to log "virt-p2v-mpt3sas-qemu-kvm-rhev-2.12.0-8"
....
qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'qcow2' '/var/tmp/v2vovld93f8f.qcow2' '/var/lib/libvirt/images/default/mpt3sas-qcow2-sda'
qemu-img: Unexpected end-of-file before all bytes were read
qemu-img: error while reading sector 1150070784: Input/output error
qemu-img: Connection closed
....

6 Check guestfs log

# cat /var/log/libvirt/qemu/guestfs-t3x85ya76lmwth9y.log 
2018-07-26 06:36:14.505+0000: starting up libvirt version: 4.5.0, package: 4.el7 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2018-07-23-09:52:46, x86-041.build.eng.bos.redhat.com), qemu version: 2.12.0qemu-kvm-rhev-2.12.0-8.el7, kernel: 3.10.0-924.el7.x86_64, hostname: hp-dl380eg8-02.lab.eng.pek2.redhat.com
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none TMPDIR=/var/tmp /usr/libexec/qemu-kvm -name guest=guestfs-t3x85ya76lmwth9y,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-10-guestfs-t3x85ya76lmw/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu host -m 2000 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 1512de3f-11b9-4806-8c0b-f90ae5194cc5 -display none -no-user-config -nodefaults -device sga -chardev socket,id=charmonitor,fd=28,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -no-acpi -boot strict=on -kernel /var/tmp/.guestfs-0/appliance.d/kernel -initrd /var/tmp/.guestfs-0/appliance.d/initrd -append 'panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 guestfs_network=1 TERM=linux guestfs_identifier=v2v' -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/var/tmp/v2vovld93f8f.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=unsafe,copy-on-read=on,discard=unmap -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1,write-cache=on -drive file=/tmp/libguestfsyTuaLD/overlay1.qcow2,format=qcow2,if=none,id=drive-scsi0-0-1-0,cache=unsafe -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0,write-cache=on -chardev socket,id=charserial0,path=/tmp/libguestfsBMzPhf/console.sock -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/tmp/libguestfsBMzPhf/guestfsd.sock -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x4 -netdev user,id=usernet,net=169.254.0.0/16 -device virtio-net-pci,netdev=usernet -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
2018-07-26 06:36:14.505+0000: Domain id=10 is tainted: custom-argv
2018-07-26 06:36:14.505+0000: Domain id=10 is tainted: host-cpu
2018-07-26T06:39:07.262136Z qemu-kvm: terminating on signal 15 from pid 1515 (<unknown process>)
2018-07-26 06:39:07.850+0000: shutting down, reason=destroyed



Hi rjones,

   Could you please help to check if above problem is still caused by qemu-kvm-rhev?

Comment 20 mxie@redhat.com 2018-07-26 09:33:04 UTC
Created attachment 1470668 [details]
virt-p2v-mpt3sas-qemu-kvm-rhev-2.12.0-8-log

Comment 21 Richard W.M. Jones 2018-07-26 11:38:29 UTC
Getting back to basics, I found a very easy way to reproduce the
original bug by creating a guest with an irregular size:

  $ virt-builder fedora-27 --size=7234567890b
  $ virt-v2v -i disk fedora-27.img -o null
  ...
  [   0.4] Opening the overlay
  virt-v2v: error: libguestfs error: guestfs_launch failed, see earlier error messages

And if you look in the ~/.cache/libvirt/qemu/log/guestfs*.log file:

  qemu-system-x86_64: /builddir/build/BUILD/qemu-2.12.0/block/io.c:1098: bdrv_co_do_copy_on_readv: Assertion `skip_bytes < pnum' failed.

I then tried different versions of qemu:

upstream qemu from git (18a398f6) => WORKING.

qemu-kvm-rhev-2.10.0-21.el7_5.4.x86_64 => WORKING.

qemu-kvm-rhev-2.12.0-3.el7.x86_64 => BUG REPRODUCED.

qemu-kvm-rhev-2.12.0-8.el7.x86_64 => WORKING.

I would say that the bug has been fixed in qemu-kvm, and the bug
you are seeing is a new problem.

Comment 22 Richard W.M. Jones 2018-07-26 11:41:22 UTC
(In reply to mxie from comment #19)
> 6 Check guestfs log
> 
> # cat /var/log/libvirt/qemu/guestfs-t3x85ya76lmwth9y.log 

One problem is that this log file does not correspond to the guest used in the
log files attached in comment 20.  (You need to look at guestfs-rcm8meh2boll2jnc.log)

Comment 23 mxie@redhat.com 2018-07-27 09:15:24 UTC
(In reply to Richard W.M. Jones from comment #22)
> (In reply to mxie from comment #19)
> > 6 Check guestfs log
> > 
> > # cat /var/log/libvirt/qemu/guestfs-t3x85ya76lmwth9y.log 
> 
> One problem is that this log file does not correspond to the guest used in
> the
> log files attached in comment 20.  (You need to look at
> guestfs-rcm8meh2boll2jnc.log)

I just found below info named "guestfs-xxx" in virt-v2v-conversion-log.txt, but didn't find info "guestfs-rcm8meh2boll2jnc.log"

# cat virt-p2v-mpt3sas-qemu-kvm-rhev-2.12.0-8/virt-v2v-conversion-log.txt |grep guestfs-
libguestfs: guest random name = guestfs-t3x85ya76lmwth9y
.....

# cat virt-p2v-mpt3sas-qemu-kvm-rhev-2.12.0-8/virt-v2v-conversion-log.txt |grep guestfs-rcm8m

nothing

Hi rjones,
    Do I need to open a new bug for the new problem?

Comment 24 mxie@redhat.com 2018-08-07 03:18:54 UTC
Verify the bug with builds:
virt-v2v-1.38.2-10.el7.x86_64
libguestfs-1.38.2-10.el7.x86_64
libvirt-4.5.0-6.el7.x86_64
qemu-kvm-rhev-2.12.0-9.el7.x86_64

Steps:
1.Install rhel7.5 on mpt3sas host
2.Boot mpt3sas host into virt-p2v client
3.Input conversion server info and pass the test connection
4 Input conversion info,then start convert 
5.But the conversion is failed during copying disk with below error
....
qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'qcow2' '/var/tmp/v2vovl72ee24.qcow2' '/tmp/v2v.GsQrfs/8f20a49b-8ed5-4d9c-b266-71b2fc12d658/images/d2ddbebd-ab17-4cab-a0a6-da89098ac598/7ad39ae5-76a8-4012-9e1e-c5306e597bb1'
qemu-img: Unexpected end-of-file before all bytes were read
qemu-img: error while reading sector 1779191808: Input/output error
qemu-img: Connection closed
qemu-img: error while reading sector 1779187712: Input/output error
qemu-img: Connection closed
....
virt-v2v: error: qemu-img command failed, see earlier errors
....

6 Check guestfs log
# cat /var/log/libvirt/qemu/guestfs-q0d16weha17c396t.log 
2018-08-06 09:30:50.843+0000: starting up libvirt version: 4.5.0, package: 6.el7 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2018-08-01-09:52:16, x86-017.build.eng.bos.redhat.com), qemu version: 2.12.0qemu-kvm-rhev-2.12.0-9.el7, kernel: 3.10.0-924.el7.x86_64, hostname: hp-dl380eg8-02.lab.eng.pek2.redhat.com
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none TMPDIR=/var/tmp /usr/libexec/qemu-kvm -name guest=guestfs-q0d16weha17c396t,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-52-guestfs-q0d16weha17c/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu host -m 2000 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 6a08f6c0-e2c6-41a4-9003-069a9f2485d5 -display none -no-user-config -nodefaults -device sga -chardev socket,id=charmonitor,fd=32,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-reboot -no-acpi -boot strict=on -kernel /var/tmp/.guestfs-0/appliance.d/kernel -initrd /var/tmp/.guestfs-0/appliance.d/initrd -append 'panic=1 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 guestfs_network=1 TERM=linux guestfs_identifier=v2v' -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 -drive file=/var/tmp/v2vovl72ee24.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=unsafe,copy-on-read=on,discard=unmap -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1,write-cache=on -drive file=/tmp/libguestfsOsvZax/overlay1.qcow2,format=qcow2,if=none,id=drive-scsi0-0-1-0,cache=unsafe -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0,write-cache=on -chardev socket,id=charserial0,path=/tmp/libguestfsETGXND/console.sock -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/tmp/libguestfsETGXND/guestfsd.sock -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x4 -netdev user,id=usernet,net=169.254.0.0/16 -device virtio-net-pci,netdev=usernet -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
2018-08-06 09:30:50.843+0000: Domain id=52 is tainted: custom-argv
2018-08-06 09:30:50.843+0000: Domain id=52 is tainted: host-cpu
2018-08-06T09:36:46.597768Z qemu-kvm: terminating on signal 15 from pid 50457 (<unknown process>)
2018-08-06 09:36:47.433+0000: shutting down, reason=destroyed


Result: 
    there is no error message"qemu-kvm: block/io.c:1098: bdrv_co_do_copy_on_readv: Assertion `skip_bytes < pnum' failed." in guestfs log, the failure of conversion is caused by unknown process on qemu-kvm which will be tracked by bug1613110.So move the bug from ON_QA to VERIFIED

Comment 25 errata-xmlrpc 2018-11-01 11:10:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443