Bug 1459801

Summary: nbd/server.c:nbd_receive_request():L706: read failed when do migration_cancel
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Suqin Huang <shuang>
Component: qemu-kvmAssignee: Eric Blake <eblake>
qemu-kvm sub component: NBD QA Contact: zixchen
Status: CLOSED WONTFIX Docs Contact:
Severity: medium    
Priority: medium CC: afrosi, aliang, chayang, coli, eblake, juzhang, knoel, rbalakri, virt-maint, xuwei
Version: ---Keywords: Triaged
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-01-15 07:37:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1458725, 1473046    

Description Suqin Huang 2017-06-08 08:22:21 UTC
Description of problem:


Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.9.0-8.el7.x86_64

How reproducible:
2/2

Steps to Reproduce:
1. Server:

# qemu-nbd -f qcow2 rhel74-64-virtio.qcow2 -p 9000 -t --share=2

2. Client:
Boot up guest

cmd:

/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga cirrus  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_dGGu2S/monitor-qmpmonitor1-20170608-041234-CUqSlEp0,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_dGGu2S/monitor-catch_monitor-20170608-041234-CUqSlEp0,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idHoC3M4  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_dGGu2S/serial-serial0-20170608-041234-CUqSlEp0,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20170608-041234-CUqSlEp0,path=/var/tmp/avocado_dGGu2S/seabios-20170608-041234-CUqSlEp0,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20170608-041234-CUqSlEp0,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=raw,file=nbd:10.73.196.131:9000 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x3 \
    -device virtio-net-pci,mac=9a:15:16:17:18:19,id=idmk5xPz,vectors=4,netdev=iddxJKCe,bus=pci.0,addr=0x4  \
    -netdev tap,id=iddxJKCe,vhost=on,vhostfd=37,fd=36 \
    -m 4096  \
    -smp 4,cores=2,threads=1,sockets=2  \
    -cpu 'Westmere',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :2  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -incoming tcp:0:5200


3. Do live migration
{'execute': 'migrate', 'arguments': {'uri': 'tcp:localhost:5200', 'blk': False, 'inc': False}, 'id': 'Ig1TYidk'}

Actual results:

nbd/server.c:nbd_receive_request():L706: read failed error show in server
guest can be migrated successfully


Expected results:


Additional info:

Comment 2 Suqin Huang 2017-06-08 09:02:54 UTC
the error happen before the migration operation.

boot up src image and boot up dst image, then the error show

Comment 5 Suqin Huang 2017-06-14 09:11:58 UTC
I can reproduce with 

qemu-kvm-rhev-2.8.0-6.el7.x86_64

do migration cancel in src guest:

src:
(qemu) migrate -d tcp:0:5200
(qemu) migrate_cancel 

result:

dst guest:
(qemu) qemu-kvm: Unknown combination of migration flags: 0
qemu-kvm: error while loading state section id 3(ram)
qemu-kvm: load of migration failed: Invalid argument


server:

# qemu-nbd -f qcow2 rhel74-64-virtio-scsi.qcow2 --share=4 -t -p 9000
nbd/server.c:nbd_receive_request():L710: read failed

Comment 6 Suqin Huang 2017-06-14 09:13:03 UTC
not the same as Bug 1458725, as it just happen when do migrate_cancel.

migration canbe do successfully if didn't cancel it.

Comment 7 Eric Blake 2018-01-17 20:53:05 UTC
I wonder if NBD drives should be a migration blocker? After all, having two separate clients (source and destination of the qemu migration) both trying to connect as read-write clients to the same NBD server may not always work right.  The NBD protocol permits servers to allow multiple clients, but does not require it to work well, and I don't know if qemu-nbd as a server is designed to handle this sort of scenario.

Comment 9 Tingting Mao 2019-03-28 12:01:08 UTC
Reproduced this bug in rhel8.

Tested with:
qemu-kvm-3.1.0-20.module+el8+2904+e658c755
kernel-4.18.0-80.el8


Steps:

Server:
# qemu-nbd -f qcow2 sn.qcow2 -t -p 9000 --share=2

Client:
1. Boot guest in source 
# /usr/libexec/qemu-kvm \
        -name 'guest' \
        -machine q35 \
        -nodefaults \
        -vga qxl \
        -vnc :1 \
        -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0,addr=0x3 \
        -blockdev driver=nbd,cache.direct=on,cache.no-flush=off,server.host=localhost,server.port=9000,server.type=inet,node-name=my_file \
        -blockdev driver=raw,node-name=my,file=my_file \
        -device scsi-hd,drive=my \
        -monitor stdio \
        -m 8192 \
        -smp 8 \
        -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b3,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0,addr=0x9  \
        -netdev tap,id=idxgXAlm \
        -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/timao/monitor-qmpmonitor1-20180220-094308-h9I6hRsI,server,nowait \
        -mon chardev=qmp_id_qmpmonitor1,mode=control  \
        -device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0 \
2. Boot guest in target
# /usr/libexec/qemu-kvm \
        -name 'guest' \
        -machine q35 \
        -nodefaults \
        -vga qxl \
        -vnc :2 \
        -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0,addr=0x3 \
        -blockdev driver=nbd,cache.direct=on,cache.no-flush=off,server.host=localhost,server.port=9000,server.type=inet,node-name=my_file \
        -blockdev driver=raw,node-name=my,file=my_file \
        -device scsi-hd,drive=my \
        -monitor stdio \
        -m 8192 \
        -smp 8 \
        -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b3,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0,addr=0x9  \
        -netdev tap,id=idxgXAlm \
        -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/timao/monitor-qmpmonitor1-20180220-094308-h9I6hRsI,server,nowait \
        -mon chardev=qmp_id_qmpmonitor1,mode=control  \
        -device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0 \
	-incoming tcp:0:5200

3. Migration and cancel in source
(qemu) migrate -d tcp:0:5200
(qemu) migrate_cancel


Result:

Target:
(qemu) qemu-kvm: check_section_footer: Read section footer failed: -5
qemu-kvm: load of migration failed: Invalid argument

Server:
# qemu-nbd -f qcow2 sn.qcow2 -t -p 9000 --share=2 
Disconnect client, due to: Unexpected end-of-file before all bytes were read
Disconnect client, due to: Unexpected end-of-file before all bytes were read
Disconnect client, due to: Unexpected end-of-file before all bytes were read

Comment 10 Ademar Reis 2020-02-05 22:44:01 UTC
QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks

Comment 12 Alice Frosi 2020-10-13 08:58:29 UTC
Reproduced this issue with :
  qemu-kvm-5.1.0-10.module+el8.3.0+8254+568ca30d
  kernel-4.18.0-234

Same steps as Comment9:
qemu-nbd -f qcow2 /var/lib/libvirt/images/f31.qcow2 -p 9000 -t --share=2

1- Boot source guest
/usr/libexec/qemu-kvm \
        -name 'guest' \
        -machine q35 \
        -nodefaults \
        -vga qxl \
        -vnc :1 \
        -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0,addr=0x3 \
        -blockdev driver=nbd,cache.direct=on,cache.no-flush=off,server.host=localhost,server.port=9000,server.type=inet,node-name=my_file \
        -blockdev driver=raw,node-name=my,file=my_file \
        -device scsi-hd,drive=my \
        -monitor stdio \
        -m 8192 \
        -smp 8 \
        -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b3,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0,addr=0x9  \
        -netdev tap,id=idxgXAlm \
        -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor,server,nowait \
        -mon chardev=qmp_id_qmpmonitor1,mode=control  \
        -device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0

2- Boot dest guest 
/usr/libexec/qemu-kvm \
        -name 'guest' \
        -machine q35 \
        -nodefaults \
        -vga qxl \
        -vnc :2 \
        -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0,addr=0x3 \
        -blockdev driver=nbd,cache.direct=on,cache.no-flush=off,server.host=localhost,server.port=9000,server.type=inet,node-name=my_file \
        -blockdev driver=raw,node-name=my,file=my_file \
        -device scsi-hd,drive=my \
        -monitor stdio \
        -m 8192 \
        -smp 8 \
        -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b3,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0,addr=0x9  \
        -netdev tap,id=idxgXAlm \
        -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-target,server,nowait \
        -mon chardev=qmp_id_qmpmonitor1,mode=control  \
        -device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0 \
	-incoming tcp:0:5200

3- Migration and cancel in source
(qemu) migrate -d tcp:0:5200
(qemu) migrate_cancel

Result:
In target: 
qemu-kvm: error while loading state section id 1(ram)
qemu-kvm: load of migration failed: Invalid argument

In server:
qemu-nbd: Disconnect client, due to: Failed to read request: Unexpected end-of-file before all bytes were read
qemu-nbd: Disconnect client, due to: Failed to read request: Unexpected end-of-file before all bytes were read
qemu-nbd: Disconnect client, due to: Failed to read request: Unexpected end-of-file before all bytes were read

Comment 15 RHEL Program Management 2021-01-15 07:37:39 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Comment 16 zixchen 2021-01-15 09:37:01 UTC
With qemu-kvm-5.2.0-1.module+el8.4.0+9091+650b220a.x86_64, it reproduce the bug, but libvirt doesn't support such option 'migrate_cancel'
Hence, agree to closed this.