Bug 1459801 - nbd/server.c:nbd_receive_request():L706: read failed when do migration_cancel
Summary: nbd/server.c:nbd_receive_request():L706: read failed when do migration_cancel
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: ---
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 8.0
Assignee: Eric Blake
QA Contact: zixchen
URL:
Whiteboard:
Depends On:
Blocks: 1458725 1473046
TreeView+ depends on / blocked
 
Reported: 2017-06-08 08:22 UTC by Suqin Huang
Modified: 2021-01-15 09:37 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-01-15 07:37:39 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Suqin Huang 2017-06-08 08:22:21 UTC
Description of problem:


Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.9.0-8.el7.x86_64

How reproducible:
2/2

Steps to Reproduce:
1. Server:

# qemu-nbd -f qcow2 rhel74-64-virtio.qcow2 -p 9000 -t --share=2

2. Client:
Boot up guest

cmd:

/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults  \
    -vga cirrus  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_dGGu2S/monitor-qmpmonitor1-20170608-041234-CUqSlEp0,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_dGGu2S/monitor-catch_monitor-20170608-041234-CUqSlEp0,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idHoC3M4  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_dGGu2S/serial-serial0-20170608-041234-CUqSlEp0,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20170608-041234-CUqSlEp0,path=/var/tmp/avocado_dGGu2S/seabios-20170608-041234-CUqSlEp0,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20170608-041234-CUqSlEp0,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=raw,file=nbd:10.73.196.131:9000 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x3 \
    -device virtio-net-pci,mac=9a:15:16:17:18:19,id=idmk5xPz,vectors=4,netdev=iddxJKCe,bus=pci.0,addr=0x4  \
    -netdev tap,id=iddxJKCe,vhost=on,vhostfd=37,fd=36 \
    -m 4096  \
    -smp 4,cores=2,threads=1,sockets=2  \
    -cpu 'Westmere',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :2  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -incoming tcp:0:5200


3. Do live migration
{'execute': 'migrate', 'arguments': {'uri': 'tcp:localhost:5200', 'blk': False, 'inc': False}, 'id': 'Ig1TYidk'}

Actual results:

nbd/server.c:nbd_receive_request():L706: read failed error show in server
guest can be migrated successfully


Expected results:


Additional info:

Comment 2 Suqin Huang 2017-06-08 09:02:54 UTC
the error happen before the migration operation.

boot up src image and boot up dst image, then the error show

Comment 5 Suqin Huang 2017-06-14 09:11:58 UTC
I can reproduce with 

qemu-kvm-rhev-2.8.0-6.el7.x86_64

do migration cancel in src guest:

src:
(qemu) migrate -d tcp:0:5200
(qemu) migrate_cancel 

result:

dst guest:
(qemu) qemu-kvm: Unknown combination of migration flags: 0
qemu-kvm: error while loading state section id 3(ram)
qemu-kvm: load of migration failed: Invalid argument


server:

# qemu-nbd -f qcow2 rhel74-64-virtio-scsi.qcow2 --share=4 -t -p 9000
nbd/server.c:nbd_receive_request():L710: read failed

Comment 6 Suqin Huang 2017-06-14 09:13:03 UTC
not the same as Bug 1458725, as it just happen when do migrate_cancel.

migration canbe do successfully if didn't cancel it.

Comment 7 Eric Blake 2018-01-17 20:53:05 UTC
I wonder if NBD drives should be a migration blocker? After all, having two separate clients (source and destination of the qemu migration) both trying to connect as read-write clients to the same NBD server may not always work right.  The NBD protocol permits servers to allow multiple clients, but does not require it to work well, and I don't know if qemu-nbd as a server is designed to handle this sort of scenario.

Comment 9 Tingting Mao 2019-03-28 12:01:08 UTC
Reproduced this bug in rhel8.

Tested with:
qemu-kvm-3.1.0-20.module+el8+2904+e658c755
kernel-4.18.0-80.el8


Steps:

Server:
# qemu-nbd -f qcow2 sn.qcow2 -t -p 9000 --share=2

Client:
1. Boot guest in source 
# /usr/libexec/qemu-kvm \
        -name 'guest' \
        -machine q35 \
        -nodefaults \
        -vga qxl \
        -vnc :1 \
        -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0,addr=0x3 \
        -blockdev driver=nbd,cache.direct=on,cache.no-flush=off,server.host=localhost,server.port=9000,server.type=inet,node-name=my_file \
        -blockdev driver=raw,node-name=my,file=my_file \
        -device scsi-hd,drive=my \
        -monitor stdio \
        -m 8192 \
        -smp 8 \
        -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b3,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0,addr=0x9  \
        -netdev tap,id=idxgXAlm \
        -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/timao/monitor-qmpmonitor1-20180220-094308-h9I6hRsI,server,nowait \
        -mon chardev=qmp_id_qmpmonitor1,mode=control  \
        -device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0 \
2. Boot guest in target
# /usr/libexec/qemu-kvm \
        -name 'guest' \
        -machine q35 \
        -nodefaults \
        -vga qxl \
        -vnc :2 \
        -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0,addr=0x3 \
        -blockdev driver=nbd,cache.direct=on,cache.no-flush=off,server.host=localhost,server.port=9000,server.type=inet,node-name=my_file \
        -blockdev driver=raw,node-name=my,file=my_file \
        -device scsi-hd,drive=my \
        -monitor stdio \
        -m 8192 \
        -smp 8 \
        -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b3,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0,addr=0x9  \
        -netdev tap,id=idxgXAlm \
        -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/timao/monitor-qmpmonitor1-20180220-094308-h9I6hRsI,server,nowait \
        -mon chardev=qmp_id_qmpmonitor1,mode=control  \
        -device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0 \
	-incoming tcp:0:5200

3. Migration and cancel in source
(qemu) migrate -d tcp:0:5200
(qemu) migrate_cancel


Result:

Target:
(qemu) qemu-kvm: check_section_footer: Read section footer failed: -5
qemu-kvm: load of migration failed: Invalid argument

Server:
# qemu-nbd -f qcow2 sn.qcow2 -t -p 9000 --share=2 
Disconnect client, due to: Unexpected end-of-file before all bytes were read
Disconnect client, due to: Unexpected end-of-file before all bytes were read
Disconnect client, due to: Unexpected end-of-file before all bytes were read

Comment 10 Ademar Reis 2020-02-05 22:44:01 UTC
QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks

Comment 12 Alice Frosi 2020-10-13 08:58:29 UTC
Reproduced this issue with :
  qemu-kvm-5.1.0-10.module+el8.3.0+8254+568ca30d
  kernel-4.18.0-234

Same steps as Comment9:
qemu-nbd -f qcow2 /var/lib/libvirt/images/f31.qcow2 -p 9000 -t --share=2

1- Boot source guest
/usr/libexec/qemu-kvm \
        -name 'guest' \
        -machine q35 \
        -nodefaults \
        -vga qxl \
        -vnc :1 \
        -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0,addr=0x3 \
        -blockdev driver=nbd,cache.direct=on,cache.no-flush=off,server.host=localhost,server.port=9000,server.type=inet,node-name=my_file \
        -blockdev driver=raw,node-name=my,file=my_file \
        -device scsi-hd,drive=my \
        -monitor stdio \
        -m 8192 \
        -smp 8 \
        -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b3,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0,addr=0x9  \
        -netdev tap,id=idxgXAlm \
        -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor,server,nowait \
        -mon chardev=qmp_id_qmpmonitor1,mode=control  \
        -device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0

2- Boot dest guest 
/usr/libexec/qemu-kvm \
        -name 'guest' \
        -machine q35 \
        -nodefaults \
        -vga qxl \
        -vnc :2 \
        -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0,addr=0x3 \
        -blockdev driver=nbd,cache.direct=on,cache.no-flush=off,server.host=localhost,server.port=9000,server.type=inet,node-name=my_file \
        -blockdev driver=raw,node-name=my,file=my_file \
        -device scsi-hd,drive=my \
        -monitor stdio \
        -m 8192 \
        -smp 8 \
        -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b3,id=idMmq1jH,vectors=4,netdev=idxgXAlm,bus=pcie.0,addr=0x9  \
        -netdev tap,id=idxgXAlm \
        -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-target,server,nowait \
        -mon chardev=qmp_id_qmpmonitor1,mode=control  \
        -device pcie-root-port,id=pcie.0-root-port-8,slot=8,chassis=8,addr=0x8,bus=pcie.0 \
	-incoming tcp:0:5200

3- Migration and cancel in source
(qemu) migrate -d tcp:0:5200
(qemu) migrate_cancel

Result:
In target: 
qemu-kvm: error while loading state section id 1(ram)
qemu-kvm: load of migration failed: Invalid argument

In server:
qemu-nbd: Disconnect client, due to: Failed to read request: Unexpected end-of-file before all bytes were read
qemu-nbd: Disconnect client, due to: Failed to read request: Unexpected end-of-file before all bytes were read
qemu-nbd: Disconnect client, due to: Failed to read request: Unexpected end-of-file before all bytes were read

Comment 15 RHEL Program Management 2021-01-15 07:37:39 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Comment 16 zixchen 2021-01-15 09:37:01 UTC
With qemu-kvm-5.2.0-1.module+el8.4.0+9091+650b220a.x86_64, it reproduce the bug, but libvirt doesn't support such option 'migrate_cancel'
Hence, agree to closed this.


Note You need to log in before you can comment on or make changes to this bug.