RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1458725 - Unable to migrate with nbd block
Summary: Unable to migrate with nbd block
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Eric Blake
QA Contact: Tingting Mao
URL:
Whiteboard:
Depends On: 1459801
Blocks: 1473046
TreeView+ depends on / blocked
 
Reported: 2017-06-05 10:29 UTC by Han Han
Modified: 2018-11-19 10:13 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-19 10:07:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Libvirtd and qemu VM logs (1006.96 KB, application/x-gzip)
2017-06-05 10:29 UTC, Han Han
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1454582 0 unspecified CLOSED Qemu crashes when start guest with qcow2 nbd image 2021-02-22 00:41:40 UTC

Internal Links: 1454582

Description Han Han 2017-06-05 10:29:05 UTC
Created attachment 1285010 [details]
Libvirtd and qemu VM logs

Description of problem:
As subject.

Version-Release number of selected component (if applicable):
libvirt-2.0.0-10.el7_3.9.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.10.x86_64


How reproducible:
100%

Steps to Reproduce:
1. Prepare a nbd server:
# qemu-nbd -t -x nbd_raw --format=qcow2 nbd.qcow2 -p 10803
2. Start a VM with nbd block like following:
<disk type='network' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source protocol='nbd' name='nbd_qcow2'>
        <host name='xx.xx.xx.xx' port='10802'/>
      </source>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </disk>

3. Migrate the VM to another host
# virsh migrate q-qcow2 qemu+ssh://xx.xx.xx.xx/system --verbose --unsafe

The migration will hang for long time.

4. Check VM state and migration:
# virsh qemu-monitor-command q-qcow2 --hmp info migrate

On src host, VM is running:
# virsh list                  
 Id    Name                           State
----------------------------------------------------
 6     q-qcow2                        running

Om dst host, VM is paused:
# virsh list
 Id    Name                           State
----------------------------------------------------
 1     q-qcow2                        paused

Actual results:
As above

Expected results:
Success migration.

Additional info:
It can be reproduced on RHEL7.4(qemu-kvm-rhev-2.9.0-7.el7.x86_64 
libvirt-3.2.0-7.el7.x86_64)
In RHEL7.4, you should reproduce it via raw format because this case is blocked by BZ1454582.

Comment 2 Han Han 2017-06-05 10:35:04 UTC
Hui qing,
I am not sure if it is a libvirt bug or a qemu bug. Could you please try it with raw qemu command? Thks
As I know, this bug can be reproduced on these migration combinations:
RHEL7.4 --> RHEL7.4
RHEL7.3 --> RHEL7.3
RHEL7.3 --> RHEL7.4

Comment 3 Eric Blake 2017-06-06 12:03:11 UTC
Based on bug 1458699, are you sure that both the raw source and destination files (the nbd.qcow2 file being served raw by qemu-nbd) are sufficiently pre-allocated enough that the migration is not trying to write qcow2 data beyond the end of the file?

Comment 4 huiqingding 2017-06-07 08:01:41 UTC
I test 7.3.z->7.4, the guest is rhel7.4 and image format is raw. 

7.3.z host:
kernel-3.10.0-514.25.2.el7.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.10.x86_64

7.4 host:
kernel-3.10.0-663.el7.x86_64
qemu-kvm-rhev-2.9.0-8.el7.x86_64

Test Steps:
1. Start nbd server as comment #0
# qemu-nbd -t -x nbd_raw --format=raw -p 30001 /home/kvm_autotest_root/images/rhel74-64-virtio.raw

2. Boot guest on source 7.3.z host:
/usr/libexec/qemu-kvm \
-machine pc-q35-rhel7.3.0,accel=kvm,usb=off,vmport=off \
-m 1024 \
-realtime mlock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid a94868ef-00d4-4f19-975c-900360c47bfc \
-no-user-config \
-nodefaults \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e \
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 \
-device ioh3420,port=0x10,chassis=3,id=pci.3,bus=pcie.0,addr=0x2 \
-device pci-bridge,chassis_nr=4,id=pci.4,bus=pci.3,addr=0x0 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 \
-device virtio-scsi-pci,id=scsi0,bus=pci.2,addr=0x4 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x3 \
-drive file=nbd:10.66.9.152:30001:exportname=nbd_raw,format=raw,if=none,id=drive-virtio-disk0 \
-device virtio-blk-pci,scsi=off,bus=pci.2,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-netdev tap,id=hostnet0,script=/etc/qemu-ifup \
-device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:89:de:fc,bus=pci.2,addr=0x1 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev spicevmc,id=charchannel0,name=vdagent \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \
-spice port=5900,addr=0.0.0.0,disable-ticketing,seamless-migration=on \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,bus=pcie.0,addr=0x1 \
-device intel-hda,id=sound0,bus=pci.2,addr=0x2 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-chardev spicevmc,id=charredir0,name=usbredir \
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 \
-chardev spicevmc,id=charredir1,name=usbredir \
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 \
-device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x5 \
-msg timestamp=on \
-monitor stdio

3. Boot guest with "-incoming tcp:0:5800" on 7.4 host

Results:
after step2, the guest can boot and check "dmesg", no error info.
after step3, qemu-kvm process hang:
# strace -f -p $(pidof qemu-kvm)
strace: Process 17349 attached with 2 threads
[pid 17350] futex(0x55eaa270e140, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 17349] recvmsg(18,

Comment 5 Han Han 2017-06-08 04:39:36 UTC
According to comment4, change to qemu-kvm-rhev

Comment 6 CongLi 2017-06-08 08:44:10 UTC
It maybe a dup, Refer:
Bug 1459801 - nbd/server.c:nbd_receive_request():L706: read failed when do migration

Comment 8 Eric Blake 2018-01-17 20:50:28 UTC
(In reply to Han Han from comment #0)

> Steps to Reproduce:
> 1. Prepare a nbd server:
> # qemu-nbd -t -x nbd_raw --format=qcow2 nbd.qcow2 -p 10803
> 2. Start a VM with nbd block like following:
> <disk type='network' device='disk'>
>       <driver name='qemu' type='qcow2'/>
>       <source protocol='nbd' name='nbd_qcow2'>
>         <host name='xx.xx.xx.xx' port='10802'/>

Is there a mismatch between the port serving the NBD image (10803) and the libvirt XML (10802), or is there a transcription error in the bug report?

Comment 9 Han Han 2018-01-18 00:49:17 UTC
(In reply to Eric Blake from comment #8)
> (In reply to Han Han from comment #0)
> 
> > Steps to Reproduce:
> > 1. Prepare a nbd server:
> > # qemu-nbd -t -x nbd_raw --format=qcow2 nbd.qcow2 -p 10803
> > 2. Start a VM with nbd block like following:
> > <disk type='network' device='disk'>
> >       <driver name='qemu' type='qcow2'/>
> >       <source protocol='nbd' name='nbd_qcow2'>
> >         <host name='xx.xx.xx.xx' port='10802'/>
> 
> Is there a mismatch between the port serving the NBD image (10803) and the
> libvirt XML (10802), or is there a transcription error in the bug report?

Well, it's a transcription error there.

Comment 11 Han Han 2018-11-19 10:07:30 UTC
Seeing manual of qemu-nbd:
       -e, --shared=num
           Allow up to num clients to share the device (default 1)


Let's set shared to 2 and try migration:
# qemu-nbd -t -p 10809 -e 2 --format=raw /var/lib/libvirt/images/PC.qcow2 -v

# virsh migrate pc qemu+ssh://10.66.4.183/system --verbose --unsafe
Migration: [100 %]  

Migration did succeed :). Obviously, the former migration failure is due to the default shared connection limits 1


Note You need to log in before you can comment on or make changes to this bug.