Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1091159

Summary: QEMU core dumped when load I/O to a data disk created by nbd server
Product: Red Hat Enterprise Linux 7 Reporter: Sibiao Luo <sluo>
Component: qemu-kvmAssignee: Hanna Czenczek <hreitz>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0CC: chayang, famz, hhuang, juzhang, kwolf, michen, pbonzini, qzhang, rbalakri, sluo, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-11-03 13:48:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sibiao Luo 2014-04-25 04:58:57 UTC
Description of problem:
Meet this issue when reproduce the bug 1090713, create a data disk created by nbd server, then load i/o to it in guest which cause guest core dumped.

Version-Release number of selected component (if applicable):
host info:
# uname -r && rpm -q qemu-kvm-rhev
3.10.0-121.el7.x86_64
qemu-kvm-rhev-1.5.3-60.el7ev.x86_64
# rpm -q nbd
nbd-2.9.20-3.el7.x86_64
guest info:
# uname -r
3.10.0-121.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.start nbd-server to export a qcow2 image with absolute path on the NBD server host.
# nbd-server 12345 /home/my-data-disk.qcow2
2.launch a KVM guest with this exported image as a data disk.
e.g:...-drive file=nbd:10.66.83.171:12345,if=none,id=drive-data-disk,format=qcow2,cache=none,aio=native -device virtio-scsi-pci,id=scsi1,addr=0x7,bus=pci.0 -device scsi-hd,bus=scsi1.0,drive=drive-data-disk,id=data-disk,bootindex=2
3.login guest and load i/o to the data disk in guest.
guest ]# dd if=/dev/zero of=/dev/sdb bs=1M

Actual results:
after step 2, guest boot up successfully.
after step 3, load i/o to the data disk in guest cause the QEMU core dumped.
(qemu) block I/O error in device 'drive-data-disk': Invalid argument (22)
block I/O error in device 'drive-data-disk': Operation not permitted (1)
block I/O error in device 'drive-data-disk': Operation not permitted (1)
block I/O error in device 'drive-data-disk': Operation not permitted (1)
Segmentation fault (core dumped)

Core was generated by `/usr/libexec/qemu-kvm -S -M pc -cpu host -enable-kvm -m 2048 -smp 2,sockets=2,c'.
Program terminated with signal 11, Segmentation fault.
#0  bdrv_co_do_rw (opaque=0x0) at block.c:4470
4470	    if (!acb->is_write) {

(gdb) bt
#0  bdrv_co_do_rw (opaque=0x0) at block.c:4470
#1  0x00007f48131ddd5a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at coroutine-ucontext.c:118
#2  0x00007f480dd89570 in ?? () from /lib64/libc.so.6
#3  0x00007fff41d2d9b0 in ?? ()
#4  0x0000000000000000 in ?? ()
(gdb) bt full
#0  bdrv_co_do_rw (opaque=0x0) at block.c:4470
        acb = 0x0
        bs = <optimized out>
#1  0x00007f48131ddd5a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at coroutine-ucontext.c:118
        self = 0x7f4814ee7c00
        co = 0x7f4814ee7c00
#2  0x00007f480dd89570 in ?? () from /lib64/libc.so.6
No symbol table info available.
#3  0x00007fff41d2d9b0 in ?? ()
No symbol table info available.
#4  0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) 

Expected results:
It should no any qemu core dumped.

Additional info:
# /usr/libexec/qemu-kvm -S -M pc -cpu SandyBridge -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo -uuid 990ea161-6b67-41b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -drive file=/home/RHEL-7.0-20140409.0_Server_x86_64.qcow2bk,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native -device virtio-scsi-pci,id=scsi0,addr=0x4,bus=pci.0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-system-disk,id=system-disk,bootindex=1 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:01:02:B6:40:11,bus=pci.0,addr=0x5 -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -serial unix:/tmp/ttyS0,server,nowait -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -vnc :1 -monitor stdio -drive file=nbd:10.66.83.171:12345,if=none,id=drive-data-disk,format=qcow2,cache=none,aio=native -device virtio-scsi-pci,id=scsi1,addr=0x7,bus=pci.0 -device scsi-hd,bus=scsi1.0,drive=drive-data-disk,id=data-disk,bootindex=2

Comment 2 juzhang 2014-05-03 23:05:22 UTC
Hi Sluo,

Can you reply comment1?

Best Regards,
Junyi

Comment 5 Hanna Czenczek 2014-06-28 21:50:34 UTC
Hi Sluo,

Could you by any chance prepare an image for me which makes qemu segfault in the way you describe it? I tried with an empty and a nearly-filled image and both gave a lot of I/O errors (because of BZ 1090713) but qemu did not crash (neither upstream nor RHEL 7's qemu).

Thanks,
Max

Comment 6 Sibiao Luo 2014-06-30 03:07:20 UTC
(In reply to Max Reitz from comment #5)
> Hi Sluo,
> 
> Could you by any chance prepare an image for me which makes qemu segfault in
> the way you describe it? I tried with an empty and a nearly-filled image and
> both gave a lot of I/O errors (because of BZ 1090713) but qemu did not crash
> (neither upstream nor RHEL 7's qemu).
> 
If specify the preallocation=metadata when use qemu-img to create the qcow2 image which work well with i/o loading in the guest.

It's hard to reproduce for me now using the same kvm cmdline in comment #0 while my host and guest image are different.

If start the qemu-kvm cmdline with gcc tool which is easily to trigger the back trace.

1.# qemu-img create -f qcow2 /home/my-data-disk.qcow2 10G
Formatting '/home/my-data-disk.qcow2', fmt=qcow2 size=10737418240 encryption=off cluster_size=65536 lazy_refcounts=off 
2.# qemu-img info /home/my-data-disk.qcow2
image: /home/my-data-disk.qcow2
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 196K
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
3.# nbd-server 12345 /home/my-data-disk.qcow2

4.# gdb /usr/libexec/qemu-kvm
(gdb) r -M pc -cpu SandyBridge -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -no-kvm-pit-reinjection -usb -device usb-tablet,id=input0 -name sluo -uuid 990ea161-6b67-41b2-b803-19fb01d30d30 -rtc base=localtime,clock=host,driftfix=slew -device virtio-serial-pci,id=virtio-serial0,max_ports=16,vectors=0,bus=pci.0,addr=0x3 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port2 -drive file=/home/RHEL-Server-7.0-64-virtio.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native -device virtio-scsi-pci,id=scsi0,addr=0x4,bus=pci.0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-system-disk,id=system-disk,bootindex=1 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:01:02:B6:40:11,bus=pci.0,addr=0x5 -device virtio-balloon-pci,id=ballooning,bus=pci.0,addr=0x6 -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -serial unix:/tmp/ttyS0,server,nowait -k en-us -boot menu=on -qmp tcp:0:4444,server,nowait -vnc :1 -monitor stdio -drive file=nbd:10.66.104.53:12345,if=none,id=drive-data-disk,format=qcow2,cache=none,aio=native -device virtio-scsi-pci,id=scsi1,addr=0x7,bus=pci.0 -device scsi-hd,bus=scsi1.0,drive=drive-data-disk,id=data-disk,bootindex=2

5.load i/o in guest with # dd if=/dev/zero of=/dev/sdb bs=1M

Results:
after step 5, qemu core dumped.
(qemu) block I/O error in device 'drive-data-disk': Invalid argument (22)
block I/O error in device 'drive-data-disk': Operation not permitted (1)

Program received signal SIGPIPE, Broken pipe.
0x00007ffff60547bd in sendmsg () from /lib64/libpthread.so.0

(gdb) bt
#0  0x00007ffff60547bd in sendmsg () from /lib64/libpthread.so.0
#1  0x0000555555854a85 in do_send_recv (do_send=true, iov_cnt=1, iov=0x5555574c8410, sockfd=22) at util/iov.c:111
#2  iov_send_recv (sockfd=sockfd@entry=22, iov=iov@entry=0x5555574c8410, iov_cnt=iov_cnt@entry=1, offset=offset@entry=0, 
    bytes=bytes@entry=28, do_send=do_send@entry=true) at util/iov.c:186
#3  0x0000555555711474 in qemu_co_sendv_recvv (sockfd=22, iov=iov@entry=0x5555574c8410, iov_cnt=iov_cnt@entry=1, 
    offset=offset@entry=0, bytes=28, do_send=true) at qemu-coroutine-io.c:37
#4  0x0000555555711503 in qemu_co_send_recv (sockfd=<optimized out>, buf=<optimized out>, bytes=<optimized out>, 
    do_send=<optimized out>) at qemu-coroutine-io.c:64
#5  0x00005555556e89d1 in write_sync (size=28, buffer=0x5555574c8440, fd=22) at nbd.c:185
#6  nbd_send_request (csock=22, request=request@entry=0x5555574c84e0) at nbd.c:681
#7  0x000055555562419e in nbd_co_send_request (s=s@entry=0x55555666de20, request=request@entry=0x5555574c84e0, 
    qiov=qiov@entry=0x5555574c86e0, offset=offset@entry=0) at block/nbd.c:350
#8  0x0000555555624a87 in nbd_co_writev_1 (bs=<optimized out>, sector_num=<optimized out>, nb_sectors=<optimized out>, 
    qiov=0x5555574c86e0, offset=0) at block/nbd.c:532
#9  0x0000555555618bca in bdrv_aligned_pwritev (flags=0, qiov=0x5555574c86e0, bytes=65536, offset=<optimized out>, 
    req=0x5555574c85d0, bs=0x55555666abf0) at block.c:3022
#10 bdrv_co_do_pwritev (bs=0x55555666abf0, offset=<optimized out>, bytes=65536, qiov=<optimized out>, flags=(unknown: 0))
    at block.c:3151
#11 0x0000555555619132 in bdrv_rw_co_entry (opaque=0x5555574c8690) at block.c:2454
#12 0x00005555556191f8 in bdrv_prwv_co (bs=bs@entry=0x55555666abf0, offset=offset@entry=131072, 
    qiov=qiov@entry=0x5555574c86e0, is_write=is_write@entry=true, flags=flags@entry=(unknown: 0)) at block.c:2490
#13 0x0000555555619549 in bdrv_pwritev (qiov=0x5555574c86e0, offset=131072, bs=0x55555666abf0) at block.c:2621
#14 bdrv_pwrite (bs=0x55555666abf0, offset=131072, buf=<optimized out>, bytes=<optimized out>) at block.c:2643
#15 0x0000555555628450 in qcow2_cache_entry_flush (bs=bs@entry=0x5555566669a0, c=c@entry=0x5555564fe780, i=i@entry=0)
    at block/qcow2-cache.c:138
#16 0x000055555562831d in qcow2_cache_flush (bs=bs@entry=0x5555566669a0, c=0x5555564fe780) at block/qcow2-cache.c:159
#17 0x000055555562932d in l2_allocate (table=0x5555574c87e8, l1_index=0, bs=0x5555566669a0) at block/qcow2-cluster.c:208
#18 get_cluster_table (bs=bs@entry=0x5555566669a0, offset=offset@entry=1032192, 
    new_l2_table=new_l2_table@entry=0x5555574c88c0, new_l2_index=new_l2_index@entry=0x5555574c88bc)
    at block/qcow2-cluster.c:564
#19 0x000055555562a0ba in handle_copied (m=0x5555574c8988, bytes=<synthetic pointer>, host_offset=<synthetic pointer>, 
    guest_offset=1032192, bs=0x5555566669a0) at block/qcow2-cluster.c:905
#20 qcow2_alloc_cluster_offset (bs=bs@entry=0x5555566669a0, offset=offset@entry=1032192, num=num@entry=0x5555574c897c, 
    host_offset=host_offset@entry=0x5555574c8980, m=m@entry=0x5555574c8988) at block/qcow2-cluster.c:1247
#21 0x000055555562f262 in qcow2_co_writev (bs=0x5555566669a0, sector_num=2016, remaining_sectors=1008, 
    qiov=0x55555691bd08) at block/qcow2.c:1129
#22 0x0000555555618bca in bdrv_aligned_pwritev (flags=0, qiov=0x55555691bd08, bytes=516096, offset=<optimized out>, 
    req=0x5555574c8a80, bs=0x5555566669a0) at block.c:3022
#23 bdrv_co_do_pwritev (bs=0x5555566669a0, offset=<optimized out>, bytes=516096, qiov=<optimized out>, 
    flags=(unknown: 0)) at block.c:3151
#24 0x0000555555619654 in bdrv_co_do_writev (flags=<optimized out>, qiov=<optimized out>, nb_sectors=<optimized out>, 
    sector_num=<optimized out>, bs=<optimized out>) at block.c:3175
#25 bdrv_co_do_rw (opaque=0x55555691cae0) at block.c:4474
#26 0x0000555555651d5a in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at coroutine-ucontext.c:118
#27 0x00007ffff2caa570 in ?? () from /lib64/libc.so.6
#28 0x00007fffffffd0c0 in ?? ()
#29 0x0000000000000000 in ?? ()
(gdb) q

Best Regards,
sluo

Comment 8 Hanna Czenczek 2014-07-19 22:07:25 UTC
Hm, I can't seem to reproduce the segfault (neither with upstream QEMU nor with RHEL7's). QEMU prints a lot of error messages regarding NBD and dd only states "No space left on device" (when it tries to write the 10241st MB); everything is as expected and there is no crash.

Comment 9 Kevin Wolf 2014-07-21 12:09:48 UTC
(In reply to Sibiao Luo from comment #6)
> Program received signal SIGPIPE, Broken pipe.
> 0x00007ffff60547bd in sendmsg () from /lib64/libpthread.so.0

This is not a segfault, but merely a signal. qemu doesn't core dump here. This
is also the reason why you don't trigger this outside the debugger, it's just
a gdb feature to stop on signals.

This should go away with 'handle SIGPIPE nostop noprint' in gdb before actually
running qemu.