Bug 1672031 - qemu-kvm segfault when read a disconnected NBD virtio disk with rerror=stop or detach a disconnected nbd virtio disk
Summary: qemu-kvm segfault when read a disconnected NBD virtio disk with rerror=stop o...
Keywords:
Status: CLOSED DUPLICATE of bug 1478227
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Eric Blake
QA Contact: Tingting Mao
URL:
Whiteboard:
Depends On: 1672028
Blocks: 1672029
TreeView+ depends on / blocked
 
Reported: 2019-02-03 10:11 UTC by Han Han
Modified: 2020-03-04 16:02 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1672028
Environment:
Last Closed: 2019-05-13 06:22:19 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Han Han 2019-02-03 10:11:11 UTC
reproduced on:
libvirt-4.10.0-1.module+el8+2317+367e35b5.x86_64
qemu-kvm-3.1.0-6.module+el8+2711+98525d2b.x86_64

+++ This bug was initially created as a clone of Bug #1672028 +++

Description of problem:
As subject

Version-Release number of selected component (if applicable):
qemu-kvm-2.12.0-60.module+el8+2749+88f75c21.x86_64
libvirt-4.5.0-20.module+el8+2724+8292f19c.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Prepare a qemu-nbd server:
# qemu-nbd -t /tmp/scsi
WARNING: Image format was not specified for '/tmp/scsi' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images,

2. Start a vm with nbd disk, set rerror=stop:
Disk xml:
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' rerror_policy='stop'/>
      <source protocol='nbd'>
        <host name='localhost' port='10809'/>
      </source>
      <target dev='sda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>


Qemu cmdline:
/usr/libexec/qemu-kvm -name guest=a1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-7-a1/master-key.aes -machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Server-IBRS,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,umip=on,pku=on,stibp=on,ssbd=on -m 1024 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 833e62e6-d894-4670-bc98-6f36f3d83f89 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=30,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x8 -drive file=/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64-clone.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=nbd:localhost:10809,format=raw,if=none,id=drive-virtio-disk1,rerror=stop -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=32,id=hostnet0,vhost=on,vhostfd=33 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:02:91:4a,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on


3. Kill the nbd-server and do some reading from the disk
# killall qemu-nbd

(vm) # dd if=/dev/vdb of=file

VM will crash

Backtrace:
(gdb) bt
#0  0x000055b39fd1e0de in aio_co_schedule (ctx=0x55b3a1b28c40, co=0x0) at util/async.c:444                                                                                 
#1  0x000055b39fc67b49 in bdrv_attach_aio_context (bs=0x55b3a1b7abc0, new_context=new_context@entry=0x55b3a1b28c40) at block.c:4969                                        
#2  0x000055b39fc67b27 in bdrv_attach_aio_context (bs=bs@entry=0x55b3a1b74580, new_context=new_context@entry=0x55b3a1b28c40) at block.c:4966                               
#3  0x000055b39fc67c51 in bdrv_set_aio_context (bs=0x55b3a1b74580, new_context=0x55b3a1b28c40) at block.c:5002                                                             
#4  0x000055b39fc98f1c in blk_set_aio_context (blk=<optimized out>, new_context=<optimized out>) at block/block-backend.c:1909                                             
#5  0x000055b39fa63a02 in virtio_blk_data_plane_stop (vdev=<optimized out>)
    at /usr/src/debug/qemu-kvm-2.12.0-60.module+el8+2749+88f75c21.x86_64/hw/block/dataplane/virtio-blk.c:286                                                               
#6  0x000055b39fc001ff in virtio_bus_stop_ioeventfd (bus=0x55b3a2c88438) at hw/virtio/virtio-bus.c:246                                                                     
#7  0x000055b39fa8bd9e in virtio_vmstate_change (opaque=0x55b3a2c884b0, running=0, state=<optimized out>)                                                                  
    at /usr/src/debug/qemu-kvm-2.12.0-60.module+el8+2749+88f75c21.x86_64/hw/virtio/virtio.c:2219                                                                           
#8  0x000055b39fb19a9f in vm_state_notify (running=0, state=RUN_STATE_IO_ERROR) at vl.c:1643                                                                               
#9  0x000055b39fa2d17a in do_vm_stop (state=RUN_STATE_IO_ERROR, send_stop=<optimized out>) at /usr/src/debug/qemu-kvm-2.12.0-60.module+el8+2749+88f75c21.x86_64/cpus.c:1012
#10 0x000055b39f9ea531 in main_loop_should_exit () at vl.c:1950
#11 main_loop () at vl.c:1968
#12 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4789  



Actual results:
As above

Expected results:
No segment fault

Additional info:
No reproduced on scsi,ide,sata disk

Comment 3 Han Han 2019-05-13 03:06:41 UTC
More easy steps to reproduce this issue: Detach a disconnect virtio nbd disk:
1. Start a vm with nbd disk
2. Close the nbd server
3. Detach the disk

Vesion: libvirt-4.5.0-16.virtcov.el7.x86_64 qemu-kvm-rhev-2.12.0-27.el7.x86_64

The backtrace:
#0  0x000055ded99c77e4 in aio_co_schedule (ctx=0x55dedc6837c0, co=0x0) at util/async.c:444
        _old = 0x0
        scheduled = <optimized out>
        __func__ = "aio_co_schedule"
#1  0x000055ded98f19a5 in bdrv_attach_aio_context (bs=0x55dedcb10000, new_context=new_context@entry=0x55dedc6837c0) at block.c:5000
        ban = <optimized out>
        ban_tmp = <optimized out>
        child = 0x0
        __PRETTY_FUNCTION__ = "bdrv_attach_aio_context"
#2  0x000055ded98f1983 in bdrv_attach_aio_context (bs=bs@entry=0x55dedcb13400, new_context=new_context@entry=0x55dedc6837c0) at block.c:4997
        ban = <optimized out>
        ban_tmp = <optimized out>
        child = 0x55dedcc60c80
        __PRETTY_FUNCTION__ = "bdrv_attach_aio_context"
#3  0x000055ded98f1a79 in bdrv_set_aio_context (bs=0x55dedcb13400, new_context=0x55dedc6837c0) at block.c:5033
        ctx = 0x0
#4  0x000055ded9933c08 in blk_set_aio_context (blk=<optimized out>, new_context=<optimized out>) at block/block-backend.c:1936
        bs = <optimized out>
        tgm = <optimized out>
#5  0x000055ded96e31a8 in virtio_blk_data_plane_stop (vdev=<optimized out>) at /usr/src/debug/qemu-2.12.0/hw/block/dataplane/virtio-blk.c:286
        vblk = 0x55dedde94170
        __func__ = "virtio_blk_data_plane_stop"
        s = 0x55dedf507bc0
        qbus = 0x55dedde940f8
        k = 0x55dedc682500
        i = <optimized out>
        nvqs = 1
#6  0x000055ded9888df5 in virtio_bus_stop_ioeventfd (bus=bus@entry=0x55dedde940f8) at hw/virtio/virtio-bus.c:246
        vdev = 0x55dedde94170
        vdc = <optimized out>
        __func__ = "virtio_bus_stop_ioeventfd"
#7  0x000055ded9886161 in virtio_pci_common_write (proxy=0x55dedde8c000) at hw/virtio/virtio-pci.c:294
        proxy = 0x55dedde8c000
        vdev = 0x55dedde94170
        __func__ = "virtio_pci_common_write"
#8  0x000055ded9886161 in virtio_pci_common_write (opaque=0x55dedde8c000, addr=<optimized out>, val=0, size=<optimized out>) at hw/virtio/virtio-pci.c:1283
        proxy = 0x55dedde8c000
        vdev = 0x55dedde94170
        __func__ = "virtio_pci_common_write"
#9  0x000055ded96bf6e3 in memory_region_write_accessor (mr=<optimized out>, addr=<optimized out>, value=<optimized out>, size=<optimized out>, shift=<optimized out>, mask=<optimized out>, attrs=...) at /usr/src/debug/qemu-2.12.0/memory.c:530
        tmp = <optimized out>
#10 0x000055ded96bd3f9 in access_with_adjusted_size (addr=addr@entry=20, value=value@entry=0x7f81014846d8, size=size@entry=1, access_size_min=<optimized out>, access_size_max=<optimized out>, access_fn=access_fn@entry=0x55ded96bf6a0 <memory_region_write_accessor>, mr=mr@entry=0x55dedde8c9d0, attrs=attrs@entry=...)
    at /usr/src/debug/qemu-2.12.0/memory.c:597
        access_mask = 255
        access_size = 1
        i = <optimized out>
        r = 0
#11 0x000055ded96c1495 in memory_region_dispatch_write (mr=mr@entry=0x55dedde8c9d0, addr=addr@entry=20, data=0, size=size@entry=1, attrs=attrs@entry=...)
    at /usr/src/debug/qemu-2.12.0/memory.c:1474
#12 0x000055ded9671ab3 in flatview_write_continue (fv=fv@entry=0x55dedcaa8d80, addr=addr@entry=4294967316, attrs=..., 
    attrs@entry=..., buf=buf@entry=0x7f8126387028 <Address 0x7f8126387028 out of bounds>, len=len@entry=1, addr1=20, l=1, mr=0x55dedde8c9d0)
    at /usr/src/debug/qemu-2.12.0/exec.c:3140
        ptr = <optimized out>
        val = <optimized out>
        result = 0
        release_lock = true
#13 0x000055ded9671bff in flatview_write (fv=0x55dedcaa8d80, addr=4294967316, attrs=..., buf=0x7f8126387028 <Address 0x7f8126387028 out of bounds>, len=1)
    at /usr/src/debug/qemu-2.12.0/exec.c:3184
        l = 1
        addr1 = 20
        mr = <optimized out>
        result = 0
#14 0x000055ded9675eff in address_space_write (as=<optimized out>, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>)
    at /usr/src/debug/qemu-2.12.0/exec.c:3300
        result = 0
        fv = <optimized out>
#15 0x000055ded9675fa5 in address_space_rw (as=<optimized out>, addr=<optimized out>, attrs=..., 
    attrs@entry=..., buf=buf@entry=0x7f8126387028 <Address 0x7f8126387028 out of bounds>, len=<optimized out>, is_write=<optimized out>)
    at /usr/src/debug/qemu-2.12.0/exec.c:3311
#16 0x000055ded96d0008 in kvm_cpu_exec (cpu=cpu@entry=0x55dedc9c2000) at /usr/src/debug/qemu-2.12.0/accel/kvm/kvm-all.c:1996
        attrs = {unspecified = 0, secure = 0, user = 0, requester_id = 0}
        run = <optimized out>
        ret = <optimized out>
        run_ret = 0
#17 0x000055ded96ad4c6 in qemu_kvm_cpu_thread_fn (arg=0x55dedc9c2000) at /usr/src/debug/qemu-2.12.0/cpus.c:1215
        cpu = 0x55dedc9c2000
        r = <optimized out>
#18 0x00007f810ce94ea5 in start_thread (arg=0x7f8101487700) at pthread_create.c:307
        __res = <optimized out>
        pd = 0x7f8101487700
        now = <optimized out>
        unwind_buf = 
              {cancel_jmp_buf = {{jmp_buf = {140192049035008, -9178915296927424559, 0, 8392704, 0, 140192049035008, 9196336952062631889, 9196325123545096145}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
#19 0x00007f810cbbd8cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Comment 4 Tingting Mao 2019-05-13 06:22:19 UTC
Hi,

From the gdb trace log, this bug should a dup of BZ #1478227.For bug 1478227 is reported earlier, I will close this bug. If there is any disagreement, comment or reopen it pls. 


Thanks,
Tingting

*** This bug has been marked as a duplicate of bug 1478227 ***


Note You need to log in before you can comment on or make changes to this bug.