Bug 1120042

Summary: Segmentation fault at block/qcow2-cluster.c:462
Product: Red Hat Enterprise Linux 6 Reporter: Xu Han <xuhan>
Component: qemu-kvmAssignee: Jeff Cody <jcody>
Status: CLOSED WORKSFORME QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.6CC: bsarathy, chayang, coli, juzhang, mkenneth, qzhang, rbalakri, scui, shu, virt-maint, xuhan
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-02-02 02:42:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
gdb bt full none

Description Xu Han 2014-07-16 06:40:12 UTC
Created attachment 918333 [details]
gdb bt full

Description of problem:
QEMU met segmentation fault during block streaming.
-----------------------------------------------------------------------------
Core was generated by `/usr/bin/qemu-kvm -S -name virt-tests-vm1 -M rhel6.5.0 -nodefaults -vga qxl -gl'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f3b9b3dc2a9 in qcow2_get_cluster_offset (bs=0x7f3b9c71d010, offset=14505934848, num=0x7f3b9f96e73c, cluster_offset=0x7f3b9f96e730)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block/qcow2-cluster.c:462
462	    if (!l2_offset) {
-----------------------------------------------------------------------------
(gdb) bt
#0  0x00007f3b9b3dc2a9 in qcow2_get_cluster_offset (bs=0x7f3b9c71d010, offset=14505934848, num=0x7f3b9f96e73c, cluster_offset=0x7f3b9f96e730)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block/qcow2-cluster.c:462
#1  0x00007f3b9b3d7fe1 in qcow2_co_readv (bs=0x7f3b9c71d010, sector_num=<value optimized out>, remaining_sectors=64, qiov=0x7f3b9f96e8b0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block/qcow2.c:544
#2  0x00007f3b9b3bb59f in bdrv_co_do_readv (bs=0x7f3b9c71d010, sector_num=28331904, nb_sectors=64, qiov=0x7f3b9f96e8b0, flags=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:2205
#3  0x00007f3b9b3bbbce in bdrv_rw_co_entry (opaque=0x7f3b9f96e880) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:1885
#4  0x00007f3b9b3bdad8 in bdrv_rw_co (bs=0x7f3b9c71d010, sector_num=<value optimized out>, buf=<value optimized out>, nb_sectors=<value optimized out>, 
    is_write=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:1929
#5  0x00007f3b9b3dd737 in qcow2_read (bs=0x7f3b9c3b1010, start_sect=<value optimized out>, cluster_offset=9894494208, n_start=0, n_end=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block/qcow2-cluster.c:339
#6  copy_sectors (bs=0x7f3b9c3b1010, start_sect=<value optimized out>, cluster_offset=9894494208, n_start=0, n_end=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block/qcow2-cluster.c:382
#7  0x00007f3b9b3dd930 in qcow2_alloc_cluster_link_l2 (bs=0x7f3b9c3b1010, m=0x7f3b9f96eab0) at /usr/src/debug/qemu-kvm-0.12.1.2/block/qcow2-cluster.c:663
#8  0x00007f3b9b3d794f in qcow2_co_writev (bs=0x7f3b9c3b1010, sector_num=<value optimized out>, remaining_sectors=520, qiov=0x7f3b9cba5f08)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block/qcow2.c:753
#9  0x00007f3b9b3bba53 in bdrv_co_do_writev (bs=0x7f3b9c3b1010, sector_num=28331968, nb_sectors=520, qiov=0x7f3b9cba5f08, flags=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:2321
#10 0x00007f3b9b3bbb11 in bdrv_co_do_rw (opaque=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block.c:4057
#11 0x00007f3b9b3c52eb in coroutine_trampoline (i0=<value optimized out>, i1=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/coroutine-ucontext.c:129
#12 0x00007f3b97e39bf0 in ?? () from /lib64/libc.so.6
#13 0x00007fff44bcb670 in ?? ()
#14 0x0000000000000000 in ?? ()

Version-Release number of selected component (if applicable):
qemu-img-rhev-0.12.1.2-2.429.el6.x86_64

How reproducible:
Sometimes

Steps to Reproduce:
1. Create a snaphot.
{"execute": "blockdev-snapshot-sync", "arguments": {"device": "drive_image1", "snapshot-file": "/usr/local/staf/test/RHEV/kvm/autotest-devel/client/tests/virt/shared/data/images/sn1", "format": "qcow2"}, "id": "hcXTpZt0"}

2. Do streaming.
{"execute": "block-stream", "arguments": {"device": "drive_image1", "speed": 1048576}, "id": "BPtZff13"}

Actual results:
Core dumped.

Expected results:
The block job should finish with no error.

Additional info:
1. Last QMP logs:
----------------
2014-07-08 00:25:05: {"execute": "query-block-jobs", "id": "Y6oPvlbL"}
2014-07-08 00:25:05: {"return": [{"device": "drive_image1", "len": 32212254720, "offset": 25239748608, "speed": 10485760, "type": "stream"}], "id": "Y6oPvlbL"}
2014-07-08 00:25:06: {"execute": "query-status", "id": "fJEPsORL"}
2014-07-08 00:25:06: {"return": {"status": "running", "singlestep": false, "running": true}, "id": "fJEPsORL"}
2014-07-08 00:25:06: {"execute": "screendump", "arguments": {"filename": "/dev/shm/scrdump-LH5dhX.ppm"}, "id": "duA43BM9"}
2014-07-08 00:25:06: {"return": {}, "id": "duA43BM9"}
2014-07-08 00:25:06: {"execute": "query-block-jobs", "id": "EYC48Yyo"}
2014-07-08 00:25:06: {"return": [{"device": "drive_image1", "len": 32212254720, "offset": 31458263040, "speed": 10485760, "type": "stream"}], "id": "EYC48Yyo"}
2014-07-08 00:25:07: {"execute": "query-block-jobs", "id": "FzL4NnTU"}

2. QEMU command line:
------------------
/usr/bin/qemu-kvm \
    -S  \
    -name 'virt-tests-vm1' \
    -M rhel6.5.0  \
    -nodefaults  \
    -vga qxl  \
    -global qxl-vga.vram_size=33554432 \
    -device AC97,bus=pci.0,addr=03  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20140707-235147-2xAI2d2T,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20140707-235147-2xAI2d2T,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20140707-235147-2xAI2d2T,path=/tmp/seabios-20140707-235147-2xAI2d2T,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20140707-235147-2xAI2d2T,iobase=0x402 \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=04 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=05 \
    -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,file=/usr/local/staf/test/RHEV/kvm/autotest-devel/client/tests/virt/shared/data/images/win7-64-sp1-virtio.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:67:68:69:6a:6b,id=idTHP0SR,vectors=4,netdev=idRVWPZu,bus=pci.0,addr=06  \
    -netdev tap,id=idRVWPZu,vhost=on,vhostfd=28,fd=27  \
    -m 4096  \
    -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
    -cpu 'Opteron_G3',hv_relaxed \
    -drive id=drive_cd1,if=none,snapshot=off,aio=native,media=cdrom,file=/usr/local/staf/test/RHEV/kvm/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso \
    -device scsi-cd,id=cd1,drive=drive_cd1 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -spice port=3000,password=123456,addr=0,image-compression=auto_glz,zlib-glz-wan-compression=auto,streaming-video=all,agent-mouse=on,playback-compression=on,ipv4  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off \
    -enable-kvm

Comment 1 Xu Han 2014-07-16 06:46:22 UTC
(gdb) p *s
$1 = {
  cluster_bits = 16, 
  cluster_size = 65536, 
  cluster_sectors = 128, 
  l2_bits = 13, 
  l2_size = 8192, 
  l1_size = 62, 
  l1_vm_state_index = 60, 
  csize_shift = 54, 
  csize_mask = 255, 
  cluster_offset_mask = 18014398509481983, 
  l1_table_offset = 196608, 
  l1_table = 0x0, 
  l2_table_cache = 0x7f3b9c3b1f20, 
  refcount_block_cache = 0x7f3b9c3aefc0, 
  cluster_cache = 0x7f3b9c4f5610 "", 
  cluster_data = 0x7f3b9153d010 "", 
  cluster_cache_offset = 18446744073709551615, 
  cluster_allocs = {
    lh_first = 0x0
  }, 
  refcount_table = 0x7f3b9c505620, 
  refcount_table_offset = 65536, 
  refcount_table_size = 8192, 
  free_cluster_index = 0, 
  free_byte_offset = 0, 
  lock = {
    locked = true, 
    queue = {
      entries = {
        tqh_first = 0x0, 
        tqh_last = 0x7f3b9c3b1a90
      }
    }
  }, 
  crypt_method = 0, 
  crypt_method_header = 0, 
  aes_encrypt_key = {
    rd_key = {0 <repeats 60 times>}, 
    rounds = 0
  }, 
  aes_decrypt_key = {
    rd_key = {0 <repeats 60 times>}, 
    rounds = 0
  }, 
  snapshots_offset = 0, 
  snapshots_size = 0, 
  nb_snapshots = 0, 
  snapshots = 0x0, 
  overlap_check = 127, 
  unknown_header_ext = {
    lh_first = 0x0
  }
}

Comment 4 juzhang 2014-07-16 23:24:43 UTC
Hi Xu,

Does rhel7.1 host hit this issue as well?

Best Regards,
Junyi

Comment 5 Xu Han 2014-07-17 10:55:32 UTC
(In reply to juzhang from comment #4)
> Hi Xu,
> 
> Does rhel7.1 host hit this issue as well?
> 
> Best Regards,
> Junyi

Tested 10 times block streaming on rhel7.1 host, did not hit this issue.

virt.qemu.smp_2.4096m.repeat1.run_test.Host_RHEL.7.1.qcow2.virtio_scsi.up.virtio_net.Win7.x86_64.sp1.io-github-autotest-qemu.block_stream.simple_test.reset_speed                             GOOD       1763     completed successfully
virt.qemu.smp_2.4096m.repeat2.run_test.Host_RHEL.7.1.qcow2.virtio_scsi.up.virtio_net.Win7.x86_64.sp1.io-github-autotest-qemu.block_stream.simple_test.reset_speed                             GOOD       1674     completed successfully
virt.qemu.smp_2.4096m.repeat3.run_test.Host_RHEL.7.1.qcow2.virtio_scsi.up.virtio_net.Win7.x86_64.sp1.io-github-autotest-qemu.block_stream.simple_test.reset_speed                             GOOD       1537     completed successfully
virt.qemu.smp_2.4096m.repeat4.run_test.Host_RHEL.7.1.qcow2.virtio_scsi.up.virtio_net.Win7.x86_64.sp1.io-github-autotest-qemu.block_stream.simple_test.reset_speed                             GOOD       1553     completed successfully
virt.qemu.smp_2.4096m.repeat5.run_test.Host_RHEL.7.1.qcow2.virtio_scsi.up.virtio_net.Win7.x86_64.sp1.io-github-autotest-qemu.block_stream.simple_test.reset_speed                             GOOD       1575     completed successfully
virt.qemu.smp_2.4096m.repeat6.run_test.Host_RHEL.7.1.qcow2.virtio_scsi.up.virtio_net.Win7.x86_64.sp1.io-github-autotest-qemu.block_stream.simple_test.reset_speed                             GOOD       1516     completed successfully
virt.qemu.smp_2.4096m.repeat7.run_test.Host_RHEL.7.1.qcow2.virtio_scsi.up.virtio_net.Win7.x86_64.sp1.io-github-autotest-qemu.block_stream.simple_test.reset_speed                             GOOD       1513     completed successfully
virt.qemu.smp_2.4096m.repeat8.run_test.Host_RHEL.7.1.qcow2.virtio_scsi.up.virtio_net.Win7.x86_64.sp1.io-github-autotest-qemu.block_stream.simple_test.reset_speed                             GOOD       1558     completed successfully
virt.qemu.smp_2.4096m.repeat9.run_test.Host_RHEL.7.1.qcow2.virtio_scsi.up.virtio_net.Win7.x86_64.sp1.io-github-autotest-qemu.block_stream.simple_test.reset_speed                             GOOD       1907     completed successfully
virt.qemu.smp_2.4096m.repeat10.run_test.Host_RHEL.7.1.qcow2.virtio_scsi.up.virtio_net.Win7.x86_64.sp1.io-github-autotest-qemu.block_stream.simple_test.reset_speed                            GOOD       1576     completed successfully