Bug 1132765

Created attachment 929415 [details]
vmcore-dmesg.txt

qemu-kvm-0.12.1.2-2.431.el6.x86_64 hit this problem.

I cannot reproduce with steps in c0. I'm using a 2.6.32-525.el6.x86_64 kernel.

Error messages are seen in guest dmesg:

scsi 2:0:0:0: Direct-Access     QEMU     QEMU HARDDISK    0.12 PQ: 0 ANSI: 5
sd 2:0:0:0: Attached scsi generic sg2 type 0
sd 2:0:0:0: [sdb] 209715200 512-byte logical blocks: (107 GB/100 GiB)
sd 2:0:0:0: [sdb] Write Protect is off
sd 2:0:0:0: [sdb] Mode Sense: 63 00 00 08
sd 2:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
 sdb: unknown partition table
sd 2:0:0:0: [sdb] Attached SCSI disk
sd 2:0:0:0: [sdb]  Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
sd 2:0:0:0: [sdb] CDB: Read(10): 28 00 00 14 6e 00 00 01 00 00
end_request: I/O error, dev sdb, sector 1338880
Buffer I/O error on device sdb, logical block 167360
Buffer I/O error on device sdb, logical block 167361
Buffer I/O error on device sdb, logical block 167362
Buffer I/O error on device sdb, logical block 167363
Buffer I/O error on device sdb, logical block 167364
Buffer I/O error on device sdb, logical block 167365
Buffer I/O error on device sdb, logical block 167366
Buffer I/O error on device sdb, logical block 167367
Buffer I/O error on device sdb, logical block 167368
Buffer I/O error on device sdb, logical block 167369

mazhang, can you test the latest kernel image?

Fam

Try reproduce this bug with latest guest kernel, but hit qemu-kvm crash.

Host:
qemu-kvm-tools-0.12.1.2-2.451.el6.x86_64
qemu-kvm-0.12.1.2-2.451.el6.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.451.el6.x86_64
gpxe-roms-qemu-0.9.7-6.12.el6.noarch
qemu-img-0.12.1.2-2.451.el6.x86_64
2.6.32-504.el6.x86_64

Guest:
kernel-2.6.32-526.el6

Result:
qemu-kvm crash.

(gdb) bt full
#0  0x00007ffff76ef6fd in write () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00007ffff724ccf1 in ?? () from /lib64/libglib-2.0.so.0
No symbol table info available.
#2  0x00007ffff71fc837 in g_io_channel_write_chars () from /lib64/libglib-2.0.so.0
No symbol table info available.
#3  0x00007ffff7e4474e in io_channel_send (fd=0x7ffff8d1ff00, buf=0x7ffff8e87960, len=16) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-char.c:736
        bytes_written = 0
        offset = <value optimized out>
        status = <value optimized out>
        __PRETTY_FUNCTION__ = "io_channel_send"
#4  0x00007ffff7dbb471 in monitor_flush (mon=0x7ffff88f1f00) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:292
        rc = <value optimized out>
        len = 16
        buf = 0x7ffff8e87960 "{\"return\": {}}\r\n"
#5  0x00007ffff7dbb5e4 in monitor_puts (mon=0x7ffff88f1f00, str=0x7ffff8f36fbf "") at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:326
        c = <value optimized out>
#6  0x00007ffff7dbb629 in monitor_json_emitter (mon=0x7ffff88f1f00, data=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:428
        json = 0x7ffff8e69fd0
        __PRETTY_FUNCTION__ = "monitor_json_emitter"
#7  0x00007ffff7dbb798 in monitor_protocol_emitter (mon=0x7ffff88f1f00, data=0x0) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:464
        qmp = 0x7fff4a22b020
#8  0x00007ffff7dbb960 in monitor_call_handler (mon=0x7ffff88f1f00, cmd=0x7ffff82bfa00, params=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:4390
        ret = <value optimized out>
        data = 0x0
#9  0x00007ffff7dbc574 in handle_qmp_command (parser=<value optimized out>, tokens=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:5003
        err = <value optimized out>
        obj = <value optimized out>
        input = <value optimized out>
        args = 0x7ffffc8eff70
        cmd = 0x7ffff82bfa00
        mon = 0x7ffff88f1f00
        cmd_name = <value optimized out>
        query_cmd = 0x0
        __func__ = "handle_qmp_command"
#10 0x00007ffff7e20f04 in json_message_process_token (lexer=0x7ffff88f1fb0, token=0x7ffff909efd0, type=JSON_OPERATOR, x=52, y=499)
    at /usr/src/debug/qemu-kvm-0.12.1.2/json-streamer.c:87
        parser = 0x7ffff88f1fa8
        dict = 0x7fff48cb7f00

Created attachment 984509 [details]
back trace

It's not a crash, because your monitor connection is disconnected for some reason, which is not related to virtio-scsi or guest kernel. Do you get this all the time? What are the steps?

Anyway it's different from results in comment 0.

Fam

(In reply to Fam Zheng from comment #7)
> It's not a crash, because your monitor connection is disconnected for some
> reason, which is not related to virtio-scsi or guest kernel. Do you get this
> all the time? What are the steps?
> 
> Anyway it's different from results in comment 0.
> 
> Fam

Not all the time, about 50%, and can't reproduce guest kernel crash.
Steps is the same as comment 0.

So I'm going to close this bug as the guest crash issue is gone. Please file new bugs if there are other issues. (Again, SIGPIPE is not a crash, it's just the other end of the monitor is closed. In your test it's possibly the bash script exited.)

Fam

(In reply to Fam Zheng from comment #9)
> So I'm going to close this bug as the guest crash issue is gone. Please file
> new bugs if there are other issues. (Again, SIGPIPE is not a crash, it's
> just the other end of the monitor is closed. In your test it's possibly the
> bash script exited.)
> 
> Fam

Set gdb SIGPIPE nostop, break didn't happen.

Hit this problem on kernel-2.6.32-544.el6.

Host:
2.6.32-544.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.457.el6.x86_64
qemu-img-rhev-0.12.1.2-2.457.el6.x86_64
qemu-kvm-rhev-debuginfo-0.12.1.2-2.457.el6.x86_64
qemu-kvm-rhev-tools-0.12.1.2-2.457.el6.x86_64

Guest:
2.6.32-544.el6.x86_64

Steps:
1. Boot vm
/usr/libexec/qemu-kvm \
-M pc \
-cpu SandyBridge \
-m 2G \
-smp 4,sockets=2,cores=2,threads=1 \
-enable-kvm \
-name rhel6 \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-smbios type=1,manufacturer='Red Hat',product='RHEV Hypervisor',version=el6,serial=koTUXQrb,uuid=feebc8fd-f8b0-4e75-abc3-e63fcdb67170 \
-k en-us \
-rtc base=localtime,driftfix=slew \
-nodefaults \
-monitor stdio \
-qmp tcp:0:6779,server,nowait \
-boot menu=on \
-bios /usr/share/seabios/bios.bin \
-serial unix:/tmp/console0,server,nowait \
-spice port=5900,disable-ticketing \
-vga qxl \
-usb -device usb-tablet,id=input0 \
-netdev tap,id=tap0 \
-device virtio-net-pci,netdev=tap0,id=net0,mac=52:54:00:11:11:15 \
-device virtio-scsi-pci,id=scsi0 \
-drive file=/home/rhel6-64.qcow2,if=none,id=drive-scsi-disk,format=qcow2,cache=none,werror=stop,rerror=stop \
-device scsi-hd,drive=drive-scsi-disk,bus=scsi0.0,scsi-id=0,lun=0,id=scsi-disk,bootindex=1 \
-device virtio-scsi-pci,id=bus1,bus=pci.0,addr=0x7 \
-drive file=/home/storage0.qcow2,if=none,media=disk,format=qcow2,rerror=stop,werror=stop,cache=none,aio=native,id=scsi-disk0 \
-device scsi-hd,bus=bus1.0,drive=scsi-disk0,id=disk \

2. Run dd test in guest.
# dd if=/dev/zero of=/dev/sdb bs=1M count=10240

3. Hotplug/unplug virtio scsi disk.
[root@dhcp-9-236 ~]# cat repeat-hotplug.sh 
#!/bin/bash
# some simply group snapshot stress testing

let i=0
exec 3<>/dev/tcp/localhost/6779
echo -e "{ 'execute': 'qmp_capabilities' }" >&3
read response <&3
echo $response
while [ $i -lt 200 ]
do
    echo -e '{"execute":"device_add","arguments":{"driver":"virtio-scsi-pci","id":"test30"}}' >&3
    read response <&3;  echo "$i: $response"
    echo -e '{"execute":"__com.redhat_drive_add", "arguments": {"file":"/tmp/storage.qcow2","format":"qcow2","id":"test30"}}' >&3
    read response <&3;  echo "$i: $response"
    echo -e '{"execute":"device_add","arguments":{"driver":"scsi-hd","drive":"test30","id":"test31"}}' >&3
    read response <&3;  echo "$i: $response"
    sleep 2
    echo -e '{"execute":"device_del","arguments":{"id":"test31"}}' >&3
    read response <&3;  echo "$i: $response"
    echo -e '{"execute":"device_del","arguments":{"id":"test30"}}' >&3
    read response <&3;  echo "$i: $response"
    let i=$i+1
    sleep 2
done

Created attachment 1002139 [details]
vmcore-dmesg

Yes. Might be the same with bz 1199421. I'm looking at it.

*** This bug has been marked as a duplicate of bug 1199421 ***

mazhang, could you test this build (fix for 1199421) to make sure this one is a duplicate?

https://brewweb.devel.redhat.com/taskinfo?taskID=8864631