Bug 987257 - [TestOnly] qemu should not crash when boot guest with 260 scsi disks
[TestOnly] qemu should not crash when boot guest with 260 scsi disks
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
7.0
Unspecified Unspecified
medium Severity medium
: rc
: ---
Assigned To: Ademar Reis
Virtualization Bugs
: TestOnly
Depends On: 1007334
Blocks: 1000340
  Show dependency treegraph
 
Reported: 2013-07-23 01:43 EDT by CongLi
Modified: 2014-06-17 23:31 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 996809 1000340 (view as bug list)
Environment:
Last Closed: 2014-06-13 05:48:09 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
core dump file (15.08 MB, application/x-xz)
2013-07-23 01:43 EDT, CongLi
no flags Details
CML (49.12 KB, application/x-shellscript)
2013-08-01 02:17 EDT, CongLi
no flags Details

  None (edit)
Description CongLi 2013-07-23 01:43:56 EDT
Created attachment 777171 [details]
core dump file

Description of problem:
qemu core dump when boot a Win7 guest with 260 scsi disks

Version-Release number of selected component (if applicable):
kernel: 3.10.0-1.el7.x86_64
qemu-kvm-1.5.1-2.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Create 260 images by command:
    /root/staf-kvm-devel/autotest-devel/client/tests/virt/qemu/qemu-img create -f qcow2 /tmp/stg0.qcow2 1M

2. Start the guest with those above 260 disks by command:
    /root/staf-kvm-devel/autotest-devel/client/tests/virt/qemu/qemu \
    -S \
    -name 'virt-tests-vm1' \
    -nodefaults \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20130722-124256-zR5T4PpC,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,id=serial_id_serial1,path=/tmp/serial-serial1-20130722-124256-zR5T4PpC,server,nowait \
    -device isa-serial,chardev=serial_id_serial1 \
    -chardev socket,id=seabioslog_id_20130722-124256-zR5T4PpC,path=/tmp/seabios-20130722-124256-zR5T4PpC,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20130722-124256-zR5T4PpC,iobase=0x402 \
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4 \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/images/win7-64-virtio.qcow2',if=none,id=drive-virtio-disk1,media=disk,cache=none,snapshot=off,format=qcow2,aio=native \
    -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,bootindex=0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,addr=0x6 \
    -drive file='/tmp/stg0.qcow2',if=none,id=virtio-scsi-id0,media=disk,cache=none,snapshot=off,format=qcow2,aio=native \
    -device scsi-disk,lun=0,drive=virtio-scsi-id0 \
    ......
    -drive file='/tmp/stg260.qcow2',if=none,id=virtio-scsi-id260,media=disk,cache=none,snapshot=off,format=qcow2,aio=native \
    -device scsi-disk,lun=16380,drive=virtio-scsi-id260 \
    -device rtl8139,netdev=id9KbU2b,mac='9a:75:76:77:78:79',bus=pci.0,addr=0x3,id='idMtu2Dc' \
    -netdev tap,id=id9KbU2b,fd=22 \
    -m 16384 \
    -smp 8,maxcpus=8,cores=4,threads=1,sockets=2 \
    -cpu 'Opteron_G4',,hv_relaxed \
    -M pc \
    -drive file='/root/staf-kvm-devel/autotest-devel/client/tests/virt/shared/data/isos/windows/winutils.iso',if=none,id=drive-ide0-0-0,media=cdrom,format=raw \
    -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -vnc :0 \
    -vga std \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off  \
    -enable-kvm

3.
Actual results:
qemu core dump

Expected results:
qemu works well.
if this is not a supported operation, qemu should raise error instead of core dump

Additional info:
1. cpuinfo:
processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           E5530  @ 2.40GHz
stepping	: 5
microcode	: 0x11
cpu MHz		: 2394.110
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 4787.21
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:


2. (gdb) bt
#0  0x00007f3cd3d95a19 in raise () from /lib64/libc.so.6
#1  0x00007f3cd3d97128 in abort () from /lib64/libc.so.6
#2  0x00007f3cd3dd5d47 in __libc_message () from /lib64/libc.so.6
#3  0x00007f3cd3ddd0e8 in _int_free () from /lib64/libc.so.6
#4  0x00007f3cd7bbd9af in g_free () from /lib64/libglib-2.0.so.0
#5  0x00007f3cd86fb724 in virtio_scsi_complete_req (req=0x7f3cf564c3e0) at /usr/src/debug/qemu-1.5.1/hw/scsi/virtio-scsi.c:70
#6  0x00007f3cd860ad7b in scsi_req_complete (req=0x7f3cf23c2e30, status=<optimized out>) at hw/scsi/scsi-bus.c:1617
#7  0x00007f3cd86fc062 in virtio_scsi_handle_cmd (vdev=0x7f3cf23d0b18, vq=0x7f3cf23cbf50) at /usr/src/debug/qemu-1.5.1/hw/scsi/virtio-scsi.c:400
#8  0x00007f3cd863e7ee in qemu_iohandler_poll (pollfds=0x7f3cdb450c00, ret=ret@entry=4) at iohandler.c:143
#9  0x00007f3cd8643ec8 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:465
#10 0x00007f3cd8544609 in main_loop () at vl.c:2029
#11 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4419
(gdb)
Comment 2 Asias He 2013-07-31 23:16:04 EDT
Can you reproduce this with linux guest, e.g. RHEL7 guest.
Also, Can you put your full qemu cmd line somewhere I can access.
Comment 3 CongLi 2013-08-01 02:15:57 EDT
(In reply to Asias He from comment #2)
> Can you reproduce this with linux guest, e.g. RHEL7 guest.
> Also, Can you put your full qemu cmd line somewhere I can access.


Hi, Asias

1. Tested with RHEL7 guest, no core dump occured.

2. CML is attached as cmd.sh
Comment 4 CongLi 2013-08-01 02:17:18 EDT
Created attachment 781454 [details]
CML
Comment 5 Asias He 2013-08-01 02:47:31 EDT
(In reply to CongLi from comment #3)
> (In reply to Asias He from comment #2)
> > Can you reproduce this with linux guest, e.g. RHEL7 guest.
> > Also, Can you put your full qemu cmd line somewhere I can access.
> 
> 
> Hi, Asias
> 
> 1. Tested with RHEL7 guest, no core dump occured.


Did you actually see 260 luns in guest?

can you paste 'ls /dev/sd*' and 'sg_scan' in guest.

> 
> 2. CML is attached as cmd.sh

Thanks. Did you choose lun randomly. I theory, we can support 256 targets and 16384 luns per target.
Comment 6 Asias He 2013-08-01 03:04:52 EDT
I tried 1) lun = 1000 to 1255 and 2) lun = 1000 to 1254

No lun can be seen in 1), 255 luns can be seen in 2)

Seems linux kernel limited the total number of luns per target to 255, though virtio-scsi support more than 255 luns per target, i.e., 16384.



echo 'gdb --args x86_64-softmmu/qemu-system-x86_64 -L /opt/qemu/share/qemu/ -netdev tap,id=hn0,vhost=off -device virtio-net-pci,netdev=hn0 -nographic -vnc :0 -enable-kvm -m 2048 -smp 4 -cpu qemu64,+x2apic -M pc  -drive file=/root/img/RHEL7.2013.4.raw,if=virtio -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 -kernel /boot/bzImage -append "root=/dev/mapper/rhel-root console=ttyS0" -initrd /boot/initramfs-3.6.0-0.29.el7.x86_64.img \' > run.scsi.multi.sh
for i in `seq 1000 1255`;do
#qemu-img create -f qcow2 /root/img/tmp/$i.qcow2 1g >/dev/null 2>&1
echo "-drive file=/root/img/tmp/$i.qcow2,if=none,id=hd$i,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop \\" >> run.scsi.multi.sh
echo "-device scsi-hd,bus=scsi0.0,drive=hd$i,id=hd$i,lun=$i \\" >> run.scsi.multi.sh
done
echo running run.scsi.multi.sh 

sh run.scsi.multi.sh
~
Comment 7 CongLi 2013-08-01 05:56:58 EDT
(In reply to Asias He from comment #5)
> (In reply to CongLi from comment #3)
> > (In reply to Asias He from comment #2)
> > > Can you reproduce this with linux guest, e.g. RHEL7 guest.
> > > Also, Can you put your full qemu cmd line somewhere I can access.
> > 
> > 
> > Hi, Asias
> > 
> > 1. Tested with RHEL7 guest, no core dump occured.
> 
> 
> Did you actually see 260 luns in guest?
> 

No.  
# cat /proc/scsi/scsi 
Attached devices:
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: QEMU     Model: QEMU HARDDISK    Rev: 1.5.
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 01 Lun: 00
  Vendor: QEMU     Model: QEMU HARDDISK    Rev: 1.5.
  Type:   Direct-Access                    ANSI  SCSI revision: 05

When scsi disks=255, there are 255 luns.
When scsi disks=256, no 256 luns. 

> can you paste 'ls /dev/sd*' and 'sg_scan' in guest.
> 

# ls /dev/sd*
/dev/sda  /dev/sda1  /dev/sda2  /dev/sdb
# sg_scan
/dev/sg0: scsi2 channel=0 id=0 lun=0
/dev/sg1: scsi2 channel=0 id=1 lun=0


> > 
> > 2. CML is attached as cmd.sh
> 
> Thanks. Did you choose lun randomly. I theory, we can support 256 targets
> and 16384 luns per target.

lun id increase from 0 to 16383 by step 63
drive_lun:range(0,16383,63)
Comment 12 mazhang 2013-12-26 22:30:55 EST
Reproduced this bug.

Host:
qemu-kvm-1.5.3-9.el7.x86_64
kernel-3.10.0-64.el7.x86_64

Guest:
win7-64

Result:
qemu-kvm core dumped.
(gdb) bt
#0  0x00007ffff625ebc8 in pthread_once () from /lib64/libpthread.so.0
#1  0x00007ffff33ad1cc in backtrace () from /lib64/libc.so.6
#2  0x00007ffff3319144 in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff3321e8c in _int_malloc () from /lib64/libc.so.6
#4  0x00007ffff332344c in malloc () from /lib64/libc.so.6
#5  0x00007ffff7de9379 in _dl_map_object_deps () from /lib64/ld-linux-x86-64.so.2
#6  0x00007ffff7def91c in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#7  0x00007ffff7deb304 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#8  0x00007ffff7def24b in _dl_open () from /lib64/ld-linux-x86-64.so.2
#9  0x00007ffff33d3a32 in do_dlopen () from /lib64/libc.so.6
#10 0x00007ffff7deb304 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#11 0x00007ffff33d3af2 in __libc_dlopen_mode () from /lib64/libc.so.6
#12 0x00007ffff33ad0b5 in init () from /lib64/libc.so.6
#13 0x00007ffff625ebd0 in pthread_once () from /lib64/libpthread.so.0
#14 0x00007ffff33ad1cc in backtrace () from /lib64/libc.so.6
#15 0x00007ffff3319144 in __libc_message () from /lib64/libc.so.6
#16 0x00007ffff33204cd in _int_free () from /lib64/libc.so.6
#17 0x00007ffff76f39af in g_free () from /lib64/libglib-2.0.so.0
#18 0x0000555555777ab4 in virtio_scsi_complete_req (req=0x55556d405400) at /usr/src/debug/qemu-1.5.3/hw/scsi/virtio-scsi.c:70
#19 0x00005555556880fb in scsi_req_complete (req=0x55556d42a7d0, status=<optimized out>) at hw/scsi/scsi-bus.c:1617
#20 0x00005555557783f2 in virtio_scsi_handle_cmd (vdev=0x55556d3dd198, vq=0x55556d43af80) at /usr/src/debug/qemu-1.5.3/hw/scsi/virtio-scsi.c:400
#21 0x00005555556b97de in qemu_iohandler_poll (pollfds=0x55555651f200, ret=ret@entry=1) at iohandler.c:143
#22 0x00005555556beeb8 in main_loop_wait (nonblocking=<optimized out>) at main-loop.c:465
#23 0x00005555555c3d51 in main_loop () at vl.c:1986
#24 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4379
Comment 13 mazhang 2013-12-27 03:41:42 EST
Verified this bug.

Host:
qemu-img-1.5.3-30.el7.x86_64
kernel-3.10.0-64.el7.x86_64

Guest:
win7-64

Result:
Boot guest with 260 scsi disks successful, guest works well, not found core dump.

So this bug has been fixed.
Comment 14 juzhang 2013-12-29 22:35:05 EST
According to comment12 and comment13, set this issue as verified.
Comment 16 Ludek Smid 2014-06-13 05:48:09 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.