Bug 841519 - virtio scsi disk could not be initialized when hotplug virtio scsi disk more than 990
virtio scsi disk could not be initialized when hotplug virtio scsi disk more ...
Status: CLOSED DEFERRED
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.3
x86_64 Linux
medium Severity medium
: rc
: ---
Assigned To: Paolo Bonzini
Virtualization Bugs
:
Depends On:
Blocks: 997736
  Show dependency treegraph
 
Reported: 2012-07-19 05:40 EDT by FuXiangChun
Modified: 2015-03-25 03:16 EDT (History)
19 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 997736 (view as bug list)
Environment:
Last Closed: 2012-07-20 09:51:38 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
990 scsi device guest cli (115.17 KB, text/plain)
2012-12-19 22:25 EST, weizhang
no flags Details
result of find /sys in guest (6.53 MB, text/plain)
2012-12-19 22:27 EST, weizhang
no flags Details

  None (edit)
Description FuXiangChun 2012-07-19 05:40:09 EDT
Description of problem:
 qemu monitor display "could not open disk image /root/images/virtio-scsi-991: Too many open files
 Device 'scsi991' could not be initialized
" when hotplug number more than 990.  not find any error message from syslog. host and guest work well. 

Version-Release number of selected component (if applicable):
#rpm -qa|grep qemu
qemu-kvm-0.12.1.2-2.295.el6.x86_64

# uname -r
2.6.32-278.el6.x86_64

guest: rhel6.3

How reproducible:
always

Steps to Reproduce:
1./usr/libexec/qemu-kvm -name 'vm1' -drive file=/root/RHEL6.3-20120613.2-Server-bak.qcow2,index=0,if=none,id=drive-virtio-disk1,media=disk,cache=none,boot=off,snapshot=off,format=qcow2,aio=native -device virtio-scsi-pci,id=bus1 -device scsi-hd,drive=drive-virtio-disk1,id=virtio-disk1 -device virtio-net-pci,netdev=idB1z4op,mac=9a:5d:72:92:0c:ea,id=ndev00idB1z4op -netdev tap,id=idB1z4op,vhost=on,script=/home/autotest-devel/client/tests/kvm/scripts/qemu-ifup-switch -m 102400 -smp 10,cores=10,threads=1,sockets=1 -cpu 'Penryn' -vnc :1 -vga std -rtc base=localtime,clock=host,driftfix=slew -M rhel6.3.0 -boot order=cdn,once=d,menu=off -enable-kvm  -monitor unix:/tmp/monitor2,server,nowait -monitor stdio

2. hotplug virtio-scsi with this script

i=1

j=0

while [ $i -lt 16388 ]

do

j=$((i%255))

sleep 2 

echo "__com.redhat_drive_add file=/root/images/virtio-scsi-$i,format=qcow2,media=disk,id=scsi$i" | nc -U /tmp/monitor2

echo "device_add scsi-hd,bus=bus1.0,drive=scsi$i,scsi-id=$j,id=hd$i" |nc -U /tmp/monitor2

i=$(($i+1))

done


3.check disks inside guest
  
Actual results:
(qemu) __com.redhat_drive_add file=/root/images/virtio-scsi-990,format=qcow2,media=disk,id=scsi990

(qemu) QEMU 0.12.1 monitor - type 'help' for more information

(qemu) device_add scsi-hd,bus=bus1.0,drive=scsi990,scsi-id=225,id=hd990

(qemu) QEMU 0.12.1 monitor - type 'help' for more information

(qemu) __com.redhat_drive_add file=/root/images/virtio-scsi-991,format=qcow2,media=disk,id=scsi991

could not open disk image /root/images/virtio-scsi-991: Too many open files

Device 'scsi991' could not be initialized


Expected results:
can hotplug 16384 virtio scsi disk.

Additional info:
guest can find 990 disks. host memory is 128G, and assign 100G memory to guest.
Comment 1 Paolo Bonzini 2012-07-20 09:51:38 EDT
You simply hit a limit on the number of open files.  These can be changed with "ulimit -n".
Comment 2 FuXiangChun 2012-07-22 21:32:03 EDT
(In reply to comment #1)
> You simply hit a limit on the number of open files.  These can be changed
> with "ulimit -n".

I have executed "ulimit -n 20000" in host and guest before hotplug. but still hit this issue.
Comment 3 FuXiangChun 2012-07-23 21:55:18 EDT
(In reply to comment #1)
> You simply hit a limit on the number of open files.  These can be changed
> with "ulimit -n".

Paolo,
  "Too many open files...." this warning message is from qemu monitor, and I have executed "ulimit -n 20000" in host and guest.  so I think it may be qemu's issue.

In addition,rhel7 hasn't this issue.
Comment 4 Paolo Bonzini 2012-07-24 07:28:13 EDT
If RHEL7 doesn't have this issue, it could be a kernel problem.  In any case, I thing 990 is good enough for RHEL6. :)
Comment 5 weizhang 2012-12-18 21:28:48 EST
Hi Paolo,

When I start with 990 disks in 1 guest, I find in guest only 512 can be recognized, is that a bug or there is limitation in it?
Comment 6 Paolo Bonzini 2012-12-19 10:24:46 EST
Can you attach the XML file?
Comment 7 Paolo Bonzini 2012-12-19 10:25:12 EST
And also please the result of "find /sys" in the guest.
Comment 8 weizhang 2012-12-19 22:25:40 EST
Created attachment 666502 [details]
990 scsi device guest cli

with (qemu)info block we can see disk990 info
Comment 9 weizhang 2012-12-19 22:27:13 EST
Created attachment 666503 [details]
result of find /sys in guest
Comment 10 zhonglinzhang 2013-09-02 01:23:45 EDT
windows2012 guest and qemu-kvm hang when when hotplug virtio scsi disk less than 1000

Version-Release number of selected component (if applicable): 
kernel: 2.6.32-414.el6.x86_64
qemu-kvm: qemu-kvm-0.12.1.2-2.398.el6.x86_64

How reproducible:
100%

Steps to Reproduce:

1. boot win2012 guest with following command line:
/usr/libexec/qemu-kvm -M pc -enable-kvm -m 200G -smp 32,sockets=4,cores=8,threads=1 -name scalability-test -rtc base=localtime,clock=host,driftfix=slew -k en-us -boot menu=on -spice disable-ticketing,port=5931 -vga qxl -monitor stdio -drive file=/home/win2012-64-virtio.qcow2,if=none,id=drive-system-disk,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-system-disk,id=system-disk,addr=0x9 -device virtio-scsi-pci,id=bus1,bus=pci.0,addr=0x5 -monitor unix:/tmp/monitor11,server,nowait

2. hotplug virtio-scsi with this script:
cat test.sh
i=0
j=0
while [ $i -lt 16666 ]
do
j=$((i%255))

sleep 5
if ((i>5000)) ;then
     sleep 5
fi

echo "__com.redhat_drive_add file=/home/storage/usb-storage$i.qcow2,format=qcow2,media=disk,id=scsi$i" | nc -U /tmp/monitor11
echo "device_add scsi-hd,bus=bus1.0,drive=scsi$i,scsi-id=$j,id=hd$i" |nc -U /tmp/monitor11
i=$(($i+1))
done

3. (qemu) info block
scsi993: removable=0 io-status=ok file=/home/storage/usb-storage993.qcow2 ro=0 drv=qcow2 encrypted=0
scsi994: removable=0 io-status=ok file=/home/storage/usb-storage994.qcow2 ro=0 drv=qcow2 encrypted=0
scsi995: removable=0 io-status=ok file=/home/storage/usb-storage995.qcow2 ro=0 drv=qcow2 encrypted=0
scsi996: removable=0 io-status=ok file=/home/storage/usb-storage996.qcow2 ro=0 drv=qcow2 encrypted=0
scsi997: removable=0 io-status=ok file=/home/storage/usb-storage997.qcow2 ro=0 drv=qcow2 encrypted=0
scsi998: removable=0 io-status=ok file=/home/storage/usb-storage998.qcow2 ro=0 drv=qcow2 encrypted=0
(qemu)

Actual results:
windows2012 guest and qemu-kvm hang when hotplug virtio scsi disk less than 1000
Comment 11 Hu Jianwei 2014-09-04 04:15:27 EDT
Recorded testing results on new versions.

Version:
libvirt-0.10.2-44.el6.x86_64
qemu-kvm-0.12.1.2-2.441.el6.x86_64
kernel-2.6.32-498.el6.x86_64


1. Prepare a shutoff guest
[root@dell-per415-03 364648]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 8     r7                             shut off

2. Attach 1011 scsi disks using below way
[root@dell-per415-03 364648]# cat run.sh 
#! /bin/sh -

for i in {0..1010}
do 
    sed -e "s/#IMAGE#/disk-$i/g" -e "s/#UNIT#/$i/g" -e "s/#ID#/$i/g" 111scsi-disk-unit.xml > scsi-disks-$i.xml
    virsh attach-device r7 scsi-disks-$i.xml --persistent
    qemu-img create /var/lib/libvirt/images/disk-$i 10M
done

[root@dell-per415-03 364648]# cat 111scsi-disk-unit.xml 
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/#IMAGE#'/>
<target dev='sd#ID#' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='#UNIT#'/>
</disk>

3. Start this domain, but no response
[root@dell-per415-03 364648]# time virsh start r7

^C

real	24m34.639s
user	0m0.044s
sys	0m0.037s

[root@dell-per415-03 364648]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 9     r7                             shut off

4. The domain is shutoff, but the qemu process still exists
[root@dell-per415-03 364648]# ps aux | grep qemu-kvm
qemu       305 99.9  0.0 1411916 32796 ?       Rl   15:46  25:03 /usr/libexec/qemu-kvm -name r7 -S -M rhel6.6.0 -enable-kvm -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 0f1001f7-ac45-9c85-d143-959a9f054c6b -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/r7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x7 -drive file=/mnt/r7.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive file=/var/lib/libvirt/images/disk-331,if=none,id=drive-scsi0-0-0-331,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=331,drive=drive-scsi0-0-0-331,id=scsi0-0-0-331 -drive file=/var/lib/libvirt/images/disk-332,if=none,id=drive-scsi0-0-0-332,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=332,drive=drive-scsi0-0-0-332,id=scsi0-0-0-332 -drive file=/var/lib/libvirt/images/disk-333,if=none,id=drive-scsi0-0-0-333,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=333,drive=drive-scsi0-0-0-333,id=scsi0-0-0-333 -drive file=/var/lib/libvirt/images/disk-334,if=none,id=drive-scsi0-0-0-334,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=334,drive=drive-scsi0-0-0-334,id=scsi0-0-0-334 -drive file=/var/lib/libvirt/images/disk-335,if=none,id=drive-scsi0-0-0-335,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=335,drive=drive-scsi0-0-0-335,id=scsi0-0-0-335 -drive file=/var/lib/libvirt/images/disk-336,if=none,id=drive-scsi0-0-0-336,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=336,drive=drive-scsi0-0-0-336,id=scsi0-0-0-336 -drive file=/var/lib/libvirt/images/disk-337,if=none,id=drive-scsi0-0-0-337,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=337,drive=drive-scsi0-0-0-337,id=scsi0-0-0-337 -drive file=/var/lib/libvirt/images/disk-338,if=none,id=drive-scsi0-0-0-338,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=338,drive=drive-scsi0-0-0-338,id=scsi0-0-0-338 -drive file=/var/lib/libvirt/images/disk-339,if=none,id=drive-scsi0-0-0-339,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=339,drive=drive-scsi0-0-0-339,id=scsi0-0-0-339 -drive file=/var/lib/libvirt/images/disk-340,if=none,id=drive-scsi0-0-0-340,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=340,drive=drive-scsi0-0-0-340,id=scsi0-0-0-340 -drive file=/var/lib/libvirt/images/disk-341,if=none,id=drive-scsi0-0-0-341,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=341,drive=drive-scsi0-0-0-341,id=scsi0-0-0-341 -drive file=/var/lib/libvirt/images/disk-342,if=none,id=drive-scsi0-0-0-342,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=342,drive=drive-scsi0-0-0-342,id=scsi0-0-0-342 -drive file=/var/lib/libvirt/images/disk-343,if=none,id=drive-scsi0-0-0-343,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=343,drive=drive-scsi0-0-0-343,id=scsi0-0-0-343 -drive file=/var/lib/libvirt/images/disk-344,if=none,id=drive-scsi0-0-0-344,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=344,drive=drive-scsi0-0-0-344,id=scsi0-0-0-344 -drive file=/var/lib/libvirt/images/disk-345,if=none,id=drive-scsi0-0-0-345,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=345,drive=drive-scsi0-0-0-345,id=scsi0-0-0-345 -drive file=/var/lib/libvirt/images/disk-346,if=none,id=drive-scsi0-0-0-346,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=346,drive=drive-scsi0-0-0-346,id=scsi0-0-0-346 -drive file=/var/lib/libvirt/images/disk-347,if=none,id=drive-scsi0-0-0-347,format=raw -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=347,drive=drive-scsi0-0-0-347,id=scsi0-0-0-347 -drive file=/var/lib/libvirt/images/disk-348,if=none,id=drive-scsi0-0-0-348,format=raw -devic
root       615  0.0  0.0 103252   856 pts/1    S+   16:11   0:00 grep qemu-kvm
Comment 12 yisun 2015-03-25 03:16:32 EDT
record test result on RHEL6.7 
Version:
libvirt-0.10.2-50.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.460.el6.x86_64
kernel-2.6.32-544.el6.x86_64


1. Prepare a shutoff guest
[root@localhost]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 8     vm1                             shut off

2. Attach 1011 scsi disks using below way
[root@localhost]# cat run.sh 
#! /bin/sh -

for i in {0..1010}
do 
    sed -e "s/#IMAGE#/disk-$i/g" -e "s/#UNIT#/$i/g" -e "s/#ID#/$i/g" 111scsi-disk-unit.xml > scsi-disks-$i.xml
    virsh attach-device vm1 scsi-disks-$i.xml --persistent
    qemu-img create /var/lib/libvirt/images/disk-$i 10M
done

[root@dell-per415-03 364648]# cat 111scsi-disk-unit.xml 
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/#IMAGE#'/>
<target dev='sd#ID#' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='#UNIT#'/>
</disk>

3. Start this domain, failed after a long time.
[root@localhost]# # virsh start vm1
error: Failed to start domain vm1
error: Unable to read from monitor: Connection reset by peer


[root@dlocalhost]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 9     vm1                             shut off

4. # tail -5 /var/log/libvirt/qemu/vm1.log
....
char device redirected to /dev/pts/0
kvm_create_vcpu: Too many open files
Failed to create vCPU. Check the -smp parameter.
2015-03-25 06:53:17.591+0000: shutting down

Note You need to log in before you can comment on or make changes to this bug.