Bug 1371932

Summary: Can not attach 1024 scsi disks by libvirt although execute "ulimit -n 20000"
Product: Red Hat Enterprise Linux 7 Reporter: Jingjing Shao <jishao>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: dyuan, jdenemar, rbalakri, xuzhang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-31 14:17:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Jingjing Shao 2016-08-31 13:00:35 UTC
Description:
Can not attach 1024 scsi disks by libvirt although execute "ulimit -n 20000"

Version:
libvirt-2.0.0-6.el7.x86_64
qemu-kvm-rhev-2.6.0-22.el7.x86_64

setup:


Steps:
1. Prepare a shutdown guest,such as vm-jishao
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     vm-jishao                      shut off

# virsh dumpxml vm-jishao
<domain type='kvm'>
  <name>vm-jishao</name>
  <uuid>9a077ad4-2de9-4c13-9c3c-abffc325b825</uuid>
  <memory unit='KiB'>8048576</memory>
  <currentMemory unit='KiB'>8048576</currentMemory>
  <vcpu placement='static'>4</vcpu>
....


2.Prepare below two templets
# cat scsi-disk-unit_1.xml
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/#IMAGE#'/>
<target dev='sd#ID#' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='#UNIT#'/>
</disk>

# cat scsi-disk-unit_2.xml
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/#IMAGE#'/>
<target dev='sd#ID#' bus='scsi'/>
<address type='drive' controller='1' bus='0' target='0' unit='#UNIT#'/>


3. Prepare below shell scripts add_disk.sh
#! /bin/sh -
for i in {1..512}
do
qemu-img create /var/lib/libvirt/images/disk_1-$i 10M
sed -e "s/#IMAGE#/disk_1-$i/g" -e "s/#UNIT#/$i/g" -e "s/#ID#/$i/g" scsi-disk-unit_1.xml > scsi-disks_1-$i.xml
virsh attach-device vm-jishao scsi-disks_1-$i.xml --persistent
done
for i in {1..512}
do
qemu-img create /var/lib/libvirt/images/disk_2-$i 10M
sed -e "s/#IMAGE#/disk_2-$i/g" -e "s/#UNIT#/$i/g" -e "s/#ID#/$(($i+512))/g" scsi-disk-unit_2.xml > scsi-disks_2-$i.xml
virsh attach-device vm-jishao scsi-disks_2-$i.xml --persistent
done

4.
# ulimit -n 20000
# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 514470
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 20000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 514470
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

5.Execute the shell scripts
# sh add_disk.sh

6.# virsh start vm-jishao
error: Failed to start domain vm-jishao
error: internal error: qemu unexpectedly closed the monitor: e-scsi1-0-0-390,id=scsi1-0-0-390 -drive file=/var/lib/libvirt/images/disk_2-391,format=raw,if=none,id=drive-scsi1-0-0-391 -device scsi-hd,bus=scsi1.0,channel=0,scsi-id=0,lun=391,drive=drive-scsi1-0-0-391,id=scsi1-0-0-391 -drive file=/var/lib/libvirt/images/disk_2-392,format=raw,if=none,id=drive-scsi1-0-0-392 -device scsi-hd,bus=scsi1.0,channel=0,scsi-id=0,lun=392,drive=drive-scsi1-0-0-392,id=scsi1-0-0-392 -drive file=/var/lib/libvirt/images/disk_2-393,format=raw,if=none,id=drive-scsi1-0-0-393 -device scsi-hd,bus=scsi1.0,channel=0,scsi-id=0,lun=393,drive=drive-scsi1-0-0-393,id=scsi1-0-0-393 -drive file=/var/lib/libvirt/images/disk_2-394,format=raw,if=none,id=drive-scsi1-0-0-394 -device scsi-hd,bus=scsi1.0,channel=0,scsi-id=0,lun=394,drive=drive-scsi1-0-0-394,id=scsi1-0-0-394 -drive file=/var/lib/libvirt/images/disk_2-395,format=raw,if=none,id=drive-scsi1-0-0-395 -device scsi-hd,bus=scsi1.0,channel=0,scsi-id=0,lun=395,drive=drive-scsi1-0-0-395,id=scsi1-0-0-395 -drive file

7.Check the /var/log/libvirt/qemu/vm-jishao.log
2016-08-31T12:46:14.880814Z qemu-kvm: -drive file=/var/lib/libvirt/images/disk_2-492,format=raw,if=none,id=drive-scsi1-0-0-492: Could not open '/var/lib/libvirt/images/disk_2-492': Too many open files
2016-08-31 12:46:14.924+0000: shutting down


Actual results
As the step 6 and 7

Expected results:
the guest should start successfully.

Additional info:
1.start the guest and hotplug disks 
it will get error when hotplug 925 disk
error: Failed to attach device from scsi-disks_2-413.xml
error: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-scsi1-0-0-413' could not be initialized


Check the /var/log/libvirt/qemu/vm-jishao.log
((null):33693): Spice-Warning **: reds.c:3041:reds_accept: accept failed, Too many open files

2.check it with qemu commend line 
it can attach 1024 scsi disks successfully

Comment 1 Jiri Denemark 2016-08-31 14:17:12 UTC
Setting limits in a shell has no impact on libvirtd, see /etc/sysconfig/libvirtd for steps you need to take to change the limits of libvrtd.

Comment 2 Jingjing Shao 2016-09-01 03:14:54 UTC
(In reply to Jiri Denemark from comment #1)
> Setting limits in a shell has no impact on libvirtd, see
> /etc/sysconfig/libvirtd for steps you need to take to change the limits of
> libvrtd.


Thanks Jiri, it works.
But I still have another question as below, If it should give a more clearly error info instead of guest'hang ?

1.Do not setting limits and keep it as default.

2. Prepare a guest as below 
# virsh dumpxml vm-jishao
<domain type='kvm'>
  <name>vm-jishao</name>
  <uuid>9a077ad4-2de9-4c13-9c3c-abffc325b825</uuid>
  <memory unit='KiB'>2048576</memory>
  <currentMemory unit='KiB'>2048576</currentMemory>
  <vcpu placement='static'>2</vcpu>
....
  <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
....

3.Prepare below two templets
# cat scsi-disk-unit_1.xml
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/#IMAGE#'/>
<target dev='sd#ID#' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='#UNIT#'/>
</disk>

# cat scsi-disk-unit_2.xml
<disk type='file' device='disk'>
<driver name='qemu' type='raw'/>
<source file='/var/lib/libvirt/images/#IMAGE#'/>
<target dev='sd#ID#' bus='scsi'/>
<address type='drive' controller='1' bus='0' target='0' unit='#UNIT#'/>


4. Prepare below shell scripts add_disk.sh
#! /bin/sh -
for i in {1..512}
do
qemu-img create /var/lib/libvirt/images/disk_1-$i 10M
sed -e "s/#IMAGE#/disk_1-$i/g" -e "s/#UNIT#/$i/g" -e "s/#ID#/$i/g" scsi-disk-unit_1.xml > scsi-disks_1-$i.xml
virsh attach-device vm-jishao scsi-disks_1-$i.xml --persistent
done
for i in {1..487}
do
qemu-img create /var/lib/libvirt/images/disk_2-$i 10M
sed -e "s/#IMAGE#/disk_2-$i/g" -e "s/#UNIT#/$i/g" -e "s/#ID#/$(($i+512))/g" scsi-disk-unit_2.xml > scsi-disks_2-$i.xml
virsh attach-device vm-jishao scsi-disks_2-$i.xml --persistent
done

4.Execute the shell scripts
# sh add_disk.sh

5.# # virsh start vm-jishao
^C


# virsh list --all
 Id    Name                           State
----------------------------------------------------
 1570  vm-jishao                      paused


6.Check the /var/log/libvirt/qemu/vm-jishao.log
it does not print anything

Comment 3 Jiri Denemark 2016-09-05 11:22:47 UTC
It probably should fail and the domain should remain running, but it's hard to see what happened without any logs...