Hide Forgot
Description: Can not attach 1024 scsi disks by libvirt although execute "ulimit -n 20000" Version: libvirt-2.0.0-6.el7.x86_64 qemu-kvm-rhev-2.6.0-22.el7.x86_64 setup: Steps: 1. Prepare a shutdown guest,such as vm-jishao # virsh list --all Id Name State ---------------------------------------------------- - vm-jishao shut off # virsh dumpxml vm-jishao <domain type='kvm'> <name>vm-jishao</name> <uuid>9a077ad4-2de9-4c13-9c3c-abffc325b825</uuid> <memory unit='KiB'>8048576</memory> <currentMemory unit='KiB'>8048576</currentMemory> <vcpu placement='static'>4</vcpu> .... 2.Prepare below two templets # cat scsi-disk-unit_1.xml <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/#IMAGE#'/> <target dev='sd#ID#' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='#UNIT#'/> </disk> # cat scsi-disk-unit_2.xml <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/#IMAGE#'/> <target dev='sd#ID#' bus='scsi'/> <address type='drive' controller='1' bus='0' target='0' unit='#UNIT#'/> 3. Prepare below shell scripts add_disk.sh #! /bin/sh - for i in {1..512} do qemu-img create /var/lib/libvirt/images/disk_1-$i 10M sed -e "s/#IMAGE#/disk_1-$i/g" -e "s/#UNIT#/$i/g" -e "s/#ID#/$i/g" scsi-disk-unit_1.xml > scsi-disks_1-$i.xml virsh attach-device vm-jishao scsi-disks_1-$i.xml --persistent done for i in {1..512} do qemu-img create /var/lib/libvirt/images/disk_2-$i 10M sed -e "s/#IMAGE#/disk_2-$i/g" -e "s/#UNIT#/$i/g" -e "s/#ID#/$(($i+512))/g" scsi-disk-unit_2.xml > scsi-disks_2-$i.xml virsh attach-device vm-jishao scsi-disks_2-$i.xml --persistent done 4. # ulimit -n 20000 # ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 514470 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 20000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 514470 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 5.Execute the shell scripts # sh add_disk.sh 6.# virsh start vm-jishao error: Failed to start domain vm-jishao error: internal error: qemu unexpectedly closed the monitor: e-scsi1-0-0-390,id=scsi1-0-0-390 -drive file=/var/lib/libvirt/images/disk_2-391,format=raw,if=none,id=drive-scsi1-0-0-391 -device scsi-hd,bus=scsi1.0,channel=0,scsi-id=0,lun=391,drive=drive-scsi1-0-0-391,id=scsi1-0-0-391 -drive file=/var/lib/libvirt/images/disk_2-392,format=raw,if=none,id=drive-scsi1-0-0-392 -device scsi-hd,bus=scsi1.0,channel=0,scsi-id=0,lun=392,drive=drive-scsi1-0-0-392,id=scsi1-0-0-392 -drive file=/var/lib/libvirt/images/disk_2-393,format=raw,if=none,id=drive-scsi1-0-0-393 -device scsi-hd,bus=scsi1.0,channel=0,scsi-id=0,lun=393,drive=drive-scsi1-0-0-393,id=scsi1-0-0-393 -drive file=/var/lib/libvirt/images/disk_2-394,format=raw,if=none,id=drive-scsi1-0-0-394 -device scsi-hd,bus=scsi1.0,channel=0,scsi-id=0,lun=394,drive=drive-scsi1-0-0-394,id=scsi1-0-0-394 -drive file=/var/lib/libvirt/images/disk_2-395,format=raw,if=none,id=drive-scsi1-0-0-395 -device scsi-hd,bus=scsi1.0,channel=0,scsi-id=0,lun=395,drive=drive-scsi1-0-0-395,id=scsi1-0-0-395 -drive file 7.Check the /var/log/libvirt/qemu/vm-jishao.log 2016-08-31T12:46:14.880814Z qemu-kvm: -drive file=/var/lib/libvirt/images/disk_2-492,format=raw,if=none,id=drive-scsi1-0-0-492: Could not open '/var/lib/libvirt/images/disk_2-492': Too many open files 2016-08-31 12:46:14.924+0000: shutting down Actual results As the step 6 and 7 Expected results: the guest should start successfully. Additional info: 1.start the guest and hotplug disks it will get error when hotplug 925 disk error: Failed to attach device from scsi-disks_2-413.xml error: internal error: unable to execute QEMU command '__com.redhat_drive_add': Device 'drive-scsi1-0-0-413' could not be initialized Check the /var/log/libvirt/qemu/vm-jishao.log ((null):33693): Spice-Warning **: reds.c:3041:reds_accept: accept failed, Too many open files 2.check it with qemu commend line it can attach 1024 scsi disks successfully
Setting limits in a shell has no impact on libvirtd, see /etc/sysconfig/libvirtd for steps you need to take to change the limits of libvrtd.
(In reply to Jiri Denemark from comment #1) > Setting limits in a shell has no impact on libvirtd, see > /etc/sysconfig/libvirtd for steps you need to take to change the limits of > libvrtd. Thanks Jiri, it works. But I still have another question as below, If it should give a more clearly error info instead of guest'hang ? 1.Do not setting limits and keep it as default. 2. Prepare a guest as below # virsh dumpxml vm-jishao <domain type='kvm'> <name>vm-jishao</name> <uuid>9a077ad4-2de9-4c13-9c3c-abffc325b825</uuid> <memory unit='KiB'>2048576</memory> <currentMemory unit='KiB'>2048576</currentMemory> <vcpu placement='static'>2</vcpu> .... <video> <model type='cirrus' vram='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> .... 3.Prepare below two templets # cat scsi-disk-unit_1.xml <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/#IMAGE#'/> <target dev='sd#ID#' bus='scsi'/> <address type='drive' controller='0' bus='0' target='0' unit='#UNIT#'/> </disk> # cat scsi-disk-unit_2.xml <disk type='file' device='disk'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/#IMAGE#'/> <target dev='sd#ID#' bus='scsi'/> <address type='drive' controller='1' bus='0' target='0' unit='#UNIT#'/> 4. Prepare below shell scripts add_disk.sh #! /bin/sh - for i in {1..512} do qemu-img create /var/lib/libvirt/images/disk_1-$i 10M sed -e "s/#IMAGE#/disk_1-$i/g" -e "s/#UNIT#/$i/g" -e "s/#ID#/$i/g" scsi-disk-unit_1.xml > scsi-disks_1-$i.xml virsh attach-device vm-jishao scsi-disks_1-$i.xml --persistent done for i in {1..487} do qemu-img create /var/lib/libvirt/images/disk_2-$i 10M sed -e "s/#IMAGE#/disk_2-$i/g" -e "s/#UNIT#/$i/g" -e "s/#ID#/$(($i+512))/g" scsi-disk-unit_2.xml > scsi-disks_2-$i.xml virsh attach-device vm-jishao scsi-disks_2-$i.xml --persistent done 4.Execute the shell scripts # sh add_disk.sh 5.# # virsh start vm-jishao ^C # virsh list --all Id Name State ---------------------------------------------------- 1570 vm-jishao paused 6.Check the /var/log/libvirt/qemu/vm-jishao.log it does not print anything
It probably should fail and the domain should remain running, but it's hard to see what happened without any logs...