Bug 739489
| Summary: | [qemu-kvm][scalability] qemu could not open disk error at 256 devices | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | David Naori <dnaori> | ||||
| Component: | qemu-kvm | Assignee: | Ronen Hod <rhod> | ||||
| Status: | CLOSED WONTFIX | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 6.1 | CC: | acathrow, ajia, areis, chayang, cwei, danken, iheim, jkt, jmoyer, juzhang, knoel, kwolf, mgoldboi, michen, mkenneth, qzhang, rhod, shuang, tburke, virt-maint, xfu, yeylon | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 740899 (view as bug list) | Environment: | |||||
| Last Closed: | 2014-07-27 11:35:39 UTC | Type: | --- | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
This is a aio limit of the kernel that can be set through the /proc interface: /proc/sys/fs/aio-max-nr my host: # cat /proc/sys/fs/aio-max-nr 65536 Hi David can you tell me how to reproduce it? how many dis you boot guests and how many disks per guest attach? this is my testing result and command line: boot 4 guests and attach 125 disks(iscsi lv) in every guest. the last boot fifth guest and attach 17 disks. a total of booting 5 guests and attach 517 disks. all guests work well. command line: /usr/libexec/qemu-kvm -m 4G -smp 4 -drive file=/dev/vg0/data519,if=none,id=drive-virtio0-0-0,format=qcow2,werror=ignore,rerror=ignore,cache=none -device virtio-blk-pci,drive=drive-virtio0-0-0,id=virti0-0-0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:56:00:94:a3:f8,bootindex=1 -uuid 9e6f04cf-2ad7-45aa-9333-2d2ee26570c6 -boot c -monitor stdio -spice id=on,disable-ticketing,port=5913 -drive file=/dev/vg0/data2,if=none,id=drive-virtio0-0-2,format=qcow2,cache=none -device virtio-blk-pci,drive=drive-virtio0-0-2,id=virti0-0-2,multifunction=on,addr=05.0 ....... -drive file=/dev/vg0/data126,if=none,id=drive-virtio0-0-126,format=qcow2,cache=none -device virtio-blk-pci,drive=drive-virtio0-0-126,id=virti0-0-126,multifunction=on,addr=019.0 (In reply to comment #3) > my host: > # cat /proc/sys/fs/aio-max-nr > 65536 > > Hi David > > can you tell me how to reproduce it? how many dis you boot guests and how many > disks per guest attach? > > this is my testing result and command line: > > boot 4 guests and attach 125 disks(iscsi lv) in every guest. the last boot > fifth guest and attach 17 disks. a total of booting 5 guests and attach 517 > disks. all guests work well. > > command line: > /usr/libexec/qemu-kvm -m 4G -smp 4 -drive > file=/dev/vg0/data519,if=none,id=drive-virtio0-0-0,format=qcow2,werror=ignore,rerror=ignore,cache=none > -device virtio-blk-pci,drive=drive-virtio0-0-0,id=virti0-0-0 -netdev > tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:56:00:94:a3:f8,bootindex=1 -uuid > 9e6f04cf-2ad7-45aa-9333-2d2ee26570c6 -boot c -monitor stdio -spice > id=on,disable-ticketing,port=5913 -drive > file=/dev/vg0/data2,if=none,id=drive-virtio0-0-2,format=qcow2,cache=none > -device > virtio-blk-pci,drive=drive-virtio0-0-2,id=virti0-0-2,multifunction=on,addr=05.0 > ....... > -drive > file=/dev/vg0/data126,if=none,id=drive-virtio0-0-126,format=qcow2,cache=none > -device > virtio-blk-pci,drive=drive-virtio0-0-126,id=virti0-0-126,multifunction=on,addr=019.0 I did it via rhevm. on boot: - 10 vms X 24-DISKS - 15 vms X 1 DISK ** other's got this problem with 512 devices. -doesn't 125 disks per vm takes more then the maximum pci slots allowed? Here's one of the 24 disks vms cmd: /usr/libexec/qemu-kvm -S -M rhel6.0.0 -cpu Conroe -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name NEW-24-DISKS-12 -uuid c2fdb1e2-9dd7-470e-b0d9-fa0aa163770d -smbios type=1,manufacturer=Red Hat,product=RHEL,version=6Server-6.2.0.2.el6,serial=38373035-3536-4247-3830-33333434394C_78:E7:D1:E4:8E:F8,uuid=c2fdb1e2-9dd7-470e-b0d9-fa0aa163770d -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/NEW-24-DISKS-12.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2011-09-13T07:26:34 -no-shutdown -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/d05ec8e9-d708-4715-9d30-d9012a94e465/063eaf2e-cde7-486d-8ae2-7c4a6bc36a4b,if=none,id=drive-virtio-disk0,format=qcow2,serial=15-9d30-d9012a94e465,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/d8449af8-8529-402b-a174-a99596953473/fe065bb9-81cc-4465-ac0d-a980d7d8d8cb,if=none,id=drive-virtio-disk1,format=qcow2,serial=2b-a174-a99596953473,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/a6af42b0-fc33-4640-a0c9-2441c61b5c6f/d11e6577-1559-4b6f-9334-b161a4b8f5db,if=none,id=drive-virtio-disk2,format=qcow2,serial=40-a0c9-2441c61b5c6f,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x8,drive=drive-virtio-disk2,id=virtio-disk2 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/f7f1390f-1172-4f6c-a140-0b513b441850/40c34e99-1375-425a-b8b0-b052935c3493,if=none,id=drive-virtio-disk3,format=qcow2,serial=6c-a140-0b513b441850,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x9,drive=drive-virtio-disk3,id=virtio-disk3 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/e1e29a21-4f65-4a40-9f23-edcbaacaa49e/df52b601-7e08-4432-b78d-4a8b2f4e7332,if=none,id=drive-virtio-disk4,format=qcow2,serial=40-9f23-edcbaacaa49e,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0xa,drive=drive-virtio-disk4,id=virtio-disk4 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/5f2e19bd-d71d-4f5e-aabc-a051bdb0fc85/08843ea0-1b53-44ca-addc-c95383e9810f,if=none,id=drive-virtio-disk5,format=qcow2,serial=5e-aabc-a051bdb0fc85,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0xb,drive=drive-virtio-disk5,id=virtio-disk5 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/64684722-37b9-4fa8-bbb2-65005f34ab6d/5a0fffde-61ff-4edc-ab7f-3c569d4bf59a,if=none,id=drive-virtio-disk6,format=qcow2,serial=a8-bbb2-65005f34ab6d,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0xc,drive=drive-virtio-disk6,id=virtio-disk6 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/cb13314a-4cbe-4960-a03d-e54e14b4bab3/5670b6ce-887a-4916-85b5-00987a6f2059,if=none,id=drive-virtio-disk7,format=qcow2,serial=60-a03d-e54e14b4bab3,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0xd,drive=drive-virtio-disk7,id=virtio-disk7 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/c524273c-9e27-44d3-bb01-d9b506a06b4b/b8269cad-7ac6-4788-b5e5-d8093a40520b,if=none,id=drive-virtio-disk8,format=qcow2,serial=d3-bb01-d9b506a06b4b,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0xe,drive=drive-virtio-disk8,id=virtio-disk8 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/3b1735e2-d8a6-4e46-a898-24a686100af5/b019823d-5773-4846-912c-d03896673859,if=none,id=drive-virtio-disk9,format=qcow2,serial=46-a898-24a686100af5,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0xf,drive=drive-virtio-disk9,id=virtio-disk9 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/1ced3ffa-eb81-4dd9-a8b2-802cf618d67b/dfa8d0ee-844b-4725-be4d-24c8bd5dd853,if=none,id=drive-virtio-disk10,format=qcow2,serial=d9-a8b2-802cf618d67b,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x10,drive=drive-virtio-disk10,id=virtio-disk10 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/30990053-f961-4ab0-a44f-0265741f4fcc/74ad9706-52bb-4182-9bd6-0482e37899b9,if=none,id=drive-virtio-disk11,format=qcow2,serial=b0-a44f-0265741f4fcc,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x11,drive=drive-virtio-disk11,id=virtio-disk11 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/1c325264-5e62-42de-a5f0-63351fa85d4c/647c4737-0613-4f94-88f5-d8874adc8908,if=none,id=drive-virtio-disk12,format=qcow2,serial=de-a5f0-63351fa85d4c,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x12,drive=drive-virtio-disk12,id=virtio-disk12 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/7dafbbb9-7251-4014-9a31-462ee9a02fc5/334cd8c0-88cf-43b1-95a0-e0535dfa2a1a,if=none,id=drive-virtio-disk13,format=qcow2,serial=14-9a31-462ee9a02fc5,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x13,drive=drive-virtio-disk13,id=virtio-disk13 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/9afa359b-3f64-4ad9-94e7-e2ff48e3b2b6/c9ec9c25-eaad-446d-a39a-295004d43538,if=none,id=drive-virtio-disk14,format=qcow2,serial=d9-94e7-e2ff48e3b2b6,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x14,drive=drive-virtio-disk14,id=virtio-disk14 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/58252625-ca0d-4e01-80a1-14fc70b568c2/18bbc950-a2c8-4803-9366-b8f4cf0e5694,if=none,id=drive-virtio-disk15,format=qcow2,serial=01-80a1-14fc70b568c2,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x15,drive=drive-virtio-disk15,id=virtio-disk15 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/5620b91b-cf1b-4d0c-b5df-ea1bb4a2f161/f8e512b1-70c6-4ac4-96d3-072f07fb78c1,if=none,id=drive-virtio-disk16,format=qcow2,serial=0c-b5df-ea1bb4a2f161,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x16,drive=drive-virtio-disk16,id=virtio-disk16 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/0cd3205e-2408-45a4-82aa-fb31c9a71cd5/6c559ca8-998d-48e1-be6a-38327c419e28,if=none,id=drive-virtio-disk17,format=qcow2,serial=a4-82aa-fb31c9a71cd5,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x17,drive=drive-virtio-disk17,id=virtio-disk17 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/492c9e82-2248-4d2f-8960-0c5427a5d8b3/0699e017-8310-4b71-af91-73987720e418,if=none,id=drive-virtio-disk18,format=qcow2,serial=2f-8960-0c5427a5d8b3,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x18,drive=drive-virtio-disk18,id=virtio-disk18 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/fbae21d6-c66b-4571-9442-cf6a936600c6/4e8745d7-52c3-4aa4-b2e7-ace7ef040c26,if=none,id=drive-virtio-disk19,format=qcow2,serial=71-9442-cf6a936600c6,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x19,drive=drive-virtio-disk19,id=virtio-disk19 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/dd830119-558b-4e57-9ee0-71de31f20e8a/a2b93794-d11d-417b-a3f2-9912454b8f05,if=none,id=drive-virtio-disk20,format=qcow2,serial=57-9ee0-71de31f20e8a,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x1a,drive=drive-virtio-disk20,id=virtio-disk20 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/b2880506-d2c6-457f-af5c-746ecd9f06c5/1105eb68-4b1e-4598-a034-4a9be0cf6e5b,if=none,id=drive-virtio-disk21,format=qcow2,serial=7f-af5c-746ecd9f06c5,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x1b,drive=drive-virtio-disk21,id=virtio-disk21 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/eb9bbf47-9218-49c2-9d77-4db58778e80d/081fc7bd-17ea-4a30-850b-58ae63f818de,if=none,id=drive-virtio-disk22,format=qcow2,serial=c2-9d77-4db58778e80d,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x1c,drive=drive-virtio-disk22,id=virtio-disk22 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/54545296-0e0b-43fd-93c4-c269baa93822/7653ce90-09f1-4035-b860-a906d5ca3b24,if=none,id=drive-virtio-disk23,format=qcow2,serial=fd-93c4-c269baa93822,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x1d,drive=drive-virtio-disk23,id=virtio-disk23 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=37 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:81,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/NEW-24-DISKS-12.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -usb -spice port=5920,tls-port=5921,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=inputs -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 (In reply to comment #3) > can you tell me how to reproduce it? how many dis you boot guests and how many > disks per guest attach? It seems that you are missing the aio=native argument, which vdsm uses only with block devices. (In reply to comment #6) > (In reply to comment #3) > > > can you tell me how to reproduce it? how many dis you boot guests and how many > > disks per guest attach? > > It seems that you are missing the aio=native argument, which vdsm uses only > with block devices. Please make sure to incorporate the usage of aio=native in libvirt acceptance tests. Is there a scalability test defined with such numbers of disks? (In reply to comment #7) > (In reply to comment #6) > > (In reply to comment #3) > > > > > can you tell me how to reproduce it? how many dis you boot guests and how many > > > disks per guest attach? > > > > It seems that you are missing the aio=native argument, which vdsm uses only > > with block devices. > > Please make sure to incorporate the usage of aio=native in libvirt acceptance > tests. > > Is there a scalability test defined with such numbers of disks? no, only pci address cann't beyond 32. this bug can be reproduced when add aio=native option to command line. when boot No.513 disk qemu appear below error message: qemu-kvm: -drive file=/dev/vg0/data513,if=none,id=drive-virtio0-0-30,format=qcow2,cache=none,aio=native: could not open disk image /dev/vg0/data513: Invalid argument This is a similar bug: https://bugzilla.redhat.com/show_bug.cgi?id=670787, it may be helpful, although it's a SR-IOV device at above bug. Technical note: "Limit number of devices that may be assigned to a VM to avoid running out of resources. Limit set to 8 devices." Alex Christoph/Kevin, is there anything to be fixed in qemu for 6.2? To me, seems that increasing the aio limit in /etc would be enough Yes, just bumping the limit sounds like the best band-aid for now. The limit will be increase by libvirt. What I like to check is why we fail instead of retrying the EAGAIN that we should get. Christoph, can you please check it out? I removed the dependency from the qemu bug that should be solved later on (6.3) according to Christoph the error path is too complicated to fix for 6.2. When we get the EAGAIN error, with today's code it is impossible to say if it happened due to this error or another and did it happen on qcow2 meta data or the actual guest data. but we still need this solved. so either the number is increased in rhel, or we need to know to increase it in vdsm. (In reply to comment #14) > I removed the dependency from the qemu bug that should be solved later on (6.3) > according to Christoph the error path is too complicated to fix for 6.2. When > we get the EAGAIN error, with today's code it is impossible to say if it > happened due to this error or another and did it happen on qcow2 meta data or > the actual guest data. My concern is that I think we could have handled the IOs even w/ low limit. If a block aio request fails w/ EAGAIN because of hitting the maximum aio limit, we could just re-issue it when other IO request complete. Of course it's not that good for performance and the limit should be 64k but a proper handling of it is to re-send the request to the OS again. (In reply to comment #20) > (In reply to comment #14) > > I removed the dependency from the qemu bug that should be solved later on (6.3) > > according to Christoph the error path is too complicated to fix for 6.2. When > > we get the EAGAIN error, with today's code it is impossible to say if it > > happened due to this error or another and did it happen on qcow2 meta data or > > the actual guest data. > > My concern is that I think we could have handled the IOs even w/ low limit. If > a block aio request fails w/ EAGAIN because of hitting the maximum aio limit, > we could just re-issue it when other IO request complete. Of course it's not > that good for performance and the limit should be 64k but a proper handling of > it is to re-send the request to the OS again. Strictly speaking, it's not a block aio request(ie, io_submit) but a procedure of preparation of AIO request(ie, io_setup). So if we want to retry in case of -EAGAIN, we have to wait until other VM exits. It doesn't make sense. More desirable thing is that we emit concrete error message like this "You should raise /proc/sys/fs/aio-max-nr", I think. What is passed in as the nr_events parameter to io_setup? I'd start by making sure that's a reasonable number. Bumping the system maximum is a fine solution, by the way, but it can result in a large amount of pinned memory for the aio completion events. This request was not resolved in time for the current release. Red Hat invites you to ask your support representative to propose this request, if still desired, for consideration in the next release of Red Hat Enterprise Linux. This request was erroneously removed from consideration in Red Hat Enterprise Linux 6.4, which is currently under development. This request will be evaluated for inclusion in Red Hat Enterprise Linux 6.4. This request was not resolved in time for the current release. Red Hat invites you to ask your support representative to propose this request, if still desired, for consideration in the next release of Red Hat Enterprise Linux. Closing this bug. Too old, and not essential for RHEL6. Might be worth checking with RHEL7. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Created attachment 523807 [details] strace Description of problem: running total number of 255 vm's disks: * 10 vms X 24-DISKS * 15 vms X 1 DISK when trying to run another vm with a single disk (total 256 disks): c32c4406074c,cache=none,werror=stop,rerror=stop,aio=native: could not open disk image /rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/5fac004a-7c46-4955-9344-c32c4406074c/13d1618d-bf9d-459c-a652-a5c2f74559ca: Inappropriate ioctl for device and sometimes this error: qemu-kvm: -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/8132287f-de0b-44b9-a1f9-c14a570a0813/4830b523-4df5-4389-853b-229794edaffb,if=none,id=drive-virtio-disk15,format=qcow2,serial=b9-a1f9-c14a570a0813,cache=none,werror=stop,rerror=stop,aio=native: could not open disk image /rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/8132287f-de0b-44b9-a1f9-c14a570a0813/4830b523-4df5-4389-853b-229794edaffb: Invalid argument *problem is that there is a global limit on the total number of aio events # cat /proc/sys/fs/aio-max-nr 65536 after upping it, i was able to run more vms. Version-Release number of selected component (if applicable): kernel-2.6.32-193.el6.x86_64 qemu-kvm-0.12.1.2-2.184.el6.x86_64 How reproducible: 100% Actual results: Expected results: Additional info: strace attached