RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 739489 - [qemu-kvm][scalability] qemu could not open disk error at 256 devices
Summary: [qemu-kvm][scalability] qemu could not open disk error at 256 devices
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm
Version: 6.1
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Ronen Hod
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-09-19 09:01 UTC by David Naori
Modified: 2023-09-14 01:25 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 740899 (view as bug list)
Environment:
Last Closed: 2014-07-27 11:35:39 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
strace (40.53 KB, application/octet-stream)
2011-09-19 09:01 UTC, David Naori 		no flags Details
View All

Description David Naori 2011-09-19 09:01:09 UTC 
Created attachment 523807 [details]
strace

Description of problem:
running total number of 255 vm's disks:
* 10 vms X 24-DISKS
* 15 vms X 1 DISK

when trying to run another vm with a single disk (total 256 disks):
c32c4406074c,cache=none,werror=stop,rerror=stop,aio=native: could not open disk image /rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/5fac004a-7c46-4955-9344-c32c4406074c/13d1618d-bf9d-459c-a652-a5c2f74559ca: Inappropriate ioctl for device

and sometimes this error:
qemu-kvm: -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/8132287f-de0b-44b9-a1f9-c14a570a0813/4830b523-4df5-4389-853b-229794edaffb,if=none,id=drive-virtio-disk15,format=qcow2,serial=b9-a1f9-c14a570a0813,cache=none,werror=stop,rerror=stop,aio=native: could not open disk image /rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/8132287f-de0b-44b9-a1f9-c14a570a0813/4830b523-4df5-4389-853b-229794edaffb: Invalid argument


*problem is that there is a global limit on the total number of aio events

# cat /proc/sys/fs/aio-max-nr
65536

after upping it, i was able to run more vms.

Version-Release number of selected component (if applicable):
kernel-2.6.32-193.el6.x86_64
qemu-kvm-0.12.1.2-2.184.el6.x86_64

How reproducible:
100%


Actual results:


Expected results:


Additional info:
strace attached
Comment 2 Dor Laor 2011-09-19 12:03:53 UTC 
This is a aio limit of the kernel that can be set through the /proc interface:
/proc/sys/fs/aio-max-nr
Comment 3 FuXiangChun 2011-09-19 12:31:49 UTC 
my host:
# cat /proc/sys/fs/aio-max-nr
65536

Hi David

can you tell me how to reproduce it? how many dis you boot guests and how many disks per guest attach? 

this is my testing result and command line:

boot 4 guests and attach 125 disks(iscsi lv) in every guest. the last boot fifth guest and attach 17 disks.  a total of booting 5 guests and attach 517 disks.  all guests work well. 

command line:
/usr/libexec/qemu-kvm -m 4G -smp 4  -drive file=/dev/vg0/data519,if=none,id=drive-virtio0-0-0,format=qcow2,werror=ignore,rerror=ignore,cache=none -device virtio-blk-pci,drive=drive-virtio0-0-0,id=virti0-0-0 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:56:00:94:a3:f8,bootindex=1 -uuid 9e6f04cf-2ad7-45aa-9333-2d2ee26570c6 -boot c -monitor stdio -spice id=on,disable-ticketing,port=5913 -drive file=/dev/vg0/data2,if=none,id=drive-virtio0-0-2,format=qcow2,cache=none -device virtio-blk-pci,drive=drive-virtio0-0-2,id=virti0-0-2,multifunction=on,addr=05.0
.......
-drive file=/dev/vg0/data126,if=none,id=drive-virtio0-0-126,format=qcow2,cache=none -device virtio-blk-pci,drive=drive-virtio0-0-126,id=virti0-0-126,multifunction=on,addr=019.0
Comment 4 David Naori 2011-09-19 12:48:04 UTC 
(In reply to comment #3)
> my host:
> # cat /proc/sys/fs/aio-max-nr
> 65536
> 
> Hi David
> 
> can you tell me how to reproduce it? how many dis you boot guests and how many
> disks per guest attach? 
> 
> this is my testing result and command line:
> 
> boot 4 guests and attach 125 disks(iscsi lv) in every guest. the last boot
> fifth guest and attach 17 disks.  a total of booting 5 guests and attach 517
> disks.  all guests work well. 
> 
> command line:
> /usr/libexec/qemu-kvm -m 4G -smp 4  -drive
> file=/dev/vg0/data519,if=none,id=drive-virtio0-0-0,format=qcow2,werror=ignore,rerror=ignore,cache=none
> -device virtio-blk-pci,drive=drive-virtio0-0-0,id=virti0-0-0 -netdev
> tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:56:00:94:a3:f8,bootindex=1 -uuid
> 9e6f04cf-2ad7-45aa-9333-2d2ee26570c6 -boot c -monitor stdio -spice
> id=on,disable-ticketing,port=5913 -drive
> file=/dev/vg0/data2,if=none,id=drive-virtio0-0-2,format=qcow2,cache=none
> -device
> virtio-blk-pci,drive=drive-virtio0-0-2,id=virti0-0-2,multifunction=on,addr=05.0
> .......
> -drive
> file=/dev/vg0/data126,if=none,id=drive-virtio0-0-126,format=qcow2,cache=none
> -device
> virtio-blk-pci,drive=drive-virtio0-0-126,id=virti0-0-126,multifunction=on,addr=019.0

I did it via rhevm.

on boot:
- 10 vms X 24-DISKS
- 15 vms X 1 DISK

** other's got this problem with 512 devices.

-doesn't 125 disks per vm takes more then the maximum pci slots allowed?



Here's one of the 24 disks vms cmd:

/usr/libexec/qemu-kvm -S -M rhel6.0.0 -cpu Conroe -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name NEW-24-DISKS-12 -uuid c2fdb1e2-9dd7-470e-b0d9-fa0aa163770d -smbios type=1,manufacturer=Red Hat,product=RHEL,version=6Server-6.2.0.2.el6,serial=38373035-3536-4247-3830-33333434394C_78:E7:D1:E4:8E:F8,uuid=c2fdb1e2-9dd7-470e-b0d9-fa0aa163770d -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/NEW-24-DISKS-12.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2011-09-13T07:26:34 -no-shutdown -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/d05ec8e9-d708-4715-9d30-d9012a94e465/063eaf2e-cde7-486d-8ae2-7c4a6bc36a4b,if=none,id=drive-virtio-disk0,format=qcow2,serial=15-9d30-d9012a94e465,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/d8449af8-8529-402b-a174-a99596953473/fe065bb9-81cc-4465-ac0d-a980d7d8d8cb,if=none,id=drive-virtio-disk1,format=qcow2,serial=2b-a174-a99596953473,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/a6af42b0-fc33-4640-a0c9-2441c61b5c6f/d11e6577-1559-4b6f-9334-b161a4b8f5db,if=none,id=drive-virtio-disk2,format=qcow2,serial=40-a0c9-2441c61b5c6f,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x8,drive=drive-virtio-disk2,id=virtio-disk2 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/f7f1390f-1172-4f6c-a140-0b513b441850/40c34e99-1375-425a-b8b0-b052935c3493,if=none,id=drive-virtio-disk3,format=qcow2,serial=6c-a140-0b513b441850,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x9,drive=drive-virtio-disk3,id=virtio-disk3 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/e1e29a21-4f65-4a40-9f23-edcbaacaa49e/df52b601-7e08-4432-b78d-4a8b2f4e7332,if=none,id=drive-virtio-disk4,format=qcow2,serial=40-9f23-edcbaacaa49e,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0xa,drive=drive-virtio-disk4,id=virtio-disk4 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/5f2e19bd-d71d-4f5e-aabc-a051bdb0fc85/08843ea0-1b53-44ca-addc-c95383e9810f,if=none,id=drive-virtio-disk5,format=qcow2,serial=5e-aabc-a051bdb0fc85,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0xb,drive=drive-virtio-disk5,id=virtio-disk5 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/64684722-37b9-4fa8-bbb2-65005f34ab6d/5a0fffde-61ff-4edc-ab7f-3c569d4bf59a,if=none,id=drive-virtio-disk6,format=qcow2,serial=a8-bbb2-65005f34ab6d,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0xc,drive=drive-virtio-disk6,id=virtio-disk6 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/cb13314a-4cbe-4960-a03d-e54e14b4bab3/5670b6ce-887a-4916-85b5-00987a6f2059,if=none,id=drive-virtio-disk7,format=qcow2,serial=60-a03d-e54e14b4bab3,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0xd,drive=drive-virtio-disk7,id=virtio-disk7 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/c524273c-9e27-44d3-bb01-d9b506a06b4b/b8269cad-7ac6-4788-b5e5-d8093a40520b,if=none,id=drive-virtio-disk8,format=qcow2,serial=d3-bb01-d9b506a06b4b,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0xe,drive=drive-virtio-disk8,id=virtio-disk8 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/3b1735e2-d8a6-4e46-a898-24a686100af5/b019823d-5773-4846-912c-d03896673859,if=none,id=drive-virtio-disk9,format=qcow2,serial=46-a898-24a686100af5,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0xf,drive=drive-virtio-disk9,id=virtio-disk9 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/1ced3ffa-eb81-4dd9-a8b2-802cf618d67b/dfa8d0ee-844b-4725-be4d-24c8bd5dd853,if=none,id=drive-virtio-disk10,format=qcow2,serial=d9-a8b2-802cf618d67b,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x10,drive=drive-virtio-disk10,id=virtio-disk10 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/30990053-f961-4ab0-a44f-0265741f4fcc/74ad9706-52bb-4182-9bd6-0482e37899b9,if=none,id=drive-virtio-disk11,format=qcow2,serial=b0-a44f-0265741f4fcc,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x11,drive=drive-virtio-disk11,id=virtio-disk11 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/1c325264-5e62-42de-a5f0-63351fa85d4c/647c4737-0613-4f94-88f5-d8874adc8908,if=none,id=drive-virtio-disk12,format=qcow2,serial=de-a5f0-63351fa85d4c,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x12,drive=drive-virtio-disk12,id=virtio-disk12 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/7dafbbb9-7251-4014-9a31-462ee9a02fc5/334cd8c0-88cf-43b1-95a0-e0535dfa2a1a,if=none,id=drive-virtio-disk13,format=qcow2,serial=14-9a31-462ee9a02fc5,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x13,drive=drive-virtio-disk13,id=virtio-disk13 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/9afa359b-3f64-4ad9-94e7-e2ff48e3b2b6/c9ec9c25-eaad-446d-a39a-295004d43538,if=none,id=drive-virtio-disk14,format=qcow2,serial=d9-94e7-e2ff48e3b2b6,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x14,drive=drive-virtio-disk14,id=virtio-disk14 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/58252625-ca0d-4e01-80a1-14fc70b568c2/18bbc950-a2c8-4803-9366-b8f4cf0e5694,if=none,id=drive-virtio-disk15,format=qcow2,serial=01-80a1-14fc70b568c2,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x15,drive=drive-virtio-disk15,id=virtio-disk15 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/5620b91b-cf1b-4d0c-b5df-ea1bb4a2f161/f8e512b1-70c6-4ac4-96d3-072f07fb78c1,if=none,id=drive-virtio-disk16,format=qcow2,serial=0c-b5df-ea1bb4a2f161,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x16,drive=drive-virtio-disk16,id=virtio-disk16 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/0cd3205e-2408-45a4-82aa-fb31c9a71cd5/6c559ca8-998d-48e1-be6a-38327c419e28,if=none,id=drive-virtio-disk17,format=qcow2,serial=a4-82aa-fb31c9a71cd5,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x17,drive=drive-virtio-disk17,id=virtio-disk17 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/492c9e82-2248-4d2f-8960-0c5427a5d8b3/0699e017-8310-4b71-af91-73987720e418,if=none,id=drive-virtio-disk18,format=qcow2,serial=2f-8960-0c5427a5d8b3,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x18,drive=drive-virtio-disk18,id=virtio-disk18 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/fbae21d6-c66b-4571-9442-cf6a936600c6/4e8745d7-52c3-4aa4-b2e7-ace7ef040c26,if=none,id=drive-virtio-disk19,format=qcow2,serial=71-9442-cf6a936600c6,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x19,drive=drive-virtio-disk19,id=virtio-disk19 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/dd830119-558b-4e57-9ee0-71de31f20e8a/a2b93794-d11d-417b-a3f2-9912454b8f05,if=none,id=drive-virtio-disk20,format=qcow2,serial=57-9ee0-71de31f20e8a,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x1a,drive=drive-virtio-disk20,id=virtio-disk20 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/b2880506-d2c6-457f-af5c-746ecd9f06c5/1105eb68-4b1e-4598-a034-4a9be0cf6e5b,if=none,id=drive-virtio-disk21,format=qcow2,serial=7f-af5c-746ecd9f06c5,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x1b,drive=drive-virtio-disk21,id=virtio-disk21 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/eb9bbf47-9218-49c2-9d77-4db58778e80d/081fc7bd-17ea-4a30-850b-58ae63f818de,if=none,id=drive-virtio-disk22,format=qcow2,serial=c2-9d77-4db58778e80d,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x1c,drive=drive-virtio-disk22,id=virtio-disk22 -drive file=/rhev/data-center/61ecb7ad-1007-4ba7-a43b-9e80569c0cb3/ccbe9419-e910-4a9a-a742-8693aa1f503c/images/54545296-0e0b-43fd-93c4-c269baa93822/7653ce90-09f1-4035-b860-a906d5ca3b24,if=none,id=drive-virtio-disk23,format=qcow2,serial=fd-93c4-c269baa93822,cache=none,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x1d,drive=drive-virtio-disk23,id=virtio-disk23 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=37 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:81,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/NEW-24-DISKS-12.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -usb -spice port=5920,tls-port=5921,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=inputs -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0
Comment 6 Dan Kenigsberg 2011-09-19 13:18:06 UTC 
(In reply to comment #3)
 
> can you tell me how to reproduce it? how many dis you boot guests and how many
> disks per guest attach? 

It seems that you are missing the aio=native argument, which vdsm uses only with block devices.
Comment 7 Ayal Baron 2011-09-19 21:32:14 UTC 
(In reply to comment #6)
> (In reply to comment #3)
> 
> > can you tell me how to reproduce it? how many dis you boot guests and how many
> > disks per guest attach? 
> 
> It seems that you are missing the aio=native argument, which vdsm uses only
> with block devices.

Please make sure to incorporate the usage of aio=native in libvirt acceptance tests.

Is there a scalability test defined with such numbers of disks?
Comment 8 FuXiangChun 2011-09-20 03:14:32 UTC 
(In reply to comment #7)
> (In reply to comment #6)
> > (In reply to comment #3)
> > 
> > > can you tell me how to reproduce it? how many dis you boot guests and how many
> > > disks per guest attach? 
> > 
> > It seems that you are missing the aio=native argument, which vdsm uses only
> > with block devices.
> 
> Please make sure to incorporate the usage of aio=native in libvirt acceptance
> tests.
> 
> Is there a scalability test defined with such numbers of disks?
  no, only pci address cann't beyond 32.

this bug can be reproduced when add aio=native option to command line. 
when boot No.513 disk qemu appear below error message:
qemu-kvm: -drive file=/dev/vg0/data513,if=none,id=drive-virtio0-0-30,format=qcow2,cache=none,aio=native: could not open disk image /dev/vg0/data513: Invalid argument
Comment 9 Alex Jia 2011-09-20 03:47:40 UTC 
This is a similar bug: https://bugzilla.redhat.com/show_bug.cgi?id=670787,
it may be helpful, although it's a SR-IOV device at above bug.

Technical note:
"Limit number of devices that may be assigned to a VM to avoid running out of
resources.  Limit set to 8 devices."

Alex
Comment 10 Dor Laor 2011-09-20 13:11:52 UTC 
Christoph/Kevin, is there anything to be fixed in qemu for 6.2? To me, seems that increasing the aio limit in /etc would be enough
Comment 11 chellwig@redhat.com 2011-09-20 14:31:21 UTC 
Yes, just bumping the limit sounds like the best band-aid for now.
Comment 12 Dor Laor 2011-09-27 08:03:30 UTC 
The limit will be increase by libvirt.
What I like to check is why we fail instead of retrying the EAGAIN that we should get. Christoph, can you please check it out?
Comment 14 Dor Laor 2011-10-10 15:38:57 UTC 
I removed the dependency from the qemu bug that should be solved later on (6.3) according to Christoph the error path is too complicated to fix for 6.2. When we get the EAGAIN error, with today's code it is impossible to say if it happened due to this error or another and did it happen on qcow2 meta data or the actual guest data.
Comment 18 Itamar Heim 2011-12-11 12:01:55 UTC 
but we still need this solved. so either the number is increased in rhel, or we need to know to increase it in vdsm.
Comment 20 Dor Laor 2011-12-18 12:51:20 UTC 
(In reply to comment #14)
> I removed the dependency from the qemu bug that should be solved later on (6.3)
> according to Christoph the error path is too complicated to fix for 6.2. When
> we get the EAGAIN error, with today's code it is impossible to say if it
> happened due to this error or another and did it happen on qcow2 meta data or
> the actual guest data.

My concern is that I think we could have handled the IOs even w/ low limit. If a block aio request fails w/ EAGAIN because of hitting the maximum aio limit, we could just re-issue it when other IO request complete. Of course it's not that good for performance and the limit should be 64k but a proper handling of it is to re-send the request to the OS again.
Comment 21 Minchan Kim 2011-12-20 02:14:58 UTC 
(In reply to comment #20)
> (In reply to comment #14)
> > I removed the dependency from the qemu bug that should be solved later on (6.3)
> > according to Christoph the error path is too complicated to fix for 6.2. When
> > we get the EAGAIN error, with today's code it is impossible to say if it
> > happened due to this error or another and did it happen on qcow2 meta data or
> > the actual guest data.
> 
> My concern is that I think we could have handled the IOs even w/ low limit. If
> a block aio request fails w/ EAGAIN because of hitting the maximum aio limit,
> we could just re-issue it when other IO request complete. Of course it's not
> that good for performance and the limit should be 64k but a proper handling of
> it is to re-send the request to the OS again.

Strictly speaking, it's not a block aio request(ie, io_submit) but a procedure of preparation of AIO request(ie, io_setup). So if we want to retry in case of -EAGAIN, we have to wait until other VM exits. It doesn't make sense. More desirable thing is that we emit concrete error message like this "You should raise /proc/sys/fs/aio-max-nr", I think.
Comment 22 Jeff Moyer 2012-03-05 16:47:06 UTC 
What is passed in as the nr_events parameter to io_setup?  I'd start by making sure that's a reasonable number.  Bumping the system maximum is a fine solution, by the way, but it can result in a large amount of pinned memory for the aio completion events.
Comment 23 RHEL Program Management 2012-07-10 08:18:31 UTC 
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 24 RHEL Program Management 2012-07-11 02:00:44 UTC 
This request was erroneously removed from consideration in Red Hat Enterprise Linux 6.4, which is currently under development.  This request will be evaluated for inclusion in Red Hat Enterprise Linux 6.4.
Comment 25 RHEL Program Management 2012-12-14 08:16:17 UTC 
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 27 Ronen Hod 2014-07-27 11:35:39 UTC 
Closing this bug. Too old, and not essential for RHEL6.
Might be worth checking with RHEL7.
Comment 28 Red Hat Bugzilla 2023-09-14 01:25:27 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days
Note You need to log in before you can comment on or make changes to this bug.