Bug 1947230

Summary: Enable QEMU support for io_uring in RHEL9
Product: Red Hat Enterprise Linux 9 Reporter: Han Han <hhan>
Component: qemu-kvmAssignee: Stefan Hajnoczi <stefanha>
qemu-kvm sub component: Storage QA Contact: qing.wang <qinwang>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: berrange, chayang, coli, jinzhao, jmoyer, jusual, juzhang, knoel, kwolf, lmen, makhomed, mrezanin, qinwang, qzhang, sgarzare, smitterl, stefanha, vgoyal, virt-maint, wquan, yama, zhenyzha
Version: 9.2Keywords: FutureFeature, Triaged
Target Milestone: beta   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-8.0.0-4.el9 Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: 1769597 Environment:
Last Closed: 2023-11-07 08:26:38 UTC Type: Feature Request
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1706143, 1758680, 1769597, 1862551, 2068237    
Bug Blocks: 1769598, 1946936    

Comment 1 Jeff Moyer 2021-06-02 17:54:26 UTC
FYI, we still do not plan to support io_uring in RHEL 9.0.  The goal is to enable support in an update release.

Comment 19 Han Han 2023-05-23 04:28:44 UTC
Tested on libvirt-9.3.0-2.el9.x86_64 qemu-kvm-8.0.0-4.el9.x86_64 liburing-0.7-7.el9.x86_64 kernel-5.14.0-316.el9.x86_64. However the io_uring doesn't work.
Steps:
1. Start a domain with the disk for io_uring
➜  ~ virsh dumpxml rhel-9.2 --xpath //disk
<disk type="file" device="disk">
  <driver name="qemu" type="qcow2" cache="none" io="io_uring" copy_on_read="on" ats="on" packed="on"/>
  <source file="/var/lib/libvirt/images/rhel-9.2.qcow2"/>
  <target dev="vda" bus="virtio"/>
  <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</disk>


➜  ~ virsh start rhel-9.2 --console       
error: Failed to start domain 'rhel-9.2'
error: internal error: process exited while connecting to monitor: 2023-05-23T04:24:28.614198Z qemu-kvm: -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/rhel-9.2.qcow2","aio":"io_uring","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}: Unable to use io_uring: failed to init linux io_uring ring: Function not implemented

It seems the issue is from kernel.
See also https://bugzilla.redhat.com/show_bug.cgi?id=2068237#c79

Comment 20 qing.wang 2023-05-23 06:35:44 UTC
(In reply to Han Han from comment #19)
> Tested on libvirt-9.3.0-2.el9.x86_64 qemu-kvm-8.0.0-4.el9.x86_64
> liburing-0.7-7.el9.x86_64 kernel-5.14.0-316.el9.x86_64. However the io_uring
> doesn't work.
> Steps:
> 1. Start a domain with the disk for io_uring
> ➜  ~ virsh dumpxml rhel-9.2 --xpath //disk
> <disk type="file" device="disk">
>   <driver name="qemu" type="qcow2" cache="none" io="io_uring"
> copy_on_read="on" ats="on" packed="on"/>
>   <source file="/var/lib/libvirt/images/rhel-9.2.qcow2"/>
>   <target dev="vda" bus="virtio"/>
>   <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
> </disk>
> 
> 
> ➜  ~ virsh start rhel-9.2 --console       
> error: Failed to start domain 'rhel-9.2'
> error: internal error: process exited while connecting to monitor:
> 2023-05-23T04:24:28.614198Z qemu-kvm: -blockdev
> {"driver":"file","filename":"/var/lib/libvirt/images/rhel-9.2.qcow2","aio":
> "io_uring","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":
> false},"auto-read-only":true,"discard":"unmap"}: Unable to use io_uring:
> failed to init linux io_uring ring: Function not implemented
> 
> It seems the issue is from kernel.
> See also Red Hathttps://bugzilla.redhat.com/show_bug.cgi?id=2068237#c79

This feature need kernel support ,please see 
https://bugzilla.redhat.com/show_bug.cgi?id=1947230#c16

Comment 21 Han Han 2023-05-23 07:08:14 UTC
(In reply to qing.wang from comment #20)
> (In reply to Han Han from comment #19)
> > Tested on libvirt-9.3.0-2.el9.x86_64 qemu-kvm-8.0.0-4.el9.x86_64
> > liburing-0.7-7.el9.x86_64 kernel-5.14.0-316.el9.x86_64. However the io_uring
> > doesn't work.
> > Steps:
> > 1. Start a domain with the disk for io_uring
> > ➜  ~ virsh dumpxml rhel-9.2 --xpath //disk
> > <disk type="file" device="disk">
> >   <driver name="qemu" type="qcow2" cache="none" io="io_uring"
> > copy_on_read="on" ats="on" packed="on"/>
> >   <source file="/var/lib/libvirt/images/rhel-9.2.qcow2"/>
> >   <target dev="vda" bus="virtio"/>
> >   <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
> > </disk>
> > 
> > 
> > ➜  ~ virsh start rhel-9.2 --console       
> > error: Failed to start domain 'rhel-9.2'
> > error: internal error: process exited while connecting to monitor:
> > 2023-05-23T04:24:28.614198Z qemu-kvm: -blockdev
> > {"driver":"file","filename":"/var/lib/libvirt/images/rhel-9.2.qcow2","aio":
> > "io_uring","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":
> > false},"auto-read-only":true,"discard":"unmap"}: Unable to use io_uring:
> > failed to init linux io_uring ring: Function not implemented
> > 
> > It seems the issue is from kernel.
> > See also Red Hathttps://bugzilla.redhat.com/show_bug.cgi?id=2068237#c79
> 
> This feature need kernel support ,please see 
> https://bugzilla.redhat.com/show_bug.cgi?id=1947230#c16

It should have been fixed in kernel. The kernel fixed version is kernel-5.14.0-315.el9, while my test version is kernel-5.14.0-316.el9.x86_64

Comment 22 Kevin Wolf 2023-05-23 10:08:07 UTC
In RHEL 9.3, io_uring still needs to be explicitly enabled in the kernel command line. See Stefan's comment in the merge request:

> Note that the "io_uring.enable=y" host kernel parameter is required because Linux io_uring will be disabled by default in CentOS Stream.

Did you set this kernel parameter when booting the host?

Comment 23 qing.wang 2023-05-24 06:13:38 UTC
(In reply to Han Han from comment #21)
> (In reply to qing.wang from comment #20)
> > (In reply to Han Han from comment #19)
> > > Tested on libvirt-9.3.0-2.el9.x86_64 qemu-kvm-8.0.0-4.el9.x86_64
> > > liburing-0.7-7.el9.x86_64 kernel-5.14.0-316.el9.x86_64. However the io_uring
> > > doesn't work.
> > > Steps:
> > > 1. Start a domain with the disk for io_uring
> > > ➜  ~ virsh dumpxml rhel-9.2 --xpath //disk
> > > <disk type="file" device="disk">
> > >   <driver name="qemu" type="qcow2" cache="none" io="io_uring"
> > > copy_on_read="on" ats="on" packed="on"/>
> > >   <source file="/var/lib/libvirt/images/rhel-9.2.qcow2"/>
> > >   <target dev="vda" bus="virtio"/>
> > >   <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
> > > </disk>
> > > 
> > > 
> > > ➜  ~ virsh start rhel-9.2 --console       
> > > error: Failed to start domain 'rhel-9.2'
> > > error: internal error: process exited while connecting to monitor:
> > > 2023-05-23T04:24:28.614198Z qemu-kvm: -blockdev
> > > {"driver":"file","filename":"/var/lib/libvirt/images/rhel-9.2.qcow2","aio":
> > > "io_uring","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":
> > > false},"auto-read-only":true,"discard":"unmap"}: Unable to use io_uring:
> > > failed to init linux io_uring ring: Function not implemented
> > > 
> > > It seems the issue is from kernel.
> > > See also Red Hathttps://bugzilla.redhat.com/show_bug.cgi?id=2068237#c79
> > 
> > This feature need kernel support ,please see 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1947230#c16
> 
> It should have been fixed in kernel. The kernel fixed version is
> kernel-5.14.0-315.el9, while my test version is kernel-5.14.0-316.el9.x86_64



It works after adding io_uring.enable=y in the kernel command line.

Red Hat Enterprise Linux release 9.3 Beta (Plow)
5.14.0-316.el9.x86_64
qemu-kvm-8.0.0-4.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230301gitf80f052277c8-4.el9.noarch
libvirt-9.3.0-2.el9.x86_64
virtio-win-prewhql-0.1-236.iso

Comment 24 Yanan Fu 2023-05-24 09:20:36 UTC
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Comment 28 qing.wang 2023-05-26 03:05:18 UTC
Psss regression test with

Red Hat Enterprise Linux release 9.3 Beta (Plow)
5.14.0-316.el9.x86_64
qemu-kvm-8.0.0-4.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230301gitf80f052277c8-4.el9.noarch
libvirt-9.3.0-2.el9.x86_64
virtio-win-prewhql-0.1-237.iso


python ConfigTest.py --category=virtual_block_device --iothread_scheme=roundrobin --nr_iothreads=2 --platform=x86_64 --guestname=RHEL.9.2.0 --driveformat=virtio_scsi --nicmodel=virtio_net --imageformat=qcow2 --machines=i440fx --customsparams="vm_mem_limit = 12G\nimage_aio=io_uring" --firmware=default_bios --netdst=virbr0

Comment 32 errata-xmlrpc 2023-11-07 08:26:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6368