Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
Use below fio command to test disk reading performance on Azure Confidential VM. And the result shows IOPS is dramatically lower than that of ordinary VM.
# fio --filename=/dev/sdb --direct=1 --rw=read --sync=0 --time_based --bs=4k --size=4096M --numjobs=1 --iodepth=1 --runtime=30 --group_reporting --name=test --thread --ioengine=libaio
...
bw ( KiB/s): min= 3296, max= 3544, per=100.00%, avg=3444.20, stdev=49.35, samples=59
iops : min= 824, max= 886, avg=861.05, stdev=12.34, samples=59
lat (usec) : 750=86.91%, 1000=11.57%
lat (msec) : 2=1.46%, 4=0.05%, 10=0.01%
...
Version-Release number of selected component (if applicable):
5.14.0-340.el9.x86_64
How reproducible:
100%
Steps to Reproduce:
The VM sizes I used are Standard_DC16ads_v5 and Standard_DC96ads_v5. Tested on the temp storage.
1. # fio --filename=/dev/sdb --direct=1 --rw=read --sync=0 --time_based --bs=4k --size=4096M --numjobs=1 --iodepth=1 --runtime=30 --group_reporting --name=test --thread --ioengine=libaio
Actual results:
iops : min= 824, max= 886, avg=861.05, stdev=12.34, samples=59
Expected results:
Tested on Standard_D4ds_v5:
iops : min=10734, max=18974, avg=13713.49, stdev=1538.76, samples=59
Additional info:
1. Bisected issue to minor version - 195.el9:
5.14.0-194.el9.x86_64
bw ( KiB/s): min=25256, max=56160, per=99.98%, avg=50932.88, stdev=6357.32, samples=59
iops : min= 6314, max=14040, avg=12733.22, stdev=1589.33, samples=59
5.14.0-195.el9.x86_64
bw ( KiB/s): min= 1072, max= 3584, per=100.00%, avg=3452.24, stdev=320.44, samples=59
iops : min= 268, max= 896, avg=863.05, stdev=80.11, samples=59
2. Latest RHEL 8.9 also has this issue but not as bad:
bw ( KiB/s): min=14816, max=36472, per=100.00%, avg=25072.03, stdev=4593.09, samples=59
iops : min= 3704, max= 9118, avg=6268.00, stdev=1148.27, samples=59
3. Pbench test suite with kernel-5.14.0-340.el9.x86_64:
http://pbench.perf.lab.eng.bos.redhat.com/users/virt-perftest-test/EC2::Standard-DC96ads-v5/fio_Azure_9.3.202307190_x86_64_gen2_localssd_quick_D230723T234533/fio_bs_4_1024_iod_1_njobs_1_read_write_rw_2023.07.24T03.45.33/result.html
4. Not sure if this BZ is related but introduced in the same version:
https://bugzilla.redhat.com/show_bug.cgi?id=2215362
This regression bisects to
commit 1d45e77775b808dcd7716f6b8219ee12f73d1855
Author: Nico Pache <npache>
Date: Wed Nov 2 08:54:46 2022 -0600
x86/PAT: Have pat_enabled() properly reflect state when running on Xen
commit 72cbc8f04fe2fa93443c0fcccb7ad91dfea3d9ce
Author: Jan Beulich <jbeulich>
Date: Thu Apr 28 16:50:29 2022 +0200
x86/PAT: Have pat_enabled() properly reflect state when running on Xen
So I think we need
commit 90b926e68f500844dff16b5bcea178dc55cf580a
Author: Juergen Gross <jgross>
Date: Tue Jan 10 07:54:27 2023 +0100
x86/pat: Fix pat_x_mtrr_type() for MTRR disabled case
to fix the problem. Let me run some tests.