RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1680231 - severe performance impact using luks format
Summary: severe performance impact using luks format
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: qemu-kvm
Version: 8.0
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: 8.1
Assignee: Daniel Berrangé
QA Contact: Tingting Mao
URL:
Whiteboard:
Depends On:
Blocks: 1680226
TreeView+ depends on / blocked
 
Reported: 2019-02-23 08:24 UTC by Yihuang Yu
Modified: 2020-11-14 15:33 UTC (History)
16 users (show)

Fixed In Version: qemu-kvm-2.12.0-69.module+el8.1.0+3143+457f984c
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1680226
Environment:
Last Closed: 2019-11-05 20:48:05 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
iostat log (iostat 1 -x -m -p vda) (16.95 KB, application/gzip)
2019-02-23 08:24 UTC, Yihuang Yu 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:3345 0 None None None 2019-11-05 20:48:35 UTC

Description Yihuang Yu 2019-02-23 08:24:19 UTC 
Created attachment 1537764 [details]
iostat log (iostat 1 -x -m -p vda)

Slow train qemu also has this problem.
qemu version: qemu-kvm-2.12.0-61.module+el8+2786+5afd5ae3.ppc64le

luks:
qemu-img create -f luks --object secret,id=secret0,data="redhat" -o key-secret=secret0 data.luks 100G

dd if=/dev/zero of=/dev/vda bs=1M count=20480
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 507.995 s, 42.3 MB/s

luks encrypted qcow2:
qemu-img create --object secret,id=secret0,data="redhat" -f qcow2 -o encrypt.format=luks,encrypt.key-secret=secret0 data.qcow2_luks 100G

dd if=/dev/zero of=/dev/vda bs=1M count=20480
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 506.761 s, 42.4 MB/s

More I/O information can be found in the log file I attached.

+++ This bug was initially created as a clone of Bug #1680226 +++

Description of problem:
Performance is severely degraded using luks formatted disks.

Version-Release number of selected component (if applicable):
qemu version: qemu-kvm-3.1.0-15.module+el8+2792+e33e01a0.ppc64le
host kernel version: 4.18.0-70.el8.ppc64le
guest kernel version: 4.18.0-70.el8.ppc64le

How reproducible:
100%

Steps to Reproduce:
1. Create images in different formats
# qemu-img create -f raw data.raw 100G
# qemu-img create -f qcow2 data.qcow2 100G
# qemu-img create -f luks --object secret,id=secret0,data="redhat" -o key-secret=secret0 data.luks 100G

2. Launch a guest with different images, luks as follows
/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine pseries  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190221-213831-uAIBCGI0,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190221-213831-uAIBCGI0,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial,server,nowait \
    -device spapr-vty,reg=0x30000000,chardev=serial_id_serial0 \
    -device qemu-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 \
    -blockdev driver=file,filename=rhel80-ppc64le-virtio-scsi.qcow2,cache.direct=off,cache.no-flush=on,node-name=image1 \
    -blockdev driver=qcow2,file=image1,node-name=drive_image1 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:24:25:26:27:28,id=idGN5fuf,vectors=4,netdev=idBTV5B1,bus=pci.0,addr=0x5  \
    -netdev tap,id=idBTV5B1,vhost=on \
    -object secret,id=secret0,data="redhat" \
    -blockdev driver=file,filename=data.luks,cache.direct=off,cache.no-flush=on,node-name=image2 \
    -blockdev driver=luks,file=image2,node-name=drive_image2,key-secret=secret0 \
    -device virtio-blk-pci,id=virtio_blk_pci1,drive=drive_image2,bus=pci.0,addr=0x6 \
    -m 51200  \
    -smp 16,maxcpus=16,cores=8,threads=1,sockets=2 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio

3. test disk performance via "dd" and "qemu-io", compare them.
*raw:
dd if=/dev/zero of=/dev/vda bs=1M count=20480
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 7.82131 s, 2.7 GB/s

qemu-io -c 'write 0 1G' --image-opts driver=raw,file.filename=data.raw
wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:04.92 (207.711 MiB/sec and 0.2028 ops/sec)

*qcow2:
dd if=/dev/zero of=/dev/vda bs=1M count=20480
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 7.75988 s, 2.8 GB/s

qemu-io -c 'write 0 1G' --image-opts driver=qcow2,file.filename=data.qcow2
wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:04.48 (228.175 MiB/sec and 0.2228 ops/sec)

Actual results:
*luks:
dd if=/dev/zero of=/dev/vda bs=1M count=20480
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 441.068 s, 48.7 MB/s

qemu-io  --object secret,id=secret0,data="redhat" -c 'write 0 1G' --image-opts driver=luks,file.filename=data.luks,key-secret=secret0
wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:27.54 (37.172 MiB/sec and 0.0363 ops/sec)

Expected results:
The luks format disk does not show a large performance degradation.

Additional info:
It doesn't seem to reproduce on x86, I don't see significant performance differences.

x86_64:
raw:
dd if=/dev/zero of=/dev/vdb bs=1M count=20480
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 114.943 s, 187 MB/s

qemu-io -c 'write 0 1G' --image-opts driver=raw,file.filename=data.raw
wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:09.62 (106.356 MiB/sec and 0.1039 ops/sec)

luks:
dd if=/dev/zero of=/dev/vdb bs=1M count=20480
20480+0 records in
20480+0 records out
21474836480 bytes (21 GB, 20 GiB) copied, 126.769 s, 169 MB/s

qemu-io  --object secret,id=secret0,data="redhat" -c 'write 0 1G' --image-opts driver=luks,file.filename=data.luks,key-secret=secret0
wrote 1073741824/1073741824 bytes at offset 0
1 GiB, 1 ops; 0:00:13.32 (76.833 MiB/sec and 0.0750 ops/sec)
Comment 1 Ademar Reis 2019-02-25 16:59:39 UTC 
See also bug 1666336
Comment 3 Daniel Berrangé 2019-04-18 16:27:07 UTC 
There is one set of upstream patches that significantly improve the performance when using AES in XTS mode, which is the default. These patches approximately double the performance for encryption/decryption:

  https://lists.gnu.org/archive/html/qemu-devel/2018-10/msg05389.html

These are quite straightforward to backport to QEMU in RHEL-7.

There will still be a delta vs the in-kernel performance, even with these patches, but it will be reduced.

Note that problem with ppc64 is different from x86_64.

On x86_64,  QEMU will use AES instructions in hardware.

AFAIK, there is no hardware accelerated impl for ppc64 so it will be significantly slower than x86_64.
Comment 13 Danilo de Paula 2019-05-03 01:48:38 UTC 
Fix included in qemu-kvm-2.12.0-69.module+el8.1.0+3143+457f984c
Comment 14 Tingting Mao 2019-05-15 07:18:23 UTC 
Hi Daniel,

According to my latest test, it seems the performance improved not that lot for luks(write:138.288 MiB/sec -> 174.160 MiB/sec, read:161.318 MiB/sec -> 198.676 MiB/sec). And it still causes almost 100% CPU usage on their hypervisor when writing data. So could you please help to check where I am wrong? Or the fix needs to update. Thanks in advance.



Tested with:
# qemu-img create -f luks --object secret,id=secret0,data="redhat" -o key-secret=secret0 data.luks 10G
Formatting 'data.luks', fmt=luks size=10737418240 key-secret=secret0

# df -Th data.luks 
Filesystem                               Type  Size  Used Avail Use% Mounted on
/dev/mapper/rhel_dell--per740xd--01-home xfs   290G  100G  190G  35% /home

# lscpu 
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              24
On-line CPU(s) list: 0-23
Thread(s) per core:  2
Core(s) per socket:  6
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Gold 6128 CPU @ 3.40GHz
Stepping:            4
CPU MHz:             1200.826
BogoMIPS:            6800.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            19712K
NUMA node0 CPU(s):   0,2,4,6,8,10,12,14,16,18,20,22
NUMA node1 CPU(s):   1,3,5,7,9,11,13,15,17,19,21,23
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke flush_l1d

# free -h
              total        used        free      shared  buff/cache   available
Mem:           62Gi       704Mi        51Gi        18Mi        10Gi        61Gi
Swap:          31Gi          0B        31Gi



Steps:

In 'qemu-kvm-2.12.0-68.module+el8.1.0+3138+c9bec3af':

Write:
qemu-io  --object secret,id=secret0,data="redhat" -c 'write 0 1.9G' --image-opts driver=luks,file.filename=data.luks,key-secret=secret0
wrote 2040109465/2040109465 bytes at offset 0
1.900 GiB, 1 ops; 0:00:14.06 (138.288 MiB/sec and 0.0711 ops/sec)

Read:
qemu-io  --object secret,id=secret0,data="redhat" -c 'read 0 1.9G' --image-opts driver=luks,file.filename=data.luks,key-secret=secret0
read 2040109465/2040109465 bytes at offset 0
1.900 GiB, 1 ops; 0:00:12.06 (161.318 MiB/sec and 0.0829 ops/sec)



In 'qemu-kvm-2.12.0-71.module+el8.1.0+3170+c76f9235':

Write:
qemu-io  --object secret,id=secret0,data="redhat" -c 'write 0 1.9G' --image-opts driver=luks,file.filename=data.luks,key-secret=secret0
wrote 2040109465/2040109465 bytes at offset 0
1.900 GiB, 1 ops; 0:00:11.17 (174.160 MiB/sec and 0.0895 ops/sec)

Read:
qemu-io  --object secret,id=secret0,data="redhat" -c 'read 0 1.9G' --image-opts driver=luks,file.filename=data.luks,key-secret=secret0
read 2040109465/2040109465 bytes at offset 0
1.900 GiB, 1 ops; 0:00:09.79 (198.676 MiB/sec and 0.1021 ops/sec)
Comment 15 Daniel Berrangé 2019-05-15 08:30:29 UTC 
Note that we're comparing different things. When I wrote

  "On my dev machine this increases perf from 327.41 MB/s to 635.68 MB/s with  AES-128 XTS"

this is referring to a microbenchmark of the crypto code run by tests/benchmark-crypto-cipher. ie this is the maximum possible AES-128 + XTS performance in a tight loop with nothing else running.

The numbers you're reporting 174.160 MiB/sec / 198.676 MiB/sec  from qemu-io are doing the combination of encryption and disk I/O, so will always be lower than what I reported, as some portion of CPU time is spent on the I/O itself.

You still saw a performance improvement, which is good as it shows the patch is working. Obviously we still need further work in future to improve perf more.
Comment 16 Tingting Mao 2019-05-15 10:08:40 UTC 
(In reply to Daniel Berrange from comment #15)
> Note that we're comparing different things. When I wrote
> 
>   "On my dev machine this increases perf from 327.41 MB/s to 635.68 MB/s
> with  AES-128 XTS"
> 
> this is referring to a microbenchmark of the crypto code run by
> tests/benchmark-crypto-cipher. ie this is the maximum possible AES-128 + XTS
> performance in a tight loop with nothing else running.
> 
> The numbers you're reporting 174.160 MiB/sec / 198.676 MiB/sec  from qemu-io
> are doing the combination of encryption and disk I/O, so will always be
> lower than what I reported, as some portion of CPU time is spent on the I/O
> itself.
> 
> You still saw a performance improvement, which is good as it shows the patch
> is working. Obviously we still need further work in future to improve perf
> more.

Thanks for you info first. And I would like to confirm the next status of this bug. Does we set it as verified then, and report new bug to trace future improvement. Or re-assign it, and trace the future improvement in this bug?
Comment 17 Daniel Berrangé 2019-05-21 10:27:24 UTC 
You've demonstrated that the patches applied for this bug result in an improvement, so this is success from QE pov IMHO.

Future performance improvements are tracked via this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1701948
Comment 19 Tingting Mao 2019-05-22 02:35:23 UTC 
Based on comment 14 and comment 17. Set this bug as verified. And thanks Daniel.
Comment 21 errata-xmlrpc 2019-11-05 20:48:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3345
Note You need to log in before you can comment on or make changes to this bug.