RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1654997 - Bad performance of umount with -o discard after removing file
Summary: Bad performance of umount with -o discard after removing file
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: kmod-kvdo
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: sclafani
QA Contact: Jakub Krysl
URL:
Whiteboard:
Depends On: 1657340
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-30 10:04 UTC by Jakub Krysl
Modified: 2021-09-06 15:22 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-20 01:44:21 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Jakub Krysl 2018-11-30 10:04:09 UTC 
Description of problem:
I have a regression test that runs iterations of mounting, copying file, umounting, mounting again, removing the file and umounting again. There seems to be significant performance drop compared to RHEL-7.6.
I used the same server for this testing and the same local disk. The disk is 10TB in size, size of the file is 1.8GB. Filesystem is XFS. Listing the last of 20 iterations and performance summary for runs of RHEL-7.6 and RHEL-8.0-20181120.0:

RHEL-7.6:
############# Running iteration 20 #############

INFO: [2018-11-29 10:21:36] Copying file /tmp/vdo_random_data.file to /mnt/vdo_test/vdo_test.file with params src = /tmp/vdo_random_data.file, dst = /mnt/vdo_test/vdo_test.file

INFO: [2018-11-29 10:21:38] Umounting filesystem with params cmd = umount /mnt/vdo_test
INFO: [2018-11-29 11:21:38] Running: 'umount /mnt/vdo_test'...


INFO: [2018-11-29 10:22:52] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test
INFO: [2018-11-29 11:22:52] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'...


INFO: [2018-11-29 10:22:53] Removing file /mnt/vdo_test/vdo_test.file with params cmd = rm -f /mnt/vdo_test/vdo_test.file
INFO: [2018-11-29 11:22:53] Running: 'rm -f /mnt/vdo_test/vdo_test.file'...


INFO: [2018-11-29 10:22:53] Umounting filesystem with params cmd = umount /mnt/vdo_test
INFO: [2018-11-29 11:22:53] Running: 'umount /mnt/vdo_test'...


INFO: [2018-11-29 10:23:19] Checking stats of vdo vdo_test
INFO: [2018-11-29 11:23:19] Running: 'vdostats'...
Device               1K-blocks      Used Available Use% Space saving%
/dev/mapper/vdo_test 9766436864  10792428 9755644436   0%           99%

INFO: [2018-11-29 10:23:20] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test
INFO: [2018-11-29 11:23:20] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'...

######### This iteration took 01m44s  #########

####### SUMMARY of 20 iterations ########
######### Average duration: 01m46s ##########
######### Minimum duration: 01m38s ##########
######### Maximum duration: 02m01s ##########


RHEL-8.0-20181120.0:
############# Running iteration 20 #############

INFO: [2018-11-29 15:43:52] Copying file /tmp/vdo_random_data.file to /mnt/vdo_test/vdo_test.file with params src = /tmp/vdo_random_data.file, dst = /mnt/vdo_test/vdo_test.file

INFO: [2018-11-29 15:43:54] Umounting filesystem with params cmd = umount /mnt/vdo_test
INFO: [2018-11-29 16:43:54] Running: 'umount /mnt/vdo_test'...


INFO: [2018-11-29 15:45:12] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test
INFO: [2018-11-29 16:45:12] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'...


INFO: [2018-11-29 15:45:12] Removing file /mnt/vdo_test/vdo_test.file with params cmd = rm -f /mnt/vdo_test/vdo_test.file
INFO: [2018-11-29 16:45:12] Running: 'rm -f /mnt/vdo_test/vdo_test.file'...


INFO: [2018-11-29 15:45:12] Umounting filesystem with params cmd = umount /mnt/vdo_test
INFO: [2018-11-29 16:45:12] Running: 'umount /mnt/vdo_test'...


INFO: [2018-11-29 15:49:18] Checking stats of vdo vdo_test
INFO: [2018-11-29 16:49:18] Running: 'vdostats'...
Device               1K-blocks      Used Available Use% Space saving%
/dev/mapper/vdo_test 9766436864  10792588 9755644276   0%           99%

INFO: [2018-11-29 15:49:18] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test
INFO: [2018-11-29 16:49:18] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'...

######### This iteration took 05m26s  #########

####### SUMMARY of 20 iterations ########
######### Average duration: 05m42s ##########
######### Minimum duration: 04m59s ##########
######### Maximum duration: 06m34s ##########


As you can see, performance of impact if quite huge, the latter is on average 3.1x slower. It seems the worst part is umounting after removing the file. As it is mounted with -o discard, I expect discards to be the issue. I tried increasing --maxDiscardSize to 4MB, but did not see any difference.


Version-Release number of selected component (if applicable):
kmod-kvdo-6.1.1.125-5.el7.x86_64
vdo-6.1.1.125-3.el7.x86_64
vs
vdo-6.2.0.273-9.el8.x86_64
kmod-kvdo-6.2.0.273-35.el8.x86_66

How reproducible:
100%

Steps to Reproduce:
in description

Actual results:
umounting after discard is much slower than before

Expected results:
no significant performance impact of discards compared to RHEL-7.6

Additional info:
Comment 1 Jakub Krysl 2018-11-30 10:09:36 UTC 
Tested also mounting without -o discard and calling fstrim, see BZ 1655000 for result.
Comment 2 Andy Walsh 2018-12-15 15:53:59 UTC 
kernel-4.18.0-53.el8 should have provided the fix for this.  Jakub, can you verify that?
Comment 3 Jakub Krysl 2018-12-17 11:15:48 UTC 
(In reply to Andy Walsh from comment #2)
> kernel-4.18.0-53.el8 should have provided the fix for this.  Jakub, can you
> verify that?

I ran on the same compose with new kernel, just had to upgrade kmod-kvdo to newer patch because of the new kernel.
# rpm -qa *vdo*
vdo-6.2.0.273-9.el8.x86_64
kmod-kvdo-6.2.0.273-36.el8.x86_64
# uname -r
4.18.0-53.el8.x86_64


############# Running iteration 20 #############

INFO: [2018-12-17 10:48:29] Copying file /tmp/vdo_random_data.file to /mnt/vdo_test/vdo_test.file with params src = /tmp/vdo_random_data.file, dst = /mnt/vdo_test/vdo_test.file

INFO: [2018-12-17 10:48:30] Umounting filesystem with params cmd = umount /mnt/vdo_test
INFO: [2018-12-17 11:48:30] Running: 'umount /mnt/vdo_test'...


INFO: [2018-12-17 10:50:06] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test
INFO: [2018-12-17 11:50:06] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'...


INFO: [2018-12-17 10:50:07] Removing file /mnt/vdo_test/vdo_test.file with params cmd = rm -f /mnt/vdo_test/vdo_test.file
INFO: [2018-12-17 11:50:07] Running: 'rm -f /mnt/vdo_test/vdo_test.file'...


INFO: [2018-12-17 10:50:07] Umounting filesystem with params cmd = umount /mnt/vdo_test
INFO: [2018-12-17 11:50:07] Running: 'umount /mnt/vdo_test'...


INFO: [2018-12-17 10:50:35] Checking stats of vdo vdo_test
INFO: [2018-12-17 11:50:35] Running: 'vdostats'...
Device               1K-blocks      Used Available Use% Space saving%
/dev/mapper/vdo_test 9766436864  10792548 9755644316   0%           99%

INFO: [2018-12-17 10:50:36] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test
INFO: [2018-12-17 11:50:36] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'...

######### This iteration took 02m07s  #########

####### SUMMARY of 20 iterations ########
######### Average duration: 02m24s ##########
######### Minimum duration: 02m07s ##########
######### Maximum duration: 03m25s ##########

It is a little bit slower than on 7.6, but much faster than before. So I'd say the fix provided in the kernel fixes this.
Comment 5 Andy Walsh 2019-02-20 01:44:21 UTC 
This was caused by a kernel bug that was resolved in 4.18.0-53.el8.x86_64.
Comment 6 Andy Walsh 2019-02-20 01:46:47 UTC 
Oops.  All supported architectures, not just x86_64.
Note You need to log in before you can comment on or make changes to this bug.