Bug 1654997 - Bad performance of umount with -o discard after removing file
Summary: Bad performance of umount with -o discard after removing file
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: kmod-kvdo
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: sclafani
QA Contact: Jakub Krysl
URL:
Whiteboard:
Depends On: 1657340
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-30 10:04 UTC by Jakub Krysl
Modified: 2019-05-22 05:13 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-20 01:44:21 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)

Description Jakub Krysl 2018-11-30 10:04:09 UTC
Description of problem:
I have a regression test that runs iterations of mounting, copying file, umounting, mounting again, removing the file and umounting again. There seems to be significant performance drop compared to RHEL-7.6.
I used the same server for this testing and the same local disk. The disk is 10TB in size, size of the file is 1.8GB. Filesystem is XFS. Listing the last of 20 iterations and performance summary for runs of RHEL-7.6 and RHEL-8.0-20181120.0:

RHEL-7.6:
############# Running iteration 20 #############

INFO: [2018-11-29 10:21:36] Copying file /tmp/vdo_random_data.file to /mnt/vdo_test/vdo_test.file with params src = /tmp/vdo_random_data.file, dst = /mnt/vdo_test/vdo_test.file

INFO: [2018-11-29 10:21:38] Umounting filesystem with params cmd = umount /mnt/vdo_test
INFO: [2018-11-29 11:21:38] Running: 'umount /mnt/vdo_test'...


INFO: [2018-11-29 10:22:52] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test
INFO: [2018-11-29 11:22:52] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'...


INFO: [2018-11-29 10:22:53] Removing file /mnt/vdo_test/vdo_test.file with params cmd = rm -f /mnt/vdo_test/vdo_test.file
INFO: [2018-11-29 11:22:53] Running: 'rm -f /mnt/vdo_test/vdo_test.file'...


INFO: [2018-11-29 10:22:53] Umounting filesystem with params cmd = umount /mnt/vdo_test
INFO: [2018-11-29 11:22:53] Running: 'umount /mnt/vdo_test'...


INFO: [2018-11-29 10:23:19] Checking stats of vdo vdo_test
INFO: [2018-11-29 11:23:19] Running: 'vdostats'...
Device               1K-blocks      Used Available Use% Space saving%
/dev/mapper/vdo_test 9766436864  10792428 9755644436   0%           99%

INFO: [2018-11-29 10:23:20] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test
INFO: [2018-11-29 11:23:20] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'...

######### This iteration took 01m44s  #########

####### SUMMARY of 20 iterations ########
######### Average duration: 01m46s ##########
######### Minimum duration: 01m38s ##########
######### Maximum duration: 02m01s ##########


RHEL-8.0-20181120.0:
############# Running iteration 20 #############

INFO: [2018-11-29 15:43:52] Copying file /tmp/vdo_random_data.file to /mnt/vdo_test/vdo_test.file with params src = /tmp/vdo_random_data.file, dst = /mnt/vdo_test/vdo_test.file

INFO: [2018-11-29 15:43:54] Umounting filesystem with params cmd = umount /mnt/vdo_test
INFO: [2018-11-29 16:43:54] Running: 'umount /mnt/vdo_test'...


INFO: [2018-11-29 15:45:12] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test
INFO: [2018-11-29 16:45:12] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'...


INFO: [2018-11-29 15:45:12] Removing file /mnt/vdo_test/vdo_test.file with params cmd = rm -f /mnt/vdo_test/vdo_test.file
INFO: [2018-11-29 16:45:12] Running: 'rm -f /mnt/vdo_test/vdo_test.file'...


INFO: [2018-11-29 15:45:12] Umounting filesystem with params cmd = umount /mnt/vdo_test
INFO: [2018-11-29 16:45:12] Running: 'umount /mnt/vdo_test'...


INFO: [2018-11-29 15:49:18] Checking stats of vdo vdo_test
INFO: [2018-11-29 16:49:18] Running: 'vdostats'...
Device               1K-blocks      Used Available Use% Space saving%
/dev/mapper/vdo_test 9766436864  10792588 9755644276   0%           99%

INFO: [2018-11-29 15:49:18] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test
INFO: [2018-11-29 16:49:18] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'...

######### This iteration took 05m26s  #########

####### SUMMARY of 20 iterations ########
######### Average duration: 05m42s ##########
######### Minimum duration: 04m59s ##########
######### Maximum duration: 06m34s ##########


As you can see, performance of impact if quite huge, the latter is on average 3.1x slower. It seems the worst part is umounting after removing the file. As it is mounted with -o discard, I expect discards to be the issue. I tried increasing --maxDiscardSize to 4MB, but did not see any difference.


Version-Release number of selected component (if applicable):
kmod-kvdo-6.1.1.125-5.el7.x86_64
vdo-6.1.1.125-3.el7.x86_64
vs
vdo-6.2.0.273-9.el8.x86_64
kmod-kvdo-6.2.0.273-35.el8.x86_66

How reproducible:
100%

Steps to Reproduce:
in description

Actual results:
umounting after discard is much slower than before

Expected results:
no significant performance impact of discards compared to RHEL-7.6

Additional info:

Comment 1 Jakub Krysl 2018-11-30 10:09:36 UTC
Tested also mounting without -o discard and calling fstrim, see BZ 1655000 for result.

Comment 2 Andy Walsh 2018-12-15 15:53:59 UTC
kernel-4.18.0-53.el8 should have provided the fix for this.  Jakub, can you verify that?

Comment 3 Jakub Krysl 2018-12-17 11:15:48 UTC
(In reply to Andy Walsh from comment #2)
> kernel-4.18.0-53.el8 should have provided the fix for this.  Jakub, can you
> verify that?

I ran on the same compose with new kernel, just had to upgrade kmod-kvdo to newer patch because of the new kernel.
# rpm -qa *vdo*
vdo-6.2.0.273-9.el8.x86_64
kmod-kvdo-6.2.0.273-36.el8.x86_64
# uname -r
4.18.0-53.el8.x86_64


############# Running iteration 20 #############

INFO: [2018-12-17 10:48:29] Copying file /tmp/vdo_random_data.file to /mnt/vdo_test/vdo_test.file with params src = /tmp/vdo_random_data.file, dst = /mnt/vdo_test/vdo_test.file

INFO: [2018-12-17 10:48:30] Umounting filesystem with params cmd = umount /mnt/vdo_test
INFO: [2018-12-17 11:48:30] Running: 'umount /mnt/vdo_test'...


INFO: [2018-12-17 10:50:06] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test
INFO: [2018-12-17 11:50:06] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'...


INFO: [2018-12-17 10:50:07] Removing file /mnt/vdo_test/vdo_test.file with params cmd = rm -f /mnt/vdo_test/vdo_test.file
INFO: [2018-12-17 11:50:07] Running: 'rm -f /mnt/vdo_test/vdo_test.file'...


INFO: [2018-12-17 10:50:07] Umounting filesystem with params cmd = umount /mnt/vdo_test
INFO: [2018-12-17 11:50:07] Running: 'umount /mnt/vdo_test'...


INFO: [2018-12-17 10:50:35] Checking stats of vdo vdo_test
INFO: [2018-12-17 11:50:35] Running: 'vdostats'...
Device               1K-blocks      Used Available Use% Space saving%
/dev/mapper/vdo_test 9766436864  10792548 9755644316   0%           99%

INFO: [2018-12-17 10:50:36] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test
INFO: [2018-12-17 11:50:36] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'...

######### This iteration took 02m07s  #########

####### SUMMARY of 20 iterations ########
######### Average duration: 02m24s ##########
######### Minimum duration: 02m07s ##########
######### Maximum duration: 03m25s ##########

It is a little bit slower than on 7.6, but much faster than before. So I'd say the fix provided in the kernel fixes this.

Comment 5 Andy Walsh 2019-02-20 01:44:21 UTC
This was caused by a kernel bug that was resolved in 4.18.0-53.el8.x86_64.

Comment 6 Andy Walsh 2019-02-20 01:46:47 UTC
Oops.  All supported architectures, not just x86_64.


Note You need to log in before you can comment on or make changes to this bug.