Hide Forgot
Description of problem: I have a regression test that runs iterations of mounting, copying file, umounting, mounting again, removing the file and umounting again. There seems to be significant performance drop compared to RHEL-7.6. I used the same server for this testing and the same local disk. The disk is 10TB in size, size of the file is 1.8GB. Filesystem is XFS. Listing the last of 20 iterations and performance summary for runs of RHEL-7.6 and RHEL-8.0-20181120.0: RHEL-7.6: ############# Running iteration 20 ############# INFO: [2018-11-29 10:21:36] Copying file /tmp/vdo_random_data.file to /mnt/vdo_test/vdo_test.file with params src = /tmp/vdo_random_data.file, dst = /mnt/vdo_test/vdo_test.file INFO: [2018-11-29 10:21:38] Umounting filesystem with params cmd = umount /mnt/vdo_test INFO: [2018-11-29 11:21:38] Running: 'umount /mnt/vdo_test'... INFO: [2018-11-29 10:22:52] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test INFO: [2018-11-29 11:22:52] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'... INFO: [2018-11-29 10:22:53] Removing file /mnt/vdo_test/vdo_test.file with params cmd = rm -f /mnt/vdo_test/vdo_test.file INFO: [2018-11-29 11:22:53] Running: 'rm -f /mnt/vdo_test/vdo_test.file'... INFO: [2018-11-29 10:22:53] Umounting filesystem with params cmd = umount /mnt/vdo_test INFO: [2018-11-29 11:22:53] Running: 'umount /mnt/vdo_test'... INFO: [2018-11-29 10:23:19] Checking stats of vdo vdo_test INFO: [2018-11-29 11:23:19] Running: 'vdostats'... Device 1K-blocks Used Available Use% Space saving% /dev/mapper/vdo_test 9766436864 10792428 9755644436 0% 99% INFO: [2018-11-29 10:23:20] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test INFO: [2018-11-29 11:23:20] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'... ######### This iteration took 01m44s ######### ####### SUMMARY of 20 iterations ######## ######### Average duration: 01m46s ########## ######### Minimum duration: 01m38s ########## ######### Maximum duration: 02m01s ########## RHEL-8.0-20181120.0: ############# Running iteration 20 ############# INFO: [2018-11-29 15:43:52] Copying file /tmp/vdo_random_data.file to /mnt/vdo_test/vdo_test.file with params src = /tmp/vdo_random_data.file, dst = /mnt/vdo_test/vdo_test.file INFO: [2018-11-29 15:43:54] Umounting filesystem with params cmd = umount /mnt/vdo_test INFO: [2018-11-29 16:43:54] Running: 'umount /mnt/vdo_test'... INFO: [2018-11-29 15:45:12] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test INFO: [2018-11-29 16:45:12] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'... INFO: [2018-11-29 15:45:12] Removing file /mnt/vdo_test/vdo_test.file with params cmd = rm -f /mnt/vdo_test/vdo_test.file INFO: [2018-11-29 16:45:12] Running: 'rm -f /mnt/vdo_test/vdo_test.file'... INFO: [2018-11-29 15:45:12] Umounting filesystem with params cmd = umount /mnt/vdo_test INFO: [2018-11-29 16:45:12] Running: 'umount /mnt/vdo_test'... INFO: [2018-11-29 15:49:18] Checking stats of vdo vdo_test INFO: [2018-11-29 16:49:18] Running: 'vdostats'... Device 1K-blocks Used Available Use% Space saving% /dev/mapper/vdo_test 9766436864 10792588 9755644276 0% 99% INFO: [2018-11-29 15:49:18] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test INFO: [2018-11-29 16:49:18] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'... ######### This iteration took 05m26s ######### ####### SUMMARY of 20 iterations ######## ######### Average duration: 05m42s ########## ######### Minimum duration: 04m59s ########## ######### Maximum duration: 06m34s ########## As you can see, performance of impact if quite huge, the latter is on average 3.1x slower. It seems the worst part is umounting after removing the file. As it is mounted with -o discard, I expect discards to be the issue. I tried increasing --maxDiscardSize to 4MB, but did not see any difference. Version-Release number of selected component (if applicable): kmod-kvdo-6.1.1.125-5.el7.x86_64 vdo-6.1.1.125-3.el7.x86_64 vs vdo-6.2.0.273-9.el8.x86_64 kmod-kvdo-6.2.0.273-35.el8.x86_66 How reproducible: 100% Steps to Reproduce: in description Actual results: umounting after discard is much slower than before Expected results: no significant performance impact of discards compared to RHEL-7.6 Additional info:
Tested also mounting without -o discard and calling fstrim, see BZ 1655000 for result.
kernel-4.18.0-53.el8 should have provided the fix for this. Jakub, can you verify that?
(In reply to Andy Walsh from comment #2) > kernel-4.18.0-53.el8 should have provided the fix for this. Jakub, can you > verify that? I ran on the same compose with new kernel, just had to upgrade kmod-kvdo to newer patch because of the new kernel. # rpm -qa *vdo* vdo-6.2.0.273-9.el8.x86_64 kmod-kvdo-6.2.0.273-36.el8.x86_64 # uname -r 4.18.0-53.el8.x86_64 ############# Running iteration 20 ############# INFO: [2018-12-17 10:48:29] Copying file /tmp/vdo_random_data.file to /mnt/vdo_test/vdo_test.file with params src = /tmp/vdo_random_data.file, dst = /mnt/vdo_test/vdo_test.file INFO: [2018-12-17 10:48:30] Umounting filesystem with params cmd = umount /mnt/vdo_test INFO: [2018-12-17 11:48:30] Running: 'umount /mnt/vdo_test'... INFO: [2018-12-17 10:50:06] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test INFO: [2018-12-17 11:50:06] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'... INFO: [2018-12-17 10:50:07] Removing file /mnt/vdo_test/vdo_test.file with params cmd = rm -f /mnt/vdo_test/vdo_test.file INFO: [2018-12-17 11:50:07] Running: 'rm -f /mnt/vdo_test/vdo_test.file'... INFO: [2018-12-17 10:50:07] Umounting filesystem with params cmd = umount /mnt/vdo_test INFO: [2018-12-17 11:50:07] Running: 'umount /mnt/vdo_test'... INFO: [2018-12-17 10:50:35] Checking stats of vdo vdo_test INFO: [2018-12-17 11:50:35] Running: 'vdostats'... Device 1K-blocks Used Available Use% Space saving% /dev/mapper/vdo_test 9766436864 10792548 9755644316 0% 99% INFO: [2018-12-17 10:50:36] Mounting XFS filesystem with discard enabled with params cmd = mount -o discard /dev/mapper/vdo_test /mnt/vdo_test INFO: [2018-12-17 11:50:36] Running: 'mount -o discard /dev/mapper/vdo_test /mnt/vdo_test'... ######### This iteration took 02m07s ######### ####### SUMMARY of 20 iterations ######## ######### Average duration: 02m24s ########## ######### Minimum duration: 02m07s ########## ######### Maximum duration: 03m25s ########## It is a little bit slower than on 7.6, but much faster than before. So I'd say the fix provided in the kernel fixes this.
This was caused by a kernel bug that was resolved in 4.18.0-53.el8.x86_64.
Oops. All supported architectures, not just x86_64.