Bug 1348601
| Summary: | GFS2: fsck.gfs2 performance regression on large empty file systems | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Nate Straz <nstraz> |
| Component: | gfs2-utils | Assignee: | Robert Peterson <rpeterso> |
| Status: | CLOSED ERRATA | QA Contact: | cluster-qe <cluster-qe> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 7.3 | CC: | cluster-maint, gfs2-maint, rpeterso |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | gfs2-utils-3.1.10-1.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-08-01 21:57:28 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1413684 | ||
| Bug Blocks: | 1385242 | ||
I have a set of patches that improve fsck.gfs2 performance since the regression. I'm not sure I'll be able to make it as fast as the original, but I can certainly make it a lot better. Reassigning to myself. It's too late to get this into RHEL7.3. Punting this out to RHEL7.4. I pushed the two patches to the upstream master branch. They were tested on system gfs-i24c-01 using a fairly large file system (294TB) that was partly populated (several million files, generated by mockup). Before the patches: Level 1 resource group check: Checking if all rgrp and rindex values are good. (level 1 passed) Starting pass1 Reconciling bitmaps. reconcile_bitmaps completed in 16m46.493s pass1 completed in 38m21.495s Starting pass1b pass1b completed in 0.000s Starting pass2 pass2 completed in 7h12m50.482s Starting pass3 pass3 completed in 21.032s Starting pass4 pass4 completed in 4m8.415s Starting check_statfs check_statfs completed in 0.167s gfs2_fsck complete With the two pushed patches applied: [root@gfs-i24c-01 /home/bob/gfs2-utils]# cat /tmp/fsck.twoperfpatches.out Initializing fsck Validating resource group index. Level 1 resource group check: Checking if all rgrp and rindex values are good. (level 1 passed) Starting pass1 Reconciling bitmaps. reconcile_bitmaps completed in 16m10.194s pass1 completed in 38m29.875s Starting pass1b pass1b completed in 0.000s Starting pass2 pass2 completed in 23m35.125s Starting pass3 pass3 completed in 19.352s Starting pass4 pass4 completed in 4m14.756s Starting check_statfs check_statfs completed in 0.204s gfs2_fsck complete Meager/no savings to pass1, but pass2 went from 7h12m to 23m35s. Changing status to POST. Pass 2 times for gfs2-utils-3.1.0-3.el7 are much better than gfs2-utils-3.1.9-3.el7.
pass2 time for fsck0
FS Size 3.1.9-3.el7/fsc 3.1.10-3.el7/fs
16G 0 0
32G 0 0
64G 0 0
128G 0 0
256G 1 0
512G 3 0
1T 7 0
2T 17 0
4T 47 0
8T 92 0
16T 271 0
32T 700 0
64T 1746 0
128T 3891
256T 8357
pass2 time for fsck80
FS Size 3.1.9-3.el7/fsc 3.1.10-3.el7/fs
16G 0 0
32G 0 0
64G 0 0
128G 1 0
256G 2 0
512G 6 1
1T 18 2
2T 37 4
4T 103 22
8T 184 39
16T 353 37
32T 925 56
64T 2232
128T 4942
256T 9904
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2226 |
Description of problem: After running performance tests on gfs2-utils-3.1.9-1.el7, I found that pass2 has gotten a lot slower compared to gfs2-utils-3.1.8-6.el7 in rhel7.2. Time for fsck.gfs2 @ empty FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0.25 0.25 0.32 32G 0.32 0.31 0.59 64G 0.40 0.42 0.93 128G 0.61 0.77 1.55 256G 2.59 1.00 2.74 512G 2.16 1.88 5.16 1T 3.82 4.98 10.77 2T 7.24 19.24 22.70 4T 14.61 43.74 54.25 8T 30.88 1:16.33 2:14.32 16T 1:11.88 1:58.48 5:48.88 32T 2:15.28 64T 4:36.60 Memory usage for fsck.gfs2 @ empty FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 1.35MB 1.35MB 1.34MB 32G 2.12MB 2.12MB 2.11MB 64G 4.42MB 4.41MB 4.41MB 128G 9.01MB 9.01MB 9.00MB 256G 18.18MB 18.18MB 18.17MB 512G 36.55MB 36.55MB 36.53MB 1T 73.27MB 73.27MB 73.23MB 2T 146.71MB 142.96MB 146.64MB 4T 293.59MB 239.57MB 293.45MB 8T 587.34MB 417.05MB 587.08MB 16T 1174.85MB 836.52MB 1174.34MB 32T 2349.86MB 64T 4699.90MB Time for fsck.gfs2 @ 80% full FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 41.52 4.41 20.51 32G 1:22.53 20.57 30.54 64G 3:25.28 54.69 5:31.78 128G 6:09.89 1:58.49 14:27.92 256G 11:35.94 3:30.89 16:30.81 512G 25:53.00 7:57.41 17:41.22 1T 2:17:17.00 18:53.12 2:18:03.00 2T 6:34:12.00 38:10.10 4:46:34.00 4T 16:23:25.00 1:15:07.00 8:45:39.00 8T 36:47:47.00 2:33:12.00 23:47:59.00 16T 66:29:01.00 5:25:51.00 32T 135:20:07.00 64T Memory usage for fsck.gfs2 @ 80% full FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 4.95MB 3.86MB 1.94MB 32G 9.16MB 7.13MB 3.30MB 64G 17.91MB 13.79MB 6.64MB 128G 35.24MB 27.05MB 12.67MB 256G 69.88MB 53.79MB 23.79MB 512G 139.40MB 106.58MB 46.09MB 1T 276.50MB 209.96MB 91.11MB 2T 553.46MB 416.84MB 180.88MB 4T 1100.69MB 833.06MB 354.77MB 8T 2186.08MB 1662.25MB 684.78MB 16T 4362.93MB 3319.53MB 32T 8714.22MB 64T pass1 time for fsck0 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0 0 0 32G 0 0 0 64G 0 0 0 128G 0 0 0 256G 0 0 0 512G 0 0 0 1T 0 0 1 2T 0 0 3 4T 0 0 5 8T 0 0 12 16T 0 20 23 32T 0 64T 1 pass1b time for fsck0 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0 0 0 32G 0 0 0 64G 0 0 0 128G 0 0 0 256G 0 0 0 512G 0 0 0 1T 0 0 0 2T 0 0 0 4T 0 0 0 8T 0 0 0 16T 0 0 0 32T 0 64T 0 pass1c time for fsck0 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0 0 32G 0 0 64G 0 0 128G 0 0 256G 0 0 512G 0 0 1T 0 0 2T 0 0 4T 0 0 8T 0 0 16T 0 0 32T 0 64T 0 pass2 time for fsck0 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0 0 0 32G 0 0 0 64G 0 0 0 128G 0 0 0 256G 0 0 1 512G 0 0 3 1T 1 1 7 2T 2 2 17 4T 5 5 43 8T 10 10 100 16T 22 20 269 32T 40 64T 80 pass3 time for fsck0 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0 0 0 32G 0 0 0 64G 0 0 0 128G 0 0 0 256G 0 0 0 512G 0 0 0 1T 0 0 0 2T 0 0 0 4T 0 0 0 8T 0 0 0 16T 0 0 0 32T 0 64T 0 pass4 time for fsck0 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0 0 0 32G 0 0 0 64G 0 0 0 128G 0 0 0 256G 0 0 0 512G 0 0 0 1T 0 0 0 2T 0 0 1 4T 0 0 2 8T 0 0 4 16T 0 0 9 32T 0 64T 0 pass5 time for fsck0 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0 0 32G 0 0 64G 0 0 128G 0 0 256G 0 0 512G 1 0 1T 1 1 2T 3 2 4T 5 5 8T 11 11 16T 23 23 32T 46 64T 92 pass1 time for fsck80 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 40 3 19 32G 81 19 29 64G 203 53 330 128G 366 116 865 256G 688 207 986 512G 1539 471 1053 1T 3113 1121 8267 2T 6162 2264 17160 4T 13123 4454 31436 8T 31541 8762 85194 16T 53887 18254 32T 112337 64T pass1b time for fsck80 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0 0 0 32G 0 0 0 64G 0 0 0 128G 0 0 0 256G 0 0 0 512G 0 0 0 1T 0 0 0 2T 0 0 0 4T 0 0 0 8T 0 0 0 16T 0 0 32T 0 64T pass1c time for fsck80 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0 0 32G 0 0 64G 0 0 128G 1 0 256G 2 0 512G 5 0 1T 5101 0 2T 17405 0 4T 45652 0 8T 100397 0 16T 184230 0 32T 372173 64T pass2 time for fsck80 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0 0 0 32G 0 0 0 64G 0 0 0 128G 0 0 1 256G 1 1 2 512G 2 2 5 1T 11 4 10 2T 67 9 23 4T 201 20 83 8T 491 380 456 16T 1161 1214 32T 2572 64T pass3 time for fsck80 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0 0 0 32G 0 0 0 64G 0 0 0 128G 0 0 0 256G 0 0 0 512G 0 0 0 1T 0 0 0 2T 0 0 0 4T 0 0 0 8T 0 0 0 16T 0 0 32T 0 64T pass4 time for fsck80 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0 0 0 32G 0 0 0 64G 0 0 0 128G 0 0 0 256G 0 0 0 512G 0 0 0 1T 0 0 0 2T 0 0 1 4T 0 0 2 8T 1 1 5 16T 3 2 32T 6 64T pass5 time for fsck80 FS Size rhel7.1/fsck-pe rhel7.2/fsck-pe 3.1.9-1.el7/fsc 16G 0 0 32G 0 0 64G 0 0 128G 0 0 256G 0 0 512G 0 0 1T 1 1 2T 2 2 4T 5 5 8T 11 11 16T 24 23 32T 49 64T Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: