RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1348601 - GFS2: fsck.gfs2 performance regression on large empty file systems
Summary: GFS2: fsck.gfs2 performance regression on large empty file systems
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: gfs2-utils
Version: 7.3
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Robert Peterson
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On: 1413684
Blocks: 1385242
TreeView+ depends on / blocked
 
Reported: 2016-06-21 14:33 UTC by Nate Straz
Modified: 2017-08-01 21:57 UTC (History)
3 users (show)

Fixed In Version: gfs2-utils-3.1.10-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 21:57:28 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2226 0 normal SHIPPED_LIVE gfs2-utils bug fix and enhancement update 2017-08-01 18:43:08 UTC

Description Nate Straz 2016-06-21 14:33:46 UTC 
Description of problem:

After running performance tests on gfs2-utils-3.1.9-1.el7, I found that pass2 has gotten a lot slower compared to gfs2-utils-3.1.8-6.el7 in rhel7.2.

Time for fsck.gfs2 @ empty
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G             0.25             0.25             0.32  
            32G             0.32             0.31             0.59  
            64G             0.40             0.42             0.93  
           128G             0.61             0.77             1.55  
           256G             2.59             1.00             2.74  
           512G             2.16             1.88             5.16  
             1T             3.82             4.98            10.77  
             2T             7.24            19.24            22.70  
             4T            14.61            43.74            54.25  
             8T            30.88          1:16.33          2:14.32  
            16T          1:11.88          1:58.48          5:48.88  
            32T          2:15.28                                    
            64T          4:36.60                                    
Memory usage for fsck.gfs2 @ empty
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G           1.35MB           1.35MB           1.34MB  
            32G           2.12MB           2.12MB           2.11MB  
            64G           4.42MB           4.41MB           4.41MB  
           128G           9.01MB           9.01MB           9.00MB  
           256G          18.18MB          18.18MB          18.17MB  
           512G          36.55MB          36.55MB          36.53MB  
             1T          73.27MB          73.27MB          73.23MB  
             2T         146.71MB         142.96MB         146.64MB  
             4T         293.59MB         239.57MB         293.45MB  
             8T         587.34MB         417.05MB         587.08MB  
            16T        1174.85MB         836.52MB        1174.34MB  
            32T        2349.86MB                                    
            64T        4699.90MB                                    
Time for fsck.gfs2 @ 80% full
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G            41.52             4.41            20.51  
            32G          1:22.53            20.57            30.54  
            64G          3:25.28            54.69          5:31.78  
           128G          6:09.89          1:58.49         14:27.92  
           256G         11:35.94          3:30.89         16:30.81  
           512G         25:53.00          7:57.41         17:41.22  
             1T       2:17:17.00         18:53.12       2:18:03.00  
             2T       6:34:12.00         38:10.10       4:46:34.00  
             4T      16:23:25.00       1:15:07.00       8:45:39.00  
             8T      36:47:47.00       2:33:12.00      23:47:59.00  
            16T      66:29:01.00       5:25:51.00                   
            32T     135:20:07.00                                    
            64T                                                     
Memory usage for fsck.gfs2 @ 80% full
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G           4.95MB           3.86MB           1.94MB  
            32G           9.16MB           7.13MB           3.30MB  
            64G          17.91MB          13.79MB           6.64MB  
           128G          35.24MB          27.05MB          12.67MB  
           256G          69.88MB          53.79MB          23.79MB  
           512G         139.40MB         106.58MB          46.09MB  
             1T         276.50MB         209.96MB          91.11MB  
             2T         553.46MB         416.84MB         180.88MB  
             4T        1100.69MB         833.06MB         354.77MB  
             8T        2186.08MB        1662.25MB         684.78MB  
            16T        4362.93MB        3319.53MB                   
            32T        8714.22MB                                    
            64T                                                     
pass1 time for fsck0
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G                0                0                0  
            32G                0                0                0  
            64G                0                0                0  
           128G                0                0                0  
           256G                0                0                0  
           512G                0                0                0  
             1T                0                0                1  
             2T                0                0                3  
             4T                0                0                5  
             8T                0                0               12  
            16T                0               20               23  
            32T                0                                    
            64T                1                                    
pass1b time for fsck0
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G                0                0                0  
            32G                0                0                0  
            64G                0                0                0  
           128G                0                0                0  
           256G                0                0                0  
           512G                0                0                0  
             1T                0                0                0  
             2T                0                0                0  
             4T                0                0                0  
             8T                0                0                0  
            16T                0                0                0  
            32T                0                                    
            64T                0                                    
pass1c time for fsck0
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G                0                0                   
            32G                0                0                   
            64G                0                0                   
           128G                0                0                   
           256G                0                0                   
           512G                0                0                   
             1T                0                0                   
             2T                0                0                   
             4T                0                0                   
             8T                0                0                   
            16T                0                0                   
            32T                0                                    
            64T                0                                    
pass2 time for fsck0
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G                0                0                0  
            32G                0                0                0  
            64G                0                0                0  
           128G                0                0                0  
           256G                0                0                1  
           512G                0                0                3  
             1T                1                1                7  
             2T                2                2               17  
             4T                5                5               43  
             8T               10               10              100  
            16T               22               20              269  
            32T               40                                    
            64T               80                                    
pass3 time for fsck0
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G                0                0                0  
            32G                0                0                0  
            64G                0                0                0  
           128G                0                0                0  
           256G                0                0                0  
           512G                0                0                0  
             1T                0                0                0  
             2T                0                0                0  
             4T                0                0                0  
             8T                0                0                0  
            16T                0                0                0  
            32T                0                                    
            64T                0                                    
pass4 time for fsck0
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G                0                0                0  
            32G                0                0                0  
            64G                0                0                0  
           128G                0                0                0  
           256G                0                0                0  
           512G                0                0                0  
             1T                0                0                0  
             2T                0                0                1  
             4T                0                0                2  
             8T                0                0                4  
            16T                0                0                9  
            32T                0                                    
            64T                0                                    
pass5 time for fsck0
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G                0                0                   
            32G                0                0                   
            64G                0                0                   
           128G                0                0                   
           256G                0                0                   
           512G                1                0                   
             1T                1                1                   
             2T                3                2                   
             4T                5                5                   
             8T               11               11                   
            16T               23               23                   
            32T               46                                    
            64T               92                                    
pass1 time for fsck80
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G               40                3               19  
            32G               81               19               29  
            64G              203               53              330  
           128G              366              116              865  
           256G              688              207              986  
           512G             1539              471             1053  
             1T             3113             1121             8267  
             2T             6162             2264            17160  
             4T            13123             4454            31436  
             8T            31541             8762            85194  
            16T            53887            18254                   
            32T           112337                                    
            64T                                                     
pass1b time for fsck80
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G                0                0                0  
            32G                0                0                0  
            64G                0                0                0  
           128G                0                0                0  
           256G                0                0                0  
           512G                0                0                0  
             1T                0                0                0  
             2T                0                0                0  
             4T                0                0                0  
             8T                0                0                0  
            16T                0                0                   
            32T                0                                    
            64T                                                     
pass1c time for fsck80
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G                0                0                   
            32G                0                0                   
            64G                0                0                   
           128G                1                0                   
           256G                2                0                   
           512G                5                0                   
             1T             5101                0                   
             2T            17405                0                   
             4T            45652                0                   
             8T           100397                0                   
            16T           184230                0                   
            32T           372173                                    
            64T                                                     
pass2 time for fsck80
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G                0                0                0  
            32G                0                0                0  
            64G                0                0                0  
           128G                0                0                1  
           256G                1                1                2  
           512G                2                2                5  
             1T               11                4               10  
             2T               67                9               23  
             4T              201               20               83  
             8T              491              380              456  
            16T             1161             1214                   
            32T             2572                                    
            64T                                                     
pass3 time for fsck80
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G                0                0                0  
            32G                0                0                0  
            64G                0                0                0  
           128G                0                0                0  
           256G                0                0                0  
           512G                0                0                0  
             1T                0                0                0  
             2T                0                0                0  
             4T                0                0                0  
             8T                0                0                0  
            16T                0                0                   
            32T                0                                    
            64T                                                     
pass4 time for fsck80
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G                0                0                0  
            32G                0                0                0  
            64G                0                0                0  
           128G                0                0                0  
           256G                0                0                0  
           512G                0                0                0  
             1T                0                0                0  
             2T                0                0                1  
             4T                0                0                2  
             8T                1                1                5  
            16T                3                2                   
            32T                6                                    
            64T                                                     
pass5 time for fsck80
        FS Size  rhel7.1/fsck-pe  rhel7.2/fsck-pe  3.1.9-1.el7/fsc  
            16G                0                0                   
            32G                0                0                   
            64G                0                0                   
           128G                0                0                   
           256G                0                0                   
           512G                0                0                   
             1T                1                1                   
             2T                2                2                   
             4T                5                5                   
             8T               11               11                   
            16T               24               23                   
            32T               49                                    
            64T                                                     


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Robert Peterson 2016-06-22 19:20:36 UTC 
I have a set of patches that improve fsck.gfs2 performance since
the regression. I'm not sure I'll be able to make it as fast as
the original, but I can certainly make it a lot better.
Reassigning to myself.
Comment 2 Robert Peterson 2016-10-05 14:02:23 UTC 
It's too late to get this into RHEL7.3. Punting this out to RHEL7.4.
Comment 3 Robert Peterson 2017-03-20 13:31:10 UTC 
I pushed the two patches to the upstream master branch.
They were tested on system gfs-i24c-01 using a fairly large
file system (294TB) that was partly populated (several million
files, generated by mockup). Before the patches:

Level 1 resource group check: Checking if all rgrp and rindex values are good.
(level 1 passed)
Starting pass1
Reconciling bitmaps.
reconcile_bitmaps completed in 16m46.493s
pass1 completed in 38m21.495s
Starting pass1b
pass1b completed in 0.000s
Starting pass2
pass2 completed in 7h12m50.482s
Starting pass3
pass3 completed in 21.032s
Starting pass4
pass4 completed in 4m8.415s
Starting check_statfs
check_statfs completed in 0.167s
gfs2_fsck complete

With the two pushed patches applied:

[root@gfs-i24c-01 /home/bob/gfs2-utils]# cat /tmp/fsck.twoperfpatches.out
Initializing fsck
Validating resource group index.
Level 1 resource group check: Checking if all rgrp and rindex values are good.
(level 1 passed)
Starting pass1
Reconciling bitmaps.
reconcile_bitmaps completed in 16m10.194s
pass1 completed in 38m29.875s
Starting pass1b
pass1b completed in 0.000s
Starting pass2
pass2 completed in 23m35.125s
Starting pass3
pass3 completed in 19.352s
Starting pass4
pass4 completed in 4m14.756s
Starting check_statfs
check_statfs completed in 0.204s
gfs2_fsck complete

Meager/no savings to pass1, but pass2 went from 7h12m to 23m35s.
Changing status to POST.
Comment 6 Nate Straz 2017-05-25 14:00:57 UTC 
Pass 2 times for gfs2-utils-3.1.0-3.el7 are much better than gfs2-utils-3.1.9-3.el7.


pass2 time for fsck0
        FS Size  3.1.9-3.el7/fsc  3.1.10-3.el7/fs  
            16G                0                0  
            32G                0                0  
            64G                0                0  
           128G                0                0  
           256G                1                0  
           512G                3                0  
             1T                7                0  
             2T               17                0  
             4T               47                0  
             8T               92                0  
            16T              271                0  
            32T              700                0  
            64T             1746                0  
           128T             3891                   
           256T             8357                   

pass2 time for fsck80
        FS Size  3.1.9-3.el7/fsc  3.1.10-3.el7/fs  
            16G                0                0  
            32G                0                0  
            64G                0                0  
           128G                1                0  
           256G                2                0  
           512G                6                1  
             1T               18                2  
             2T               37                4  
             4T              103               22  
             8T              184               39  
            16T              353               37  
            32T              925               56  
            64T             2232                   
           128T             4942                   
           256T             9904
Comment 7 errata-xmlrpc 2017-08-01 21:57:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2226
Note You need to log in before you can comment on or make changes to this bug.