Bug 1153316
| Summary: | GFS2: fsck.gfs2 requires too much memory on large file systems | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Nate Straz <nstraz> | ||||||||||
| Component: | gfs2-utils | Assignee: | Robert Peterson <rpeterso> | ||||||||||
| Status: | CLOSED ERRATA | QA Contact: | cluster-qe <cluster-qe> | ||||||||||
| Severity: | unspecified | Docs Contact: | |||||||||||
| Priority: | medium | ||||||||||||
| Version: | 7.1 | CC: | agruenba, cluster-maint, gfs2-maint, rpeterso, swhiteho | ||||||||||
| Target Milestone: | rc | ||||||||||||
| Target Release: | --- | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | gfs2-utils-3.1.8-1.el7 | Doc Type: | Bug Fix | ||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | |||||||||||||
| : | 1268045 (view as bug list) | Environment: | |||||||||||
| Last Closed: | 2015-11-19 03:52:42 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Bug Depends On: | 1184482 | ||||||||||||
| Bug Blocks: | 1111393, 1165285, 1268045, 1497636 | ||||||||||||
| Attachments: |
|
||||||||||||
|
Description
Nate Straz
2014-10-15 18:32:41 UTC
I finally got a fsck.gfs2 run in on an empty 250TB file system. [root@mckinley-04 fsck]# /usr/bin/time fsck.gfs2 -y /dev/250TB/gfs2 Initializing fsck Validating Resource Group index. Level 1 rgrp check: Checking if all rgrp and rindex values are good. (level 1 passed) Starting pass1 pass1 completed in 13.524s Starting pass1b pass1b completed in 0.000s Starting pass1c pass1c completed in 0.000s Starting pass2 pass2 completed in 13m38.589s Starting pass3 pass3 completed in 0.000s Starting pass4 pass4 completed in 0.000s Starting pass5 pass5 completed in 19m45.703s Starting check_statfs check_statfs completed in 0.121s gfs2_fsck complete 1628.86user 510.71system 36:12.72elapsed 98%CPU (0avgtext+0avgdata 17819152maxresident)k 33820072inputs+272outputs (0major+20585653minor)pagefaults 0swaps Snapshot from top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 90648 root 20 0 47.850g 0.017t 872 R 100.0 13.6 7:30.55 fsck.gfs2 Created attachment 972515 [details] Patch 1 of 3. Too-long filename check patch This is the first of 3 patches for bz#1153316. Its purpose is to check directory entries for file names that are too long to physically fit within the contraints of the block. Created attachment 972516 [details]
Patch 2 of 3: print out block number in pass3
This patch changes pass3 so that it prints out the block number
when it finds a bad directory.
Created attachment 972518 [details]
Patch 3 of 3: move to small blockmap
This patch change fsck.gfs2 so that it uses a blockmap that
has a direct 1:1 correlation with the rgrp bitmap. So instead
of using 8 bits for the block state, we only use 2 bits.
That reduces the memory to 1/4 of its previous requirements.
This patch is still being tested and is likely to have bugs
that still need fixing. So far, it passes the hardest tests
though.
Created attachment 991126 [details]
Tarball of the most recent patches
This tarball contains all the latest/greatest RHEL7 patches for
this bug. All of them have gone upstream except for the last two.
They've all been extensively tested on system gfs-i24c-01.lab.bos.
0001-fsck.gfs2-Change-basic-dentry-checks-for-too-long-of.patch
0002-fsck.gfs2-Print-out-block-number-when-pass3-finds-a-.patch
0003-fsck.gfs2-Adjust-when-hash-table-is-doubled.patch
0004-fsck.gfs2-Revise-undo-processing.patch
0005-fsck.gfs2-remove-duplicate-designation-during-undo.patch
0006-fsck.gfs2-Fix-a-use-after-free-in-pass2.patch
0007-fsck.gfs2-fix-double-free-bug.patch
0008-fsck.gfs2-rgrp-block-count-reform.patch
0009-fsck.gfs2-Change-block_map-to-match-bitmap.patch
0010-fsck.gfs2-Fix-journal-sequence-number-reporting-prob.patch
All ten of these patches now appear in the master branch of the upstream gfs2-utils git repo. So it's just a matter of pushing them to the RHEL7 branch at this point. fsck.gfs2 memory usage from gfs2-utils-3.1.8-4.el7.x86_64 on an empty file system. 32MB Resource Groups FS Size max resident in GB 1T 0.199 10T 1.985 100T 19.839 250T 49.596 256MB Resource Groups FS Size max resident in GB 1T 0.143 10T 1.434 100T 14.343 250T 35.858 2048MB Resource Groups ` FS Size max resident in GB 1T 0.128 10T 1.284 100T 12.848 250T 32.121 Testing on gfs2-utils-3.1.8-4.el7.x86_64 with an 80% full file system ran into a runtime scalability issue in pass1c (bz 1257625) above 1TB so I stopped after 16TB. Size Elapsed Max resident 16G 00:41 0.00GB 32G 01:22 0.01GB 64G 03:25 0.02GB 128G 06:09 0.03GB 256G 11:35 0.07GB 512G 25:53 0.14GB 1T 2:17:17 0.27GB 2T 6:34:12 0.54GB 4T 16:23:25 1.07GB 8T 36:47:47 2.13GB 16T 66:29:01 4.26GB Bob handed me a test binary which I used to keep going and I was able to get up to 64TB. Size Elapsed Max resident 512G 0:28:56 0.10GB 1T 0:58:18 0.20GB 2T 2:11:11 0.40GB 4T 4:23:17 0.80GB 8T 9:09:03 1.61GB 16T 16:37:00 3.20GB 32T 34:46:32 6.39GB 64T 81:03:57 12.78GB At the current trend I estimate that a 256TB file system to take about two weeks and use 52GB of RAM. 250TB empty GFS2 file system w/ default 256MB RGs gfs2-utils-3.1.7-6.el7.x86_64 maxresident 53999408k gfs2-utils-3.1.8-6.el7.x86_64 maxresident 37615632k Definitely an improvement on empty file systems, but there is still a lot more work to do. I will clone this bug to continue the work in the next release. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-2178.html |