Bug 1553004
| Summary: | original resize2fs command hang on RHEL7.4 | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Rong Zhao <zhao_rong> | ||||||||||
| Component: | e2fsprogs | Assignee: | Lukáš Czerner <lczerner> | ||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Boyang Xue <bxue> | ||||||||||
| Severity: | medium | Docs Contact: | |||||||||||
| Priority: | urgent | ||||||||||||
| Version: | 7.4 | CC: | dwysocha, fsorenso, hfukumot, kdudka, lczerner, myamazak, nmarjano, rhandlin, xzhou, zhao_rong | ||||||||||
| Target Milestone: | rc | Keywords: | Patch, Reproducer, ZStream | ||||||||||
| Target Release: | --- | ||||||||||||
| Hardware: | x86_64 | ||||||||||||
| OS: | Linux | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | e2fsprogs-1.42.9-12.el7 | Doc Type: | If docs needed, set a value | ||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | |||||||||||||
| : | 1562018 1563170 (view as bug list) | Environment: | |||||||||||
| Last Closed: | 2018-10-30 11:38:55 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Bug Depends On: | |||||||||||||
| Bug Blocks: | 1477664, 1546181, 1562018, 1563170 | ||||||||||||
| Attachments: |
|
||||||||||||
|
Description
Rong Zhao
2018-03-08 02:18:27 UTC
The filesystem RPM package has nothing to do with the resize2fs command. I am switching the component to e2fsprogs... Thanks for the report, can you please specify the file system sizes before and after each resize ? That is, from what size and to what size you resized the file system. -Lukas Hi Lukas, I did twice. First time: from 29T --> 34T Second time: from 35T --> 36T the attached dumpe2fs info is second time. Thanks. This is most likely the same as internal bz1273049, and fixed by this upstream e2fsprogs commit: commit 1e33a8b408123a4e02a6b9135807f6fd61f3e235 Author: Theodore Ts'o <tytso> Date: 2014-07-26 07:40:36 -0400 Fix 32/64-bit overflow when multiplying by blocks/clusters per group There are a number of places where we need convert groups to blocks or clusters by multiply the groups by blocks/clusters per group. Unfortunately, both quantities are 32-bit, but the result needs to be 64-bit, and very often the cast to 64-bit gets lost. Fix this by adding new macros, EXT2_GROUPS_TO_BLOCKS() and EXT2_GROUPS_TO_CLUSTERS(). This should fix a bug where resizing a 64bit file system can result in calculate_minimum_resize_size() looping forever. Addresses-Launchpad-Bug: #1321958 Signed-off-by: Theodore Ts'o <tytso> The issue occurs when the number of required data blocks can no longer be expressed in a 32-bit number, such as seen in the dumpe2fs attached to this case (number of inodes also plays into the equation, but the problem can be demonstrated without): Filesystem features: has_journal ext_attr dir_index filetype meta_bg extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Default mount options: user_xattr acl Inode count: 36700160 Block count: 9395240960 Reserved block count: 467077427 Free blocks: 2066197447 Free inodes: 36698061 here, the full filesystem has 9395240960 blocks, and 2066197447 are free, so current in-use data blocks: $ echo $((9395240960 - 2066197447)) 7329043513 $ printf %x\\n 7329043513 1b4d85439 which is larger than 0xffffffff I have an e2image provided by a customer which also experiences this issue: Filesystem features: has_journal ext_attr dir_index filetype meta_bg extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Default mount options: user_xattr acl Inode count: 670756864 Block count: 5366026240 Reserved block count: 241481761 Free blocks: 496314629 Free inodes: 667982046 block count: 5366026240 free blocks: 496314629 $ echo $((5366026240 - 496314629)) 4869711611 $ printf %x\\n 4869711611 12241e6fb ffffffff << maximum blocks represented in 32 bits when running, resize2fs is actually running not hung, but is busy attempting to calculate the required number of blocks; however, it will run for a very long time. No data is moved, and the filesystem size does not change at all; resize2fs is merely calculating the size. Enabling debugging for the size calculation, we see the following: # resize2fs -d 32 -P /dev/loop1 resize2fs 1.42.9 (28-Dec-2013) fs has 2774818 inodes, 678 groups required. fs requires 4827409032 data blocks. With 678 group(s), we have 22013094 blocks available. Added 146650 extra group(s), data_needed 4827415688, data_blocks 494613997, last_start 4789548784 Added 132227 extra group(s), data_needed 4827419016, data_blocks 498340270, last_start 9088242352 Added 132113 extra group(s), data_needed 4827422088, data_blocks 498360411, last_start 13383229789 Added 132113 extra group(s), data_needed 4827424904, data_blocks 498380552, last_start 17678217226 Added 132112 extra group(s), data_needed 4827427720, data_blocks 498368184, last_start 21973172154 Added 132113 extra group(s), data_needed 4827430280, data_blocks 498388323, last_start 26268159589 Added 132112 extra group(s), data_needed 4827432840, data_blocks 498375954, last_start 30563114516 Added 132113 extra group(s), data_needed 4827435144, data_blocks 498396096, last_start 34858101954 ... (will run for a very long time...I don't know if it will eventually stop or not) after taking the most-recently-released e2fsprogs, and applying the patch: # resize/resize2fs -d 32 -P /dev/loop1 resize2fs 1.42.9 (28-Dec-2013) fs has 2774818 inodes, 678 groups required. fs requires 4827409032 data blocks. With 678 group(s), we have 22013094 blocks available. Added 146650 extra group(s), data_needed 4827415688, data_blocks 4789581293, last_start 4789548784 Added 1155 extra group(s), data_needed 4827419016, data_blocks 4827130287, last_start 4827097777 Added 9 extra group(s), data_needed 4827419016, data_blocks 4827422877, last_start 4827390367 Last group's overhead is 258 Need 28649 data blocks in last group Final size of last group is 28907 Estimated blocks needed: 4865781995 Extents safety margin: 1000488 Estimated minimum size of the filesystem: 4866782483 Rong Zhao, Can you please run the resize2fs calculation with debugging enabled, as above, and provide the first 15-20 lines or so? It should be obvious fairly quickly whether this is the bug you are seeing as well: resize2fs -d 32 -P /dev/mapper/your_device Here are first 38 lines, I think you are right. Thanks. # resize2fs -d 32 -P /dev/billdb/billdb resize2fs 1.42.9 (28-Dec-2013) fs has 2099 inodes, 17 groups required. fs requires 7055128671 data blocks. With 17 group(s), we have 534608 blocks available. Added 215290 extra group(s), data_needed 7055128831, data_blocks 2757962872, last_start 7052897410 Added 131140 extra group(s), data_needed 7055128839, data_blocks 2758874591, last_start 11348776425 Added 131112 extra group(s), data_needed 7055128911, data_blocks 2758868046, last_start 15643737176 Added 131112 extra group(s), data_needed 7055128919, data_blocks 2758861498, last_start 19938697924 Added 131112 extra group(s), data_needed 7055128991, data_blocks 2758854951, last_start 24233658673 Added 131112 extra group(s), data_needed 7055128999, data_blocks 2758848406, last_start 28528619424 Added 131113 extra group(s), data_needed 7055129063, data_blocks 2758874617, last_start 32823612931 Added 131112 extra group(s), data_needed 7055129191, data_blocks 2758868072, last_start 37118573683 Added 131112 extra group(s), data_needed 7055129255, data_blocks 2758861526, last_start 41413534432 Added 131112 extra group(s), data_needed 7055129383, data_blocks 2758854979, last_start 45708495181 Added 131112 extra group(s), data_needed 7055129447, data_blocks 2758848435, last_start 50003455933 Added 131113 extra group(s), data_needed 7055129567, data_blocks 2758874645, last_start 54298449439 Added 131112 extra group(s), data_needed 7055129623, data_blocks 2758868098, last_start 58593410188 Added 131112 extra group(s), data_needed 7055129743, data_blocks 2758861554, last_start 62888370940 Added 131112 extra group(s), data_needed 7055129799, data_blocks 2758855006, last_start 67183331688 Added 131112 extra group(s), data_needed 7055129919, data_blocks 2758848460, last_start 71478292439 Added 131113 extra group(s), data_needed 7055129967, data_blocks 2758874673, last_start 75773285947 Added 131112 extra group(s), data_needed 7055130079, data_blocks 2758868126, last_start 80068246696 Added 131112 extra group(s), data_needed 7055130127, data_blocks 2758861582, last_start 84363207448 Added 131112 extra group(s), data_needed 7055130239, data_blocks 2758855035, last_start 88658168197 Added 131112 extra group(s), data_needed 7055130287, data_blocks 2758848488, last_start 92953128946 Added 131113 extra group(s), data_needed 7055130391, data_blocks 2758874702, last_start 97248122456 Added 131112 extra group(s), data_needed 7055130431, data_blocks 2758868155, last_start 101543083205 Added 131112 extra group(s), data_needed 7055130535, data_blocks 2758861608, last_start 105838043954 Added 131112 extra group(s), data_needed 7055130575, data_blocks 2758855064, last_start 110133004706 Added 131112 extra group(s), data_needed 7055130679, data_blocks 2758848517, last_start 114427965455 Added 131113 extra group(s), data_needed 7055130711, data_blocks 2758874731, last_start 118722958965 Added 131112 extra group(s), data_needed 7055130807, data_blocks 2758868184, last_start 123017919714 Added 131112 extra group(s), data_needed 7055130839, data_blocks 2758861637, last_start 127312880463 Added 131112 extra group(s), data_needed 7055130935, data_blocks 2758855093, last_start 131607841215 Added 131112 extra group(s), data_needed 7055130967, data_blocks 2758848546, last_start 135902801964 Added 131113 extra group(s), data_needed 7055131055, data_blocks 2758874757, last_start 140197795471 Added 131112 extra group(s), data_needed 7055131079, data_blocks 2758868213, last_start 144492756223 Added 131112 extra group(s), data_needed 7055131167, data_blocks 2758861666, last_start 148787716972 Created attachment 1410819 [details]
script to create filesystem image
the attached script will create a filesystem image that requires more blocks than addressable with 32-bit number, leading to resize2fs calculating the minimum possible filesystem size forever.
creates a 20 TB filesystem image (occupying approximately 800 MiB on disk)
(the script requires a debugfs containing 'fallocate', which is not available in the RHEL 7 e2fsprogs... could potentially be rewritten with other debugfs functions in order to remove this requirement)
Created attachment 1411230 [details]
updated reproducer
Donald Douwsma pointed out that the native fallocate command will also allocate the disk blocks without requiring the disk space (I was thinking more along the lines of 'truncate' behavior, creating either sparse files or fully allocated). This updated version uses the native fallocate command, and debugfs is no longer required at all
Created attachment 1411445 [details]
straight port of upstream commit
here is a port of the upstream e2fsprogs patch onto the RHEL 7.4 e2fsprogs-1.42.9-11.el7 source, fixing this resize2fs bug
Only very lightly tested beyond resize2fs
Thanks Boyang, just to clarify the '--large-fs" option is for the xfstests. It will fill the file system to run test on "higher block numbers". It's especially usefull to run on scratch device that's larger than 16TB. It may require some tinkering to make it work correctly though, I do not think many people are running it. -Lukas Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3276 |