RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1553004 - original resize2fs command hang on RHEL7.4
Summary: original resize2fs command hang on RHEL7.4
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: e2fsprogs
Version: 7.4
Hardware: x86_64
OS: Linux
urgent
medium
Target Milestone: rc
: ---
Assignee: Lukáš Czerner
QA Contact: Boyang Xue
URL:
Whiteboard:
Depends On:
Blocks: 1477664 1546181 1562018 1563170
TreeView+ depends on / blocked
 
Reported: 2018-03-08 02:18 UTC by Rong Zhao
Modified: 2021-09-09 13:21 UTC (History)
10 users (show)

Fixed In Version: e2fsprogs-1.42.9-12.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1562018 1563170 (view as bug list)
Environment:
Last Closed: 2018-10-30 11:38:55 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
dump2fs result for the issue filesystem. (13.19 MB, application/x-gzip)
2018-03-08 02:18 UTC, Rong Zhao 		no flags Details
script to create filesystem image (1.73 KB, application/x-shellscript)
2018-03-20 21:39 UTC, Frank Sorenson 		no flags Details
updated reproducer (1.17 KB, application/x-shellscript)
2018-03-21 14:43 UTC, Frank Sorenson 		no flags Details
straight port of upstream commit (6.81 KB, patch)
2018-03-21 20:31 UTC, Frank Sorenson 		no flags Details | Diff
Show Obsolete (1) View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3276 0 None None None 2018-10-30 11:39:15 UTC

Description Rong Zhao 2018-03-08 02:18:27 UTC 
Created attachment 1405663 [details]
dump2fs result for the issue filesystem.

Description of problem:

Related Environment:
- OS: Redhat 7.4 64bit, Kernel: 3.10.0-693.11.6.el7.x86_64
- Storage: connected 3PAR storage with multipath, create LVM on LUNs
- Filesystem: ext4

Operations:
 1. create new filesystem, format ext4
 2. tried to extend LVM, and resize2fs online, it worked indeed
 3. I copied 27T data to the new filesystem, the data is Oracle's data file
 4. Then, I try to extend LVM and resize2fs online again, resize2fs hang at read function, the strace records are below:
 
----there are more lines above, do not paste here -------
lseek(3, 38465727102976, SEEK_SET)      = 38465727102976
read(3, "\1\0\300/\21\0\300/!\0\300/_\177\200\0\0\0\5\0\0\0\0\0\0\0\0\0\200\0\330\326"..., 4096) = 4096
lseek(3, 38474317037568, SEEK_SET)      = 38474317037568
read(3, "\1\0\340/\21\0\340/!\0\340/_\177\200\0\0\0\5\0\0\0\0\0\0\0\0\0\200\0000\214"..., 4096) = 4096
--- hang here

the result of dumpe2fs is attached.

BTW, I requested new LUNs and create new filesystem, after copying data, the problem still exists.


Version-Release number of selected component (if applicable):

The original e2fsprogs version is "e2fsprogs-1.42.9-10.el7.x86_64"


How reproducible:

Because it works well for other folders (same or even larger data), and it can be reproduced after create new filesystem and copy data. So, I think it should be some file caused this issue, maybe cannot be reproduced by others, I keep the old filesystem there, and if what information needed, please let me know.


Actual results:

resize2fs hangs when extending ext4 filesystem online.


Expected results:

it should work.


Additional info:

I downloaded the latest e2fsprogs source code from kernel.org, compile and install it, it works, the new version is: https://www.kernel.org/pub/linux/kernel/people/tytso/e2fsprogs/v1.43.9/e2fsprogs-1.43.9.tar.gz
Comment 2 Kamil Dudka 2018-03-08 06:39:11 UTC 
The filesystem RPM package has nothing to do with the resize2fs command.  I am switching the component to e2fsprogs...
Comment 3 Lukáš Czerner 2018-03-08 10:24:30 UTC 
Thanks for the report,

can you please specify the file system sizes before and after each resize ? That is, from what size and to what size you resized the file system.

-Lukas
Comment 4 Rong Zhao 2018-03-09 00:55:14 UTC 
Hi Lukas,
  I did twice.

  First time: from 29T --> 34T

  Second time: from 35T --> 36T

  the attached dumpe2fs info is second time.

Thanks.
Comment 5 Frank Sorenson 2018-03-19 13:32:22 UTC 
This is most likely the same as internal bz1273049, and fixed by this upstream e2fsprogs commit:

commit 1e33a8b408123a4e02a6b9135807f6fd61f3e235
Author: Theodore Ts'o <tytso>
Date:   2014-07-26 07:40:36 -0400

    Fix 32/64-bit overflow when multiplying by blocks/clusters per group
    
    There are a number of places where we need convert groups to blocks or
    clusters by multiply the groups by blocks/clusters per group.
    Unfortunately, both quantities are 32-bit, but the result needs to be
    64-bit, and very often the cast to 64-bit gets lost.
    
    Fix this by adding new macros, EXT2_GROUPS_TO_BLOCKS() and
    EXT2_GROUPS_TO_CLUSTERS().
    
    This should fix a bug where resizing a 64bit file system can result in
    calculate_minimum_resize_size() looping forever.
    
    Addresses-Launchpad-Bug: #1321958
    
    Signed-off-by: Theodore Ts'o <tytso>


The issue occurs when the number of required data blocks can no longer be expressed in a 32-bit number, such as seen in the dumpe2fs attached to this case (number of inodes also plays into the equation, but the problem can be demonstrated without):

Filesystem features:      has_journal ext_attr dir_index filetype meta_bg extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Default mount options:    user_xattr acl
Inode count:              36700160
Block count:              9395240960
Reserved block count:     467077427
Free blocks:              2066197447
Free inodes:              36698061

here, the full filesystem has 9395240960 blocks, and 2066197447 are free, so current in-use data blocks:

$ echo $((9395240960 - 2066197447))
7329043513

$ printf %x\\n 7329043513
1b4d85439

which is larger than 0xffffffff


I have an e2image provided by a customer which also experiences this issue:

Filesystem features:      has_journal ext_attr dir_index filetype meta_bg extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Default mount options:    user_xattr acl
Inode count:              670756864
Block count:              5366026240
Reserved block count:     241481761
Free blocks:              496314629
Free inodes:              667982046

block count: 5366026240
free blocks: 496314629

$ echo $((5366026240 - 496314629))
4869711611

$ printf %x\\n 4869711611
12241e6fb
 ffffffff << maximum blocks represented in 32 bits

when running, resize2fs is actually running not hung, but is busy attempting to calculate the required number of blocks; however, it will run for a very long time.  No data is moved, and the filesystem size does not change at all; resize2fs is merely calculating the size.

Enabling debugging for the size calculation, we see the following:

# resize2fs -d 32 -P /dev/loop1
resize2fs 1.42.9 (28-Dec-2013)
fs has 2774818 inodes, 678 groups required.
fs requires 4827409032 data blocks.
With 678 group(s), we have 22013094 blocks available.
Added 146650 extra group(s), data_needed 4827415688, data_blocks 494613997, last_start 4789548784
Added 132227 extra group(s), data_needed 4827419016, data_blocks 498340270, last_start 9088242352
Added 132113 extra group(s), data_needed 4827422088, data_blocks 498360411, last_start 13383229789
Added 132113 extra group(s), data_needed 4827424904, data_blocks 498380552, last_start 17678217226
Added 132112 extra group(s), data_needed 4827427720, data_blocks 498368184, last_start 21973172154
Added 132113 extra group(s), data_needed 4827430280, data_blocks 498388323, last_start 26268159589
Added 132112 extra group(s), data_needed 4827432840, data_blocks 498375954, last_start 30563114516
Added 132113 extra group(s), data_needed 4827435144, data_blocks 498396096, last_start 34858101954
...
(will run for a very long time...I don't know if it will eventually stop or not)


after taking the most-recently-released e2fsprogs, and applying the patch:

# resize/resize2fs -d 32 -P /dev/loop1
resize2fs 1.42.9 (28-Dec-2013)
fs has 2774818 inodes, 678 groups required.
fs requires 4827409032 data blocks.
With 678 group(s), we have 22013094 blocks available.
Added 146650 extra group(s), data_needed 4827415688, data_blocks 4789581293, last_start 4789548784
Added 1155 extra group(s), data_needed 4827419016, data_blocks 4827130287, last_start 4827097777
Added 9 extra group(s), data_needed 4827419016, data_blocks 4827422877, last_start 4827390367
Last group's overhead is 258
Need 28649 data blocks in last group
Final size of last group is 28907
Estimated blocks needed: 4865781995
Extents safety margin: 1000488
Estimated minimum size of the filesystem: 4866782483


Rong Zhao,
Can you please run the resize2fs calculation with debugging enabled, as above, and provide the first 15-20 lines or so?  It should be obvious fairly quickly whether this is the bug you are seeing as well:

resize2fs -d 32 -P /dev/mapper/your_device
Comment 6 Rong Zhao 2018-03-20 02:09:41 UTC 
Here are first 38 lines, I think you are right.

Thanks.

# resize2fs -d 32 -P /dev/billdb/billdb
resize2fs 1.42.9 (28-Dec-2013)
fs has 2099 inodes, 17 groups required.
fs requires 7055128671 data blocks.
With 17 group(s), we have 534608 blocks available.
Added 215290 extra group(s), data_needed 7055128831, data_blocks 2757962872, last_start 7052897410
Added 131140 extra group(s), data_needed 7055128839, data_blocks 2758874591, last_start 11348776425
Added 131112 extra group(s), data_needed 7055128911, data_blocks 2758868046, last_start 15643737176
Added 131112 extra group(s), data_needed 7055128919, data_blocks 2758861498, last_start 19938697924
Added 131112 extra group(s), data_needed 7055128991, data_blocks 2758854951, last_start 24233658673
Added 131112 extra group(s), data_needed 7055128999, data_blocks 2758848406, last_start 28528619424
Added 131113 extra group(s), data_needed 7055129063, data_blocks 2758874617, last_start 32823612931
Added 131112 extra group(s), data_needed 7055129191, data_blocks 2758868072, last_start 37118573683
Added 131112 extra group(s), data_needed 7055129255, data_blocks 2758861526, last_start 41413534432
Added 131112 extra group(s), data_needed 7055129383, data_blocks 2758854979, last_start 45708495181
Added 131112 extra group(s), data_needed 7055129447, data_blocks 2758848435, last_start 50003455933
Added 131113 extra group(s), data_needed 7055129567, data_blocks 2758874645, last_start 54298449439
Added 131112 extra group(s), data_needed 7055129623, data_blocks 2758868098, last_start 58593410188
Added 131112 extra group(s), data_needed 7055129743, data_blocks 2758861554, last_start 62888370940
Added 131112 extra group(s), data_needed 7055129799, data_blocks 2758855006, last_start 67183331688
Added 131112 extra group(s), data_needed 7055129919, data_blocks 2758848460, last_start 71478292439
Added 131113 extra group(s), data_needed 7055129967, data_blocks 2758874673, last_start 75773285947
Added 131112 extra group(s), data_needed 7055130079, data_blocks 2758868126, last_start 80068246696
Added 131112 extra group(s), data_needed 7055130127, data_blocks 2758861582, last_start 84363207448
Added 131112 extra group(s), data_needed 7055130239, data_blocks 2758855035, last_start 88658168197
Added 131112 extra group(s), data_needed 7055130287, data_blocks 2758848488, last_start 92953128946
Added 131113 extra group(s), data_needed 7055130391, data_blocks 2758874702, last_start 97248122456
Added 131112 extra group(s), data_needed 7055130431, data_blocks 2758868155, last_start 101543083205
Added 131112 extra group(s), data_needed 7055130535, data_blocks 2758861608, last_start 105838043954
Added 131112 extra group(s), data_needed 7055130575, data_blocks 2758855064, last_start 110133004706
Added 131112 extra group(s), data_needed 7055130679, data_blocks 2758848517, last_start 114427965455
Added 131113 extra group(s), data_needed 7055130711, data_blocks 2758874731, last_start 118722958965
Added 131112 extra group(s), data_needed 7055130807, data_blocks 2758868184, last_start 123017919714
Added 131112 extra group(s), data_needed 7055130839, data_blocks 2758861637, last_start 127312880463
Added 131112 extra group(s), data_needed 7055130935, data_blocks 2758855093, last_start 131607841215
Added 131112 extra group(s), data_needed 7055130967, data_blocks 2758848546, last_start 135902801964
Added 131113 extra group(s), data_needed 7055131055, data_blocks 2758874757, last_start 140197795471
Added 131112 extra group(s), data_needed 7055131079, data_blocks 2758868213, last_start 144492756223
Added 131112 extra group(s), data_needed 7055131167, data_blocks 2758861666, last_start 148787716972
Comment 7 Frank Sorenson 2018-03-20 21:39:45 UTC 
Created attachment 1410819 [details]
script to create filesystem image

the attached script will create a filesystem image that requires more blocks than addressable with 32-bit number, leading to resize2fs calculating the minimum possible filesystem size forever.

creates a 20 TB filesystem image (occupying approximately 800 MiB on disk)

(the script requires a debugfs containing 'fallocate', which is not available in the RHEL 7 e2fsprogs...  could potentially be rewritten with other debugfs functions in order to remove this requirement)
Comment 8 Frank Sorenson 2018-03-21 14:43:33 UTC 
Created attachment 1411230 [details]
updated reproducer

Donald Douwsma pointed out that the native fallocate command will also allocate the disk blocks without requiring the disk space (I was thinking more along the lines of 'truncate' behavior, creating either sparse files or fully allocated).  This updated version uses the native fallocate command, and debugfs is no longer required at all
Comment 9 Frank Sorenson 2018-03-21 20:31:46 UTC 
Created attachment 1411445 [details]
straight port of upstream commit

here is a port of the upstream e2fsprogs patch onto the RHEL 7.4 e2fsprogs-1.42.9-11.el7 source, fixing this resize2fs bug


Only very lightly tested beyond resize2fs
Comment 39 Lukáš Czerner 2018-04-10 09:07:39 UTC 
Thanks Boyang,

just to clarify the '--large-fs" option is for the xfstests. It will fill the file system to run test on "higher block numbers". It's especially usefull to run on scratch device that's larger than 16TB. It may require some tinkering to make it work correctly though, I do not think many people are running it.

-Lukas
Comment 57 errata-xmlrpc 2018-10-30 11:38:55 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3276
Note You need to log in before you can comment on or make changes to this bug.