Bug 1612449 - inode 7 frequently have problem when I run fsck
Summary: inode 7 frequently have problem when I run fsck
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: e2fsprogs
Version: 28
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Lukáš Czerner
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-04 14:34 UTC by Patrick Dung
Modified: 2019-04-18 22:19 UTC (History)
7 users (show)

Fixed In Version: e2fsprogs-1.44.6-1.fc29
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-18 22:19:31 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Patrick Dung 2018-08-04 14:34:51 UTC
Description of problem:
inode 7 sometimes have problem when fsck is run

Version-Release number of selected component (if applicable):
kernel: 4.17.3-200.fc28.x86_64
e2fsprogs-1.44.2-0.fc28.x86_64

How reproducible:
It usually appear after the disk is not used for a while.
I have configured tune2fs to check a disk for a maximum of 30 days.
Usually fsck won't report problem if the disk is not used for one day.

Additional info:
$ sudo fsck.ext4 /dev/sdb1
e2fsck 1.44.2 (14-May-2018)
storage1 has gone 62 days without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 7 has illegal block(s).  Clear<y>? yes
Illegal block #1042 (1252450111) in inode 7.  CLEARED.
Illegal block #1043 (1934033907) in inode 7.  CLEARED.
Illegal block #1075 (2131231744) in inode 7.  CLEARED.
Illegal block #1107 (1288373248) in inode 7.  CLEARED.
Illegal block #1123 (3089105920) in inode 7.  CLEARED.
Illegal block #1155 (2389050368) in inode 7.  CLEARED.
Illegal block #1171 (3824747520) in inode 7.  CLEARED.
Illegal block #1187 (2595292160) in inode 7.  CLEARED.
Illegal block #1203 (2943026176) in inode 7.  CLEARED.
Illegal block #1235 (1430389760) in inode 7.  CLEARED.
Illegal block #1251 (2182349824) in inode 7.  CLEARED.
Too many illegal blocks in inode 7.
Clear inode<y>? yes
Restarting e2fsck from the beginning...
Resize inode not valid.  Recreate<y>? yes
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong for group #1 (420, counted=421).
Fix<y>? yes
Free blocks count wrong (341766565, counted=341766566).
Fix<y>? yes

storage1: ***** FILE SYSTEM WAS MODIFIED *****
storage1: 105794/244189184 files (2.2% non-contiguous),
634987610/976754176 blocks

Output of dumpe2fs
Filesystem volume name:   storage1
Last mounted on:          /mnt1
Filesystem UUID:          7184ae75-3871-4c69-9a10-bc93449207dc
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index
filetype meta_bg extent 64bit flex_bg sparse_super large_file
huge_file dir_nlink extra_isize metadata_csum
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              244189184
Block count:              976754176
Reserved block count:     9767541
Free blocks:              341766566
Free inodes:              244083390
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Reserved GDT blocks:      1024
Blocks per group:         4096
Fragments per group:      4096
Inodes per group:         1024
Inode blocks per group:   64
Flex block group size:    16
Filesystem created:       Sun Jul 30 12:55:21 2017
Last mount time:          Sun May 27 12:16:43 2018
Last write time:          Sun Jul 29 10:33:17 2018
Mount count:              0
Maximum mount count:      -1
Last checked:             Sun Jul 29 10:33:17 2018
Check interval:           3888000 (1 month, 2 weeks, 1 day)
Next check after:         Wed Sep 12 10:33:17 2018
Lifetime writes:          3437 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      21c957bc-ae01-49d4-a87e-e46a31812fd1
Journal backup:           inode blocks
Checksum type:            crc32c
Checksum:                 0x8a2b32da
Journal features:         journal_incompat_revoke journal_64bit
journal_checksum_v3
Journal size:             1024M
Journal length:           262144
Journal sequence:         0x00009564
Journal start:            0
Journal checksum type:    crc32c
Journal checksum:         0x26cc12b4

I had also upload the full output of dumpe2fs in here:
https://drive.google.com/open?id=1GYHviU29jCxVzuXfy9Y0BP0OerPTYQYj

Note:
For the hardware, I am using a SATA hard drive over an Avago 9361-8i
storage card. The SATA hard drive is used as an JBOD disk with write cache enabled in the hard drive. I had performed a long test with smartmontools and found no problem on the hard drive.

After searching the web. I found Cisco WebEX Node SPA have very
similar error message that I had encountered.
https://www.cisco.com/c/en/us/td/docs/interfaces_modules/shared_port_adapters/install_upgrade/ASR1000/asr_sip_spa_hw/ASRtrbl.pdf
Please check page 7-8 or search for "Inode 7" in the document.

Comment 1 Eric Sandeen 2018-08-04 21:40:26 UTC
Inode 7 is the resize inode.  Have you done any resizes of this filesystem?

Very weird that the cisco doc references this specific type of corruption.

Comment 2 Patrick Dung 2018-08-05 03:44:35 UTC
When I perform the setup, I remember I created a single partition (/dev/sdb1) on this hard drive. As far as I could remember, I have not resize the filesystem.

The problem had happened a few times in the past. I had also tried to remove the journal and add back the journal, the problem persisted.

Comment 3 Lukáš Czerner 2018-08-06 07:40:30 UTC
Hi,

as discussed on the ext4 list, this does look like a weird HW issue. Please check your system logs for file system/storage related errors/warnings.

-Lukas

Comment 4 Patrick Dung 2018-08-06 08:24:58 UTC
(In reply to Lukáš Czerner from comment #3)
> Hi,
> 
> as discussed on the ext4 list, this does look like a weird HW issue. Please
> check your system logs for file system/storage related errors/warnings.
> 
> -Lukas

I have been using the storage adapter for more than two years. Besides have that inode 7 on a specific hard drive, I also had other file systems (xfs/btrfs/ext4) on other hard drives. And I don't have problem with other hard drives.

I did not notice any error or warnings for the storage adapter or file system.

If it is an hardware issue, why it is corrupting the resize inode but not other inode(s)?

Comment 5 Lukáš Czerner 2018-08-06 10:22:15 UTC
(In reply to Patrick Dung from comment #4)
> (In reply to Lukáš Czerner from comment #3)
> > Hi,
> > 
> > as discussed on the ext4 list, this does look like a weird HW issue. Please
> > check your system logs for file system/storage related errors/warnings.
> > 
> > -Lukas
> 
> I have been using the storage adapter for more than two years. Besides have
> that inode 7 on a specific hard drive, I also had other file systems
> (xfs/btrfs/ext4) on other hard drives. And I don't have problem with other
> hard drives.
> 
> I did not notice any error or warnings for the storage adapter or file
> system.
> 
> If it is an hardware issue, why it is corrupting the resize inode but not
> other inode(s)?

So the hardware does not have any idea about inodes, so it does not really care it's 'inode 7' blocks. But it is likely that inode 7 indirect blocks will be stored in the same location. It just might be the location the HW has a problem with.

The truth is we do not know. There is not really much information about what has happend. There is nothing in the logs, you have not resized the file system so the inode and its blocks would not have been touched at all. And as Ted already told you we do have an extensive regression testing for both ext4 in kernel and e2fsprogs and we've never seen this.

I did made some suggestions on the list - report next time when you see this and try to capture the ext4 image before e2fsck fixes it so we can possibly gather more information.

All of this and the fact that you have more ext4 file systems and you only see this problem in this one specific case (always on the same drive I assume) makes me think that it is HW issue. I may be wrong, but at the moment I do not have enough information to do anything about it.

Comment 6 Patrick Dung 2018-08-06 19:18:07 UTC
(In reply to Lukáš Czerner from comment #5)
> (In reply to Patrick Dung from comment #4)
> > (In reply to Lukáš Czerner from comment #3)
> > > Hi,
> > > 
> > > as discussed on the ext4 list, this does look like a weird HW issue. Please
> > > check your system logs for file system/storage related errors/warnings.
> > > 
> > > -Lukas
> > 
> > I have been using the storage adapter for more than two years. Besides have
> > that inode 7 on a specific hard drive, I also had other file systems
> > (xfs/btrfs/ext4) on other hard drives. And I don't have problem with other
> > hard drives.
> > 
> > I did not notice any error or warnings for the storage adapter or file
> > system.
> > 
> > If it is an hardware issue, why it is corrupting the resize inode but not
> > other inode(s)?
> 
> So the hardware does not have any idea about inodes, so it does not really
> care it's 'inode 7' blocks. But it is likely that inode 7 indirect blocks
> will be stored in the same location. It just might be the location the HW
> has a problem with.
> 
> The truth is we do not know. There is not really much information about what
> has happend. There is nothing in the logs, you have not resized the file
> system so the inode and its blocks would not have been touched at all. And
> as Ted already told you we do have an extensive regression testing for both
> ext4 in kernel and e2fsprogs and we've never seen this.
> 
> I did made some suggestions on the list - report next time when you see this
> and try to capture the ext4 image before e2fsck fixes it so we can possibly
> gather more information.
> 
> All of this and the fact that you have more ext4 file systems and you only
> see this problem in this one specific case (always on the same drive I
> assume) makes me think that it is HW issue. I may be wrong, but at the
> moment I do not have enough information to do anything about it.

Good to know ext4 goes through extensive regression testing. But it may still have blind spot.

For my usage on this hard drive about once every two months. So it is usually after the check interval. I don't have problem manually mounting the filesystem and there is no warning or errors. But if I run fsck, it would report the inode 7 had problem.

From wiki: The direct/indirect/triple-indirect block maps are not targeted for checksums. To my limited knowledge, the resize inode 7 should be not protected by checksum. If the inode 7 is protected by checksum then checksum error should be reported during fsck, right?

Personally I do not think it is a HW issue. I had used smarmontools for long test and I believed the hard disk sectors have LDPC error detection (according to the manual).

Anyway, if the problem occurs again, I would try to dump the metadata.

Comment 7 Eric Sandeen 2018-08-06 20:16:32 UTC
(In reply to Patrick Dung from comment #6)

> For my usage on this hard drive about once every two months. So it is
> usually after the check interval. I don't have problem manually mounting the
> filesystem and there is no warning or errors. But if I run fsck, it would
> report the inode 7 had problem.

That makes sense, becaue the inode is never used at runtime unless you are resizing.  So nothing would notice any corruption until it gets validated by fsck.
 
> From wiki: The direct/indirect/triple-indirect block maps are not targeted
> for checksums. To my limited knowledge, the resize inode 7 should be not
> protected by checksum. If the inode 7 is protected by checksum then checksum
> error should be reported during fsck, right?

Yes, but I don't know if you have checksumming enabled in any case.

> Personally I do not think it is a HW issue. I had used smarmontools for long
> test and I believed the hard disk sectors have LDPC error detection
> (according to the manual).

It's unlikely that a disk error detectable by SMART tools would lead to data corruption, in any case.  It's usually a case of an unreadable part of the disk.

> Anyway, if the problem occurs again, I would try to dump the metadata.

FWIW, I looked at the hex of the bad blocks contained in the inode, and didn't notice any pattern.  Running e2fsck -fn (to detect errors but not change) and then e2image to gather the corrupted state is a good next step, although without a recognizeable pattern of corruption, it will still be hard to work backwards to a root cause.

Comment 8 Eric Sandeen 2018-08-06 20:23:22 UTC
When we talk about a hardware problem it's not limited to a disk failure detectable by smartmontools - raid cards have been known to be corruption vectors as well.  While there's no hard evidence that your card is at fault, it's a possibility; there are corruption fixes in the firmware updates logs for example.  The chances of hitting only this inode seems slim, though, but remember that e2fsck only checks metadata and would have no idea if file data got corrupted.

If you really want to go sleuthing, you might consider storing md5sums of all files and re-check their integrity when the inode 7 corruption is detected.  (Not sure what you have stored on this filesystem or how feasible that may be).

Even if you've been using the hardware for a year, a change in IO patterns sometimes tickles bugs.  But sure, it's just one theory at this point.

Comment 9 Patrick Dung 2018-08-07 02:52:44 UTC
(In reply to Eric Sandeen from comment #7)
> (In reply to Patrick Dung from comment #6)
> 
> > From wiki: The direct/indirect/triple-indirect block maps are not targeted
> > for checksums. To my limited knowledge, the resize inode 7 should be not
> > protected by checksum. If the inode 7 is protected by checksum then checksum
> > error should be reported during fsck, right?
> 
> Yes, but I don't know if you have checksumming enabled in any case.

I enabled checkum in metatdata journal checksum, please see my output in my first post.

Good news is that after I used the disk a few days ago. I perform fsck just now and the problem appeared again.

# fsck.ext4 -nvf /dev/sdb1
e2fsck 1.44.2 (14-May-2018)
Pass 1: Checking inodes, blocks, and sizes
Inode 7 has illegal block(s).  Clear? no

Illegal block #1042 (1252450111) in inode 7.  IGNORED.
Illegal block #1043 (1934033907) in inode 7.  IGNORED.
Illegal block #1059 (3556246528) in inode 7.  IGNORED.
Illegal block #1075 (3397846016) in inode 7.  IGNORED.
Illegal block #1091 (1709835264) in inode 7.  IGNORED.
Illegal block #1107 (1288373248) in inode 7.  IGNORED.
Illegal block #1123 (3089105920) in inode 7.  IGNORED.
Illegal block #1155 (2389050368) in inode 7.  IGNORED.
Illegal block #1171 (3824747520) in inode 7.  IGNORED.
Illegal block #1187 (2595292160) in inode 7.  IGNORED.
Illegal block #1235 (1430389760) in inode 7.  IGNORED.
Too many illegal blocks in inode 7.
Clear inode? no

Suppress messages? no

Illegal block #1251 (2182349824) in inode 7.  IGNORED.
Illegal block #1267 (2542208000) in inode 7.  IGNORED.
Illegal block #1299 (2146501632) in inode 7.  IGNORED.
Illegal block #1315 (4228842496) in inode 7.  IGNORED.
Illegal block #1331 (2325218304) in inode 7.  IGNORED.
Illegal block #1379 (2919171072) in inode 7.  IGNORED.
Illegal block #1395 (1900479488) in inode 7.  IGNORED.
Illegal block #1427 (3666805760) in inode 7.  IGNORED.
Illegal block #1459 (1079116800) in inode 7.  IGNORED.
Illegal block #1507 (3395879936) in inode 7.  IGNORED.
Illegal block #1523 (1913324544) in inode 7.  IGNORED.
Illegal block #1539 (3211527168) in inode 7.  IGNORED.
Too many illegal blocks in inode 7.
Clear inode? no

Suppress messages? no

Illegal block #1555 (3594060800) in inode 7.  IGNORED.
Illegal block #1571 (2069890048) in inode 7.  IGNORED.
Illegal block #1603 (3345875968) in inode 7.  IGNORED.
Illegal block #1635 (2418803712) in inode 7.  IGNORED.
Illegal block #1651 (1748567040) in inode 7.  IGNORED.
Illegal block #1699 (2600338432) in inode 7.  IGNORED.
Illegal block #1715 (1908868096) in inode 7.  IGNORED.
Illegal block #1731 (2336490496) in inode 7.  IGNORED.
Illegal block #1747 (1844970496) in inode 7.  IGNORED.
Illegal block #1779 (3663070208) in inode 7.  IGNORED.
Illegal block #1811 (1624310784) in inode 7.  IGNORED.
Illegal block #1827 (4138599424) in inode 7.  IGNORED.
Too many illegal blocks in inode 7.
Clear inode? no

Suppress messages? no

Illegal block #1859 (3619488768) in inode 7.  IGNORED.
Illegal block #1875 (2060059648) in inode 7.  IGNORED.
Illegal block #1891 (3116434432) in inode 7.  IGNORED.
Illegal block #1907 (3660579840) in inode 7.  IGNORED.
Illegal block #1923 (1492911104) in inode 7.  IGNORED.
Illegal block #1939 (2451768320) in inode 7.  IGNORED.
Illegal block #1955 (2997617664) in inode 7.  IGNORED.
Illegal block #2003 (1245709312) in inode 7.  IGNORED.
Inode 7, i_blocks is 213000, should be 216552.  Fix? no

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

storage1: ********** WARNING: Filesystem still has errors **********


      107137 inodes used (0.04%, out of 244189184)
        2315 non-contiguous files (2.2%)
          41 non-contiguous directories (0.0%)
             # of inodes with ind/dind/tind blocks: 0/0/0
             Extent depth histogram: 105234/1891/2
   641099968 blocks used (65.64%, out of 976754176)
           0 bad blocks
         228 large files

       97434 regular files
        9692 directories
           0 character device files
           0 block device files
           0 fifos
           0 links
           2 symbolic links (2 fast symbolic links)
           0 sockets
------------
      107128 files

# fdisk -l /dev/sdb
Disk /dev/sdb: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 9D3F145A-9C71-4924-834C-984581A1A75C

Device     Start        End    Sectors  Size Type
/dev/sdj1   2048 7814035455 7814033408  3.7T Linux filesystem

More info would come.

Comment 10 Patrick Dung 2018-08-07 03:20:32 UTC
Alright, for my comment #9, I would paste the correct fdisk output in here:

# fdisk -l /dev/sdb
Disk /dev/sdb: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 9D3F145A-9C71-4924-834C-984581A1A75C

Device     Start        End    Sectors  Size Type
/dev/sdb1   2048 7814035455 7814033408  3.7T Linux filesystem

For the meta of the file system, due to privacy concern, I used -Qs option with e2image (However, the -s option will prevent analysis of problems  related  to  hash-tree indexed directories).
Please download in here:
https://drive.google.com/open?id=1ErphLwHbPfms1t3dLqD5JhK70qWVQwST
Filename is bug1612449-20180807-1105-Qs-fsmetadata.xz

I had also performed another metadata dump with the -Q option only. In case it is necessary to use it, I may send to the specific developer(s).

Comment 11 Patrick Dung 2018-08-07 04:03:08 UTC
For comment #10, on second thought.
The metadata dump with -Q is a privacy concern for me. Just like exposing the filename/directory listing of a drive to the web.
If e2image -Q is necessary, I would prefer I clean up the files hard drive and wait for the problem to occur again.

Comment 12 Lukáš Czerner 2018-08-07 06:30:56 UTC
Hi Patrick,

I understand your concern. The e2image provides an '-s' option

"This will cause e2image to  scramble  directory  entries and  zero  out  any  unused  portions of the directory blocks before writing the image file."

Thanks!
-Lukas

Comment 13 Patrick Dung 2018-08-07 07:11:06 UTC
(In reply to Lukáš Czerner from comment #12)
> Hi Patrick,
> 
> I understand your concern. The e2image provides an '-s' option
> 
> "This will cause e2image to  scramble  directory  entries and  zero  out 
> any  unused  portions of the directory blocks before writing the image file."
> 
> Thanks!
> -Lukas

Yes, it's the '-s' option.

The e2image output with -Qs is already uploaded (in comment #10)
Please download it in here:
https://drive.google.com/open?id=1ErphLwHbPfms1t3dLqD5JhK70qWVQwST

Comment 14 Lukáš Czerner 2018-08-07 09:06:03 UTC
Thanks Patrick,

something I didn't really notice before with your file system is this:

Blocks per group:         4096

this unusual and definitelly not the default. Are you aware of this ? Do you have any reason using such non-standard layout ?

Anyway it turns out that it indeed was not a HW issue, it's a problem with e2fsprogs itself. With this unusual layout the resize inode ends up referencing blocks outside of the file system.

mke2fs -g4096 -T ext4 /mnt/test/new 3T

e2fsck will try to remove and recreate the resize_inode, however it will recreate it in exactly the same way so even if you run e2fsck immediately afterwards it will find the same problem again.

Now to see how to fix it.

-Lukas

Comment 15 Patrick Dung 2018-08-07 09:23:43 UTC
Hi Lukas,

Oh, I think I had mixed up -g and -G when I create the filesystem.
My intention was to use -G 4096 , not -g 4096.

Thanks,
Patrick

Comment 16 Lukáš Czerner 2018-08-07 09:47:02 UTC
Patrick,

turns out that the fact that you specified -g4096 made the ratio of metadata+reserved metadata to block in group so small that e2fsprogs decided it was better to use meta_bg instead.

Unfortunately it did not remove the resize_inode feature (which is incompatible with meta_bg) which led to this problem. The good news is that this was fixed upstream with the latest release 1.44.3 with commit:

commit 42e77d5db53e3ec09b5dc507169d15de219799e3
Author: Jan Kara

    libext2fs: don't create filesystems with meta_bg and resize_inode
    
    ext2fs_initialize() may end up enabling meta_bg feature for filesystem
    which have resize_inode. Such combination is invalid to make sure we
    disable resize_inode when enabling meta_bg.


Unfortunately e2fsck still fails to recognize that this combination is not valid and will attempt to recreate the resize_inode. This needs to be fixed. Meanwhile you can remove the resize_inode yourself by using tune2fs

tune2fs -O ^resize_inode <device>

then you'll need to run e2fsck to free the blocks resize_inode had allocated.

However in your case, having suboptimal fs layout because of -g4096 I'd recommend recreating the file system. This time I'd also recommend to leave it to defaults unless you really know exactly what you're doing and how it's going to affect the functionality and performance.

Thanks!
-Lukas

Comment 17 Patrick Dung 2018-08-07 11:23:01 UTC
Lukas, 

Thanks for the update.

I was going to test the flexible block group feature. I would re-create the file system again.

For the -g option, it had warning in the man page. But it may be good to display a warning if users are specifying -g option when mkfs.ext4 is run.

Thanks,
Patrick

Comment 18 Eric Sandeen 2018-08-07 14:15:25 UTC
Nice job spotting this, Lukas!  Someone may want to update the upstream thread  with this discovery as well.

-Eric

Comment 19 Fedora Update System 2019-04-04 05:43:28 UTC
e2fsprogs-1.44.6-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-b4207428d3

Comment 20 Fedora Update System 2019-04-05 03:25:52 UTC
e2fsprogs-1.44.6-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-b4207428d3

Comment 21 Fedora Update System 2019-04-18 22:19:31 UTC
e2fsprogs-1.44.6-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.