Bug 1041563

I guess this is probably a kernel issue, not an xfsprogs problem.

Did the repair shrink your fs again?  :(  I hope not.  This reminds me of a bug upstream, let me find the details.

Do you have logs which indicate anything the kernel said when you ran growfs, and got:

xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Structure needs cleaning

?

It did not shrink it.  The kernel logged this error:

Dec 09 10:22:54 ti124 kernel: ffff88022ea1c800: 58 46 53 42 00 00 10 00 00 00 00 00 a5 80 00 00  XFSB............
Dec 09 10:22:54 ti124 kernel: ffff88022ea1c810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
Dec 09 10:22:54 ti124 kernel: ffff88022ea1c820: 8f e6 58 84 36 fd 4e 62 98 9a cb 68 ff 71 b8 f3  ..X.6.Nb...h.q..
Dec 09 10:22:54 ti124 kernel: ffff88022ea1c830: 00 00 00 00 40 00 00 08 00 00 00 00 00 00 08 00  ....@...........
Dec 09 10:22:54 ti124 kernel: [113B blob data]
Dec 09 10:22:54 ti124 kernel: CPU: 6 PID: 416 Comm: kworker/6:1H Not tainted 3.10.11-200.fc19.x86_64 #1
Dec 09 10:22:54 ti124 kernel: Hardware name: Supermicro X8DTH-i/6/iF/6F/X8DTH, BIOS 2.1b       05/04/12  
Dec 09 10:22:54 ti124 kernel: Workqueue: xfslogd xfs_buf_iodone_work [xfs]
Dec 09 10:22:54 ti124 kernel:  0000000000000001 ffff880333dcbd68 ffffffff816396c7 ffff880333dcbd80
Dec 09 10:22:54 ti124 kernel:  ffffffffa03731ab ffffffffa0370b05 ffff880333dcbdb8 ffffffffa0373205
Dec 09 10:22:54 ti124 kernel:  000002da35670780 ffff880141c63000 0000000000000075 ffff880335b49800
Dec 09 10:22:54 ti124 kernel: Call Trace:
Dec 09 10:22:54 ti124 kernel:  [<ffffffff816396c7>] dump_stack+0x19/0x1b
Dec 09 10:22:54 ti124 kernel:  [<ffffffffa03731ab>] xfs_error_report+0x3b/0x40 [xfs]
Dec 09 10:22:54 ti124 kernel:  [<ffffffffa0370b05>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]
Dec 09 10:22:54 ti124 kernel:  [<ffffffffa0373205>] xfs_corruption_error+0x55/0x80 [xfs]
Dec 09 10:22:54 ti124 kernel:  [<ffffffffa03c3f14>] xfs_sb_read_verify+0x114/0x130 [xfs]
Dec 09 10:22:54 ti124 kernel:  [<ffffffffa0370b05>] ? xfs_buf_iodone_work+0x85/0xf0 [xfs]
Dec 09 10:22:54 ti124 kernel:  [<ffffffffa0370b05>] xfs_buf_iodone_work+0x85/0xf0 [xfs]
Dec 09 10:22:54 ti124 kernel:  [<ffffffff81079ba5>] process_one_work+0x175/0x430
Dec 09 10:22:54 ti124 kernel:  [<ffffffff8107a7cb>] worker_thread+0x11b/0x3a0
Dec 09 10:22:54 ti124 kernel:  [<ffffffff8107a6b0>] ? rescuer_thread+0x340/0x340
Dec 09 10:22:54 ti124 kernel:  [<ffffffff81080b70>] kthread+0xc0/0xd0
Dec 09 10:22:55 ti124 kernel:  [<ffffffff81080ab0>] ? insert_kthread_work+0x40/0x40
Dec 09 10:22:55 ti124 kernel:  [<ffffffff81647c2c>] ret_from_fork+0x7c/0xb0
Dec 09 10:22:55 ti124 kernel:  [<ffffffff81080ab0>] ? insert_kthread_work+0x40/0x40
Dec 09 10:22:55 ti124 kernel: XFS (dm-3): Corruption detected. Unmount and run xfs_repair
Dec 09 10:22:55 ti124 kernel: XFS (dm-3): metadata I/O error: block 0x43fff7800 ("xfs_trans_read_buf_map") error 117 
Dec 09 10:22:55 ti124 kernel: XFS (dm-3): error 117 reading secondary superblock for ag 34

It was pretty much the same on the other server.

Should I upgrade the kernel to 3.11.10?

Thanks,
Andy

I'm guessing that this fs has also been grown before - prior to v3.8?


See http://thread.gmane.org/gmane.comp.file-systems.xfs.general/54428 and the "Failure growing xfs with linux 3.10.5" thread on the list as well.

one thing we need is this, in xfsprogs, to make repair handle things better after the fact:

commit cbd7508db4c9597889ad98d5f027542002e0e57c
Author: Eric Sandeen <sandeen>
Date:   Thu Aug 15 02:26:40 2013 +0000

    xfs_repair: zero out unused parts of superblocks
    
    Prior to:
    1375cb65 xfs: growfs: don't read garbage for new secondary superblocks
    
    we ran the risk of allowing garbage in secondary superblocks
    beyond the in-use sb fields.  With kernels 3.10 and beyond, the
    verifiers will kick these out as invalid, but xfs_repair does
    not detect or repair this condition.
    
    There is superblock stale-data zeroing code, but it is under a
    narrow conditional - the bug addressed in the above commit did not
    meet that conditional.  So change this to check unconditionally.
    
    Further, the checking code was looking at the in-memory
    superblock buffer, which was zeroed prior to population, and
    would therefore never possibly show any stale data beyond the
    last up-rev superblock field.
    
    So instead, check the disk buffer for this garbage condition.
    
    If we detect garbage, we must zero out both the in-memory sb
    and the disk buffer; the former may contain unused data
    in up-rev sb fields which will be written back out; the latter
    may contain garbage beyond all fields, which won't be updated
    when we translate the in-memory sb back to disk.
    
    The V4 superblock case was zeroing out the sb_bad_features2
    field; we also fix that to leave that field alone.
    
    Lastly, use offsetof() instead of the tortured (__psint_t)
    casts & pointer math.
    
    Reported-by: Michael Maier <m1278468>
    Signed-off-by: Eric Sandeen <sandeen>
    Reviewed-by: Rich Johnston <rjohnston>
    Signed-off-by: Rich Johnston <rjohnston>

and we also need this in the kernel to avoid it in the first place; it's been sent to -stable but not in 3.10.y yet:

commit 10e6e65dfcedff63275c3d649d329c044caa8e26
Author: Eric Sandeen <sandeen>
Date:   Mon Sep 9 15:33:29 2013 -0500

    xfs: be more forgiving of a v4 secondary sb w/ junk in v5 fields
    
    Today, if xfs_sb_read_verify encounters a v4 superblock
    with junk past v4 fields which includes data in sb_crc,
    it will be treated as a failing checksum and a significant
    corruption.
    
    There are known prior bugs which leave junk at the end
    of the V4 superblock; we don't need to actually fail the
    verification in this case if other checks pan out ok.
    
    So if this is a secondary superblock, and the primary
    superblock doesn't indicate that this is a V5 filesystem,
    don't treat this as an actual checksum failure.
    
    We should probably check the garbage condition as
    we do in xfs_repair, and possibly warn about it
    or self-heal, but that's a different scope of work.
    
    Stable folks: This can go back to v3.10, which is what
    introduced the sb CRC checking that is tripped up by old,
    stale, incorrect V4 superblocks w/ unzeroed bits.
    
    Cc: stable.org
    Signed-off-by: Eric Sandeen <sandeen>
    Acked-by: Dave Chinner <david>
    Reviewed-by: Mark Tinguely <tinguely>
    Signed-off-by: Ben Myers <bpm>

It has been grown many times under Fedora 14, 16, and 19.  This is the second time I have had to repair it under F19.  Are these fixes by any chance in the latest F19 3.11.10 kernel?

Thanks,
Andy

Just a note that 3.10.y is irrelevant for Fedora, as all releases are on 3.11.10 or newer by this point.  If we need fixes, we can add patches to 3.11.10 manually, or get them into 3.12.y upstream

Ok, FWIW on 12/6 I got:

"3.11.10.1 -stable review patch." for "xfs: be more forgiving of a v4 secondary sb w/ junk in v5 fields"

-Eric

Andrew, it was a bit of a perfect storm of old (previously harmless) bugs and new metadata verifiers.  All fixed upstream now, AFAIK...

Heh.  3.11.10.1 is something Canonical is running on their own.  That's not to say the patches they grab aren't useful, but it's not an upstream 3.11.y release.  That is dead.  So we'll either grab things manually, or rebase to 3.12.y which was already planned.  Is it queued in 3.12?

Oh.  God, I'm out of touch, sorry.

It's in 3.12.y:

http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/fs/xfs?h=linux-3.12.y&id=31fcbef62d8142c0c173b0b8255e9b0c28a7a038

Thanks for the link.  Do you know if 3.12.y will be pushed out for Fedora 19 any time soon?  I am wondering if I should apply this patch to 3.11.10 myself.  If 3.12 (or a patched 3.11 kernel) is coming, I will wait.

Thanks,
Andy

3.12.y should be coming to f19 likely within a week or so.  It's in F20 updates-testing right now and we roll it out to the older releases a bit more slowly.

*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 19 kernel bugs.

Fedora 19 has now been rebased to 3.12.6-200.fc19.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 20, and are still experiencing this issue, please change the version to Fedora 20.

If you experience different issues, please open a new bug report for those.

Closing as ERRATA per the previous comments now that F19 is on 3.12.6.