Bug 1029820

Description of problem:
File server with XFS fs, 11TB usable (RAID6), 92% full, 9.2 million inodes used.
While deleting a large directory (1.5TB with about 1M files, some of them quite small), the delete ran for over 14 hours, the fs started becoming quite sluggish especially for writing (daemons and interactive) and eventually blew up:

Nov  2 02:36:53 piles kernel: [3582267.113743] XFS (md4): Corruption detected. Unmount and run xfs_repair
Nov  2 02:36:53 piles kernel: [3582267.329675] XFS (md4): xfs_do_force_shutdown(0x8) called from line 1365 of file fs/xfs/xfs_buf.c.  Return address = 0xf8ec8542
Nov  2 02:36:54 piles kernel: [3582268.457035] XFS (md4): Corruption of in-memory data detected.  Shutting down filesystem
Nov  2 02:36:54 piles kernel: [3582268.512041] XFS (md4): Please umount the filesystem and rectify the problem(s)
Nov  2 02:36:54 piles kernel: [3582268.585285] XFS (md4): xfs_difree: xfs_inobt_lookup() returned error 5.
Nov  2 02:36:54 piles kernel: [3582268.618031] XFS (md4): xfs_log_force: error 5 returned.
[...]
(See next attachment for full log including call trace.)


Was unable to umount the fs no matter what I tried.  lsof just blew up and since that fs is shared via NFS and SMB, etc, it was impossible to clean up for umount as most things just errored, froze or died.  Was able to get a pseudo-clean reboot (to not destroy the RAID6) with some alt-sysrq syncs and reboot.  (After disabling the fs from mounting on next boot.)

Ran:
xfs_repair -v -t 60 -m 1550 /dev/md4

(it wouldn't run with anything other than around -m 1550 without dying)

It ran for about 30mins and found zero errors!  I thought for sure there would be some on disk corruption that led to the fs blow up.

So I mounted it and it was all good, like nothing ever happened.

I checked my 16 hour rm results and found it had only deleted around half the files.  So I did the rm again but this time with severe ionice and nice and I also did the rm in smaller batches.  It rm'd all ok that way: quite quickly and with no blow ups.

This fs gets hammered pretty good most days for over 6 years and I've never seen anything like this.  Granted, I probably never have rm'd such a big dir in one go before.  This certainly seemed to me to be related to a massive rm occuring while normal other things run in the background, like recording mythtv and running some light rsync's, etc.

This bug may be hard to reproduce, and it may be quite nerve wracking to intentionally reproduce as restoring 10TB from backup would not be fun.

The computer is older (P-D) but has ECC and reliable-brand components and has never given any errors of any sort before.  It is not showing any bad caps on board or PS.

I should note during the entire process the RAID6 array was all OK with all U's in /proc/mdstat and no errors in the logs about dma,scsi,sda/etc,disk.


Version-Release number of selected component (if applicable):
kernel-PAE-3.10.11-200.fc19.i686
(Probably, it could have been 3.10.9 or 3.10.10, the computer hadn't been rebooted in a while and the boot-time logs spooled off, from what I can see.)

How reproducible:
once, so far
perhaps hard to reproduce

Steps to Reproduce:
1. Run a massive rm -rf on a dir on a massive XFS fs under decent load
2. Wait a day

Actual results:
fs gets very slow.  XFS crashes and disables the fs.  System still runs ok (no panic), you just can't access or umount the crashed fs.

Expected results:
Should delete the directory and do its normal processing happily

Additional info: