Description of problem: An XFS partition become unavailable after it became full. Version-Release number of selected component (if applicable): kernel-3.14.1-200.fc20.x86_64 How reproducible: Once Steps to Reproduce: 1. ABRT wrote something to /var/tmp making it full Actual results: Apr 30 00:18:11 hermes kernel: XFS (sdc6): xlog_write: reservation summary: trans type = FSYNC_TS (36) unit res = 9640 bytes current res = -4 bytes total reg = 0 bytes (o/flow = 0 bytes) ophdrs = 0 (ophdr space = 0 bytes) ophdr + reg = 0 bytes num regions = 0 Apr 30 00:18:11 hermes kernel: XFS (sdc6): xlog_write: reservation ran out. Need to up reservation Apr 30 00:18:11 hermes kernel: XFS (sdc6): xfs_do_force_shutdown(0x2) called from line 1999 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa01c36e8 Apr 30 00:18:11 hermes abrt-hook-ccpp[10684]: Write error: Input/output error Apr 30 00:18:11 hermes abrt-hook-ccpp[10684]: Error writing '/var/tmp/abrt/ccpp-2014-04-30-00:18:04-8659.new/coredump' Apr 30 00:18:11 hermes kernel: XFS (sdc6): Log I/O Error Detected. Shutting down filesystem Apr 30 00:18:11 hermes kernel: XFS (sdc6): Please umount the filesystem and rectify the problem(s) Apr 30 00:18:11 hermes kernel: Buffer I/O error on device sdc6, logical block 225044 Apr 30 00:18:11 hermes kernel: lost page write due to I/O error on sdc6 Apr 30 00:18:11 hermes kernel: Buffer I/O error on device sdc6, logical block 225009 Apr 30 00:18:11 hermes kernel: lost page write due to I/O error on sdc6 Apr 30 00:18:11 hermes kernel: Buffer I/O error on device sdc6, logical block 225010 Apr 30 00:18:11 hermes kernel: lost page write due to I/O error on sdc6 Apr 30 00:18:11 hermes kernel: Buffer I/O error on device sdc6, logical block 225011 Apr 30 00:18:11 hermes kernel: lost page write due to I/O error on sdc6 Apr 30 00:18:11 hermes kernel: Buffer I/O error on device sdc6, logical block 225012 Apr 30 00:18:11 hermes kernel: lost page write due to I/O error on sdc6 Apr 30 00:18:11 hermes kernel: Buffer I/O error on device sdc6, logical block 225053 Apr 30 00:18:11 hermes kernel: lost page write due to I/O error on sdc6 Apr 30 00:18:11 hermes kernel: Buffer I/O error on device sdc6, logical block 225054 Apr 30 00:18:11 hermes kernel: lost page write due to I/O error on sdc6 Apr 30 00:18:11 hermes kernel: Buffer I/O error on device sdc6, logical block 225055 Apr 30 00:18:11 hermes kernel: lost page write due to I/O error on sdc6 Apr 30 00:18:11 hermes kernel: Buffer I/O error on device sdc6, logical block 225056 Apr 30 00:18:11 hermes kernel: lost page write due to I/O error on sdc6 Apr 30 00:18:40 hermes kernel: XFS (sdc6): xfs_log_force: error 5 returned. Apr 30 00:19:11 hermes kernel: XFS (sdc6): xfs_log_force: error 5 returned. Apr 30 00:19:41 hermes kernel: XFS (sdc6): xfs_log_force: error 5 returned. Apr 30 00:20:11 hermes kernel: XFS (sdc6): xfs_log_force: error 5 returned. Apr 30 00:20:41 hermes kernel: XFS (sdc6): xfs_log_force: error 5 returned. Expected results: No errors Additional info: SMART showed no errors for the hard-disk.
dchinner: This commit in 3.15-rc1: fe4c224 xfs: inode log reservations are still too small
Description of problem: An XFS partition become unavailable witout previous warning. Version-Release number of selected component (if applicable): kernel-3.14.4-200.fc20.x86_64 How reproducible: Once Steps to reproduce: 1. Was downloading some torrent files and then suddenly I was bitten by this problem too. In my case, the only difference was that my XFS file system had plenty of space: [pfessel@wotan ~]$ LANG=C df -h /home Filesystem Size Used Avail Use% Mounted on /dev/md126p1 1.9T 1.2T 637G 66% /home [pfessel@wotan ~]$ LANG=C df -i /home Filesystem Inodes IUsed IFree IUse% Mounted on /dev/md126p1 390676224 412656 390263568 1% /home [pfessel@wotan ~]$ mount | grep home /dev/md126p1 on /home type xfs (rw,noatime,nodiratime,seclabel,attr2,inode64,sunit=512,swidth=512,noquota) Jun 2 22:12:48 wotan kernel: [23518.275534] XFS (md126p1): xlog_write: reservation summary: Jun 2 22:12:48 wotan kernel: [23518.275534] trans type = FSYNC_TS (36) Jun 2 22:12:48 wotan kernel: [23518.275534] unit res = 9640 bytes Jun 2 22:12:48 wotan kernel: [23518.275534] current res = -4 bytes Jun 2 22:12:48 wotan kernel: [23518.275534] total reg = 0 bytes (o/flow = 0 bytes) Jun 2 22:12:48 wotan kernel: [23518.275534] ophdrs = 0 (ophdr space = 0 bytes) Jun 2 22:12:48 wotan kernel: [23518.275534] ophdr + reg = 0 bytes Jun 2 22:12:48 wotan kernel: [23518.275534] num regions = 0 Jun 2 22:12:48 wotan kernel: [23518.275534] Jun 2 22:12:48 wotan kernel: [23518.275556] XFS (md126p1): xlog_write: reservation ran out. Need to up reservation Jun 2 22:12:48 wotan kernel: [23518.275566] XFS (md126p1): xfs_do_force_shutdown(0x2) called from line 1999 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa0d346e8 Jun 2 22:12:48 wotan kernel: [23518.275577] XFS (md126p1): Log I/O Error Detected. Shutting down filesystem Jun 2 22:12:48 wotan kernel: [23518.275582] XFS (md126p1): Please umount the filesystem and rectify the problem(s) Jun 2 22:12:48 wotan kernel: XFS (md126p1): xlog_write: reservation summary: trans type = FSYNC_TS (36) unit res = 9640 bytes current res = -4 bytes total reg = 0 bytes (o/flow = 0 bytes) ophdrs = 0 (ophdr space = 0 bytes) ophdr + reg = 0 bytes num regions = 0 Jun 2 22:12:48 wotan kernel: XFS (md126p1): xlog_write: reservation ran out. Need to up reservation Jun 2 22:12:48 wotan kernel: XFS (md126p1): xfs_do_force_shutdown(0x2) called from line 1999 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa0d346e8 Jun 2 22:12:48 wotan kernel: XFS (md126p1): Log I/O Error Detected. Shutting down filesystem Jun 2 22:12:48 wotan kernel: XFS (md126p1): Please umount the filesystem and rectify the problem(s) Jun 2 22:13:04 wotan kernel: [23535.018581] device p22p1 left promiscuous mode Jun 2 22:13:04 wotan kernel: device p22p1 left promiscuous mode Jun 2 22:13:17 wotan kernel: [23548.129223] XFS (md126p1): xfs_log_force: error 5 returned. Jun 2 22:13:17 wotan kernel: XFS (md126p1): xfs_log_force: error 5 returned. Jun 2 22:13:48 wotan kernel: [23578.192475] XFS (md126p1): xfs_log_force: error 5 returned. (...) Dismounted fs, remounted to replay log and then xfs_repair'd it. It appears to have recovered after this. This volume resides on a RAID volume which had one of its disks replaced less than one week ago when I moved the contents from an ext4 fs to this new xfs fs. Disks are OK and have no SMART errors. Here is /proc/mdstat for /home: md126 : active raid1 sde1[2] sdd1[0] 1953382464 blocks super 1.2 [2/2] [UU] bitmap: 0/15 pages [0KB], 65536KB chunk
This was supposedly fixed in commit fe4c224aa1ffa4352849ac5f452de7132739bee2 * * https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=fe4c224aa1ffa4352849ac5f452de7132739bee2
It just exploded on me again: Jun 15 01:11:22 wotan kernel: [43299.625792] XFS (md126p1): xlog_write: reservation summary: Jun 15 01:11:22 wotan kernel: [43299.625792] trans type = FSYNC_TS (36) Jun 15 01:11:22 wotan kernel: [43299.625792] unit res = 9640 bytes Jun 15 01:11:22 wotan kernel: [43299.625792] current res = -4 bytes Jun 15 01:11:22 wotan kernel: [43299.625792] total reg = 0 bytes (o/flow = 0 bytes) Jun 15 01:11:22 wotan kernel: [43299.625792] ophdrs = 0 (ophdr space = 0 bytes) Jun 15 01:11:22 wotan kernel: [43299.625792] ophdr + reg = 0 bytes Jun 15 01:11:22 wotan kernel: [43299.625792] num regions = 0 Jun 15 01:11:22 wotan kernel: [43299.625792] Jun 15 01:11:22 wotan kernel: [43299.625811] XFS (md126p1): xlog_write: reservation ran out. Need to up reservation Jun 15 01:11:22 wotan kernel: [43299.625817] XFS (md126p1): xfs_do_force_shutdown(0x2) called from line 1999 of file fs/xfs/xfs_log.c. R eturn address = 0xffffffffa044b6e8 Jun 15 01:11:22 wotan kernel: [43299.625823] XFS (md126p1): Log I/O Error Detected. Shutting down filesystem Jun 15 01:11:22 wotan kernel: [43299.625825] XFS (md126p1): Please umount the filesystem and rectify the problem(s) Jun 15 01:11:22 wotan kernel: XFS (md126p1): xlog_write: reservation summary: trans type = FSYNC_TS (36) unit res = 9640 bytes current res = -4 bytes total reg = 0 bytes (o/flow = 0 bytes) ophdrs = 0 (ophdr space = 0 bytes) ophdr + reg = 0 bytes num regions = 0 Since Cristian reports that it's been fixed on the above commit, is it possible to backport it to the current mainline fc20 kernel? Until now it seems to be harmless, but I do really fear losing something one of these days.
I have the same problem 3 times now on my /home. Anything I can do except compiling kernel 3.15.7? I use xfs_repair to fix it for a couple of minutes and removed some directories/files but it keeps happening and makes my system unworkable. kernel-3.14.7-200.fc20.x86_64 Jun 15 21:01:22 mobieltje kernel: [45049.282218] XFS (sdb1): xlog_write: reservation summary: Jun 15 21:01:22 mobieltje kernel: [45049.282218] trans type = FSYNC_TS (36) Jun 15 21:01:22 mobieltje kernel: [45049.282218] unit res = 9640 bytes Jun 15 21:01:22 mobieltje kernel: [45049.282218] current res = -4 bytes Jun 15 21:01:22 mobieltje kernel: [45049.282218] total reg = 0 bytes (o/flow = 0 bytes) Jun 15 21:01:22 mobieltje kernel: [45049.282218] ophdrs = 0 (ophdr space = 0 bytes) Jun 15 21:01:22 mobieltje kernel: [45049.282218] ophdr + reg = 0 bytes Jun 15 21:01:22 mobieltje kernel: [45049.282218] num regions = 0 Jun 15 21:01:22 mobieltje kernel: [45049.282218] Jun 15 21:01:22 mobieltje kernel: [45049.282238] XFS (sdb1): xlog_write: reservation ran out. Need to up reservation Jun 15 21:01:22 mobieltje kernel: [45049.282247] XFS (sdb1): xfs_do_force_shutdown(0x2) called from line 1999 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa080c6e8 Jun 15 21:01:22 mobieltje kernel: [45049.282364] XFS (sdb1): Log I/O Error Detected. Shutting down filesystem Jun 15 21:01:22 mobieltje kernel: [45049.282368] XFS (sdb1): Please umount the filesystem and rectify the problem(s) Jun 15 21:01:22 mobieltje kernel: XFS (sdb1): xlog_write: reservation summary: trans type = FSYNC_TS (36) unit res = 9640 bytes current res = -4 bytes total reg = 0 bytes (o/flow = 0 bytes) ophdrs = 0 (ophdr space = 0 bytes) ophdr + reg = 0 bytes num regions = 0 Jun 15 21:01:22 mobieltje kernel: XFS (sdb1): xlog_write: reservation ran out. Need to up reservation Jun 15 21:01:22 mobieltje kernel: XFS (sdb1): xfs_do_force_shutdown(0x2) called from line 1999 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa080c6e8 Jun 15 21:01:22 mobieltje kernel: XFS (sdb1): Log I/O Error Detected. Shutting down filesystem Jun 15 21:01:22 mobieltje kernel: XFS (sdb1): Please umount the filesystem and rectify the problem(s) Jun 15 21:01:25 mobieltje kernel: [45052.507045] XFS (sdb1): xfs_log_force: error 5 returned. Jun 15 21:01:25 mobieltje kernel: XFS (sdb1): xfs_log_force: error 5 returned.
Netbulae, good to see we aren't alone on this. What was your workload at the time when you've got the errors? Was it something write-intensive or just normal usage? In my case, I don't know but I always seem to get it when download torrents which have gigabyte-sized files inside them.
I was torrenting Fedora dvd iso's and when I stopped downloading, I don't get the error.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 20 kernel bugs. Fedora 20 has now been rebased to 3.17.2-200.fc20. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 21, and are still experiencing this issue, please change the version to Fedora 21. If you experience different issues, please open a new bug report for those.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 20 kernel bugs. Fedora 20 has now been rebased to 3.18.7-100.fc20. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 21, and are still experiencing this issue, please change the version to Fedora 21. If you experience different issues, please open a new bug report for those.
This bug seems to be fixed in F21.
Thanks!