Hide Forgot
Description of problem: Aio from/to btrfs is very slow. (The aiocp is from ltp/testcases/kernel/io/ltp-aiodio/aiocp) # tar cf module.tar /lib/modules/2.6.32-193.el6.x86_64/ # dd if=./module.tar of=/mnt/btrfs/junkfile bs=8192 conv=block,sync # dd if=/mnt/btrfs/junkfile of=/mnt/btrfs/ff2 bs=1024 conv=block,sync # time aiocp -b 8k -n 64 -f DIRECT -f SYNC /mnt/btrfs/junkfile /mnt/btrfs/ff2 real 8m9.587s user 0m0.013s sys 0m43.117s The same operation under xfs needs only 6s # time aiocp -b 8k -n 64 -f DIRECT -f SYNC /mnt/xfs/junkfile /mnt/xfs/ff2 real 0m6.543s user 0m0.005s sys 0m0.228s Ext4 needs 2mins # time aiocp -b 8k -n 64 -f DIRECT -f SYNC ./junkfile ./ff2 real 1m53.617s user 0m0.009s sys 0m0.551s Version-Release number of selected component (if applicable): kernel-2.6.32-191.el6 How reproducible: 100% Steps to Reproduce: 1. see description above 2. or you can run task /kernel/filesystems/ltp-aiodio, FSTYP=btrfs SCRATCH_DEV=/dev/sda5 make run this will run aio-stress then aiocp, xfs or ext4 can finish this run within 6 hours, btrfs just seems take forever to finish 3. Actual results: Slow aio Expected results: better aio performance Additional info: If the block size is 1K, aiocp will take 90+ mins to finish time aiocp -b 1k -n 4 -f SYNC /mnt/scratch/aiodio/junkfile /mnt/scratch/aiodio/ff2 real 95m4.673s user 0m0.070s sys 2m20.322s Seems the smaller block size is the longer it will take
This is actually an xfs bug. I ran this test on my box and I got this btrfs - 21 minutes xfs - 17 seconds then I took out the SYNC option and I got btrfs - 20 seconds xfs - 17 seconds turns out xfs has this in its .aio_write if (unlikely(file->f_flags & O_DIRECT)) ret = xfs_file_dio_aio_write(iocb, iovp, nr_segs, pos, ocount, &iolock); else ret = xfs_file_buffered_aio_write(iocb, iovp, nr_segs, pos, ocount, &iolock); xfs_aio_write_isize_update(inode, &iocb->ki_pos, ret); if (ret <= 0) <==== THERE IT IS goto out_unlock; /* Handle various SYNC-type writes */ if ((file->f_flags & O_DSYNC) || IS_SYNC(inode)) { loff_t end = pos + ret - 1; int error; xfs_rw_iunlock(ip, iolock); error = xfs_file_fsync(file, pos, end, (file->f_flags & __O_SYNC) ? 0 : 1); xfs_rw_ilock(ip, iolock); if (error) ret = error; } that should be if (ret <= 0 && ret != -EIOCBQUEUED) goto out_unlock; But I'm going to cc' Dave on this to get a second opinion.
Adding Christoph to the cc list. This behaviour was changes in upstream XFS in 2.6.33 by commit c355c65 ("xfs: remove IO_ISAIO") which removed this hunk: @@ -774,9 +772,6 @@ write_retry: current->backing_dev_info = NULL; - if (ret == -EIOCBQUEUED && !(ioflags & IO_ISAIO)) - ret = wait_on_sync_kiocb(iocb); - isize = i_size_read(inode); if (unlikely(ret < 0 && ret != -EFAULT && *offset > isize)) *offset = isize; Hence Xfs has behaved like this since RHEL6.0. As it is, I'm not sure what the semantics of "synchronous async IO" are supposed to be (seems rather silly to me), that question needs to be answered as well.
The behaviour post the commit Dave pointed out is defintively incorrect, but just adding the EIOCBQUEUED check like it's done in generic_file_aio_write is not correct either (and thus at least ext4 gets it wrong as well) because we may convert unwritten extents in the I/O completion handler, which this type of O_SYNC I/O won't force out either. Given that we do not support buffered aio thus we need to do a non-data fsync from the I/O completion handler. Doing a real log flush from there might be a bit nasty, so I suspect the right answer is to delay the aio complete even further to a callback from the log write I/O completion handler.
(In reply to comment #3) > As it is, I'm not sure what the semantics of "synchronous async IO" are > supposed to be (seems rather silly to me), that question needs to be answered > as well. Asynchronous I/O simply means we've separated the submission path from the completion path. It does actually make sense to have O_SYNC async I/O.
Since RHEL 6.2 External Beta has begun, and this bug remains unresolved, it has been rejected as it is not proposed as exception or blocker. Red Hat invites you to ask your support representative to propose this request, if appropriate and relevant, in the next release of Red Hat Enterprise Linux.
The tech preview of btrfs in RHEL6 is being ended. Red Hat will not be providing any further updates or security fixes for btrfs in RHEL6, therefore this bug is being closed as "WONTFIX". The tech preview of btrfs in RHEL7 is continuing, and an update will be provided in RHEL 7.1, in due course. If you are interested in the btrfs tech preview in RHEL7, then please do get in touch to share your use case(s) via the normal channels (e.g. your partner manager, TAM or via a support ticket depending upon which is most approporiate). We are very keen to collect as much information as we can about use cases for btrfs and other future developments in filesystems/storage.