Bug 734339 - btrfs: poor aio performance
Summary: btrfs: poor aio performance
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.3
Hardware: x86_64
OS: Linux
Target Milestone: rc
: ---
Assignee: fs-maint
QA Contact: Red Hat Kernel QE team
Depends On:
Blocks: 1177880
TreeView+ depends on / blocked
Reported: 2011-08-30 06:31 UTC by Eryu Guan
Modified: 2014-12-31 03:02 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1177880 (view as bug list)
Last Closed: 2014-10-14 11:42:00 UTC

Attachments (Terms of Use)

Description Eryu Guan 2011-08-30 06:31:46 UTC
Description of problem:
Aio from/to btrfs is very slow. (The aiocp is from ltp/testcases/kernel/io/ltp-aiodio/aiocp)

# tar cf module.tar /lib/modules/2.6.32-193.el6.x86_64/
# dd if=./module.tar of=/mnt/btrfs/junkfile bs=8192 conv=block,sync
# dd if=/mnt/btrfs/junkfile of=/mnt/btrfs/ff2 bs=1024 conv=block,sync
# time aiocp -b 8k -n 64 -f DIRECT -f SYNC /mnt/btrfs/junkfile /mnt/btrfs/ff2

real    8m9.587s
user    0m0.013s
sys     0m43.117s

The same operation under xfs needs only 6s
# time aiocp -b 8k -n 64 -f DIRECT -f SYNC /mnt/xfs/junkfile /mnt/xfs/ff2

real    0m6.543s
user    0m0.005s
sys     0m0.228s

Ext4 needs 2mins
# time aiocp -b 8k -n 64 -f DIRECT -f SYNC ./junkfile ./ff2

real    1m53.617s
user    0m0.009s
sys     0m0.551s

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. see description above
2. or you can run task /kernel/filesystems/ltp-aiodio, 
FSTYP=btrfs SCRATCH_DEV=/dev/sda5 make run
this will run aio-stress then aiocp, xfs or ext4 can finish this run within 6 hours, btrfs just seems take forever to finish
Actual results:
Slow aio

Expected results:
better aio performance

Additional info:
If the block size is 1K, aiocp will take 90+ mins to finish
time aiocp -b 1k -n 4 -f SYNC /mnt/scratch/aiodio/junkfile /mnt/scratch/aiodio/ff2

real    95m4.673s
user    0m0.070s
sys     2m20.322s

Seems the smaller block size is the longer it will take

Comment 2 Josef Bacik 2011-09-02 18:59:32 UTC
This is actually an xfs bug.  I ran this test on my box and I got this

btrfs - 21 minutes
xfs - 17 seconds

then I took out the SYNC option and I got

btrfs - 20 seconds
xfs - 17 seconds

turns out xfs has this in its .aio_write

        if (unlikely(file->f_flags & O_DIRECT))
                ret = xfs_file_dio_aio_write(iocb, iovp, nr_segs, pos,
                                                ocount, &iolock);
                ret = xfs_file_buffered_aio_write(iocb, iovp, nr_segs, pos,
                                                ocount, &iolock);

        xfs_aio_write_isize_update(inode, &iocb->ki_pos, ret);

        if (ret <= 0)     <==== THERE IT IS
                goto out_unlock;

        /* Handle various SYNC-type writes */
        if ((file->f_flags & O_DSYNC) || IS_SYNC(inode)) {
                loff_t end = pos + ret - 1;
                int error;

                xfs_rw_iunlock(ip, iolock);
                error = xfs_file_fsync(file, pos, end,
                                      (file->f_flags & __O_SYNC) ? 0 : 1);
                xfs_rw_ilock(ip, iolock);
                if (error)
                        ret = error;

that should be

if (ret <= 0 && ret != -EIOCBQUEUED)
        goto out_unlock;

But I'm going to cc' Dave on this to get a second opinion.

Comment 3 Dave Chinner 2011-09-04 23:57:48 UTC
Adding Christoph to the cc list. This behaviour was changes in upstream XFS in 2.6.33 by commit c355c65 ("xfs: remove IO_ISAIO") which removed this hunk:

@@ -774,9 +772,6 @@ write_retry:
        current->backing_dev_info = NULL;
-       if (ret == -EIOCBQUEUED && !(ioflags & IO_ISAIO))
-               ret = wait_on_sync_kiocb(iocb);
        isize = i_size_read(inode);
        if (unlikely(ret < 0 && ret != -EFAULT && *offset > isize))
                *offset = isize;

Hence Xfs has behaved like this since RHEL6.0.

As it is, I'm not sure what the semantics of "synchronous async IO" are supposed to be (seems rather silly to me), that question needs to be answered as well.

Comment 4 chellwig@redhat.com 2011-09-05 08:37:52 UTC
The behaviour post the commit Dave pointed out is defintively incorrect, but just adding the EIOCBQUEUED check like it's done in generic_file_aio_write is not correct either (and thus at least ext4 gets it wrong as well) because we may convert unwritten extents in the I/O completion handler, which this type of O_SYNC I/O won't force out either.

Given that we do not support buffered aio thus we need to do a non-data fsync from the I/O completion handler.  Doing a real log flush from there might be a bit nasty, so I suspect the right answer is to delay the aio complete even further to a callback from the log write I/O completion handler.

Comment 5 Jeff Moyer 2011-09-30 19:15:20 UTC
(In reply to comment #3)

> As it is, I'm not sure what the semantics of "synchronous async IO" are
> supposed to be (seems rather silly to me), that question needs to be answered
> as well.

Asynchronous I/O simply means we've separated the submission path from the completion path.  It does actually make sense to have O_SYNC async I/O.

Comment 6 RHEL Product and Program Management 2011-10-07 15:46:39 UTC
Since RHEL 6.2 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Comment 7 Steve Whitehouse 2014-10-14 11:42:00 UTC
The tech preview of btrfs in RHEL6 is being ended. Red Hat will not be providing any further updates or security fixes for btrfs in RHEL6, therefore this bug is being closed as "WONTFIX".

The tech preview of btrfs in RHEL7 is continuing, and an update will be provided in RHEL 7.1, in due course. If you are interested in the btrfs tech preview in RHEL7, then please do get in touch to share your use case(s) via the normal channels (e.g. your partner manager, TAM or via a support ticket depending upon which is most approporiate). We are very keen to collect as much information as we can about use cases for btrfs and other future developments in filesystems/storage.

Note You need to log in before you can comment on or make changes to this bug.