Bug 458936

There is no generic check in RHEL5 for barrier IO, only some targets return NOTSUPPORTED.

So, barrier is sent to underlying device and in some configurations it should work.

Unfortunatelly, if the request need to be split, it will not behave correctly.
(IO will finish, but without barrier flag, so there is possible corruption if power is lost for example.)

I think we should disable barriers for all device-mapper mappings all in RHEL5 kernel... (the same like upstream). (Until we have proper barrie implementation in device-mapper.)

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Milan - so, in RHEL5 we actually do pass barriers to the underlying devices, but, it will be incorrect/incomplete at times due to splits based on the underlying geometry?

But, if an LV is on a single physical device, it actually does pass the barrier correctly?  It sounds like on the one hand, current RHEL5 behavior gives a false sense of security, but it actually does  work in some cases.  If you reject all barriers as upstream does, then it will work in no cases?  Seems like a difficult trade-off.

After mkfs'ing it with the correct filesystem i mounted /dev/sdb several times, which was a iscsi-target backed by drbd backed by scsi.

xfs on /dev/sdb issued no warning
ext3 with barrier=0 issued no warning 
ext3 with barrier=1 issued no warning

Then i created a volume-group containing sdb as its only physical volume. I created a lv in it and retested:

xfs issued a warning
ext3 issued no warning for both settings of barrier

Then i obtained the kernel-source and wanted to see, where ext3 looses the track of the failed barrier requests. I found fs/jbd/commit.c (search for "ordered") where a barrier-protected commit-request for the journal is issued. However the warning issued for failed barrier requests is never issued to the kernel log. If you google for the expected error message you end up with lots of hits regarding suse.

So the question remains: does the device-mapper in RHEL support barriers? In most of the targets (of which i only use "linear") EOPNOTSUPP is returned for barrier-protected requests.

ext3 should be doing the right thing here; I was going to paste this earlier:

        /* is it possible for another commit to fail at roughly
         * the same time as this one?  If so, we don't want to
         * trust the barrier flag in the super, but instead want
         * to remember if we sent a barrier request
         */
        if (ret == -EOPNOTSUPP && barrier_done) {
                char b[BDEVNAME_SIZE];

                printk(KERN_WARNING
                        "JBD: barrier-based sync failed on %s - "
                        "disabling barriers\n",
                        bdevname(journal->j_dev, b));


so *if* the underlying storage rejects the barrier, jbd will do the right thing, warn, and turn barriers off, on the first attempted barrier IO issued from jbd.

But if lvm/dm is not issuing the EOPNOTSUPP, then there is no way for ext3 to know.

If you try this on an upstream (say, fedora) kernel you'll see the warning from jbd, because lvm rejects *all* barrier requests upstream.  :(

p.s. the difference with xfs is that xfs does more pedantic checks on mount, and first checks queue flags*, then attempts a single barrier IO, before mount even completes.  So any warning will come immediately.

*the flag check is removed in most recent kernels and xfs only relies on the barrier IO test.

xfs warns here and everything in drivers/md/dm-linear.c suggests, that EOPNOTSUPP is returned. However ext3/jbd does *not* complain. Of course I testet writing of a new file, so the commit-journal should have been called at least once and the warning should have been displayed, shouldn't it?

Eric: Does that mean jbd and xfs rely on the same occurance of EOPNOTSUPP in dm-linear.c for their check?

For xfs, look at xfs_mountfs_check_barriers(), it checks a few conditions & queue flags than actually tests a barrier IO.

You'll see ""Disabling barriers, not supported by the underlying device"" if the queue does not have flags that advertise the presence of barrier capability (this test has actually been removed upstream), and ""Disabling barriers, trial barrier write failed"" if the barrier IO test actually fails (upstream still does a trial IO).

upstream, dm does:

static int dm_request(struct request_queue *q, struct bio *bio)
{
        int r = -EIO;
        int rw = bio_data_dir(bio);
        struct mapped_device *md = q->queuedata;

        /*
         * There is no use in forwarding any barrier request since we can't
         * guarantee it is (or can be) handled by the targets correctly.
         */
        if (unlikely(bio_barrier(bio))) {
                bio_endio(bio, -EOPNOTSUPP);
                return 0;
        }

so pretty much any barrier request will get back EOPNOTSUPP; the RHEL5 kernel does not have this code.  I see no reference to EOPNOTSUPP in rhel5's dm-linear.c or upstream; how dm behaves is probably best left to Milan though.

In any case I would love to see lvm support barriers as well as it can in RHEL5, I agree.

Eric: Thanks for pointing that up. I was only looking in linear.c which does:

        if (unlikely(bio_barrier(bio))) {
                bio_endio(bio, bio->bi_size, -EOPNOTSUPP);
                return 0;
        }


Is this file used inside dm? Or is it only used for linear raid personalities?

linear.c is part of MD subsystem, not device-mapper.

Cuerrently, there is not such line in RHEL5 kernel, is doesn't return EOPNOTSUPP for dm linear mapping.

This is wrong for some situation (but not all). Situation is not so simple, because device-mapper can reload table during operation, so even if there is one PV (good, barrir should work) it can change (lvextend, pvmove, ...) and later configuration is not barrier safe (for example request which need split between two PVs...)

IOW see this upstream patch http://tinyurl.com/62blpl

In upstream device-mapper entry point (dm_request) will handle NOTSUPP barrier request, in RHEL only some targets return error.
(upstream == 2.6.25, currently 26 has some other problem so it not sending barriers now for JBD),

This bugzilla is exactly about adding patch above to rhel5 kernel.

in kernel-2.6.18-111.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-0225.html