Bug 1084508

to assess priority, how many folks are using open-behind volume option?  Open-behind translator is an optimization for small-file workloads, correct?  Has anyone measured performance of open-behind on vs off?  Does it help?

@Poornima/Anuradha,

Can you take a look at this bug?

regards,
Raghavendra

I think the issue is because of open-behind using anonymous-fd. See the following option in open-behind:

        { .key  = {"read-after-open"},
          .type = GF_OPTION_TYPE_BOOL,
          .default_value = "no",
	  .description = "read is sent only after actual open happens and real "
          "fd is obtained, instead of doing on anonymous fd (similar to write)",
        },

The read-ahead cache is per-fd and stored in the context of fd. If open-behind is using anonymous fds for doing reads, read is never sent on the fd which read-ahead has seen (during open from application). So, there is no read-ahead cache.

Can you retry the tests by setting option "read-after-open" in open-behind to "yes"?

[root@unused glusterfs]# gluster volume set dist-rep performance.read-after-open on
volume set: success
[root@unused glusterfs]# gluster volume info
 
Volume Name: dist-rep
Type: Distributed-Replicate
Volume ID: 201492ff-9eb8-48f9-a647-59b89853e3d3
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: booradley:/home/export-2/dist-rep1
Brick2: booradley:/home/export-2/dist-rep2
Brick3: booradley:/home/export-2/dist-rep3
Brick4: booradley:/home/export-2/dist-rep4
Options Reconfigured:
performance.read-after-open: on
performance.readdir-ahead: on

(In reply to Raghavendra G from comment #3)
> I think the issue is because of open-behind using anonymous-fd. See the
> following option in open-behind:
> 
>         { .key  = {"read-after-open"},
>           .type = GF_OPTION_TYPE_BOOL,
>           .default_value = "no",
> 	  .description = "read is sent only after actual open happens and real "
>           "fd is obtained, instead of doing on anonymous fd (similar to
> write)",
>         },
> 
> The read-ahead cache is per-fd and stored in the context of fd. If
> open-behind is using anonymous fds for doing reads, read is never sent on
> the fd which read-ahead has seen (during open from application). So, there
> is no read-ahead cache.

This RCA is not valid. The reason is during read-request, fd is stored in local and in response cache is stored on the fd stored in local. So, even though open-behind sends read on anonymous fd, read-ahead stores the cache in the fd passed to application/kernel.

> 
> Can you retry the tests by setting option "read-after-open" in open-behind
> to "yes"?
> 
> [root@unused glusterfs]# gluster volume set dist-rep
> performance.read-after-open on
> volume set: success
> [root@unused glusterfs]# gluster volume info
>  
> Volume Name: dist-rep
> Type: Distributed-Replicate
> Volume ID: 201492ff-9eb8-48f9-a647-59b89853e3d3
> Status: Created
> Number of Bricks: 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: booradley:/home/export-2/dist-rep1
> Brick2: booradley:/home/export-2/dist-rep2
> Brick3: booradley:/home/export-2/dist-rep3
> Brick4: booradley:/home/export-2/dist-rep4
> Options Reconfigured:
> performance.read-after-open: on
> performance.readdir-ahead: on

(In reply to Raghavendra G from comment #4)
> (In reply to Raghavendra G from comment #3)
> > I think the issue is because of open-behind using anonymous-fd. See the
> > following option in open-behind:
> > 
> >         { .key  = {"read-after-open"},
> >           .type = GF_OPTION_TYPE_BOOL,
> >           .default_value = "no",
> > 	  .description = "read is sent only after actual open happens and real "
> >           "fd is obtained, instead of doing on anonymous fd (similar to
> > write)",
> >         },
> > 
> > The read-ahead cache is per-fd and stored in the context of fd. If
> > open-behind is using anonymous fds for doing reads, read is never sent on
> > the fd which read-ahead has seen (during open from application). So, there
> > is no read-ahead cache.
> 
> This RCA is not valid. The reason is during read-request, fd is stored in
> local and in response cache is stored on the fd stored in local. So, even
> though open-behind sends read on anonymous fd, read-ahead stores the cache
> in the fd passed to application/kernel.

Well, the core of the RCA - read-ahead is disabled because of open-behind using anonymous fds - is still valid :). What was wrong was the mechanism through which read-ahead is turned off. In our current configuration read-ahead is loaded below open-behind. So, with "read-after-open" turned off, read-ahead never receives an open. Without an open, read-ahead doesn't create a context in fd, which is where all the cache is stored.

There are two solutions to this problem:
1. Load read-ahead as one of the ancestors of open-behind. This way read-ahead witnesses the open sent by application before open-behind stopping it.
2. Turn "read-after-open" option on, so that open behind does an open.

https://review.gluster.org/#/c/20511/

The above patch is merged in mainline, and would be in release-5.0