Description of problem:
The direct-io-mode=enable will not be applied in situations involving multiple recurring opens of the same file.  This is more important than it otherwise might be because direct_io is the only workaround to avoid this recently discovered fuse cache coherency issue:
http://comments.gmane.org/gmane.comp.file-systems.fuse.devel/11819
http://comments.gmane.org/gmane.comp.file-systems.fuse.devel/11823

Actual bug and fix in the additional info: section.

Version-Release number of selected component (if applicable):
3.3.0

How reproducible:
Perfectly reproducible under scenario described here

Steps to Reproduce: (single-threaded)
1.  Open a fd for writing
2.  Open 2nd fd for reading (problem actually occurs here, but not aware of any log that shows whether the FOPEN_DIRECT_IO is applied or not)
3.  Do sequence of writes to file through write fd
4.  Do a sequence of reads  from file through read fd - observe through gluster trace logs that reads issued to gluster exhibit readahead, caching, etc. (reads issued to gluster will not correspond to application read calls() as expected with direct_io)

NOTE: without direct_io enabled for the read fd, this scenario also triggers the fuse coherency issue referenced above if there is a subsequent rewrite and reread of a location through the write fd and read fd respectively.
  
Actual results:
(see steps to reproduce)

Expected results:
(see steps to reproduce)

Additional info:

In fuse_bridge.c:fuse_fd_inherit_directio(), it appears to be trying to set the current open_flags to a prior open_flags from the same file.  The problem is on line 629 - it looked up the prior fd_t with fd_lookup but then turns around and passes the current fd_t in to get the context.  The net effect is that for multiple gluster file opens, the first one is the only one that could ever use FOPEN_DIRECT_IO since any subsequent ones trip into the error case here and never get the flag set. 

*** fuse-bridge.c    2012-07-31 17:05:54.273141548 -0500
--- fuse-bridge.c-orig    2012-07-31 17:05:48.289216778 -0500
***************
*** 626,632 ****
 
          tmp_fd = fd_lookup (fd->inode, 0);
          if (tmp_fd) {
!                 tmp_fdctx = fuse_fd_ctx_get (this, tmp_fd);
                  if (tmp_fdctx) {
                          foo->open_flags &= ~FOPEN_DIRECT_IO;
                          foo->open_flags |= (tmp_fdctx->open_flags
--- 626,632 ----
 
          tmp_fd = fd_lookup (fd->inode, 0);
          if (tmp_fd) {
!                 tmp_fdctx = fuse_fd_ctx_get (this, fd);
                  if (tmp_fdctx) {
                          foo->open_flags &= ~FOPEN_DIRECT_IO;
                          foo->open_flags |= (tmp_fdctx->open_flags