Bug 1325843 - [HC] Add disk in a Hyper-converged environment fails when glusterfs is running in directIO mode
Summary: [HC] Add disk in a Hyper-converged environment fails when glusterfs is runnin...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: sharding
Version: 3.7.10
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
Assignee: Krutika Dhananjay
QA Contact: bugs@gluster.org
URL:
Whiteboard:
Depends On: 1314421 1322214 1335284
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-11 10:47 UTC by Krutika Dhananjay
Modified: 2016-06-28 12:14 UTC (History)
8 users (show)

Fixed In Version: glusterfs-3.7.12
Doc Type: Bug Fix
Doc Text:
Clone Of: 1322214
Environment:
Last Closed: 2016-06-28 12:14:18 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Krutika Dhananjay 2016-04-11 10:47:34 UTC
+++ This bug was initially created as a clone of Bug #1322214 +++

+++ This bug was initially created as a clone of Bug #1314421 +++

Description of problem:
In a oVirt-Gluster hyperconverged environment, adding disk to VM from a glusterfs storage pool fails when glusterfs is running in posix/directio mode

The gluster volume is configured to run in directIO mode by adding 

option o-direct on 

in the /var/lib/glusterd/vols/gl_01/*.vol files. Example below

volume gl_01-posix
    type storage/posix
    option o-direct on
    option brick-gid 36
    option brick-uid 36
    option volume-id c131155a-d40c-4d9e-b056-26c61b924c26
    option directory /bricks/b01/g
end-volume

When the option is removed and the volume is restarted, disks can be added to the VM from the glusterfs pool.


Version-Release number of selected component (if applicable):

ovirt version is 3.6

glusterfs-client-xlators-3.7.5-11.el7rhgs.x86_64
glusterfs-cli-3.7.5-11.el7rhgs.x86_64
glusterfs-libs-3.7.5-11.el7rhgs.x86_64
glusterfs-3.7.5-11.el7rhgs.x86_64
glusterfs-api-3.7.5-11.el7rhgs.x86_64
glusterfs-fuse-3.7.5-11.el7rhgs.x86_64
glusterfs-server-3.7.5-11.el7rhgs.x86_64



How reproducible:
Easily reproducible

Steps to Reproduce:
1. Create a GlusterFS storage pool in an oVirt environment 
2. Configure GlusterFS in a posix/directIO mode
3. Create a new VM or add disk to an existing VM. The add disk part fails

Actual results:


Expected results:


Additional info:

--- Additional comment from Krutika Dhananjay on 2016-03-17 08:11:14 EDT ---

Hi Sanjay,

In light of the recent discussion we had wrt direct-io behavior on a mail thread, I have the following question:

Assuming the 'cache=none' command line option implies that the vm image files will all be opened with O_DIRECT flag (which means that the write buffers will already be aligned with the "sector size of the underlying block device", the only layer in the combined client-server stack that could prevent us from achieving o-direct-like behavior because of caching would be the write-behind translator.

Therefore, I am wondering if it is sufficient to enable 'performance.strict-o-direct' to achieve the behavior you expect to see with o-direct?

-Krutika

--- Additional comment from Sanjay Rao on 2016-03-17 08:20:02 EDT ---

I have tested with different options. The only option that enabled true directIO on the glusterfs server was the posix setting.

I can verify again with the performance.strict-o-direct with the recent glusterfs version (glusterfs-server-3.7.5-18.33) installed on my system just to be sure.

--- Additional comment from Vijay Bellur on 2016-03-29 23:27:59 EDT ---

REVIEW: http://review.gluster.org/13846 (features/shard: Make o-direct writes work with sharding) posted (#1) for review on master by Krutika Dhananjay (kdhananj@redhat.com)

--- Additional comment from Vijay Bellur on 2016-04-06 05:50:14 EDT ---

REVIEW: http://review.gluster.org/13846 (features/shard: Make o-direct writes work with sharding) posted (#2) for review on master by Krutika Dhananjay (kdhananj@redhat.com)

--- Additional comment from Vijay Bellur on 2016-04-11 02:01:27 EDT ---

REVIEW: http://review.gluster.org/13846 (features/shard: Make o-direct writes work with sharding) posted (#3) for review on master by Krutika Dhananjay (kdhananj@redhat.com)

--- Additional comment from Vijay Bellur on 2016-04-11 05:07:14 EDT ---

REVIEW: http://review.gluster.org/13846 (features/shard: Make o-direct writes work with sharding) posted (#4) for review on master by Krutika Dhananjay (kdhananj@redhat.com)

Comment 1 Vijay Bellur 2016-04-11 10:49:35 UTC
REVIEW: http://review.gluster.org/13966 (features/shard: Make o-direct writes work with sharding) posted (#1) for review on release-3.7 by Krutika Dhananjay (kdhananj@redhat.com)

Comment 2 Vijay Bellur 2016-04-13 05:51:04 UTC
REVIEW: http://review.gluster.org/13966 (features/shard: Make o-direct writes work with sharding) posted (#2) for review on release-3.7 by Krutika Dhananjay (kdhananj@redhat.com)

Comment 3 Vijay Bellur 2016-04-17 05:43:54 UTC
COMMIT: http://review.gluster.org/13966 committed in release-3.7 by Pranith Kumar Karampuri (pkarampu@redhat.com) 
------
commit bd8ab0567017fd4dddfea738eae0bccc40b85527
Author: Krutika Dhananjay <kdhananj@redhat.com>
Date:   Tue Mar 29 18:36:08 2016 +0530

    features/shard: Make o-direct writes work with sharding
    
            Backport of: http://review.gluster.org/#/c/13846/
    
    With files opened with o-direct, the expectation is that
    the IO performed on the fds is byte aligned wrt the sector size
    of the underlying device. With files getting sharded, a single
    write from the application could be broken into more than one write
    falling on different shards which _might_ cause the original byte alignment
    property to be lost. To get around this, shard translator will send fsync
    on odirect writes to emulate o-direct-like behavior in the backend.
    
    Change-Id: I992e10162afcca17a19d9cba3bcb187a31c618ae
    BUG: 1325843
    Signed-off-by: Krutika Dhananjay <kdhananj@redhat.com>
    Reviewed-on: http://review.gluster.org/13966
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>

Comment 4 Kaushal 2016-06-28 12:14:18 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.12, please open a new bug report.

glusterfs-3.7.12 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-devel/2016-June/049918.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.