Description of problem: Gluster uses a POSIX filesystem to store files on the bricks. The atime and mtime can be set with a SETATTR call, which translates to the utimes() systemcall. Some applications that do attribute caching (like NFS-clients, FS-Cache, ...) invalidate the caches too often, because the ctime returned by a replicated volume can be different depending on the subvolumes/bricks that the attributes are read from. Some user application have checks for ctime as well. One of the commonly used applications that spits out a warning about different ctimes is "tar". When a tar archive is created, the process does: 1. check ctime 2. read file (and store it in the archive) 3. check ctime again If the ctime is different in step (1) and (3), tar returns a warning: tar: <filename> file changed as we read it To prevent this, Gluster should always return the ctime that was set by the client-side. One possible way to do this, is by storing the ctime in an extended attribute (like "trusted.gluster.ctime"). In addition, there are several Gluster internal processes that cause the ctime (on the brick, XFS filesystem) to be changed. Self-heal, rebalance and promotion/demotion with Tiering are two of the most common ones.
In the addition to ctime (change time), we can also add the btime (birth time, time of creation).
See also: https://bugzilla.redhat.com/show_bug.cgi?id=1426548 Could we increase severity and get this done? Seems rather like a bug to me than RFE. Its not just elasticsearch which checks ctime/mtime/atime.
I have started a mail thread in gluster-devel for design discussion [1]. I will brief out the design once it finalized from gluste-devel mailing list. [1] : http://lists.gluster.org/pipermail/gluster-devel/2017-February/052190.html
Fantastic! Thanks Rafi! (In reply to Mohammed Rafi KC from comment #3) > I have started a mail thread in gluster-devel for design discussion [1]. I > will brief out the design once it finalized from gluste-devel mailing list. > > > [1] : > http://lists.gluster.org/pipermail/gluster-devel/2017-February/052190.html
Hey Mohammed, what's the status on this?
(In reply to Magnus Glantz from comment #7) > Hey Mohammed, what's the status on this? I have completed the feature with basic functionality, there are three patches 1) A new client side module to handle the and co-ordinate the time attributes https://review.gluster.org/#/c/18138/ 2) Changes to the management code to add the new xlator module to the graph https://review.gluster.org/#/c/18154/ 3) Changes to the posix layer where the times are stored as extended attributes https://review.gluster.org/#/c/17224/ All of these patches are under review. And it seems to me that the priority has dropped for this. Let me know if you have any queries or requirements.
Hello, I seem to be hit by exactly this issue (GlusterFS 3.8.4 using OpenShift 3.9, Elasticsearch 6.5.4); I see that those patches have been adandoned. It seems such a fundamental issue that I would assume would impact a lot more than just Elasticsearch/Lucene/Solr users; has this ever been fixed in later upstream versions of GlusterFS? Are there any mount options that can help avoid the problem?
This is already fixed in the latest releases starting from gluster-5. The patches you mentioned got automatically abandoned because of 90 days inactivity. But later this was added with other change-id. One among them is https://review.gluster.org/#/c/glusterfs/+/19857/.
@Rubin, considering you are using OpenShift, can you test if things would work fine with containers from https://github.com/gluster/gcs project? They have latest glusterfs image (nightly), and you should be able to pick a RWX PV without issues right now.
Hi Mohammed: Are you sure this is only since GlusterFS 5.x? I ask because of this: https://github.com/amarts/glusterfs/commit/0d1dbf034a4a75ff0ebd74b7218193c00b506247 Which seems to mention GD_OP_VERSION_4_1_0 for the (then) new ctime feature, of which the description is: "When this option is enabled, time attributes (ctime,mtime,atime) are stored in xattr to keep it consistent across replica and distribute set. The time attributes stored at the backend are not considered."
Hi Rubin, We did get the feature in 4_1_0, but it was not enabled by default. In 5_0 version, it was enabled by default. In glusterfs-5.0 version, at least we have made more testing etc (http://hrkscribbles.blogspot.com/2018/11/elastic-search-on-gluster.html)
Hi Amar and Mohammed; I'm trying to see how the glusterfs version spoken about above, i.e, version 4.1 and 5.x relates to Red Hat's GlusterFS package version. For example, the latest supported package version for the Red Hat GlusterFS product is 3.12.2-25 (that's for "Red Hat Cluster Storage version 3.4 Batch 1 Update", source: https://access.redhat.com/articles/2356261) Am I right in understanding that Red Hat's supported GlusterFS product is nowhere near version 4.1 (I hope not)?