Bug 1463513 - [geo-rep]: extended attributes are not synced if the entry and extended attributes are done within changelog roleover/or entry sync
Summary: [geo-rep]: extended attributes are not synced if the entry and extended attri...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: geo-replication
Version: 3.11
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Mohammed Rafi KC
QA Contact:
URL:
Whiteboard:
Depends On: 1426952 1448914
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-06-21 07:36 UTC by Mohammed Rafi KC
Modified: 2017-08-12 13:07 UTC (History)
8 users (show)

Fixed In Version: glusterfs-3.11.2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1448914
Environment:
Last Closed: 2017-08-12 13:07:04 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Mohammed Rafi KC 2017-06-21 07:36:13 UTC
+++ This bug was initially created as a clone of Bug #1448914 +++

+++ This bug was initially created as a clone of Bug #1426952 +++

Description of problem:
=======================

Have observed a scenario where extended attributes are not syncing to slave via rsync/fuse if the entry and setxattr is done within 15 seconds. 

Master:
=======

[root@dj master]# touch r.1 ; setfattr  -n user.rtest1 -v 100112 r.1 
[root@dj master]# getfattr -d -e hex -m . r.1
# file: r.1
security.selinux=0x73797374656d5f753a6f626a6563745f723a6675736566735f743a733000
user.rtest1=0x313030313132

[root@dj master]# touch r.2 ; sleep 20 ; setfattr  -n user.rtest1 -v 100113 r.2
[root@dj master]# touch r.3 ; sleep 20 ; setfattr  -n user.rtest1 -v 100113 r.3
[root@dj master]# touch r.4 ; setfattr  -n user.rtest1 -v 100112 r.4
[root@dj master]# 


Slave:
======

[root@dj slave]# getfattr -d -e hex -m . r.1
# file: r.1
security.selinux=0x73797374656d5f753a6f626a6563745f723a6675736566735f743a733000

#root@dj slave]# 
[root@dj slave]# getfattr -d -e hex -m . r.2
# file: r.2
security.selinux=0x73797374656d5f753a6f626a6563745f723a6675736566735f743a733000

[root@dj slave]# 
[root@dj slave]# 
[root@dj slave]# getfattr -d -e hex -m . r.2
# file: r.2
security.selinux=0x73797374656d5f753a6f626a6563745f723a6675736566735f743a733000
user.rtest1=0x313030313133

[root@dj slave]# getfattr -d -e hex -m . r.1
# file: r.1
security.selinux=0x73797374656d5f753a6f626a6563745f723a6675736566735f743a733000

[root@dj slave]# getfattr -d -e hex -m . r.3
# file: r.3
security.selinux=0x73797374656d5f753a6f626a6563745f723a6675736566735f743a733000
user.rtest1=0x313030313133

[root@dj slave]# getfattr -d -e hex -m . r.4
# file: r.4
security.selinux=0x73797374656d5f753a6f626a6563745f723a6675736566735f743a733000

[root@dj slave]# 
[root@dj slave]# getfattr -d -e hex -m . r.4
# file: r.4
security.selinux=0x73797374656d5f753a6f626a6563745f723a6675736566735f743a733000

[root@dj slave]# 

Looked into the .processed directory, it doesn't contain any information yet. Looked into the changelogs and .processing directory. When extended attribute are applied after 15 sec, it is rightly logged into the new changelog which is in processing directory with (SETXATTR). If it is set with 15 secs, it is not recorded in the changelog but only Entry and SETATTR is only recorded and no record for SETXATTR.

Version-Release number of selected component (if applicable):
=============================================================

glusterfs-geo-replication-3.8.4-15.el7rhgs.x86_64


How reproducible:
=================

Always

Steps to Reproduce:
===================
1. Create geo-rep session between master and slave volume
2. Mount the volumes 
3. Create a file and wait for 20 secs
4. Set the extended attribute on the file. It will properly sync to slave.
5. Create another file and immediately set the extended attribute on the file

Actual results:
===============

extended attributes are not synced to the slave


Expected results:
=================

extended attributes should be synced to the slave



--- Additional comment from Rahul Hinduja on 2017-02-26 13:38:35 EST ---



I dont have data to prove if this is a regression. But looking at the design this doesn't seem to be a regression, would definitely need Arvinda/kotresh comments on this. 



--- Additional comment from Rahul Hinduja on 2017-02-26 13:40:10 EST ---

> I dont have data to prove if this is a regression. But looking at the design
> this doesn't seem to be a regression, would definitely need Arvinda/kotresh
> comments on this. 
+ Also, the testing of extended attributes were done in earlier releases but not within 15 sec. So do not have testing data.

--- Additional comment from Kotresh HR on 2017-02-27 04:01:08 EST ---

Analysis:

This is not a regression. This is a day one bug. There is an optimization in changelog to not record multiple METADATA ops on single gfid with in a changelog.
I think this was because there was no extended attribute support during that time and only supported SETATTR. Now since we also support SETXATTR, if SETATTR is already recorded for gfid1, we would miss recording SETXATTR on gfid1. Hence
setxattr would not get synced. This bug will affect the user set extended attributes if any.

The solution would be to remove the optimization for METADATA fops and keep it only for DATA fops.

--- Additional comment from Worker Ant on 2017-05-08 11:59:18 EDT ---

REVIEW: https://review.gluster.org/17205 (geo-replication: Add xattr as special type in changelog entry) posted (#1) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Worker Ant on 2017-05-10 02:29:03 EDT ---

REVIEW: https://review.gluster.org/17205 (features/changelog: Add xattr as special type in changelog entry) posted (#2) for review on master by mohammed rafi  kc (rkavunga)

--- Additional comment from Worker Ant on 2017-05-10 08:55:49 EDT ---

COMMIT: https://review.gluster.org/17205 committed in master by Jeff Darcy (jeff.us) 
------
commit ed27add635263eafb581e5dd1599478d82f0f28a
Author: Mohammed Rafi KC <rkavunga>
Date:   Mon May 8 21:10:50 2017 +0530

    features/changelog: Add xattr as special type in changelog entry
    
    When both SETATTR and SETXATTR fops are happening on gfid within the
    rollover time then, SETXATTR were not logged.
    In Which case we will miss the xattr fop in slave.
    
    This patch will be fix the same
    
    Change-Id: Ia75538ad1fd2797dbcf90d20dfa89f756009243d
    BUG: 1448914
    Signed-off-by: Mohammed Rafi KC <rkavunga>
    Reviewed-on: https://review.gluster.org/17205
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Kotresh HR <khiremat>
    Smoke: Gluster Build System <jenkins.org>

Comment 1 Worker Ant 2017-06-21 07:36:55 UTC
REVIEW: https://review.gluster.org/17587 (features/changelog: Add xattr as special type in changelog entry) posted (#1) for review on release-3.11 by mohammed rafi  kc (rkavunga)

Comment 2 Worker Ant 2017-07-03 12:50:35 UTC
COMMIT: https://review.gluster.org/17587 committed in release-3.11 by Shyamsundar Ranganathan (srangana) 
------
commit a8d7d4dac391c74ef3e34512c387565ab30c0a90
Author: Mohammed Rafi KC <rkavunga>
Date:   Mon May 8 21:10:50 2017 +0530

    features/changelog: Add xattr as special type in changelog entry
    
    When both SETATTR and SETXATTR fops are happening on gfid within the
    rollover time then, SETXATTR were not logged.
    In Which case we will miss the xattr fop in slave.
    
    This patch will be fix the same
    
    Backport of>
    >Change-Id: Ia75538ad1fd2797dbcf90d20dfa89f756009243d
    >BUG: 1448914
    >Signed-off-by: Mohammed Rafi KC <rkavunga>
    >Reviewed-on: https://review.gluster.org/17205
    >NetBSD-regression: NetBSD Build System <jenkins.org>
    >CentOS-regression: Gluster Build System <jenkins.org>
    >Reviewed-by: Kotresh HR <khiremat>
    >Smoke: Gluster Build System <jenkins.org>
    
    Change-Id: Ia75538ad1fd2797dbcf90d20dfa89f756009243d
    BUG: 1463513
    Signed-off-by: Mohammed Rafi KC <rkavunga>
    Reviewed-on: https://review.gluster.org/17587
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Kotresh HR <khiremat>

Comment 3 Shyamsundar 2017-08-12 13:07:04 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.11.2, please open a new bug report.

glusterfs-3.11.2 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-July/031908.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.