Bug 1554077

Summary:	4.0 clients may fail to convert iatt in dict when recieving the same from older (< 4.0) servers
Product:	[Community] GlusterFS	Reporter:	Raghavendra G <rgowdapp>
Component:	protocol	Assignee:	bugs <bugs>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.0	CC:	atumball, bugs, rgowdapp, srangana
Target Milestone:	---	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-4.0.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1554053	Environment:
Last Closed:	2018-03-15 11:27:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1554053
Bug Blocks:	1554018

Description Raghavendra G 2018-03-11 10:02:30 UTC

+++ This bug was initially created as a clone of Bug #1554053 +++

+++ This bug was initially created as a clone of Bug #1554018 +++

Description of problem:

Based on review comments in https://review.gluster.org/#/c/19690 as follows,

Du: Do we support the case of older bricks/servers and newer clients? If yes, we would face a similar problem. The only difference would be clients expecting a newer iatt when an old iatt is sent.

Shyam: The problem is similar to the disadvantage/improvement that I had mentioned in the review comments. If a client has the newer iatt implementation and connects to an older server, the mapping will fail again. The reasons this is not a major concern yet is as follows.

Our upgrade procedure and guide states servers first and then clients. So the environment will get servers upgraded first and then the clients.

Now, a server would be upgraded in a rolling fashion, so say 1 server is upgraded, then its service clients (quotad, selfheald, others...) would be the latest, but connect with the older protocol version to the older servers.

Till DHT is not in the stack (as this problem is localized to the iatt in the dict that DHT requests), we are fine, if any of these service daemons use DHT in their graph, it may start facing the same/similar problem (rebalance comes to mind, but we may need to add that to the upgrade guide).

The fix going forward (possibly we need this in 4.1) is to apply a similar fix to the client stack as well.

Resolution:

The client protocol xlator also needs a translation of iatt in dict to actual runtime definition of iatt, based on the protocol dialect in question. This change should be similar to the change in the review mentioned above.

--- Additional comment from Shyamsundar on 2018-03-10 21:14:15 EST ---

Protocol end points (NFS, SAMBA/GFAPI) may get started on the upgraded servers, which involves DHT in the graph, and can have issues if this is not addressed for the clients as well.

--- Additional comment from Worker Ant on 2018-03-10 23:14:33 EST ---

REVIEW: https://review.gluster.org/19695 (protocol: Fix 4.0 client, parsing older iatt in dict) posted (#1) for review on master by Shyamsundar Ranganathan

Comment 1 Worker Ant 2018-03-11 10:24:42 UTC

REVIEW: https://review.gluster.org/19696 (protocol: Fix 4.0 client, parsing older iatt in dict) posted (#1) for review on release-4.0 by Raghavendra G

Comment 2 Worker Ant 2018-03-11 14:10:28 UTC

COMMIT: https://review.gluster.org/19696 committed in release-4.0 by "Raghavendra G" <rgowdapp> with a commit message- protocol: Fix 4.0 client, parsing older iatt in dict

In a mixed mode cluster involving 4.0 and older 3.x bricks, if
clients are newer, then the iatt encoded in the dictionary can be
of the older iatt format, which a newer client will map incorrectly
to the newer structure.

This causes failures in FOPs that depend on this iatt for some
functionality (seen in mkdir operations failing as EIO, when DHT
hits its internal setxattr call).

The fix provided is to convert the iatt in the dict, based on which
RPC version is used to communicate with the server.

IOW, this is the reverse of change in commit "b966c7790e"

Tested using a mixed mode cluster (i.e bricks in 3.12 and 4.0 versions)
and a mixed set of clients, 3.12 and 4.0 clients.

There is no regression test provided, as this needs a mixed mode cluster
to test and validate.

>Change-Id: I454e54651ca836b9f7c28f45f51d5956106aefa9
>BUG: 1554053
>Signed-off-by: ShyamsundarR <srangana>

Change-Id: I454e54651ca836b9f7c28f45f51d5956106aefa9
BUG: 1554077
Signed-off-by: ShyamsundarR <srangana>
Signed-off-by: Raghavendra G <rgowdapp>

Comment 3 Shyamsundar 2018-03-15 11:27:41 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.0.0, please open a new bug report.

glusterfs-4.0.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2018-March/000092.html
[2] https://www.gluster.org/pipermail/gluster-users/