Bug 1544699

Summary:	Rolling upgrade to 4.0 is broken
Product:	[Community] GlusterFS	Reporter:	Anoop C S <anoopcs>
Component:	protocol	Assignee:	Anoop C S <anoopcs>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	anoopcs, bugs, pkarampu, ravishankar, srangana
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-v4.1.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1544366
Clones:	1551112 (view as bug list)		Environment:
Last Closed:	2018-06-20 17:59:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1544366, 1551112

Description Anoop C S 2018-02-13 10:33:23 UTC

+++ This bug was initially created as a clone of Bug #1544366 +++

Description of problem:
Was trying to test https://review.gluster.org/#/c/19538/ when I found that rolling upgrade from glusterfs-3.13 to the yet to be released glusterfs 4.0 is broken.  In a  2 node setup, when one node is upgraded, clients mounted on each node can only see the local bricks and not the ones on the other node. I see the following errors in the client logs.:

E [MSGID: 114044] [client-handshake.c:1093:client_setvolume_cbk] 0-testvol-client-1: SETVOLUME on remote-host failed: lock state version not supplied [Invalid argument]


Steps to Reproduce:
1. Create a 2 node 1x2 volume and mount locally on each node, all on glustefs-3.13.
2. Upgrade one of the nodes to 4.0 branch
3. Clients can see only local bricks.

--- Additional comment from Anoop C S on 2018-02-12 17:12:17 IST ---

I think I failed to handle the scenario where clients are upgraded before servers.

RCA:(new clients[>=4.0] and old servers[<=3.13])
SETVOLUME request from client post-upgrade does not contain its lk-version in dictionary that is being passed onto server(https://review.gluster.org/#/c/12363/). This means that the server side check for "clnt-lk-version" inside the received dictionary would fail and error is returned back to new client.

--- Additional comment from Ravishankar N on 2018-02-12 19:37:17 IST ---

(In reply to Anoop C S from comment #1)
> I think I failed to handle the scenario where clients are upgraded before
> servers.
> 
FWIW, it is not just the mount but things like self-heal-daemon and glfsheal (gfapi based program that is used to display 'heal info') were also affected.

Comment 1 Worker Ant 2018-02-13 10:35:19 UTC

REVIEW: https://review.gluster.org/19560 (protcol/client: Insert dummy clnt-lk-version to avoid upgrade failure) posted (#2) for review on master by Anoop C S

Comment 2 Worker Ant 2018-02-14 17:05:21 UTC

COMMIT: https://review.gluster.org/19560 committed in master by "Amar Tumballi" <amarts> with a commit message- protcol/client: Insert dummy clnt-lk-version to avoid upgrade failure

With https://review.gluster.org/#/c/12363/ being merged, we no longer
send client's lk-version to server side and the corresponding check on
server is also removed. But when clients are upgraded prior to servers,
the check for lk-version at server side fails and is reported back to
clients resulting in disconnection.

Since we don't have lock-recovery (lk-version and grace-timeout) logic
anymore in code base our best bet would be to add client's default
lk-version i.e, 1, into the dictionary just to make server side check
pass and continue with remaining SETVOLUME operations.

Change-Id: I441b67bd271d1e9ba9a7c08703e651c7a6bd945b
BUG: 1544699
Signed-off-by: Anoop C S <anoopcs>

Comment 3 Worker Ant 2018-03-02 08:23:09 UTC

REVIEW: https://review.gluster.org/19656 (protocol/server: Insert dummy clnt-lk-version to avoid upgrade failure) posted (#1) for review on master by Anoop C S

Comment 4 Worker Ant 2018-03-02 18:55:05 UTC

COMMIT: https://review.gluster.org/19656 committed in master by "Shyamsundar Ranganathan" <srangana> with a commit message- protocol/server: Insert dummy clnt-lk-version to avoid upgrade failure

This is required as we check for 'clnt-lk-version' in SETVOLUME callback
with older clients in place against newer servers. Change is similar to
what we have done via https://review.gluster.org/#/c/19560/.

Change-Id: If333c20cf9503f40687ec926c44c7e50222c05b5
BUG: 1544699
Signed-off-by: Anoop C S <anoopcs>

Comment 5 Worker Ant 2018-03-09 21:55:07 UTC

REVIEW: https://review.gluster.org/19690 (protocol: Added iatt conversion to older format) posted (#1) for review on master by Shyamsundar Ranganathan

Comment 6 Shyamsundar 2018-03-09 22:28:05 UTC

There was a further bug in the upgrade process, post any server is upgraded, directory creation from clients fail with EIO, but the directories are created on the bricks.

The issue was root caused to iatt corruption.

The reason for the corruption is that, DHT requests and receives iatt information in the dictionary. This is done for calls in (f)setxattr, (f)removexattr and unlink. The server brick process encodes this into the xdata dictionary in the new iatt format, whereas the client maps this to the older iatt structure, thus mangling the actual values (as the structure definition is not backward compatible).

The patch posted at comment #5 addresses this issue, by taking action in the protocol layer, to ensure we send back the right iatt format to the client based on the protocol dialect.

The downside of this approach is that, at a future time, we cannot add an option that can force a client to speak the 3.x dialect (akin to a mount option that can force the client to use the 3.x protocol version), as the client interpretation of the structure at that point would become mangled.

Comment 7 Worker Ant 2018-03-10 18:09:25 UTC

COMMIT: https://review.gluster.org/19690 committed in master by "Raghavendra G" <rgowdapp> with a commit message- protocol: Added iatt conversion to older format

Added iatt conversion to an older format, when dealing with
older RPC versions. This enables iatt structure conformance
when dealing with older clients.

This helps fix rolling upgrade from 3.x versions to 4.0 version
of gluster by sending the right iatt in the dictionary when DHT
requests the same.

Change-Id: Ieaf925f81f8c7798a8fba1e90a59fa9dec82856c
BUG: 1544699
Signed-off-by: ShyamsundarR <srangana>

Comment 8 Shyamsundar 2018-06-20 17:59:24 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-v4.1.0, please open a new bug report.

glusterfs-v4.1.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2018-June/000102.html
[2] https://www.gluster.org/pipermail/gluster-users/