Bug 1457179

Summary:	[Ganesha] : Grace period is not being adhered to on RHEL 7.4; Clients continue running IO even during grace.
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Ambarish <asoman>
Component:	common-ha	Assignee:	Kaleb KEITHLEY <kkeithle>
Status:	CLOSED ERRATA	QA Contact:	Ambarish <asoman>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.3	CC:	amukherj, bturner, dang, jthottan, kgaillot, kkeithle, mbenjamin, msaini, nchilaka, rhinduja, rhs-bugs, skoduri, storage-qa-internal
Target Milestone:	---
Target Release:	RHGS 3.3.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	glusterfs-3.8.4-29	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1461019 1465601 (view as bug list)		Environment:
Last Closed:	2017-09-21 04:45:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1417151, 1461019, 1465601

Description Ambarish 2017-05-31 09:23:07 UTC

Description of problem:
-----------------------

4 node cluster,4 clients accessing the export via v4.

Kill NFS-Ganesha on any node.

Grace period should be entered and any and all IO should halt for 90 seconds.

I observed that other clients continued running their IO,which is unusual.

This may be a regression introduced in latest pacemaker/corosync bits. 


Version-Release number of selected component (if applicable):
-------------------------------------------------------------

[root@gqas013 tmp]# rpm -qa|grep ganesha
glusterfs-ganesha-3.8.4-26.el7rhgs.x86_64
nfs-ganesha-gluster-2.4.4-6.el7rhgs.x86_64
nfs-ganesha-debuginfo-2.4.4-6.el7rhgs.x86_64
nfs-ganesha-2.4.4-6.el7rhgs.x86_64
[root@gqas013 tmp]# 
[root@gqas013 tmp]# 
[root@gqas013 tmp]# rpm -qa|grep libnti
libntirpc-1.4.3-1.el7rhgs.x86_64
libntirpc-devel-1.4.3-1.el7rhgs.x86_64
[root@gqas013 tmp]# 

[root@gqas013 tmp]# rpm -qa|grep pacem
pacemaker-cluster-libs-1.1.16-10.el7.x86_64
pacemaker-cli-1.1.16-10.el7.x86_64
pacemaker-1.1.16-10.el7.x86_64
pacemaker-libs-1.1.16-10.el7.x86_64
[root@gqas013 tmp]# 

[root@gqas013 tmp]# rpm -qa|grep coros
corosynclib-2.4.0-9.el7.x86_64
corosync-2.4.0-9.el7.x86_64
[root@gqas013 tmp]# 

[root@gqas013 tmp]# rpm -qa|grep resource-ag
resource-agents-3.9.5-100.el7.x86_64

[root@gqas013 tmp]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 Beta (Maipo)
[root@gqas013 tmp]# 


How reproducible:
-----------------

Easily,every time.

Steps to Reproduce:
-------------------

1. Run IO continuously and kill Ganesha
2. Observe IO on mount


Actual results:
---------------

IO continues inspite of the grace period.

Expected results:
-----------------

Writes should be stopped for 90 secs when grace is entered.

Comment 3 Manisha Saini 2017-06-08 08:44:24 UTC

qa_acking as discussed in triage meet

Comment 10 Kaleb KEITHLEY 2017-06-12 22:04:21 UTC

Indeed changing the resource create from

  `pcs resource create nfs-grace ocf:heartbeat:ganesha_grace --clone meta
 notify=true`

to

  `pcs resource create nfs-grace ocf:heartbeat:ganesha_grace --clone notify=true`

restores the original behavior seen in RHEL 7.3 and earlier.

Please check with the pcs devs and we will confirm that the changed command also works correctly on RHEL 7.3.

Comment 11 Atin Mukherjee 2017-06-13 11:52:44 UTC

upstream patch : https://review.gluster.org/17534

Comment 12 Ken Gaillot 2017-06-13 14:56:50 UTC

I confirmed with the pcs developers that this was a behavioral change from 7.3 to 7.4, as the fix for Bug 1395226.

The changed command in Comment 10 has the correct syntax, which will work on both 7.3 and 7.4.

Comment 13 Atin Mukherjee 2017-06-16 04:47:37 UTC

downstream patch : https://code.engineering.redhat.com/gerrit/#/c/108924/

Comment 15 Ambarish 2017-06-29 15:09:12 UTC

Verified on nfs-ganesha-2.4.4-10.el7rhgs.x86_64,glusterfs-ganesha-3.8.4-31.el7rhgs.x86_64

Comment 17 errata-xmlrpc 2017-09-21 04:45:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2774

Comment 18 errata-xmlrpc 2017-09-21 04:58:49 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2774