Bug 1476563

Summary:	[Stress] : Ganesha v4 mounts timed out during MTSH
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Ambarish <asoman>
Component:	nfs-ganesha	Assignee:	Kaleb KEITHLEY <kkeithle>
Status:	CLOSED ERRATA	QA Contact:	Manisha Saini <msaini>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.3	CC:	amukherj, bturner, dang, ffilz, jthottan, kkeithle, mbenjamin, msaini, rhinduja, rhs-bugs, sheggodu, skoduri, storage-qa-internal
Target Milestone:	---
Target Release:	RHGS 3.4.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	nfs-ganesha-2.5.4-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-09-04 06:53:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1503134

Description Ambarish 2017-07-30 11:05:26 UTC

Description of problem:
-----------------------

4 Node cluster,with MTSH in progress and continous IO from 3 mounts.


v4 mount timed out on one of my clients :

[root@gqac007 /]# mount -t nfs -o vers=4.0 192.168.97.161:/testvol /gluster-mount/ -v
mount.nfs: timeout set for Sun Jul 30 06:36:57 2017
mount.nfs: trying text-based options 'vers=4.0,addr=192.168.97.161,clientaddr=192.168.97.147'
mount.nfs: mount(2): Connection timed out
mount.nfs: Connection timed out
[root@gqac007 /]# 

v3 succeeded though :

[root@gqac007 /]# mount -t nfs -o vers=3 192.168.97.161:/testvol /gluster-mount/ -v
mount.nfs: timeout set for Sun Jul 30 06:59:38 2017
mount.nfs: trying text-based options 'vers=3,addr=192.168.97.161'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 192.168.97.161 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 192.168.97.161 prog 100005 vers 3 prot UDP port 20048
mount.nfs: portmap query retrying: RPC: Timed out
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 192.168.97.161 prog 100005 vers 3 prot TCP port 20048
[root@gqac007 /]# 


Mounts from other servers succeeded as well :

[root@gqac007 /]# mount -t nfs -o vers=4.0 192.168.97.162:/testvol /gluster-mount/ -v
mount.nfs: timeout set for Sun Jul 30 07:02:06 2017
mount.nfs: trying text-based options 'vers=4.0,addr=192.168.97.162,clientaddr=192.168.97.147'
[root@gqac007 /]# 


Version-Release number of selected component (if applicable):
---------------------------------------------------------------

nfs-ganesha-gluster-2.4.4-16.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-36.el7rhgs.x86_64



How reproducible:
----------------

1/1


Additional info:
----------------
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 41c5aa32-ec60-4591-ae6d-f93a0b13b47c
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gqas013.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0
Brick2: gqas005.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1
Brick3: gqas006.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2
Brick4: gqas008.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3
Options Reconfigured:
cluster.shd-wait-qlength: 655536
cluster.shd-max-threads: 64
client.event-threads: 4
server.event-threads: 4
cluster.lookup-optimize: on
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
ganesha.enable: on
features.cache-invalidation: on
server.allow-insecure: on
performance.stat-prefetch: off
transport.address-family: inet
nfs.disable: on
nfs-ganesha: enable
cluster.enable-shared-storage: enable

Comment 4 Ambarish 2017-07-30 11:17:28 UTC

Each time I try to mount , I see this in messages :

Jul 30 07:16:53 gqas013 lrmd[19750]:  notice: gqas013.sbu.lab.eng.bos.redhat.com-nfs_unblock_monitor_10000:28728:stderr [ 0+1 records in ]
Jul 30 07:16:53 gqas013 lrmd[19750]:  notice: gqas013.sbu.lab.eng.bos.redhat.com-nfs_unblock_monitor_10000:28728:stderr [ 0+1 records out ]
Jul 30 07:16:53 gqas013 lrmd[19750]:  notice: gqas013.sbu.lab.eng.bos.redhat.com-nfs_unblock_monitor_10000:28728:stderr [ 390 bytes (390 B) copied, 0.00407114 s, 95.8 kB/s ]

Comment 5 Ambarish 2017-07-30 12:50:29 UTC

I have gluster v heal info running periodially on that server too,to see when heal completes.

Comment 9 Frank Filz 2017-07-31 20:52:00 UTC

what are the other threads doing?

I assume this is a deadlock, it could be fixed by patches in 2.5 already.

Comment 10 Frank Filz 2017-07-31 20:55:08 UTC

Is both this bug and https://bugzilla.redhat.com/show_bug.cgi?id=1476559 occurring at the same time? If so, I think it's one deadlock bug...

Comment 11 Jiffin 2017-08-02 05:17:10 UTC

(In reply to Frank Filz from comment #10)
> Is both this bug and https://bugzilla.redhat.com/show_bug.cgi?id=1476559
> occurring at the same time? If so, I think it's one deadlock bug...

Yeah both bug is occurring on same server for two different clients

Comment 16 Kaleb KEITHLEY 2017-10-05 11:25:57 UTC

POST with rebase to nfs-ganesha-2.5.x

Comment 22 errata-xmlrpc 2018-09-04 06:53:35 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2610