Bug 1632960

Description of problem:

I have had this issue come up twice in RHEL 7.5 in the last three months. Both events occurred when NFS shares failed during normal operation. The NFS failures were from shared folders on two different RHEL servers.

In the first event, the problem occurred after an exported NFS3 share on a local el6.8 server was removed from service after the server was decommissioned but an entry for the share was left in the /etc/fstab file on the primary node. After some time the gvfsd-trash process on the primary node continuously respawned for multiple users and eventually crashed the node. The entry in the fstab file was removed and the server restarted to fix the issue.

In the second event, the issue occurred after two NFS3 shares from a local el5.8 node stopped responding on the primary node. The gvfsd-trash process on the primary node continuously respawned for a single user and eventually crashed the node. The node was not recoverable and had to be restarted. The restart took around 15-20 minutes probably due to NFS timeouts. The el5.8 server was not restarted.

Both of these issues were limited to the primary node on a cluster of 12 servers (eight running el7.5 and four running el6.8). The same shared folders on the other nodes continued to work as normal.

There were no errors logged on either the el7.5 or el5.8 servers relating to the second NFS share failures but any type of access to the shares from the primary el7.5 node seemed to hang (probably due to NFS timeouts).

The second event also broke the root login. It appeared to hang during the login process; however, a ctrl-c produces a bash login prompt but the shell was broken and kept throwing vte errors until I manually sourced the .bashrc file in /root. Subsequent logins also failed the same way. The actual error displayed from the shell was: 

bash: __vte_prompt_command: command not found.

An additional NFS shared folder mapped from Dell EMC Isilon storage on the main network also continued to work across all nodes including the primary node, i.e., only the two local el5.8 shares failed on the primary node. Note also that the user home folders are located on the Isilon storage that continued to work ok during both events.

The cause of why the NFS shares mapped from the el5.8 server stopped working in the second event is unknown at this point but seemed to be an NFS failure on the el7.5 node rather than any issue on the el5.8 server since the problem was limited to the primary node. Prior to el7.5 we have had no issues with the NFS shared folders on any of the nodes.

Version-Release number of selected component (if applicable):

The gvfs components on the el7 server are:
gvfs-1.30.4-5.el7.x86_64
gvfs-fuse-1.30.4-5.el7.x86_64
gvfs-gphoto2-1.30.4-5.el7.x86_64
gvfs-afc-1.30.4-5.el7.x86_64
gvfs-client-1.30.4-5.el7.x86_64
gvfs-mtp-1.30.4-5.el7.x86_64
gvfs-smb-1.30.4-5.el7.x86_64
gvfs-goa-1.30.4-5.el7.x86_64
gvfs-archive-1.30.4-5.el7.x86_64
gvfs-afp-1.30.4-5.el7.x86_64

The NFS components on the EL5 server are:
nfs-utils-lib-1.0.8-7.9.el5
nfs-utils-1.0.9-60.el5

The server VNC components installed are:
tigervnc-server-1.8.0-5.el7.x86_64
tigervnc-icons-1.8.0-5.el7.noarch
tigervnc-license-1.8.0-5.el7.noarch
tigervnc-1.8.0-5.el7.x86_64
gvnc-0.7.0-3.el7.x86_64
tigervnc-server-minimal-1.8.0-5.el7.x86_64
gtk-vnc2-0.7.0-3.el7.x86_64

How reproducible:
Not reproducible at this point.

Steps to Reproduce:
1. Unable to test on the production system
2.
3.

Actual results:


Expected results:


Additional info:

After each failure, the primary el7.5 node was not recoverable and had to be restarted. 

The el7.5 nodes are all all VMWare-based running on an ESXi 6.5 environment. CPU and memory are mapped on a one-to-one basis with the physical hardware (one VM per ESXi server) so there is no over-subscription of resources. 

The remaining el6.5 nodes as also all based on a master VM and resources assigned as with the el7.5 nodes. The el5.8 server is not part of the cluster environment and only provides the legacy VNX shares (40TB of data).

The users connect to the primary node via Putty, set up a TurboVNC client 2.1/2.2 session then work from there on the primary node. The nodes all have the TigerVNC server components installed.

The primary node also runs Ganglia, Torque PBS 6.1.2, Open MPI and MPICH3 servers. There are some local apps installed on each of the nodes as well as on the Isilon shared folder. The el5.8 shared folders are use for project data only and are on Dell EMC VNX backend hardware.

cat /etc/redhat-release: 
Red Hat Enterprise Linux Server release 7.5 (Maipo)

uname -a
Linux u3rb09s1 3.10.0-862.11.6.el7.x86_64 #1 SMP Fri Aug 10 16:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux