Bug 1269557 - FUSE clients in a container environment hang and do not recover post losing connections to all bricks
Summary: FUSE clients in a container environment hang and do not recover post losing c...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: core
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: RHGS 3.1.2
Assignee: Bug Updates Notification Mailing List
QA Contact: Shruti Sampat
URL:
Whiteboard:
Depends On: 1273387 1276550
Blocks: 1260783
TreeView+ depends on / blocked
 
Reported: 2015-10-07 14:34 UTC by Shyamsundar
Modified: 2023-09-14 03:06 UTC (History)
17 users (show)

Fixed In Version: glusterfs-3.7.5-6
Doc Type: Bug Fix
Doc Text:
Previously, inode or entry invalidation requests to FUSE by gluster caused a hang when FUSE had outstanding operations on that inode/entry. Additionally the cause of the hang is due to the gluster epoll threads blocked, attempting to notify FUSE, and consequently not processing the outstanding requests. Due to this, the FUSE mount point appeared unresponsive or returned "not connected" errors for a period of time. The client also loses connections to all gluster brick processes, with ping timeout errors in the gluster logs. With this fix, the epoll threads do not block on invalidation requests to FUSE, and hence break the deadlock that manifests currently. The FUSE mount hang or unwarranted connection errors are not observed.
Clone Of:
: 1273387 (view as bug list)
Environment:
Last Closed: 2016-03-01 05:37:40 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0193 0 normal SHIPPED_LIVE Red Hat Gluster Storage 3.1 update 2 2016-03-01 10:20:36 UTC

Description Shyamsundar 2015-10-07 14:34:50 UTC
Description of problem:
FUSE clients ping-timeout on all connections to bricks in the cluster, and post this keep returning ENOTCONN on operations or commands from a shell on the mount hangs.

Also, the FUSE clients do not seem to be reconnecting to the bricks on detecting this failure, i.e there are no connections seen with 'ss' or 'netstat' to the said bricks from the container.

Further, the container uses host based networking and is based of Docker and there are intermittent network interface loss on the container.

A newer FUSE mount against the same volume from the same client container, works clean.

Version-Release number of selected component (if applicable):
RHGS 3.1 bits running on top of a CentOS container, running on CoreOS base.

How reproducible:
Quite reproducible, takes some time (say ~2-3 hours) to reproduce, but occurs in the cu environment quite consistently.

Steps to Reproduce:
At the cu setup, the test is as simple as having the environment setup as above, and running fio on the FUSE mount, post some duration of time the problem manifests itself.

<<These steps seem to be the relevant ones at present, other than recreating the cu environment in total in house>>
NOTE: These steps are not tested.
- bricks on different containers across different nodes (could be even the same node)
- client on a different container in a different node
- Some client operations ongoing (simple operations are fine, say a simple fio workload)
- client container needs to *lose* its network and react with a ping timeout
  - In the cu environment we see from the logs that after the first ping timeout, the client recovers after 30 minutes, and then very soon after this another ping timeout occurs, and the client does not recover post this point.
  - How to make this network *loss* on the client could be through iptables, blocking all traffic to and from the client, till the ping timeout is observed in the logs against all bricks, and then disabling the iptables filter, and repeating this exercise a few times. 

Actual results:
Post some time the FUSE mount hangs and does not show any evidence of connections being made to the bricks.

Expected results:
FUSE client should recover from the ping-timeout and the network failure, when the network is up and running again, and reconnect to the bricks.

Comment 16 Raghavendra G 2015-11-02 06:26:48 UTC
Ignore previous link. Correct one is below.

https://code.engineering.redhat.com/gerrit/#/c/60584/1

Comment 20 Vivek Agarwal 2015-11-03 09:42:33 UTC
Can we get the doc text for this?

Comment 21 Shyamsundar 2015-11-03 14:15:43 UTC
Updated the doc text as requested.

Comment 23 Shruti Sampat 2016-02-09 05:37:52 UTC
Verified as fixed in RHGS container image rhgs-server-rhel7:3.1.2-6 (having glusterfs-3.7.5-16). Repeated the steps described in comment #0. Fuse client hang is no longer observed.

Comment 30 errata-xmlrpc 2016-03-01 05:37:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0193.html

Comment 32 Red Hat Bugzilla 2023-09-14 03:06:23 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.