Bug 1273387 - FUSE clients in a container environment hang and do not recover post losing connections to all bricks
FUSE clients in a container environment hang and do not recover post losing c...
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: fuse (Show other bugs)
mainline
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: bugs@gluster.org
: ZStream
Depends On:
Blocks: 1308899 1269557 1276550
  Show dependency treegraph
 
Reported: 2015-10-20 06:55 EDT by Raghavendra G
Modified: 2016-06-16 09:40 EDT (History)
10 users (show)

See Also:
Fixed In Version: glusterfs-3.8rc2
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1269557
: 1276550 (view as bug list)
Environment:
Last Closed: 2016-06-16 09:40:47 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Comment 1 Vijay Bellur 2015-10-20 07:09:00 EDT
REVIEW: http://review.gluster.org/12402 (mount/fuse: use a queue instead of pipe to communicate with thread doing inode/entry invalidations.) posted (#1) for review on master by Raghavendra G (rgowdapp@redhat.com)
Comment 2 Vijay Bellur 2015-10-26 06:43:11 EDT
REVIEW: http://review.gluster.org/12402 (mount/fuse: use a queue instead of pipe to communicate with thread doing inode/entry invalidations.) posted (#2) for review on master by Raghavendra G (rgowdapp@redhat.com)
Comment 3 Vijay Bellur 2015-10-26 06:45:10 EDT
COMMIT: http://review.gluster.org/12402 committed in master by Raghavendra G (rgowdapp@redhat.com) 
------
commit 4f65f894ab1c19618383ba212dc0f0df48675823
Author: Raghavendra G <rgowdapp@redhat.com>
Date:   Tue Oct 20 16:27:14 2015 +0530

    mount/fuse: use a queue instead of pipe to communicate with thread
    doing inode/entry invalidations.
    
    Writing to pipe can block if pipe is full. This can lead to deadlocks
    in some situations. Consider following situation:
    
    1. Kernel sends a write on an inode. Client is waiting for a response
       to write from brick.
    2. A lookup happens on behalf of different application/thread on the
       same inode. In response, mdc tries to invalidate the inode.
    3. fuse_invalidate_inode is called. It writes a invalidation request
       to pipe. Another thread which reads from this pipe writes the
       request to /dev/fuse. The invalidate code in fuse-kernel-module,
       tries to acquire lock on all pages for the inode and is blocked as
       a write is in progress on same inode (step 1)
    4. Now, poller thread is blocked in invalidate notification and cannot
       receive any messages from same socket (on which lookup response
       came). But client is expecting a response for write from same
       socket (again step1) and we've a deadlock.
    
    The deadlock can be solved in two ways:
    1. Use a queue (and a conditional variable for notifications) to pass
       invalidation requests from poller to invalidate thread. This is a
       variant of using non-blocking pipe, but doesn't have any limit on the
       amount of data (worst case we run out of memory and error out).
    
    2. Allow events from sockets, immediately after we read one
       rpc-msg. Currently we disallow events till that rpc-msg is read from
       socket, processed and handled by higher layers. That way we won't run
       into these kind of issues. Also, it'll increase parallelism in way of
       reading from sockets.
    
    This patch implements solution 1 above.
    
    Change-Id: I8e8199fd7f4da9eab46a719d9292f35c039967e1
    BUG: 1273387
    Signed-off-by: Raghavendra G <rgowdapp@redhat.com>
    Reviewed-on: http://review.gluster.org/12402
Comment 4 Vijay Bellur 2015-10-30 02:04:27 EDT
REVIEW: http://review.gluster.org/12466 (mount/fuse: use a queue instead of pipe to communicate with thread doing inode/entry invalidations.) posted (#1) for review on release-3.7 by Raghavendra G (rgowdapp@redhat.com)
Comment 5 Niels de Vos 2016-06-16 09:40:47 EDT
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Note You need to log in before you can comment on or make changes to this bug.