Bug 1230612
Summary: | Disperse volume : NFS and Fuse mounts hung with plain IO | |||
---|---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Bhaskarakiran <byarlaga> | |
Component: | disperse | Assignee: | Pranith Kumar K <pkarampu> | |
Status: | CLOSED ERRATA | QA Contact: | Bhaskarakiran <byarlaga> | |
Severity: | unspecified | Docs Contact: | ||
Priority: | medium | |||
Version: | rhgs-3.1 | CC: | annair, byarlaga, kparthas, mzywusko, nsathyan, pkarampu, rcyriac, rhs-bugs, skoduri, srangana, storage-qa-internal | |
Target Milestone: | --- | |||
Target Release: | RHGS 3.1.0 | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | glusterfs-3.7.1-7.el6 | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1234768 (view as bug list) | Environment: | ||
Last Closed: | 2015-07-29 05:02:05 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | 1227197 | |||
Bug Blocks: | 1202842, 1234768 |
Description
Bhaskarakiran
2015-06-11 08:57:53 UTC
correction: How reproducible: ================= 100%. rebooted the client, mounted the volume and ran IO and its hung. Will pickup these builds in a day or two and try to reproduce. Fuse mount too hung but with taking down 2 of the bricks. I have taken up the debug builds and trying to reproduce. (In reply to Bhaskarakiran from comment #6) > Fuse mount too hung but with taking down 2 of the bricks. I have taken up > the debug builds and trying to reproduce. Could you check if this issue is observed on volume type(s) other than disperse(erasure-coded) ? The hang is still seen on the fuse mount. [root@rhs-client29 ~]# mount /dev/mapper/vg_rhsclient29-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0") /dev/sda1 on /boot type ext4 (rw) /dev/mapper/vg_rhsclient29-lv_home on /home type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) transformers:/vol2 on /mnt/fuse type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072) [root@ninja ~]# gluster v status vol2 Status of volume: vol2 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick ninja:/rhs/brick1/vol2-1 49157 0 Y 2731 Brick ninja:/rhs/brick2/vol2-2 49158 0 Y 2740 Brick ninja:/rhs/brick3/vol2-3 49159 0 Y 2747 Brick ninja:/rhs/brick4/vol2-4 49160 0 Y 2754 Brick vertigo:/rhs/brick1/vol2-5 49156 0 Y 27613 Brick vertigo:/rhs/brick2/vol2-6 49157 0 Y 19504 Brick vertigo:/rhs/brick3/vol2-7 49158 0 Y 19511 Brick ninja:/rhs/brick1/vol2-8 49161 0 Y 2765 Brick ninja:/rhs/brick2/vol2-9 49162 0 Y 2770 Brick ninja:/rhs/brick3/vol2-10 49163 0 Y 2779 Brick ninja:/rhs/brick4/vol2-11 49164 0 Y 2786 Snapshot Daemon on localhost 49165 0 Y 2855 NFS Server on localhost 2049 0 Y 10459 Self-heal Daemon on localhost N/A N/A Y 10486 Snapshot Daemon on 10.70.34.56 49160 0 Y 19539 NFS Server on 10.70.34.56 2049 0 Y 27648 Self-heal Daemon on 10.70.34.56 N/A N/A Y 27670 Snapshot Daemon on transformers 49162 0 Y 12992 NFS Server on transformers 2049 0 Y 46858 Self-heal Daemon on transformers N/A N/A Y 46881 Snapshot Daemon on interstellar 49166 0 Y 14480 NFS Server on interstellar 2049 0 Y 48872 Self-heal Daemon on interstellar N/A N/A Y 48882 Task Status of Volume vol2 ------------------------------------------------------------------------------ There are no active volume tasks [root@ninja ~]# [root@ninja ~]# gluster --version glusterfs 3.7.1 built on Jun 28 2015 11:01:17 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@ninja ~]# [root@rhs-client29 ~]# rpm -qa |grep gluster glusterfs-fuse-3.7.1-6.el6rhs.x86_64 glusterfs-client-xlators-3.7.1-6.el6rhs.x86_64 glusterfs-3.7.1-6.el6rhs.x86_64 glusterfs-api-3.7.1-6.el6rhs.x86_64 glusterfs-libs-3.7.1-6.el6rhs.x86_64 [root@rhs-client29 ~]# The fuse mount logs shows below continuously though the volume is up. [2015-06-29 12:21:23.253607] W [MSGID: 122002] [ec-common.c:122:ec_heal_report] 0-vol2-disperse-0: Heal failed [Input/output error] [2015-06-29 12:21:23.253934] W [rpc-clnt.c:1571:rpc_clnt_submit] 0-vol2-client-0: failed to submit rpc-request (XID: 0x5fab0 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (vol2-client-0) [2015-06-29 12:21:23.253972] W [MSGID: 114031] [client-rpc-fops.c:2971:client3_3_lookup_cbk] 0-vol2-client-0: remote operation failed. Path: /dirs./dir.31618 (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected] [2015-06-29 12:21:23.254944] W [MSGID: 122053] [ec-common.c:166:ec_check_status] 0-vol2-disperse-0: Operation failed on some subvolumes (up=7FF, mask=7FF, remaining=0, good=7EE, bad=11) Is this suppose to work by disabling client side heal? Has run IO for sufficient time and didn't see the hangs with client side heal disabled. Moving this bug to fixed. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1495.html |