Bug 999944
Summary: | `Transport end point not connected errors' upon creating files on 2.1 clients mounting U5 volume | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Sachidananda Urs <surs> | ||||||||
Component: | fuse | Assignee: | Amar Tumballi <amarts> | ||||||||
Status: | CLOSED ERRATA | QA Contact: | Sachidananda Urs <surs> | ||||||||
Severity: | urgent | Docs Contact: | |||||||||
Priority: | urgent | ||||||||||
Version: | 2.1 | CC: | aavati, csaba, jclift, rhs-bugs, sdharane, shaines, vraman | ||||||||
Target Milestone: | --- | ||||||||||
Target Release: | --- | ||||||||||
Hardware: | x86_64 | ||||||||||
OS: | Linux | ||||||||||
Whiteboard: | |||||||||||
Fixed In Version: | glusterfs-3.4.0.32rhs-1 | Doc Type: | Bug Fix | ||||||||
Doc Text: | Story Points: | --- | |||||||||
Clone Of: | Environment: | ||||||||||
Last Closed: | 2013-09-23 22:38:34 UTC | Type: | Bug | ||||||||
Regression: | --- | Mount Type: | --- | ||||||||
Documentation: | --- | CRM: | |||||||||
Verified Versions: | Category: | --- | |||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||
Embargoed: | |||||||||||
Attachments: |
|
Description
Sachidananda Urs
2013-08-22 12:11:06 UTC
As a data point, please include the /var/log/gluster* and /var/lib/gluster* files (tarred) with this report, from the client with the problem and the gluster volume servers. This will let people look through the configuration, and any errors in the log files. Jclift, I'm in the process of attaching sosreports. Attach is failing, give me a moment. Client logs that could not mount the volume: mount just hangs: [2013-08-19 05:46:42.100949] I [glusterfsd.c:1970:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0.20rhs (/usr/sbin/glusterfs --volfile-id=/testvol --volfile-server=storage-qe10.lab.eng.rdu2.redhat.com /tmp) [2013-08-19 05:46:42.103896] I [socket.c:3487:socket_init] 0-glusterfs: SSL support is NOT enabled [2013-08-19 05:46:42.103940] I [socket.c:3502:socket_init] 0-glusterfs: using system polling thread [2013-08-19 05:55:45.582236] E [glusterfsd-mgmt.c:1644:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:/testvol) [2013-08-19 05:55:45.585232] W [glusterfsd.c:1062:cleanup_and_exit] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x148) [0x7f61a276bf38] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5) [0x7f61a276af35] (-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x3b8) [0x40bbb8]))) 0-: received signum (0), shutting down [2013-08-19 05:55:45.585258] I [fuse-bridge.c:6289:fini] 0-fuse: Unmounting '/tmp'. [2013-08-19 05:55:45.585280] W [glusterfsd.c:1062:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x7f61a1ce690d] (-->/lib64/libpthread.so.0(+0x7851) [0x7f61a2332851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052dd]))) 0-: received signum (15), shutting down [2013-08-19 05:55:45.596353] W [glusterfsd.c:1062:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x7f61a1ce690d] (-->/lib64/libpthread.so.0(+0x7851) [0x7f61a2332851] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x4052dd]))) 0-: received signum (15), shutting down Created attachment 789174 [details]
sosreports for nfs client hang
Created attachment 789175 [details]
sosreports for nfs client hang - 2
Volume info: Volume Name: hybrid Type: Distribute Volume ID: a6cc8192-41c7-441b-8c7c-1e4bd4289f3b Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 10.70.37.155:/data/re-0 Brick2: 10.70.37.193:/data/re-0 Options Reconfigured: diagnostics.client-log-level: DEBUG ==================================================== Attaching sosreports for the NFS hang. Trying to reproduce the ENOTCONN error, will update the bug as I progress. Volume for distributed-replicate setup: Volume Name: hybrid-1 Type: Replicate Volume ID: 2b9b2e81-189c-47e0-bc42-b79d3705d3a5 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 10.70.37.155:/data/re-1 Brick2: 10.70.37.193:/data/re-1 ================================================== In distributed-replicate setup, mount succeeds and data creation fails. [root@bob-the-minion hy-1]# cp /tmp/linux-3.10.3.tar.xz . cp: writing `./linux-3.10.3.tar.xz': Transport endpoint is not connected cp: closing `./linux-3.10.3.tar.xz': Transport endpoint is not connected ============================ [2013-08-22 14:41:34.077513] I [fuse-bridge.c:4706:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13 [2013-08-22 14:41:34.078794] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-hybrid-1 -replicate-0: added root inode [2013-08-22 14:41:48.714858] W [client-rpc-fops.c:866:client3_3_writev_cbk] 0-hybrid-1-client-1: remote operation failed: Transport endpoint is not connected [2013-08-22 14:41:48.717916] W [client-rpc-fops.c:866:client3_3_writev_cbk] 0-hybrid-1-client-0: remote operation failed: Transport endpoint is not connected [2013-08-22 14:41:48.718036] W [fuse-bridge.c:2695:fuse_writev_cbk] 0-glusterfs-fuse: 32: WRITE => -1 (Transport endpoint is not connected) [2013-08-22 14:41:48.718721] W [client-rpc-fops.c:866:client3_3_writev_cbk] 0-hybrid-1-client-1: remote operation failed: Transport endpoint is not connected [2013-08-22 14:41:48.721054] W [client-rpc-fops.c:866:client3_3_writev_cbk] 0-hybrid-1-client-0: remote operation failed: Transport endpoint is not connected [2013-08-22 14:41:48.721974] W [client-rpc-fops.c:866:client3_3_writev_cbk] 0-hybrid-1-client-1: remote operation failed: Transport endpoint is not connected [2013-08-22 14:41:48.724112] W [client-rpc-fops.c:866:client3_3_writev_cbk] 0-hybrid-1-client-0: remote operation failed: Transport endpoint is not connected Created attachment 789192 [details]
Replicate logs
<snip> Internal Whiteboard: Big Bend → Big Bend U1 Flags: rhs-2.1.0+ → rhs-2.1.z? </snip> Removing the blocker flag as it seem to be targetted for Update1 now. Most probable fix posted upstream for review @ http://review.gluster.org/#/c/5803/ Verified mounting U5 volume with 2.1 clients. Both NFS and Fuse transports tested, working fine. Verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1262.html Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1262.html |