1133861 – [SNAPSHOT]: snapshoted bricks failed to start

Bug 1133861 - [SNAPSHOT]: snapshoted bricks failed to start

Summary: [SNAPSHOT]: snapshoted bricks failed to start

Keywords:
Status:	CLOSED WORKSFORME
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	snapshot
Sub Component:
Version:	rhgs-3.0
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	urgent
Target Milestone:	---
Target Release:	---
Assignee:	Bug Updates Notification Mailing List
QA Contact:	Rahul Hinduja
Docs Contact:
URL:
Whiteboard:	SNAPSHOT
Depends On:
Blocks:	1087818
TreeView+	depends on / blocked

Reported:	2014-08-26 10:20 UTC by Rahul Hinduja
Modified:	2018-04-07 19:29 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	Known Issue
Doc Text:	New snap bricks fails to start if the total snapshot brick count in a node goes beyond 1K. Workaround (if any): Deactivate unused snapshots.
Clone Of:
Environment:
Last Closed:	2018-04-07 19:29:38 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Rahul Hinduja 2014-08-26 10:20:50 UTC

Description of problem:
=======================

Created 4 node cluster and 4 volumes. While creating snapshots in a loop{1..256} for all the volumes simultaneously, observed that the snapshoted bricks fails to come online.

Brick log snippet:
==================

[2014-08-25 13:31:57.911090] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.6.0.27 (args: /usr/sbin/glusterfsd -s inception.lab.eng.blr.redhat.com --volfile-id /snaps/c249/86e1633c6c4c476fb4823dcbe2fd4e4e.inception.lab.eng.blr.redhat.com.var-run-gluster-snaps-86e1633c6c4c476fb4823dcbe2fd4e4e-brick1-b3 -p /var/lib/glusterd/snaps/c249/86e1633c6c4c476fb4823dcbe2fd4e4e/run/inception.lab.eng.blr.redhat.com-var-run-gluster-snaps-86e1633c6c4c476fb4823dcbe2fd4e4e-brick1-b3.pid -S /var/run/e7ceb620651d6583725dc67cde7b51d1.socket --brick-name /var/run/gluster/snaps/86e1633c6c4c476fb4823dcbe2fd4e4e/brick1/b3 -l /var/log/glusterfs/bricks/var-run-gluster-snaps-86e1633c6c4c476fb4823dcbe2fd4e4e-brick1-b3.log --xlator-option *-posix.glusterd-uuid=456cfdb5-b488-4354-9aa2-e2d0cee58454 --brick-port 50169 --xlator-option 86e1633c6c4c476fb4823dcbe2fd4e4e-server.listen-port=50169)
[2014-08-25 13:31:58.080444] W [socket.c:529:__socket_rwv] 0-glusterfs: readv on 10.70.34.50:24007 failed (No data available)
[2014-08-25 13:31:58.080669] E [rpc-clnt.c:362:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x3ea160fe6d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91) [0x3ea160f8a1] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3ea160f7ee]))) 0-glusterfs: forced unwinding frame type(GlusterFS Handshake) op(GETSPEC(2)) called at 2014-08-25 13:31:57.917490 (xid=0x1)
[2014-08-25 13:31:58.080689] E [glusterfsd-mgmt.c:1596:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file (key:/snaps/c249/86e1633c6c4c476fb4823dcbe2fd4e4e.inception.lab.eng.blr.redhat.com.var-run-gluster-snaps-86e1633c6c4c476fb4823dcbe2fd4e4e-brick1-b3)
[2014-08-25 13:31:58.080720] W [glusterfsd.c:1182:cleanup_and_exit] (--> 0-: received signum (0), shutting down
[2014-08-25 13:31:58.083896] I [socket.c:3132:socket_submit_request] 0-glusterfs: not connected (priv->connected = 0)
[2014-08-25 13:31:58.083915] W [rpc-clnt.c:1562:rpc_clnt_submit] 0-glusterfs: failed to submit rpc-request (XID: 0x2 Program: Gluster Portmap, ProgVers: 1, Proc: 5) to rpc-transport (glusterfs)



Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.6.0.27-1.el6rhs.x86_64


How reproducible:
=================
2 in 2 attempt


Steps to Reproduce:
===================
1. Create 4 node cluster
2. Create 4 volumes (vol0 to vol3)
3. Mount the volumes to client (fuse and nfs)
4. Start creating some data from fuse and nfs mount
5. Start creating snapshots in a loop for volumes from different nodes in cluster use following cli:

Node1:
for i in {1..256}; do time gluster snapshot create a$i vol0; done

Node2:
for i in {1..256}; do time gluster snapshot create b$i vol1; done

Node3:
for i in {1..256}; do time gluster snapshot create b$i vol2; done

Node4:
for i in {1..256}; do time gluster snapshot create d$i vol3; done


Actual results:
===============

Brick process failed to start

Expected results:
=================

Brick process should be up and runnin

Additional info:
================

stopped the glusterd and restarted from all the nodes in cluster, the subsequent gluster cli hungs for some time with following errors:


[2014-08-26 10:13:00.640132] I [MSGID: 106006] [glusterd-handler.c:4280:__glusterd_nodesvc_rpc_notify] 0-management: glustershd has disconnected from glusterd.
[2014-08-26 10:13:00.649245] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/5722edcc12424afe9bb5ca8ba8503ad1/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:13:00.654635] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/23856d58546a43f6a1b6ff2d28750477/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.507355] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/38f73153cf71e9bb5f4a71671f1e365c.socket failed (Invalid argument)
[2014-08-26 10:14:19.510330] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/29d8e9e083c34551a99c9e2f2621fea4/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.513275] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/4bcdd4cbbb4effbaab87042de2b695b0.socket failed (Invalid argument)
[2014-08-26 10:14:19.516054] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/81014b0af2184cc1a5b626790b96376e/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.518729] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/869a38aa51049452d3cda4936aa8e649.socket failed (Invalid argument)
[2014-08-26 10:14:19.522024] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/d478ccc1d178451a95784c9e69d0577f/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.524873] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/cb188e9bc3f242e6f0173f74eca75cac.socket failed (Invalid argument)
[2014-08-26 10:14:19.527942] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/8bb9b8b8948d45b1b75aad91d5fa6636/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.530959] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/c3df211a55ae2442acb7b5faf9afd498.socket failed (Invalid argument)
[2014-08-26 10:14:19.533674] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/4eee2f546621442aa1b87bf98ac71f5b/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.536790] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/3fc516462992e8ded92285ac336fd5bf.socket failed (Invalid argument)
[2014-08-26 10:14:19.539766] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/7b43a248b46c4d48ab54cafa9227d0b5/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.542487] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/e13ecc2b4182992922b40baa7feb0f99.socket failed (Invalid argument)
[2014-08-26 10:14:19.545322] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/0ffff4902ef844759e37a64452835bd5/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.548246] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/29cc060bed609e48692e0035ef44b973.socket failed (Invalid argument)
[2014-08-26 10:14:19.550953] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/f8aa81cdf24247feab4d3a767e833f00/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.553659] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/e7ceb620651d6583725dc67cde7b51d1.socket failed (Invalid argument)
[2014-08-26 10:14:19.556221] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/86e1633c6c4c476fb4823dcbe2fd4e4e/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.559131] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/07c2cc84966aa322ecd188e77d342f35.socket failed (Invalid argument)
[2014-08-26 10:14:19.561723] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/ddab0270c79847daa15c04b1f6f8792c/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.564491] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/ecbd2ad1325242313e1b0f0fdc213e9c.socket failed (Invalid argument)
[2014-08-26 10:14:19.567013] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/51ea15e69d0d4469a2a1e9860e308e81/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.569724] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/f5d36b518e4117492d8f4b43e4f7c5a1.socket failed (Invalid argument)
[2014-08-26 10:14:19.572450] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/3003d0d4f65b46ef9ab503ebd8929a87/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.575265] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/53352de07207732d20971673af76bd72.socket failed (Invalid argument)
[2014-08-26 10:14:19.577992] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/0226947e5a7b49fb8ad3e5d0bf4d660e/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.580778] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/eae23c7c6ebbfd4ff7679a04cc56eacb.socket failed (Invalid argument)
[2014-08-26 10:14:19.583518] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/b735dc4facd34abe859ae392262906cb/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.586288] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/f039bb3e1b4741cbacabc908b0ce9643.socket failed (Invalid argument)
[2014-08-26 10:14:19.588582] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/4352089bff6a46feb8bfffa00be4a30d/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.591020] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/4b7b307e1d0e410e87c2ae109de29643.socket failed (Invalid argument)
[2014-08-26 10:14:19.593362] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/923d5302b545476d90b3ee60cc61d334/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.595771] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/6b9834ade79059fa90697889b02422bc.socket failed (Invalid argument)
[2014-08-26 10:14:19.598050] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/e7e0b3096b9149a99afc5c28ec1b88af/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.600370] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/a7efc60a0f3a791933e4731654932d6f.socket failed (Invalid argument)
[2014-08-26 10:14:19.602657] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/322c27c6087c4073bd35be4d45d4a9e5/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.605218] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/3c7d45c7bf89a9a4371f0e532a3770ed.socket failed (Invalid argument)
[2014-08-26 10:14:19.607826] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/260f1938dac641c4ac6ca0962bf9de66/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.610671] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/e7c4396af4fc6b50124c23daa0f3286f.socket failed (Invalid argument)
[2014-08-26 10:14:19.613467] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/09ce53a4b18445deb5f3ceb2c12d03f3/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.616295] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/e0c32417aed7dbd18145988372b94640.socket failed (Invalid argument)
[2014-08-26 10:14:19.618996] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/2be14e5616574fc38f2d6c960cd01c03/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.621758] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/6b35143e695a079e7f54d9ef7432bcdf.socket failed (Invalid argument)
[2014-08-26 10:14:19.624536] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/ee573786073d4c5186b73b20516d1132/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.627424] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/e08f7b5aa8151256446e1c12587d6f07.socket failed (Invalid argument)
[2014-08-26 10:14:19.630134] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/311edc38a5b54ef49c45194b28492fcc/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.632950] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/cc260accc630c111243b2e3f1d698eb9.socket failed (Invalid argument)
[2014-08-26 10:14:19.635804] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/b17aadf29d7249faac8bde6eddd9ffe7/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.638531] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/e1ae815a513bb8fa0f6ebdaf5de91280.socket failed (Invalid argument)
[2014-08-26 10:14:19.641219] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/7244f178dd2041a6b771b17ebbacc82f/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.644127] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/5db2d23273205d1e562460bb68e2d0b9.socket failed (Invalid argument)
[2014-08-26 10:14:19.646811] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/ad19ec441e5d4cbfa54b3b162fea3239/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.649618] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/c44591f201924b362866d07380717fbc.socket failed (Invalid argument)
[2014-08-26 10:14:19.652194] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/336e6c82117d4dbcbb83436277281da7/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.655125] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/1f74f7f3610bfe856fe94d18ad93abe5.socket failed (Invalid argument)
[2014-08-26 10:14:19.657714] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/5228f255b1a244eab24b67cce88cef2a/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.660522] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/028b16a3387aedd214608d3ea7e02d60.socket failed (Invalid argument)
[2014-08-26 10:14:19.663673] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/c7f93f796d9e44b4938bd88152c93244/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.663737] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/47be8069da44e747693a5f2368eb2385.socket failed (Invalid argument)
[2014-08-26 10:14:19.663757] I [MSGID: 106006] [glusterd-handler.c:4280:__glusterd_nodesvc_rpc_notify] 0-management: glustershd has disconnected from glusterd.
[2014-08-26 10:14:19.666751] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/0e687e114c697ba9cbcbb912ae4636d6.socket failed (Invalid argument)
[2014-08-26 10:14:19.669691] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/5722edcc12424afe9bb5ca8ba8503ad1/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.672577] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/2c636579d894b26bac6f2acf7b1b6e67.socket failed (Invalid argument)
[2014-08-26 10:14:19.675295] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/23856d58546a43f6a1b6ff2d28750477/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.678077] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/1294bb28ba0b462816d6301f29705fdc.socket failed (Invalid argument)
[2014-08-26 10:14:19.680930] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/9d668ac3807a40f48163c5c6346c4177/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.683768] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/6550f17c37a9cef6c7dd159f727aa894.socket failed (Invalid argument)
[2014-08-26 10:14:19.686545] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/9bcb0eb635b44584badf5fe8ef52b4c9/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.689500] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/ac6771897e19ab6d87a4d049f64931bc.socket failed (Invalid argument)
[2014-08-26 10:14:19.692248] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/c790632bf20f4718a78627fbec10cd69/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.695088] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/ad019d33eb34aa609ce22ae04ac5d01a.socket failed (Invalid argument)
[2014-08-26 10:14:19.697770] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/e8de0ee5273449518bec3dbc5a8f78f3/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.700490] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/7e78f15da0160220a410bbb78d4d4e07.socket failed (Invalid argument)
[2014-08-26 10:14:19.703520] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/dcc1df7d301c4c408954d346b6ad8360/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.706920] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/74cd6c507846c6a68621a136392fc81b.socket failed (Invalid argument)
[2014-08-26 10:14:19.709760] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/629a14190d0841a8ab7c1b14f261ddce/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.712545] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/6f1eb5cefbd1f3b2eca4cbb47e58e860.socket failed (Invalid argument)
[2014-08-26 10:14:19.715314] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/6edc8302d10b41f9bb825cc5f9a8db3f/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.718049] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/7272bb4ff2842b88154eef1a1d7f5b95.socket failed (Invalid argument)
[2014-08-26 10:14:19.720900] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/89653c521b1f4d399c8b7c6fddd4f9b4/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.723635] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/da2f2a9c430f7aff2bf630bca201f2da.socket failed (Invalid argument)
[2014-08-26 10:14:19.726352] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/9bcb375c8e794e63b30d4ac5c299f628/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.729023] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/2fbb11d334afc0267a5d5030c8c4bbd9.socket failed (Invalid argument)
[2014-08-26 10:14:19.731747] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/874be8d1e30f483e9fe8439751badd0e/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.734561] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/70c03a220e486a366d9fc4acd8b99380.socket failed (Invalid argument)
[2014-08-26 10:14:19.737320] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/e490b9ecfdc44b809f9e09f359ad6cdc/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.740039] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/1c88f8ce46538602f7a9ef21de9c32a5.socket failed (Invalid argument)
[2014-08-26 10:14:19.742670] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/a96c81a91e6945a28661a8671bbfa039/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.743989] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request
[2014-08-26 10:14:19.746795] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/38f73153cf71e9bb5f4a71671f1e365c.socket failed (Invalid argument)
[2014-08-26 10:14:19.752187] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/4bcdd4cbbb4effbaab87042de2b695b0.socket failed (Invalid argument)
[2014-08-26 10:14:19.757772] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/869a38aa51049452d3cda4936aa8e649.socket failed (Invalid argument)
[2014-08-26 10:14:19.763222] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/cb188e9bc3f242e6f0173f74eca75cac.socket failed (Invalid argument)
[2014-08-26 10:14:19.768763] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/c3df211a55ae2442acb7b5faf9afd498.socket failed (Invalid argument)
[2014-08-26 10:14:19.766022] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/8bb9b8b8948d45b1b75aad91d5fa6636/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.771537] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/4eee2f546621442aa1b87bf98ac71f5b/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.774195] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/3fc516462992e8ded92285ac336fd5bf.socket failed (Invalid argument)
[2014-08-26 10:14:19.776945] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/7b43a248b46c4d48ab54cafa9227d0b5/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.779691] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/e13ecc2b4182992922b40baa7feb0f99.socket failed (Invalid argument)
[2014-08-26 10:14:19.782325] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/0ffff4902ef844759e37a64452835bd5/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.785121] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/29cc060bed609e48692e0035ef44b973.socket failed (Invalid argument)
[2014-08-26 10:14:19.788500] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/f8aa81cdf24247feab4d3a767e833f00/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.791365] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/e7ceb620651d6583725dc67cde7b51d1.socket failed (Invalid argument)
[2014-08-26 10:14:19.793980] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/86e1633c6c4c476fb4823dcbe2fd4e4e/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.796630] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/07c2cc84966aa322ecd188e77d342f35.socket failed (Invalid argument)
[2014-08-26 10:14:19.799304] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/ddab0270c79847daa15c04b1f6f8792c/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.801969] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/ecbd2ad1325242313e1b0f0fdc213e9c.socket failed (Invalid argument)
[2014-08-26 10:14:19.804512] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/51ea15e69d0d4469a2a1e9860e308e81/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.807203] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/f5d36b518e4117492d8f4b43e4f7c5a1.socket failed (Invalid argument)
[2014-08-26 10:14:19.809703] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/3003d0d4f65b46ef9ab503ebd8929a87/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.812338] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/53352de07207732d20971673af76bd72.socket failed (Invalid argument)
[2014-08-26 10:14:19.814824] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/0226947e5a7b49fb8ad3e5d0bf4d660e/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.817347] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/eae23c7c6ebbfd4ff7679a04cc56eacb.socket failed (Invalid argument)
[2014-08-26 10:14:19.819928] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/b735dc4facd34abe859ae392262906cb/brick1/b2 has disconnected from glusterd.
[2014-08-26 10:14:19.822630] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/f039bb3e1b4741cbacabc908b0ce9643.socket failed (Invalid argument)
[2014-08-26 10:14:19.825039] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/4352089bff6a46feb8bfffa00be4a30d/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.827538] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/4b7b307e1d0e410e87c2ae109de29643.socket failed (Invalid argument)
[2014-08-26 10:14:19.829968] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/923d5302b545476d90b3ee60cc61d334/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.832597] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/6b9834ade79059fa90697889b02422bc.socket failed (Invalid argument)
[2014-08-26 10:14:19.835012] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/e7e0b3096b9149a99afc5c28ec1b88af/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.837519] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/a7efc60a0f3a791933e4731654932d6f.socket failed (Invalid argument)
[2014-08-26 10:14:19.839944] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/322c27c6087c4073bd35be4d45d4a9e5/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.842576] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/3c7d45c7bf89a9a4371f0e532a3770ed.socket failed (Invalid argument)
[2014-08-26 10:14:19.845158] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/260f1938dac641c4ac6ca0962bf9de66/brick1/b3 has disconnected from glusterd.
[2014-08-26 10:14:19.847610] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/e7c4396af4fc6b50124c23daa0f3286f.socket failed (Invalid argument)
[2014-08-26 10:14:19.850211] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/09ce53a4b18445deb5f3ceb2c12d03f3/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.852832] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/e0c32417aed7dbd18145988372b94640.socket failed (Invalid argument)
[2014-08-26 10:14:19.855547] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/2be14e5616574fc38f2d6c960cd01c03/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.858254] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/6b35143e695a079e7f54d9ef7432bcdf.socket failed (Invalid argument)
[2014-08-26 10:14:19.860976] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/ee573786073d4c5186b73b20516d1132/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.863517] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/e08f7b5aa8151256446e1c12587d6f07.socket failed (Invalid argument)
[2014-08-26 10:14:19.866122] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/311edc38a5b54ef49c45194b28492fcc/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.868525] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/cc260accc630c111243b2e3f1d698eb9.socket failed (Invalid argument)
[2014-08-26 10:14:19.871329] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/b17aadf29d7249faac8bde6eddd9ffe7/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.873959] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/e1ae815a513bb8fa0f6ebdaf5de91280.socket failed (Invalid argument)
[2014-08-26 10:14:19.876558] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/7244f178dd2041a6b771b17ebbacc82f/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.879022] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/5db2d23273205d1e562460bb68e2d0b9.socket failed (Invalid argument)
[2014-08-26 10:14:19.881546] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/ad19ec441e5d4cbfa54b3b162fea3239/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.884045] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/c44591f201924b362866d07380717fbc.socket failed (Invalid argument)
[2014-08-26 10:14:19.886761] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/336e6c82117d4dbcbb83436277281da7/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.889510] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/1f74f7f3610bfe856fe94d18ad93abe5.socket failed (Invalid argument)
[2014-08-26 10:14:19.892301] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/5228f255b1a244eab24b67cce88cef2a/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.895315] W [socket.c:529:__socket_rwv] 0-management: readv on /var/run/028b16a3387aedd214608d3ea7e02d60.socket failed (Invalid argument)
[2014-08-26 10:14:19.898295] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/c7f93f796d9e44b4938bd88152c93244/brick1/b4 has disconnected from glusterd.
[2014-08-26 10:14:19.898622] I [socket.c:2246:socket_event_handler] 0-transport: disconnecting now
[2014-08-26 10:14:19.898659] I [glusterd-handler.c:1322:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2014-08-26 10:14:19.898807] I [socket.c:3206:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
[2014-08-26 10:14:19.898840] E [rpcsvc.c:1247:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1, Program: GlusterD svc cli, ProgVers: 2, Proc: 3) to rpc-transport (socket.management)
[2014-08-26 10:14:19.898865] E [glusterd-utils.c:396:glusterd_submit_reply] 0-: Reply submission failed
[2014-08-26 10:14:19.899082] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request
[2014-08-26 10:14:19.899241] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request
[2014-08-26 10:14:19.899324] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request
[2014-08-26 10:14:19.899413] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request
[2014-08-26 10:14:19.899518] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request
[2014-08-26 10:14:19.899637] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request
[2014-08-26 10:14:19.899700] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request



Also, volume status fails with Even when glusterd is running on all nodes

[root@inception ~]# gluster v status vol0
Locking failed on rhs-arch-srv4.lab.eng.blr.redhat.com. Please check log file for details.
Locking failed on rhs-arch-srv2.lab.eng.blr.redhat.com. Please check log file for details.
[root@inception ~]#

Comment 2 Vijaikumar Mallikarjuna 2014-08-27 08:08:17 UTC

This looks like bricks,glusterd from 4 nodes are all stuck waiting for the big-lock.
We will investigate more on this issue.

Comment 3 Vijaikumar Mallikarjuna 2014-08-27 08:45:54 UTC

From the below logs, when the brick started it was immediately disconnected by the glusterd and hence brick failed to start.


From brick log:

[2014-08-25 13:31:57.911090] I [MSGID: 100030] [glusterfsd.c:1998:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.6.0.27 (args: /usr/sbin/glusterfsd -s inception.lab.eng.blr.redhat.com --volfile-id /snaps/c249/86e1633c6c4c476fb4823dcbe2fd4e4e.inception.lab.eng.blr.redhat.com.var-run-gluster-snaps-86e1633c6c4c476fb4823dcbe2fd4e4e-brick1-b3 -p /var/lib/glusterd/snaps/c249/86e1633c6c4c476fb4823dcbe2fd4e4e/run/inception.lab.eng.blr.redhat.com-var-run-gluster-snaps-86e1633c6c4c476fb4823dcbe2fd4e4e-brick1-b3.pid -S /var/run/e7ceb620651d6583725dc67cde7b51d1.socket --brick-name /var/run/gluster/snaps/86e1633c6c4c476fb4823dcbe2fd4e4e/brick1/b3 -l /var/log/glusterfs/bricks/var-run-gluster-snaps-86e1633c6c4c476fb4823dcbe2fd4e4e-brick1-b3.log --xlator-option *-posix.glusterd-uuid=456cfdb5-b488-4354-9aa2-e2d0cee58454 --brick-port 50169 --xlator-option 86e1633c6c4c476fb4823dcbe2fd4e4e-server.listen-port=50169)
[2014-08-25 13:31:58.080444] W [socket.c:529:__socket_rwv] 0-glusterfs: readv on 10.70.34.50:24007 failed (No data available)
[2014-08-25 13:31:58.080669] E [rpc-clnt.c:362:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15d) [0x3ea160fe6d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91) [0x3ea160f8a1] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3ea160f7ee]))) 0-glusterfs: forced unwinding frame type(GlusterFS Handshake) op(GETSPEC(2)) called at 2014-08-25 13:31:57.917490 (xid=0x1)
[2014-08-25 13:31:58.080689] E [glusterfsd-mgmt.c:1596:mgmt_getspec_cbk] 0-mgmt: failed to fetch volume file 


From glusterd log:

 7809 [2014-08-25 13:31:58.077509] I [socket.c:2246:socket_event_handler] 0-transport: disconnecting now
 7810 [2014-08-25 13:31:58.080258] I [MSGID: 106005] [glusterd-handler.c:4165:__glusterd_brick_rpc_notify] 0-management: Brick inception.lab.eng.blr.redhat.com:/var/run/gluster/snaps/86e1633c6c4c476fb4823dcbe2fd4e4e/brick1/b3 has       disconnected from glusterd.

Comment 4 Vijaikumar Mallikarjuna 2014-08-27 09:09:40 UTC

We see below error in the logs. There may be more number of bricks running on the machine and running out of privileged ports, hence the issue.


[2014-08-25 13:31:44.625617] E [rpcsvc.c:617:rpcsvc_handle_rpc_call] 0-glusterd: Request received from non-privileged port. Failing request                                                                                     
 7500

Comment 5 rjoseph 2014-08-28 10:43:55 UTC

Upstream patch: http://review.gluster.org/#/c/8554

Comment 10 Shalaka 2014-09-21 04:12:26 UTC

Please review and sign-off edited doc text.

Comment 11 Vijaikumar Mallikarjuna 2014-09-22 06:07:56 UTC

Doc text looks good to me

Comment 12 rjoseph 2014-09-22 09:26:10 UTC

Updated the doc text.

Note You need to log in before you can comment on or make changes to this bug.