1002014 – Unable to start glusterd due to empty peer file in peers directory

Bug 1002014 - Unable to start glusterd due to empty peer file in peers directory

Summary: Unable to start glusterd due to empty peer file in peers directory

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	glusterd
Sub Component:
Version:	2.0
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	krishnan parthasarathi
QA Contact:	storage-qa-internal@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2013-08-28 10:43 UTC by Sachidananda Urs
Modified:	2015-11-03 23:05 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2015-03-23 07:36:51 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Sachidananda Urs 2013-08-28 10:43:31 UTC

Description of problem:

During peer probe operations (possibly), glusterd ended up creating a .tmp file in peers directory. And in such cases, `service glusterd start' say started but glusterd is not started.

[root@anshi-0 ~]# service glusterd status
glusterd dead but subsys locked
[root@anshi-0 ~]#

[root@anshi-0 ~]# ls /var/lib/glusterd/peers/
ee7bfd7b-c149-484a-95df-398ac2c90cd6      ee7bfd7b-c149-484a-95df-398ac2c90cd6.tmp

Only way to bring up glusterd is to delete one of the .tmp file and restart glusterd. We should be filtering out reading the peer files other than the UUID filenames

Version-Release number of selected component (if applicable):

[root@anshi-0 ~]# gluster --version
glusterfs 3.3.0.11rhs built on Jul  3 2013 05:17:12


How reproducible:
Rarely

Steps to Reproduce:
1.
2.
3.

Actual results:

glusterd is not started

Expected results:

glusterd should be started

Additional info:

glusterd --debug output:
========================
[2013-08-28 11:52:24.504899] I [glusterfsd.c:1817:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd
 version 3.3.0.11rhs (/usr/sbin/glusterd --debug)
[2013-08-28 11:52:24.505384] D [glusterfsd.c:550:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glu
sterd.vol
[2013-08-28 11:52:24.507601] I [glusterd.c:849:init] 0-management: Using /var/lib/glusterd as working directory
[2013-08-28 11:52:24.507716] D [glusterd.c:273:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 128
[2013-08-28 11:52:24.507987] D [rpcsvc.c:1872:rpcsvc_init] 0-rpc-service: RPC service inited.
[2013-08-28 11:52:24.508028] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: GF
-DUMP, Num: 123451501, Ver: 1, Port: 0
[2013-08-28 11:52:24.508084] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /u
sr/lib64/glusterfs/3.3.0.11rhs/rpc-transport/socket.so
[2013-08-28 11:52:24.508438] D [name.c:555:server_fill_address_family] 0-socket.management: option address-famil
y not specified, defaulting to inet/inet6
[2013-08-28 11:52:24.508753] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /u
sr/lib64/glusterfs/3.3.0.11rhs/rpc-transport/rdma.so
[2013-08-28 11:52:24.509330] C [rdma.c:3960:gf_rdma_init] 0-rpc-transport/rdma: Failed to get IB devices
[2013-08-28 11:52:24.509427] E [rdma.c:4842:init] 0-rdma.management: Failed to initialize IB Device
[2013-08-28 11:52:24.509453] E [rpc-transport.c:316:rpc_transport_load] 0-rpc-transport: 'rdma' initialization f
ailed
[2013-08-28 11:52:24.509473] W [rpcsvc.c:1356:rpcsvc_transport_create] 0-rpc-service: cannot create listener, in
iting the transport failed
[2013-08-28 11:52:24.509495] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
usterD svc peer, Num: 1238437, Ver: 2, Port: 0
[2013-08-28 11:52:24.509516] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
usterD svc cli, Num: 1238463, Ver: 2, Port: 0
[2013-08-28 11:52:24.509536] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
usterD svc mgmt, Num: 1238433, Ver: 2, Port: 0
[2013-08-28 11:52:24.509555] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
uster Portmap, Num: 34123456, Ver: 1, Port: 0
[2013-08-28 11:52:24.509574] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
usterFS Handshake, Num: 14398633, Ver: 2, Port: 0
[2013-08-28 11:52:24.509610] D [glusterd-utils.c:5036:glusterd_sm_tr_log_init] 0-: returning 0
[2013-08-28 11:52:24.509669] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:24.509690] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:24.509751] D [glusterd-store.c:1210:glusterd_store_retrieve_value] 0-: key UUID read
[2013-08-28 11:52:24.509773] D [glusterd-store.c:1213:glusterd_store_retrieve_value] 0-: key UUID found
[2013-08-28 11:52:24.509805] D [glusterd-store.c:1460:glusterd_retrieve_uuid] 0-: Returning 0
[2013-08-28 11:52:24.509832] I [glusterd.c:95:glusterd_uuid_init] 0-glusterd: retrieved UUID: 2503f2f3-7080-4733
-9528-cb8c676e397d
[2013-08-28 11:52:24.570455] D [glusterd.c:328:glusterd_check_gsync_present] 0-glusterd: Returning 0
[2013-08-28 11:52:24.570522] D [glusterd.c:434:glusterd_crt_georep_folders] 0-: Returning 0
[2013-08-28 11:52:25.517152] D [glusterd-utils.c:586:glusterd_volinfo_new] 0-: Returning 0
[2013-08-28 11:52:25.517244] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.517268] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.517319] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.517373] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517411] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517435] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517466] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517492] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517514] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517534] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517555] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517576] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517602] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517627] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517649] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517688] D [glusterd-store.c:2101:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.flush-behind,value:on
[2013-08-28 11:52:25.517713] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517737] D [glusterd-store.c:2101:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.write-behind,value:on
[2013-08-28 11:52:25.517758] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517781] D [glusterd-store.c:2101:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.read-ahead,value:on
[2013-08-28 11:52:25.517802] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517821] E [glusterd-store.c:2089:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2013-08-28 11:52:25.517842] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517860] E [glusterd-store.c:2089:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2013-08-28 11:52:25.517881] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.517975] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.518009] D [glusterd-utils.c:728:glusterd_brickinfo_new] 0-: Returning 0
[2013-08-28 11:52:25.518042] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518110] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518133] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518152] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518172] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518192] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518211] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518231] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518251] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518271] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518291] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518311] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518331] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518350] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518370] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518409] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.518428] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.518476] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.518507] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518528] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518549] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969258] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969341] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969368] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.969476] D [glusterd-utils.c:728:glusterd_brickinfo_new] 0-: Returning 0
[2013-08-28 11:52:25.969503] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969552] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.969573] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.969616] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.969649] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969672] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969693] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969715] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969735] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969755] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.969795] D [glusterd-store.c:1818:glusterd_store_retrieve_bricks] 0-: Returning with 0
[2013-08-28 11:52:25.970334] D [glusterd-utils.c:1586:glusterd_volume_compute_cksum] 0-management: Returning with 0
[2013-08-28 11:52:25.970364] D [glusterd-store.c:2169:glusterd_store_retrieve_volume] 0-: Returning with 0
[2013-08-28 11:52:25.970378] D [glusterd-utils.c:947:glusterd_volinfo_find] 0-: Volume upgrade found
[2013-08-28 11:52:25.970407] D [glusterd-utils.c:954:glusterd_volinfo_find] 0-: Returning 0
[2013-08-28 11:52:25.970445] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.970463] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.970499] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.970533] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.970556] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.970587] D [glusterd-store.c:1904:glusterd_store_retrieve_rbstate] 0-: Returning with 0
[2013-08-28 11:52:25.970613] D [glusterd-utils.c:947:glusterd_volinfo_find] 0-: Volume upgrade found
[2013-08-28 11:52:25.970631] D [glusterd-utils.c:954:glusterd_volinfo_find] 0-: Returning 0
[2013-08-28 11:52:25.970665] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.970683] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.970714] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.970743] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.970763] D [glusterd-store.c:1966:glusterd_store_retrieve_node_state] 0-: Returning with 0
[2013-08-28 11:52:25.970787] D [glusterd-store.c:2334:glusterd_store_retrieve_volumes] 0-: Returning with 0
[2013-08-28 11:52:25.970844] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.970865] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.970949] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.970991] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.971006] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.971017] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.971028] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.971075] D [glusterd-utils.c:5036:glusterd_sm_tr_log_init] 0-: returning 0
[2013-08-28 11:52:25.971093] D [glusterd-utils.c:5128:glusterd_peerinfo_new] 0-: returning 0
[2013-08-28 11:52:25.971103] I [glusterd-handler.c:2286:glusterd_friend_add] 0-management: connect returned 0
[2013-08-28 11:52:25.971125] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.971136] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.971153] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.971166] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.971177] D [glusterd-store.c:2688:glusterd_store_retrieve_peers] 0-: Returning with -1
[2013-08-28 11:52:25.971187] D [glusterd-store.c:2744:glusterd_restore] 0-: Returning -1
[2013-08-28 11:52:25.971202] E [xlator.c:385:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2013-08-28 11:52:25.971216] E [graph.c:294:glusterfs_graph_init] 0-management: initializing translator failed
[2013-08-28 11:52:25.971226] E [graph.c:483:glusterfs_graph_activate] 0-graph: init failed
[2013-08-28 11:52:25.971456] W [glusterfsd.c:965:cleanup_and_exit] (-->/usr/sbin/glusterd(main+0x62e) [0x4077be] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0x145) [0x405b35] (-->/usr/sbin/glusterd(glusterfs_process_volfp+0x198) [0x4059e8]))) 0-: received signum (0), shutting down
[2013-08-28 11:52:25.971502] D [glusterfsd-mgmt.c:2157:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given

Comment 2 Vivek Agarwal 2015-03-23 07:36:51 UTC

The product version of Red Hat Storage on which this issue was reported has reached End Of Life (EOL) [1], hence this bug report is being closed. If the issue is still observed on a current version of Red Hat Storage, please file a new bug report on the current version.







[1] https://rhn.redhat.com/errata/RHSA-2014-0821.html

Comment 3 Vivek Agarwal 2015-03-23 07:39:11 UTC

The product version of Red Hat Storage on which this issue was reported has reached End Of Life (EOL) [1], hence this bug report is being closed. If the issue is still observed on a current version of Red Hat Storage, please file a new bug report on the current version.







[1] https://rhn.redhat.com/errata/RHSA-2014-0821.html

Note You need to log in before you can comment on or make changes to this bug.