Bug 1002014

Summary: Unable to start glusterd due to empty peer file in peers directory
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Sachidananda Urs <surs>
Component: glusterdAssignee: krishnan parthasarathi <kparthas>
Status: CLOSED WONTFIX QA Contact: storage-qa-internal <storage-qa-internal>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.0CC: nsathyan, rhs-bugs, vbellur
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-03-23 07:36:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sachidananda Urs 2013-08-28 10:43:31 UTC 
Description of problem:

During peer probe operations (possibly), glusterd ended up creating a .tmp file in peers directory. And in such cases, `service glusterd start' say started but glusterd is not started.

[root@anshi-0 ~]# service glusterd status
glusterd dead but subsys locked
[root@anshi-0 ~]#

[root@anshi-0 ~]# ls /var/lib/glusterd/peers/
ee7bfd7b-c149-484a-95df-398ac2c90cd6      ee7bfd7b-c149-484a-95df-398ac2c90cd6.tmp

Only way to bring up glusterd is to delete one of the .tmp file and restart glusterd. We should be filtering out reading the peer files other than the UUID filenames

Version-Release number of selected component (if applicable):

[root@anshi-0 ~]# gluster --version
glusterfs 3.3.0.11rhs built on Jul  3 2013 05:17:12


How reproducible:
Rarely

Steps to Reproduce:
1.
2.
3.

Actual results:

glusterd is not started

Expected results:

glusterd should be started

Additional info:

glusterd --debug output:
========================
[2013-08-28 11:52:24.504899] I [glusterfsd.c:1817:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd
 version 3.3.0.11rhs (/usr/sbin/glusterd --debug)
[2013-08-28 11:52:24.505384] D [glusterfsd.c:550:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glu
sterd.vol
[2013-08-28 11:52:24.507601] I [glusterd.c:849:init] 0-management: Using /var/lib/glusterd as working directory
[2013-08-28 11:52:24.507716] D [glusterd.c:273:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 128
[2013-08-28 11:52:24.507987] D [rpcsvc.c:1872:rpcsvc_init] 0-rpc-service: RPC service inited.
[2013-08-28 11:52:24.508028] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: GF
-DUMP, Num: 123451501, Ver: 1, Port: 0
[2013-08-28 11:52:24.508084] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /u
sr/lib64/glusterfs/3.3.0.11rhs/rpc-transport/socket.so
[2013-08-28 11:52:24.508438] D [name.c:555:server_fill_address_family] 0-socket.management: option address-famil
y not specified, defaulting to inet/inet6
[2013-08-28 11:52:24.508753] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /u
sr/lib64/glusterfs/3.3.0.11rhs/rpc-transport/rdma.so
[2013-08-28 11:52:24.509330] C [rdma.c:3960:gf_rdma_init] 0-rpc-transport/rdma: Failed to get IB devices
[2013-08-28 11:52:24.509427] E [rdma.c:4842:init] 0-rdma.management: Failed to initialize IB Device
[2013-08-28 11:52:24.509453] E [rpc-transport.c:316:rpc_transport_load] 0-rpc-transport: 'rdma' initialization f
ailed
[2013-08-28 11:52:24.509473] W [rpcsvc.c:1356:rpcsvc_transport_create] 0-rpc-service: cannot create listener, in
iting the transport failed
[2013-08-28 11:52:24.509495] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
usterD svc peer, Num: 1238437, Ver: 2, Port: 0
[2013-08-28 11:52:24.509516] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
usterD svc cli, Num: 1238463, Ver: 2, Port: 0
[2013-08-28 11:52:24.509536] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
usterD svc mgmt, Num: 1238433, Ver: 2, Port: 0
[2013-08-28 11:52:24.509555] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
uster Portmap, Num: 34123456, Ver: 1, Port: 0
[2013-08-28 11:52:24.509574] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
usterFS Handshake, Num: 14398633, Ver: 2, Port: 0
[2013-08-28 11:52:24.509610] D [glusterd-utils.c:5036:glusterd_sm_tr_log_init] 0-: returning 0
[2013-08-28 11:52:24.509669] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:24.509690] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:24.509751] D [glusterd-store.c:1210:glusterd_store_retrieve_value] 0-: key UUID read
[2013-08-28 11:52:24.509773] D [glusterd-store.c:1213:glusterd_store_retrieve_value] 0-: key UUID found
[2013-08-28 11:52:24.509805] D [glusterd-store.c:1460:glusterd_retrieve_uuid] 0-: Returning 0
[2013-08-28 11:52:24.509832] I [glusterd.c:95:glusterd_uuid_init] 0-glusterd: retrieved UUID: 2503f2f3-7080-4733
-9528-cb8c676e397d
[2013-08-28 11:52:24.570455] D [glusterd.c:328:glusterd_check_gsync_present] 0-glusterd: Returning 0
[2013-08-28 11:52:24.570522] D [glusterd.c:434:glusterd_crt_georep_folders] 0-: Returning 0
[2013-08-28 11:52:25.517152] D [glusterd-utils.c:586:glusterd_volinfo_new] 0-: Returning 0
[2013-08-28 11:52:25.517244] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.517268] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.517319] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.517373] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517411] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517435] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517466] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517492] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517514] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517534] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517555] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517576] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517602] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517627] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517649] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517688] D [glusterd-store.c:2101:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.flush-behind,value:on
[2013-08-28 11:52:25.517713] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517737] D [glusterd-store.c:2101:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.write-behind,value:on
[2013-08-28 11:52:25.517758] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517781] D [glusterd-store.c:2101:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.read-ahead,value:on
[2013-08-28 11:52:25.517802] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517821] E [glusterd-store.c:2089:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2013-08-28 11:52:25.517842] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517860] E [glusterd-store.c:2089:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2013-08-28 11:52:25.517881] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.517975] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.518009] D [glusterd-utils.c:728:glusterd_brickinfo_new] 0-: Returning 0
[2013-08-28 11:52:25.518042] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518110] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518133] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518152] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518172] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518192] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518211] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518231] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518251] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518271] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518291] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518311] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518331] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518350] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518370] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518409] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.518428] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.518476] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.518507] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518528] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518549] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969258] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969341] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969368] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.969476] D [glusterd-utils.c:728:glusterd_brickinfo_new] 0-: Returning 0
[2013-08-28 11:52:25.969503] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969552] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.969573] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.969616] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.969649] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969672] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969693] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969715] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969735] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969755] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.969795] D [glusterd-store.c:1818:glusterd_store_retrieve_bricks] 0-: Returning with 0
[2013-08-28 11:52:25.970334] D [glusterd-utils.c:1586:glusterd_volume_compute_cksum] 0-management: Returning with 0
[2013-08-28 11:52:25.970364] D [glusterd-store.c:2169:glusterd_store_retrieve_volume] 0-: Returning with 0
[2013-08-28 11:52:25.970378] D [glusterd-utils.c:947:glusterd_volinfo_find] 0-: Volume upgrade found
[2013-08-28 11:52:25.970407] D [glusterd-utils.c:954:glusterd_volinfo_find] 0-: Returning 0
[2013-08-28 11:52:25.970445] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.970463] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.970499] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.970533] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.970556] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.970587] D [glusterd-store.c:1904:glusterd_store_retrieve_rbstate] 0-: Returning with 0
[2013-08-28 11:52:25.970613] D [glusterd-utils.c:947:glusterd_volinfo_find] 0-: Volume upgrade found
[2013-08-28 11:52:25.970631] D [glusterd-utils.c:954:glusterd_volinfo_find] 0-: Returning 0
[2013-08-28 11:52:25.970665] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.970683] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.970714] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.970743] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.970763] D [glusterd-store.c:1966:glusterd_store_retrieve_node_state] 0-: Returning with 0
[2013-08-28 11:52:25.970787] D [glusterd-store.c:2334:glusterd_store_retrieve_volumes] 0-: Returning with 0
[2013-08-28 11:52:25.970844] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.970865] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.970949] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.970991] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.971006] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.971017] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.971028] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.971075] D [glusterd-utils.c:5036:glusterd_sm_tr_log_init] 0-: returning 0
[2013-08-28 11:52:25.971093] D [glusterd-utils.c:5128:glusterd_peerinfo_new] 0-: returning 0
[2013-08-28 11:52:25.971103] I [glusterd-handler.c:2286:glusterd_friend_add] 0-management: connect returned 0
[2013-08-28 11:52:25.971125] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.971136] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.971153] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.971166] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.971177] D [glusterd-store.c:2688:glusterd_store_retrieve_peers] 0-: Returning with -1
[2013-08-28 11:52:25.971187] D [glusterd-store.c:2744:glusterd_restore] 0-: Returning -1
[2013-08-28 11:52:25.971202] E [xlator.c:385:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2013-08-28 11:52:25.971216] E [graph.c:294:glusterfs_graph_init] 0-management: initializing translator failed
[2013-08-28 11:52:25.971226] E [graph.c:483:glusterfs_graph_activate] 0-graph: init failed
[2013-08-28 11:52:25.971456] W [glusterfsd.c:965:cleanup_and_exit] (-->/usr/sbin/glusterd(main+0x62e) [0x4077be] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0x145) [0x405b35] (-->/usr/sbin/glusterd(glusterfs_process_volfp+0x198) [0x4059e8]))) 0-: received signum (0), shutting down
[2013-08-28 11:52:25.971502] D [glusterfsd-mgmt.c:2157:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given
Comment 2 Vivek Agarwal 2015-03-23 07:36:51 UTC 
The product version of Red Hat Storage on which this issue was reported has reached End Of Life (EOL) [1], hence this bug report is being closed. If the issue is still observed on a current version of Red Hat Storage, please file a new bug report on the current version.







[1] https://rhn.redhat.com/errata/RHSA-2014-0821.html
Comment 3 Vivek Agarwal 2015-03-23 07:39:11 UTC 
The product version of Red Hat Storage on which this issue was reported has reached End Of Life (EOL) [1], hence this bug report is being closed. If the issue is still observed on a current version of Red Hat Storage, please file a new bug report on the current version.







[1] https://rhn.redhat.com/errata/RHSA-2014-0821.html