Bug 1002014 - Unable to start glusterd due to empty peer file in peers directory
Unable to start glusterd due to empty peer file in peers directory
Status: CLOSED WONTFIX
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd (Show other bugs)
2.0
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: krishnan parthasarathi
storage-qa-internal@redhat.com
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-28 06:43 EDT by Sachidananda Urs
Modified: 2015-11-03 18:05 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-03-23 03:36:51 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Sachidananda Urs 2013-08-28 06:43:31 EDT
Description of problem:

During peer probe operations (possibly), glusterd ended up creating a .tmp file in peers directory. And in such cases, `service glusterd start' say started but glusterd is not started.

[root@anshi-0 ~]# service glusterd status
glusterd dead but subsys locked
[root@anshi-0 ~]#

[root@anshi-0 ~]# ls /var/lib/glusterd/peers/
ee7bfd7b-c149-484a-95df-398ac2c90cd6      ee7bfd7b-c149-484a-95df-398ac2c90cd6.tmp

Only way to bring up glusterd is to delete one of the .tmp file and restart glusterd. We should be filtering out reading the peer files other than the UUID filenames

Version-Release number of selected component (if applicable):

[root@anshi-0 ~]# gluster --version
glusterfs 3.3.0.11rhs built on Jul  3 2013 05:17:12


How reproducible:
Rarely

Steps to Reproduce:
1.
2.
3.

Actual results:

glusterd is not started

Expected results:

glusterd should be started

Additional info:

glusterd --debug output:
========================
[2013-08-28 11:52:24.504899] I [glusterfsd.c:1817:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd
 version 3.3.0.11rhs (/usr/sbin/glusterd --debug)
[2013-08-28 11:52:24.505384] D [glusterfsd.c:550:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glu
sterd.vol
[2013-08-28 11:52:24.507601] I [glusterd.c:849:init] 0-management: Using /var/lib/glusterd as working directory
[2013-08-28 11:52:24.507716] D [glusterd.c:273:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 128
[2013-08-28 11:52:24.507987] D [rpcsvc.c:1872:rpcsvc_init] 0-rpc-service: RPC service inited.
[2013-08-28 11:52:24.508028] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: GF
-DUMP, Num: 123451501, Ver: 1, Port: 0
[2013-08-28 11:52:24.508084] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /u
sr/lib64/glusterfs/3.3.0.11rhs/rpc-transport/socket.so
[2013-08-28 11:52:24.508438] D [name.c:555:server_fill_address_family] 0-socket.management: option address-famil
y not specified, defaulting to inet/inet6
[2013-08-28 11:52:24.508753] D [rpc-transport.c:248:rpc_transport_load] 0-rpc-transport: attempt to load file /u
sr/lib64/glusterfs/3.3.0.11rhs/rpc-transport/rdma.so
[2013-08-28 11:52:24.509330] C [rdma.c:3960:gf_rdma_init] 0-rpc-transport/rdma: Failed to get IB devices
[2013-08-28 11:52:24.509427] E [rdma.c:4842:init] 0-rdma.management: Failed to initialize IB Device
[2013-08-28 11:52:24.509453] E [rpc-transport.c:316:rpc_transport_load] 0-rpc-transport: 'rdma' initialization f
ailed
[2013-08-28 11:52:24.509473] W [rpcsvc.c:1356:rpcsvc_transport_create] 0-rpc-service: cannot create listener, in
iting the transport failed
[2013-08-28 11:52:24.509495] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
usterD svc peer, Num: 1238437, Ver: 2, Port: 0
[2013-08-28 11:52:24.509516] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
usterD svc cli, Num: 1238463, Ver: 2, Port: 0
[2013-08-28 11:52:24.509536] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
usterD svc mgmt, Num: 1238433, Ver: 2, Port: 0
[2013-08-28 11:52:24.509555] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
uster Portmap, Num: 34123456, Ver: 1, Port: 0
[2013-08-28 11:52:24.509574] D [rpcsvc.c:1636:rpcsvc_program_register] 0-rpc-service: New program registered: Gl
usterFS Handshake, Num: 14398633, Ver: 2, Port: 0
[2013-08-28 11:52:24.509610] D [glusterd-utils.c:5036:glusterd_sm_tr_log_init] 0-: returning 0
[2013-08-28 11:52:24.509669] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:24.509690] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:24.509751] D [glusterd-store.c:1210:glusterd_store_retrieve_value] 0-: key UUID read
[2013-08-28 11:52:24.509773] D [glusterd-store.c:1213:glusterd_store_retrieve_value] 0-: key UUID found
[2013-08-28 11:52:24.509805] D [glusterd-store.c:1460:glusterd_retrieve_uuid] 0-: Returning 0
[2013-08-28 11:52:24.509832] I [glusterd.c:95:glusterd_uuid_init] 0-glusterd: retrieved UUID: 2503f2f3-7080-4733
-9528-cb8c676e397d
[2013-08-28 11:52:24.570455] D [glusterd.c:328:glusterd_check_gsync_present] 0-glusterd: Returning 0
[2013-08-28 11:52:24.570522] D [glusterd.c:434:glusterd_crt_georep_folders] 0-: Returning 0
[2013-08-28 11:52:25.517152] D [glusterd-utils.c:586:glusterd_volinfo_new] 0-: Returning 0
[2013-08-28 11:52:25.517244] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.517268] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.517319] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.517373] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517411] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517435] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517466] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517492] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517514] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517534] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517555] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517576] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517602] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517627] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517649] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517688] D [glusterd-store.c:2101:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.flush-behind,value:on
[2013-08-28 11:52:25.517713] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517737] D [glusterd-store.c:2101:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.write-behind,value:on
[2013-08-28 11:52:25.517758] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517781] D [glusterd-store.c:2101:glusterd_store_retrieve_volume] 0-: Parsed as Volume-set:key=performance.read-ahead,value:on
[2013-08-28 11:52:25.517802] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517821] E [glusterd-store.c:2089:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2013-08-28 11:52:25.517842] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.517860] E [glusterd-store.c:2089:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2013-08-28 11:52:25.517881] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.517975] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.518009] D [glusterd-utils.c:728:glusterd_brickinfo_new] 0-: Returning 0
[2013-08-28 11:52:25.518042] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518110] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518133] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518152] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518172] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518192] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518211] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518231] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518251] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518271] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518291] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518311] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518331] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518350] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518370] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518409] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.518428] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.518476] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.518507] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518528] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.518549] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969258] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969341] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969368] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.969476] D [glusterd-utils.c:728:glusterd_brickinfo_new] 0-: Returning 0
[2013-08-28 11:52:25.969503] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969552] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.969573] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.969616] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.969649] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969672] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969693] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969715] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969735] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.969755] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.969795] D [glusterd-store.c:1818:glusterd_store_retrieve_bricks] 0-: Returning with 0
[2013-08-28 11:52:25.970334] D [glusterd-utils.c:1586:glusterd_volume_compute_cksum] 0-management: Returning with 0
[2013-08-28 11:52:25.970364] D [glusterd-store.c:2169:glusterd_store_retrieve_volume] 0-: Returning with 0
[2013-08-28 11:52:25.970378] D [glusterd-utils.c:947:glusterd_volinfo_find] 0-: Volume upgrade found
[2013-08-28 11:52:25.970407] D [glusterd-utils.c:954:glusterd_volinfo_find] 0-: Returning 0
[2013-08-28 11:52:25.970445] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.970463] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.970499] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.970533] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.970556] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.970587] D [glusterd-store.c:1904:glusterd_store_retrieve_rbstate] 0-: Returning with 0
[2013-08-28 11:52:25.970613] D [glusterd-utils.c:947:glusterd_volinfo_find] 0-: Volume upgrade found
[2013-08-28 11:52:25.970631] D [glusterd-utils.c:954:glusterd_volinfo_find] 0-: Returning 0
[2013-08-28 11:52:25.970665] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.970683] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.970714] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.970743] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.970763] D [glusterd-store.c:1966:glusterd_store_retrieve_node_state] 0-: Returning with 0
[2013-08-28 11:52:25.970787] D [glusterd-store.c:2334:glusterd_store_retrieve_volumes] 0-: Returning with 0
[2013-08-28 11:52:25.970844] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.970865] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.970949] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.970991] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.971006] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.971017] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with 0
[2013-08-28 11:52:25.971028] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.971075] D [glusterd-utils.c:5036:glusterd_sm_tr_log_init] 0-: returning 0
[2013-08-28 11:52:25.971093] D [glusterd-utils.c:5128:glusterd_peerinfo_new] 0-: returning 0
[2013-08-28 11:52:25.971103] I [glusterd-handler.c:2286:glusterd_friend_add] 0-management: connect returned 0
[2013-08-28 11:52:25.971125] D [glusterd-store.c:1315:glusterd_store_handle_new] 0-: Returning 0
[2013-08-28 11:52:25.971136] D [glusterd-store.c:1333:glusterd_store_handle_retrieve] 0-: Returning 0
[2013-08-28 11:52:25.971153] D [glusterd-store.c:1506:glusterd_store_iter_new] 0-: Returning with 0
[2013-08-28 11:52:25.971166] D [glusterd-store.c:1624:glusterd_store_iter_get_next] 0-: Returning with -1
[2013-08-28 11:52:25.971177] D [glusterd-store.c:2688:glusterd_store_retrieve_peers] 0-: Returning with -1
[2013-08-28 11:52:25.971187] D [glusterd-store.c:2744:glusterd_restore] 0-: Returning -1
[2013-08-28 11:52:25.971202] E [xlator.c:385:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2013-08-28 11:52:25.971216] E [graph.c:294:glusterfs_graph_init] 0-management: initializing translator failed
[2013-08-28 11:52:25.971226] E [graph.c:483:glusterfs_graph_activate] 0-graph: init failed
[2013-08-28 11:52:25.971456] W [glusterfsd.c:965:cleanup_and_exit] (-->/usr/sbin/glusterd(main+0x62e) [0x4077be] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0x145) [0x405b35] (-->/usr/sbin/glusterd(glusterfs_process_volfp+0x198) [0x4059e8]))) 0-: received signum (0), shutting down
[2013-08-28 11:52:25.971502] D [glusterfsd-mgmt.c:2157:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given
Comment 2 Vivek Agarwal 2015-03-23 03:36:51 EDT
The product version of Red Hat Storage on which this issue was reported has reached End Of Life (EOL) [1], hence this bug report is being closed. If the issue is still observed on a current version of Red Hat Storage, please file a new bug report on the current version.







[1] https://rhn.redhat.com/errata/RHSA-2014-0821.html
Comment 3 Vivek Agarwal 2015-03-23 03:39:11 EDT
The product version of Red Hat Storage on which this issue was reported has reached End Of Life (EOL) [1], hence this bug report is being closed. If the issue is still observed on a current version of Red Hat Storage, please file a new bug report on the current version.







[1] https://rhn.redhat.com/errata/RHSA-2014-0821.html

Note You need to log in before you can comment on or make changes to this bug.