Bug 1573129 - Systemctl status glusterd is showing failed but able to execute all gluster commands from the node
Summary: Systemctl status glusterd is showing failed but able to execute all gluster c...
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd
Version: rhgs-3.4
Hardware: x86_64
OS: Linux
Target Milestone: ---
: ---
Assignee: Atin Mukherjee
QA Contact: Bala Konda Reddy M
Depends On:
TreeView+ depends on / blocked
Reported: 2018-04-30 09:34 UTC by Bala Konda Reddy M
Modified: 2019-01-06 13:01 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-01-06 13:01:01 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Bala Konda Reddy M 2018-04-30 09:34:00 UTC
Description of problem: 
On a brick-mux enabled three node cluster with more 20 volumes. After executing few gluster commands in a loop, glusterd went to failed state but able to execute gluster commands

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Enable brick mux and set max bricks per process to 5
2. Created few volumes more than 20
3. Continuously execute few glusterd commands. as below
while true
gluster volume status
gluster snapshot list 2cross3_10
gluster volume quota 2cross3_10 list
gluster snapshot info 2cross3_10
gluster volume rebalance 2cross3_10 status
gluster snapshot status 2cross3_10
gluster v status

Actual results:
systemctl status glusterd is showing glusterd in failed state but able to execute gluster commands successfully

Expected results:
When glusterd is down, gluster commands should fail.

Additional info:

Comment 5 Bala Konda Reddy M 2018-05-03 09:35:00 UTC
Glusterd shutdown happened, please check the below log snippet. 

[2018-04-26 05:47:41.478279] W [socket.c:593:__socket_rwv] 0-socket.management: writev on  failed (Broken pipe)
[2018-04-26 05:47:41.491385] W [socket.c:593:__socket_rwv] 0-socket.management: writev on  failed (Broken pipe)
[2018-04-26 05:47:49.186852] I [socket.c:3672:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
[2018-04-26 05:47:49.187982] E [rpcsvc.c:1364:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, ProgVers: 2, Proc: 12) to rpc-transport (socket.management)
[2018-04-26 05:47:49.188034] E [MSGID: 106430] [glusterd-utils.c:568:glusterd_submit_reply] 0-glusterd: Reply submission failed
[2018-04-26 05:47:37.429615] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2018-04-26 05:47:37.429645] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: quotad service is stopped
[2018-04-26 05:47:37.456899] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2018-04-26 07:24:32.163945] I [MSGID: 100030] [glusterfsd.c:2504:main] 0-glusterd: Started running glusterd version 3.12.2 (args: glusterd)
[2018-04-26 07:24:32.265244] I [MSGID: 106478] [glusterd.c:1451:init] 0-management: Maximum allowed open file descriptors set to 65536
[2018-04-26 07:24:32.265379] I [MSGID: 106479] [glusterd.c:1509:init] 0-management: Using /var/lib/glusterd as working directory
[2018-04-26 07:24:32.265404] I [MSGID: 106479] [glusterd.c:1514:init] 0-management: Using /var/run/gluster as pid file working directory
[2018-04-26 07:24:32.308892] E [socket.c:796:__socket_server_bind] 0-socket.management: binding to  failed: Address already in use
[2018-04-26 07:24:32.308935] E [socket.c:799:__socket_server_bind] 0-socket.management: Port is already in use
[2018-04-26 07:24:32.308969] W [rpcsvc.c:1689:rpcsvc_create_listener] 0-rpc-service: listening on transport failed
[2018-04-26 07:24:32.309005] E [MSGID: 106243] [glusterd.c:1790:init] 0-management: creation of listener failed
[2018-04-26 07:24:32.309026] E [MSGID: 101019] [xlator.c:503:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2018-04-26 07:24:32.309054] E [MSGID: 101066] [graph.c:367:glusterfs_graph_init] 0-management: initializing translator failed
[2018-04-26 07:24:32.309069] E [MSGID: 101176] [graph.c:738:glusterfs_graph_activate] 0-graph: init failed
[2018-04-26 07:24:32.310010] W [glusterfsd.c:1367:cleanup_and_exit] (-->glusterd(glusterfs_volumes_init+0xfd) [0x557b7818b02d] -->glusterd(glusterfs_process_volfp+0x163) [0x557b7818aed3] -->glusterd(cleanup_and_exit+0x6b) [0x557b7818a44b] ) 0-: received signum (-1), shutting down
[2018-04-26 07:25:02.637219] I [MSGID: 106163] [glusterd-handshake.c:1319:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31302
[2018-04-26 07:25:33.997440] I [MSGID: 106490] [glusterd-handler.c:2617:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 403950cb-d92d-4d4f-963a-9153a70ed045
[2018-04-26 07:28:18.569623] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 403950cb-d92d-4d4f-963a-9153a70ed045, host: dhcp37-44.lab.eng.blr.redhat.com, port: 0
[2018-04-26 07:28:18.661875] I [MSGID: 106492] [glusterd-handler.c:2795:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 403950cb-d92d-4d4f-963a-9153a70ed045
[2018-04-26 07:28:18.661995] I [MSGID: 106502] [glusterd-handler.c:2840:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-04-26 07:32:06.721184] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 403950cb-d92d-4d4f-963a-9153a70ed045

Note You need to log in before you can comment on or make changes to this bug.