Hide Forgot
Description of problem: On a brick-mux enabled three node cluster with more 20 volumes. After executing few gluster commands in a loop, glusterd went to failed state but able to execute gluster commands Version-Release number of selected component (if applicable): 3.12.2-8 How reproducible: 1/1 Steps to Reproduce: 1. Enable brick mux and set max bricks per process to 5 2. Created few volumes more than 20 3. Continuously execute few glusterd commands. as below while true do gluster volume status gluster snapshot list 2cross3_10 gluster volume quota 2cross3_10 list gluster snapshot info 2cross3_10 gluster volume rebalance 2cross3_10 status gluster snapshot status 2cross3_10 gluster v status done Actual results: systemctl status glusterd is showing glusterd in failed state but able to execute gluster commands successfully Expected results: When glusterd is down, gluster commands should fail. Additional info:
Glusterd shutdown happened, please check the below log snippet. [2018-04-26 05:47:41.478279] W [socket.c:593:__socket_rwv] 0-socket.management: writev on failed (Broken pipe) [2018-04-26 05:47:41.491385] W [socket.c:593:__socket_rwv] 0-socket.management: writev on failed (Broken pipe) [2018-04-26 05:47:49.186852] I [socket.c:3672:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1) [2018-04-26 05:47:49.187982] E [rpcsvc.c:1364:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x2, Program: GlusterD svc cli, ProgVers: 2, Proc: 12) to rpc-transport (socket.management) [2018-04-26 05:47:49.188034] E [MSGID: 106430] [glusterd-utils.c:568:glusterd_submit_reply] 0-glusterd: Reply submission failed [2018-04-26 05:47:37.429615] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2018-04-26 05:47:37.429645] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: quotad service is stopped [2018-04-26 05:47:37.456899] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2018-04-26 07:24:32.163945] I [MSGID: 100030] [glusterfsd.c:2504:main] 0-glusterd: Started running glusterd version 3.12.2 (args: glusterd) [2018-04-26 07:24:32.265244] I [MSGID: 106478] [glusterd.c:1451:init] 0-management: Maximum allowed open file descriptors set to 65536 [2018-04-26 07:24:32.265379] I [MSGID: 106479] [glusterd.c:1509:init] 0-management: Using /var/lib/glusterd as working directory [2018-04-26 07:24:32.265404] I [MSGID: 106479] [glusterd.c:1514:init] 0-management: Using /var/run/gluster as pid file working directory [2018-04-26 07:24:32.308892] E [socket.c:796:__socket_server_bind] 0-socket.management: binding to failed: Address already in use [2018-04-26 07:24:32.308935] E [socket.c:799:__socket_server_bind] 0-socket.management: Port is already in use [2018-04-26 07:24:32.308969] W [rpcsvc.c:1689:rpcsvc_create_listener] 0-rpc-service: listening on transport failed [2018-04-26 07:24:32.309005] E [MSGID: 106243] [glusterd.c:1790:init] 0-management: creation of listener failed [2018-04-26 07:24:32.309026] E [MSGID: 101019] [xlator.c:503:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again [2018-04-26 07:24:32.309054] E [MSGID: 101066] [graph.c:367:glusterfs_graph_init] 0-management: initializing translator failed [2018-04-26 07:24:32.309069] E [MSGID: 101176] [graph.c:738:glusterfs_graph_activate] 0-graph: init failed [2018-04-26 07:24:32.310010] W [glusterfsd.c:1367:cleanup_and_exit] (-->glusterd(glusterfs_volumes_init+0xfd) [0x557b7818b02d] -->glusterd(glusterfs_process_volfp+0x163) [0x557b7818aed3] -->glusterd(cleanup_and_exit+0x6b) [0x557b7818a44b] ) 0-: received signum (-1), shutting down [2018-04-26 07:25:02.637219] I [MSGID: 106163] [glusterd-handshake.c:1319:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31302 [2018-04-26 07:25:33.997440] I [MSGID: 106490] [glusterd-handler.c:2617:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 403950cb-d92d-4d4f-963a-9153a70ed045 [2018-04-26 07:28:18.569623] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 403950cb-d92d-4d4f-963a-9153a70ed045, host: dhcp37-44.lab.eng.blr.redhat.com, port: 0 [2018-04-26 07:28:18.661875] I [MSGID: 106492] [glusterd-handler.c:2795:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 403950cb-d92d-4d4f-963a-9153a70ed045 [2018-04-26 07:28:18.661995] I [MSGID: 106502] [glusterd-handler.c:2840:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2018-04-26 07:32:06.721184] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 403950cb-d92d-4d4f-963a-9153a70ed045