Red Hat Bugzilla – Bug 1612098
Brick not coming up on a volume after rebooting the node
Last modified: 2018-10-02 11:42:35 EDT
Description of problem: ======================= If we reboot a node which has a brick which is a part of a volume. The brick does not come up after reboot Version-Release number of selected component (if applicable): =========================================================== server-3.12.2-15.el7rhgs.x86_64 How reproducible: ================= 3/3 Steps to Reproduce: ================== 1.Create a Volume 2.Reboot a node which has a brick which is part of that volume 3.after reboot brick does not come up Actual results: ============== Brick should come up after reboot Expected results: ================= Brick not coming up after reboot Additional info: ================ On a EC volume [root@dhcp35-56 ~]# gluster v status Status of volume: dispersed Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.56:/gluster/brick1/vol2 N/A N/A N N/A Brick 10.70.35.228:/gluster/brick1/vol2 49152 0 Y 23860 Brick 10.70.35.17:/gluster/brick3/vol2 49152 0 Y 3190 Brick 10.70.35.3:/gluster/brick1/vol2 49152 0 Y 22153 Brick 10.70.35.27:/gluster/brick1/vol2 49152 0 Y 13528 Brick 10.70.35.130:/gluster/brick1/vol2 N/A N/A N N/A Self-heal Daemon on localhost N/A N/A Y 2040 Self-heal Daemon on 10.70.35.130 N/A N/A Y 22985 Self-heal Daemon on dhcp35-3.lab.eng.blr.re dhat.com N/A N/A Y 8224 Self-heal Daemon on 10.70.35.228 N/A N/A Y 10154 Self-heal Daemon on 10.70.35.27 N/A N/A Y 32318 Self-heal Daemon on 10.70.35.17 N/A N/A Y 22964 Task Status of Volume dispersed ------------------------------------------------------------------------------ There are no active volume tasks On a replica volume Status of volume: test Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.35.56:/gluster/brick1/r1 N/A N/A N N/A Brick 10.70.35.228:/gluster/brick1/r1 49153 0 Y 10131 Brick 10.70.35.17:/gluster/brick3/r1 49153 0 Y 22941 Self-heal Daemon on localhost N/A N/A Y 2040 Self-heal Daemon on dhcp35-3.lab.eng.blr.re dhat.com N/A N/A Y 8224 Self-heal Daemon on 10.70.35.130 N/A N/A Y 22985 Self-heal Daemon on 10.70.35.17 N/A N/A Y 22964 Self-heal Daemon on 10.70.35.27 N/A N/A Y 32318 Self-heal Daemon on 10.70.35.228 N/A N/A Y 10154 Task Status of Volume test Logs - ==== [2018-08-03 12:46:56.304320] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2018-08-03 12:46:56.312524] I [MSGID: 106163] [glusterd-handshake.c:1319:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31302 [2018-08-03 12:46:56.766187] I [MSGID: 106163] [glusterd-handshake.c:1319:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31302 [2018-08-03 12:46:56.793152] I [MSGID: 106490] [glusterd-handler.c:2627:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 8725052e-568b-4123-ac75-21e9574c923e [2018-08-03 12:46:56.810897] I [MSGID: 106493] [glusterd-handler.c:3890:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.70.35.228 (0), ret: 0, op_ret: 0 [2018-08-03 12:46:56.885420] E [glusterd-utils.c:6135:glusterd_brick_start] 0-management: fsid comparison is failed it means Brick root path /gluster/brick1/vol2 is not created by glusterd, start/attach will als o fail [2018-08-03 12:46:56.885976] E [glusterd-utils.c:6135:glusterd_brick_start] 0-management: fsid comparison is failed it means Brick root path /gluster/brick1/r1 is not created by glusterd, start/attach will also fail [2018-08-03 12:46:57.082261] I [MSGID: 106163] [glusterd-handshake.c:1319:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31302 [2018-08-03 12:46:57.118469] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 8725052e-568b-4123-ac75-21e9574c923e, host: 10.70.35.228, port: 0 [2018-08-03 12:46:57.155205] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2018-08-03 12:46:57.155478] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: nfs already stopped [2018-08-03 12:46:57.155529] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: nfs service is stopped [2018-08-03 12:46:57.156890] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2018-08-03 12:46:57.159529] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: glustershd already stopped [2018-08-03 12:46:57.159570] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: glustershd service is stopped [2018-08-03 12:46:57.159732] I [MSGID: 106567] [glusterd-svc-mgmt.c:211:glusterd_svc_start] 0-management: Starting glustershd service [2018-08-03 12:46:58.168512] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2018-08-03 12:46:58.169517] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: quotad already stopped [2018-08-03 12:46:58.169584] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: quotad service is stopped [2018-08-03 12:46:58.169687] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2018-08-03 12:46:58.170082] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: bitd already stopped [2018-08-03 12:46:58.170121] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: bitd service is stopped [2018-08-03 12:46:58.170189] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2018-08-03 12:46:58.170471] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: scrub already stopped [2018-08-03 12:46:58.170496] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: scrub service is stopped [2018-08-03 12:46:58.170609] E [glusterd-utils.c:6135:glusterd_brick_start] 0-management: fsid comparison is failed it means Brick root path /gluster/brick1/vol2 is not created by glusterd, start/attach will als o fail [2018-08-03 12:46:58.452086] E [glusterd-utils.c:6135:glusterd_brick_start] 0-management: fsid comparison is failed it means Brick root path /gluster/brick1/r1 is not created by glusterd, start/attach will also fail [2018-08-03 12:46:58.804420] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2018-08-03 12:46:58.804871] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2018-08-03 12:46:58.806954] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 9248003d-418e-41d5-acd8-de2808bc6191, host: 10.70.35.130, port: 0 [2018-08-03 12:46:58.849347] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: cf5cf97a-fa28-4107-8a01-1ead0965eab4, host: 10.70.35.17, port: 0 [2018-08-03 12:46:58.876069] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 8d650b81-ca20-409d-8721-621fe166f2bd, host: dhcp35-3.lab.eng.blr.redhat.com, port: 0 [2018-08-03 12:46:58.922170] I [MSGID: 106492] [glusterd-handler.c:2805:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: cf5cf97a-fa28-4107-8a01-1ead0965eab4 [2018-08-03 12:46:59.003237] I [MSGID: 106502] [glusterd-handler.c:2850:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2018-08-03 12:46:59.158677] I [MSGID: 106492] [glusterd-handler.c:2805:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8d650b81-ca20-409d-8721-621fe166f2bd [2018-08-03 12:46:59.158827] I [MSGID: 106502] [glusterd-handler.c:2850:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2018-08-03 12:46:59.327880] I [MSGID: 106492] [glusterd-handler.c:2805:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 9248003d-418e-41d5-acd8-de2808bc6191
upstream patch : https://review.gluster.org/20638
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607