Hide Forgot
Describe the issue: ------------------- On a 4 node cluster setup, the peers are in 'connected' state, then enabled the SSL management encryption, post that, the peers go into 'disconnected' state. Is this issue reproducible? If yes, share more details.: -------------------------------------------------------- The issue is reproducible on the setup under test multiple times. Steps to Reproduce: ------------------ 1. Stop the glusterd process on the server nodes. 2. Generate Self-signed certificate using the steps in: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-network_encryption#chap-Network_Encryption-Preparing_Certificates 3. Enable the management encryption using the steps in: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/ch20s04 4. Start glusterd process on all server nodes. Actual results: -------------- The peers get disconnected after enabling the encryption. Expected results: ----------------- Peers should not get disconnected. Additional info: --------------- * Sosreports will be provided. * No volumes are present on the cluster. * Snippet from glusterd.log: [2020-10-12 01:30:31.754542] I [socket.c:4322:ssl_setup_connection_params] 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 10.70.36.54:1023 [2020-10-12 01:30:31.758089] I [socket.c:4322:ssl_setup_connection_params] 0-management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 10.70.36.54:24007 [2020-10-12 01:30:31.760988] I [socket.c:4322:ssl_setup_connection_params] 0-management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 10.70.36.56:24007 [2020-10-12 01:30:31.762828] I [socket.c:4322:ssl_setup_connection_params] 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 10.70.36.56:1021 [2020-10-12 01:30:31.768526] E [socket.c:246:ssl_dump_error_stack] 0-management: error:140840FF:SSL routines:ssl3_connect:unknown state [2020-10-12 01:30:31.768602] I [MSGID: 106004] [glusterd-handler.c:6533:__glusterd_peer_rpc_notify] 0-management: Peer <10.70.36.56> (<e3f06e22-9244-41d2-a6b1-8e3e8156bc46>), in state <Peer in Cluster>, has disconnected from glusterd. [2020-10-12 01:30:31.768882] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f5c2f0a6e6b] (--> /lib64/libgfrpc.so.0(+0xd814)[0x7f5c2ee4c814] (--> /lib64/libgfrpc.so.0(+0xd92e)[0x7f5c2ee4c92e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3)[0x7f5c2ee4d9e3] (--> /lib64/libgfrpc.so.0(+0xf578)[0x7f5c2ee4e578] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2020-10-12 01:30:31.768545 (xid=0x4) [2020-10-12 01:30:31.768901] E [MSGID: 106167] [glusterd-handshake.c:2262:__glusterd_peer_dump_version_cbk] 0-management: Error through RPC layer, retry again later [2020-10-12 01:30:31.772000] E [socket.c:246:ssl_dump_error_stack] 0-management: error:140840FF:SSL routines:ssl3_connect:unknown state [2020-10-12 01:30:31.772076] I [MSGID: 106004] [glusterd-handler.c:6533:__glusterd_peer_rpc_notify] 0-management: Peer <10.70.36.54> (<ef7e3db2-14ed-4333-9cfc-aa425744828c>), in state <Peer in Cluster>, has disconnected from glusterd. [2020-10-12 01:30:31.772344] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f5c2f0a6e6b] (--> /lib64/libgfrpc.so.0(+0xd814)[0x7f5c2ee4c814] (--> /lib64/libgfrpc.so.0(+0xd92e)[0x7f5c2ee4c92e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3)[0x7f5c2ee4d9e3] (--> /lib64/libgfrpc.so.0(+0xf578)[0x7f5c2ee4e578] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2020-10-12 01:30:31.772019 (xid=0x4) [2020-10-12 01:30:31.772362] E [MSGID: 106167] [glusterd-handshake.c:2262:__glusterd_peer_dump_version_cbk] 0-management: Error through RPC layer, retry again later