Bug 1887332 - [SSL] Peers get disconnected after enabling management encryption
Summary: [SSL] Peers get disconnected after enabling management encryption
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterd
Version: rhgs-3.5
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Srijan Sivakumar
QA Contact: Bala Konda Reddy M
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-12 07:13 UTC by Sayalee
Modified: 2021-07-12 04:06 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-14 09:53:00 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Sayalee 2020-10-12 07:13:30 UTC
Describe the issue:
-------------------
On a 4 node cluster setup, the peers are in 'connected' state, then enabled the SSL management encryption, post that, the peers go into 'disconnected' state. 


Is this issue reproducible? If yes, share more details.:
--------------------------------------------------------
The issue is reproducible on the setup under test multiple times.


Steps to Reproduce:
------------------
1. Stop the glusterd process on the server nodes.
2. Generate Self-signed certificate using the steps in: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-network_encryption#chap-Network_Encryption-Preparing_Certificates
3. Enable the management encryption using the steps in:
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/ch20s04
4. Start glusterd process on all server nodes.


Actual results:
--------------
The peers get disconnected after enabling the encryption.

 
Expected results:
-----------------
Peers should not get disconnected.
 

Additional info:
--------------- 
* Sosreports will be provided.

* No volumes are present on the cluster.

* Snippet from glusterd.log:

[2020-10-12 01:30:31.754542] I [socket.c:4322:ssl_setup_connection_params] 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 10.70.36.54:1023
[2020-10-12 01:30:31.758089] I [socket.c:4322:ssl_setup_connection_params] 0-management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 10.70.36.54:24007
[2020-10-12 01:30:31.760988] I [socket.c:4322:ssl_setup_connection_params] 0-management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 10.70.36.56:24007
[2020-10-12 01:30:31.762828] I [socket.c:4322:ssl_setup_connection_params] 0-socket.management: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 1 for peer 10.70.36.56:1021
[2020-10-12 01:30:31.768526] E [socket.c:246:ssl_dump_error_stack] 0-management:   error:140840FF:SSL routines:ssl3_connect:unknown state
[2020-10-12 01:30:31.768602] I [MSGID: 106004] [glusterd-handler.c:6533:__glusterd_peer_rpc_notify] 0-management: Peer <10.70.36.56> (<e3f06e22-9244-41d2-a6b1-8e3e8156bc46>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-10-12 01:30:31.768882] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f5c2f0a6e6b] (--> /lib64/libgfrpc.so.0(+0xd814)[0x7f5c2ee4c814] (--> /lib64/libgfrpc.so.0(+0xd92e)[0x7f5c2ee4c92e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3)[0x7f5c2ee4d9e3] (--> /lib64/libgfrpc.so.0(+0xf578)[0x7f5c2ee4e578] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2020-10-12 01:30:31.768545 (xid=0x4)
[2020-10-12 01:30:31.768901] E [MSGID: 106167] [glusterd-handshake.c:2262:__glusterd_peer_dump_version_cbk] 0-management: Error through RPC layer, retry again later
[2020-10-12 01:30:31.772000] E [socket.c:246:ssl_dump_error_stack] 0-management:   error:140840FF:SSL routines:ssl3_connect:unknown state
[2020-10-12 01:30:31.772076] I [MSGID: 106004] [glusterd-handler.c:6533:__glusterd_peer_rpc_notify] 0-management: Peer <10.70.36.54> (<ef7e3db2-14ed-4333-9cfc-aa425744828c>), in state <Peer in Cluster>, has disconnected from glusterd.
[2020-10-12 01:30:31.772344] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f5c2f0a6e6b] (--> /lib64/libgfrpc.so.0(+0xd814)[0x7f5c2ee4c814] (--> /lib64/libgfrpc.so.0(+0xd92e)[0x7f5c2ee4c92e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3)[0x7f5c2ee4d9e3] (--> /lib64/libgfrpc.so.0(+0xf578)[0x7f5c2ee4e578] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2020-10-12 01:30:31.772019 (xid=0x4)
[2020-10-12 01:30:31.772362] E [MSGID: 106167] [glusterd-handshake.c:2262:__glusterd_peer_dump_version_cbk] 0-management: Error through RPC layer, retry again later


Note You need to log in before you can comment on or make changes to this bug.