Bug 1224289
| Summary: | NFS server crashed in one of the node after enabling the management SSL | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | ssamanta | ||||
| Component: | glusterd | Assignee: | Kaushal <kaushal> | ||||
| Status: | CLOSED WORKSFORME | QA Contact: | storage-qa-internal <storage-qa-internal> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | rhgs-3.1 | CC: | dshetty, mzywusko, nlevinki, rcyriac, vagarwal, vbellur | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2015-05-26 12:35:36 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1223636 | ||||||
| Attachments: |
|
||||||
|
Description
ssamanta
2015-05-22 12:58:58 UTC
After the above issue the gluster command fails with "Request timeout". [root@gqas003 ~]# gluster volume info Error : Request timed out No volumes present [root@gqas003 ~]# [root@gqas007 ~]# service glusterd status glusterd (pid 6157) is running... [root@gqas007 ~]# gluster peer status Error : Request timed out [root@gqas007 ~]# Can you please provide the coredump for the crashed nfs process? Sobhan provided me with access to the systems he faces the issues on. What I found was that the bricks were left running when the switch to management encryption was done. This is incorrect. When enabling or disabling management encryption, all GlusterFS processes - GlusterD, bricks, clients etc. - need to be stopped and started. This is needed because, 1. Interactions between processes trying to use encrypted connections and processes using unencrypted connections is undefined, and will lead to failures as observed here. 2. All GlusterFS processes communicate with GlusterD, so changing management encryption's state affects all of them 3. It is not possible to do a dynamically switch an unencrypted connection to encrypted or vice-versa. Sobhan was following [1], which isn't complete with respect to upgrade procedures. This lack of documentation was one of the issues we found when I got involved with the GlusterFS network encryption and Manila. As a result, I've written up a guide on how-to use network encryption with GlusterFS at [2], which covers many different scenarios of enabling network encryption in GlusterFS, including enabling management encryption on an existing cluster (as is the case here). I'll work with the documentation team to provide proper official documentation for RHGS based on [2]. But till we get the official documentation, please refer to [2] for network encryption guidance. Sobhan, could you please re-run your tests following the guidelines given in [2]. You shouldn't be facing any issues if you follow it. In case you do hit issues even when following the guidelines, please let met know. As this is issue is not really a bug with GlusterFS, but arose because of incorrect setup/steps followed, I suggest closing this bug. I'll do the same if there are no objections. [1]: https://github.com/gluster/glusterfs/blob/master/doc/admin-guide/en-US/markdown/admin_ssl.md [2]: https://kshlm.in/network-encryption-in-glusterfs/ Created attachment 1029922 [details]
verification logs
|