Bug 1447920
| Summary: | [Brick MUX]: Tier daemons in failed state on a setup where brick-multiplexing was on-and-put-off-later | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Sweta Anandpara <sanandpa> |
| Component: | core | Assignee: | hari gowtham <hgowtham> |
| Status: | CLOSED ERRATA | QA Contact: | Bala Konda Reddy M <bmekala> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | rhgs-3.3 | CC: | amukherj, nchilaka, rhinduja, rhs-bugs, storage-qa-internal, vbellur |
| Target Milestone: | --- | ||
| Target Release: | RHGS 3.3.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | brick-multiplexing | ||
| Fixed In Version: | glusterfs-3.8.4-26 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-09-21 04:41:45 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1417151 | ||
|
Description
Sweta Anandpara
2017-05-04 08:55:36 UTC
[qe@rhsqe-repo 1447920]$ hostname rhsqe-repo.lab.eng.blr.redhat.com [qe@rhsqe-repo 1447920]$ pwd /home/repo/sosreports/1447920 [qe@rhsqe-repo 1447920]$ ll total 279940 -rwxr-xr-x. 1 qe qe 47589384 May 4 15:48 sosreport-sysreg-prod_dhcp47-113-20170504044728.tar.xz -rwxr-xr-x. 1 qe qe 48654484 May 4 15:48 sosreport-sysreg-prod_dhcp47-114-20170504044728.tar.xz -rwxr-xr-x. 1 qe qe 48254232 May 4 15:48 sosreport-sysreg-prod_dhcp47-115-20170504044728.tar.xz -rwxr-xr-x. 1 qe qe 45685132 May 4 15:48 sosreport-sysreg-prod_dhcp47-116-20170504044728.tar.xz -rwxr-xr-x. 1 qe qe 46089020 May 4 15:48 sosreport-sysreg-prod_dhcp47-117-20170504044728.tar.xz -rwxr-xr-x. 1 qe qe 50375272 May 4 15:48 sosreport-sysreg-prod_dhcp47-121-20170504044728.tar.xz [qe@rhsqe-repo 1447920]$ RCA: From the logs, I was able to see that the brick multiplexing was enabled, then a volume was created and it was converted into a tiered volume. then multiplexing was disabled after which the upgrade was done. After the upgrade, the tierd didn't come up on the node as it wasn't able to connect to the subvolumes [2017-04-25 08:23:46.587380] E [MSGID: 114058] [client-handshake.c:1537:client_query_portmap_cbk] 0-disp-client-6: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. glusterd kept trying for a while and as it was unable to connect, it got a child down event. [2017-04-25 08:23:25.458734] E [socket.c:2318:socket_connect_finish] 0-glusterfs: connection to ::1:24007 failed (Connection refused); disconnecting socket This patch (https://review.gluster.org/#/c/17101/) fixes this socket issue. Upstream patches : https://review.gluster.org/#/q/topic:bug-1444596 Downstream patches: https://code.engineering.redhat.com/gerrit/#/c/105595/ https://code.engineering.redhat.com/gerrit/#/c/105596/ Build : 3.8.4-36 Enabled brick mux, created a volume and attached tier. Disabled brick mux, and rebooted one node. tier daemons are coming up after that. @rahul, switching brick-mux on and off when volume is present is it recommended? BUILD: 3.8.4-38 1. Enabled brick-mux 2. created a volume and made it tiered volume 3. Disabled brick-mux (with and without this step) 4. restarted glusterd 5. tier daemons are coming up (visible in status too, gluster vol tier <vol> status) Hence marking it as verified Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2774 |