Bug 1211264
| Summary: | Data Tiering: glusterd(management) communication issues seen on tiering setup | |||
|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Nag Pavan Chilakam <nchilaka> | |
| Component: | tiering | Assignee: | Mohammed Rafi KC <rkavunga> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | bugs <bugs> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | mainline | CC: | bugs, dlambrig, vagarwal | |
| Target Milestone: | --- | Keywords: | Reopened, Triaged | |
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.8rc2 | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1219846 (view as bug list) | Environment: | ||
| Last Closed: | 2016-06-16 12:50:21 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1186580, 1219846, 1260923 | |||
|
Description
Nag Pavan Chilakam
2015-04-13 13:12:35 UTC
CLI executed logs: ================== [root@yarrow glusterfs]# gluster v info disperse Volume Name: disperse Type: Disperse Volume ID: a6e4f8dd-bbf8-484f-9d1c-c6267899bb0a Status: Created Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: yarrow:/yarrow_200G_7/disperse Brick2: yarrow:/yarrow_200G_8/disperse Brick3: rhs-client6:/brick15/disperse [root@yarrow glusterfs]# gluster v start disperse volume start: disperse: success [root@yarrow glusterfs]# gluster v attach-tier disperse replica 2 yarrow:/yarrow_ssd_75G_2/disperse rhs-client6:/brick16/disperse volume add-brick: failed: Commit failed on localhost. Please check the log file for more details. [root@yarrow glusterfs]# gluster v info disperse Volume Name: disperse Type: Tier Volume ID: a6e4f8dd-bbf8-484f-9d1c-c6267899bb0a Status: Started Number of Bricks: 2 x 2 = 5 Transport-type: tcp Bricks: Brick1: rhs-client6:/brick16/disperse Brick2: yarrow:/yarrow_ssd_75G_2/disperse Brick3: yarrow:/yarrow_200G_7/disperse Brick4: yarrow:/yarrow_200G_8/disperse Brick5: rhs-client6:/brick15/disperse [root@yarrow glusterfs]# gluster v detach-tier disperse volume remove-brick unknown: failed: Commit failed on localhost. Please check the log file for more details. [root@yarrow glusterfs]# gluster v info disperse Volume Name: disperse Type: Distributed-Disperse Volume ID: a6e4f8dd-bbf8-484f-9d1c-c6267899bb0a Status: Started Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: yarrow:/yarrow_200G_7/disperse Brick2: yarrow:/yarrow_200G_8/disperse Brick3: rhs-client6:/brick15/disperse [root@yarrow glusterfs]# gluster v attach-tier disperse replica 2 yarrow:/yarrow_ssd_75G_2/disperse rhs-client6:/brick16/disperse volume add-brick: failed: /yarrow_ssd_75G_2/disperse is already part of a volume [root@yarrow glusterfs]# gluster v attach-tier disperse replica 2 yarrow:/yarrow_ssd_75G_2/disperse rhs-client6:/brick16/disperse force volume add-brick: failed: /yarrow_ssd_75G_2/disperse is already part of a volume [root@yarrow glusterfs]# gluster v info disperse Volume Name: disperse Type: Tier Volume ID: a6e4f8dd-bbf8-484f-9d1c-c6267899bb0a Status: Started Number of Bricks: 2 x 2 = 5 Transport-type: tcp Bricks: Brick1: rhs-client6:/brick16/disperse Brick2: yarrow:/yarrow_ssd_75G_2/disperse Brick3: yarrow:/yarrow_200G_7/disperse Brick4: yarrow:/yarrow_200G_8/disperse Brick5: rhs-client6:/brick15/disperse [root@yarrow glusterfs]# gluster v detach-tier disperse volume remove-brick unknown: failed: Commit failed on localhost. Please check the log file for more details. [root@yarrow glusterfs]# gluster v attach-tier disperse replica 2 yarrow:/yarrow_ssd_75G_2/disperse rhs-client6:/brick16/disperse force volume add-brick: failed: /yarrow_ssd_75G_2/disperse is already part of a volume [root@yarrow glusterfs]# sosreports@rhsqe-repo:/home/repo/sosreports/1211264 We have submitted fix 10108, which is not merged. The issues with detach-tier may no longer exist (I do not see them). Returning to QE to retest. Dan, We are stilling seeing issues with glusted communication b/w nodes with a tiered volume as of 28th April. Kindly put it "ON_QA" only when the fix is availble for testing REVIEW: http://review.gluster.org/10449 (glusterd/tiering: Exchange tier info during glusted handshake) posted (#1) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/10449 (glusterd/tiering: Exchange tier info during glusted handshake) posted (#2) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/10449 (glusterd/tiering: Exchange tier info during glusted handshake) posted (#3) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/10449 (glusterd/tiering: Exchange tier info during glusted handshake) posted (#4) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/10449 (glusterd/tiering: Exchange tier info during glusted handshake) posted (#5) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/10449 (glusterd/tiering: Exchange tier info during glusted handshake) posted (#6) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/10449 (glusterd/tiering: Exchange tier info during glusted handshake) posted (#7) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/10449 (glusterd/tiering: Exchange tier info during glusted handshake) posted (#8) for review on master by Vijay Bellur (vbellur) REVIEW: http://review.gluster.org/10449 (glusterd/tiering: Exchange tier info during glusted handshake) posted (#9) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/10768 (tiering: Correct errors in cli and glusterd) posted (#1) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/10768 (tiering: Correct errors in cli and glusterd) posted (#2) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/10768 (tiering: Correct errors in cli and glusterd) posted (#3) for review on master by Vijay Bellur (vbellur) REVIEW: http://review.gluster.org/10768 (tiering: Correct errors in cli and glusterd) posted (#4) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/10449 (glusterd/tiering: Exchange tier info during glusted handshake) posted (#10) for review on master by mohammed rafi kc (rkavunga) REVIEW: http://review.gluster.org/10768 (tiering: Correct errors in cli and glusterd) posted (#5) for review on master by mohammed rafi kc (rkavunga) COMMIT: http://review.gluster.org/10449 committed in master by Kaushal M (kaushal) ------ commit d133071e7ced1794e09ffe4ef8cb14cf5b9f7e75 Author: Mohammed Rafi KC <rkavunga> Date: Wed Apr 29 12:00:40 2015 +0530 glusterd/tiering: Exchange tier info during glusted handshake Change-Id: Ibc2f8eeb32d3e5dfd6945ca8a6d5f0f80a78ebac BUG: 1211264 Signed-off-by: Mohammed Rafi KC <rkavunga> Reviewed-on: http://review.gluster.org/10449 Reviewed-by: Kaushal M <kaushal> Tested-by: Gluster Build System <jenkins.com> Tested-by: NetBSD Build System COMMIT: http://review.gluster.org/10768 committed in master by Kaushal M (kaushal) ------ commit f1fb71bbf677be40b7bab997221f832c7fa7527a Author: Mohammed Rafi KC <rkavunga> Date: Wed May 13 16:53:22 2015 +0530 tiering: Correct errors in cli and glusterd Problem 1: volume info shows Cold Bricks instead of Tier type eg: Volume Name: patchy2 Type: Tier Volume ID: 28c25b8d-b8a1-45dc-b4b7-cbd0b344f58f Status: Started Number of Bricks: 3 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 1 Brick1: 10.70.1.35:/home/brick43 Cold Bricks: Cold Tier Type : Distribute Number of Bricks: 2 Brick2: 10.70.1.35:/home/brick19 Brick3: 10.70.1.35:/home/brick16 Options Reconfigured: Problem 2: Detach-tier sending enums of Rebalance detach-tier has it's own Enum to send with detach-tier command, using that enums will make more appropriate. Problem 3: Wrongly sets hot_brick count during the dictionary copying for response Change-Id: Icc054a999a679456881bc70511470d32ff8a86e4 BUG: 1211264 Signed-off-by: Mohammed Rafi KC <rkavunga> Reviewed-on: http://review.gluster.org/10768 Reviewed-by: Atin Mukherjee <amukherj> Reviewed-by: Kaushal M <kaushal> Tested-by: Gluster Build System <jenkins.com> Tested-by: NetBSD Build System Fix for this BZ is already present in a GlusterFS release. You can find clone of this BZ, fixed in a GlusterFS release and closed. Hence closing this mainline BZ as well. This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user |