Description of problem: ======================= Have a 4 node cluster with eventing enabled. Have a 1*(4+2) volume as cold tier and 1*4 as hot tier. Try to detach using the command 'gluster volume tier detach start' followed with 'gluster volume tier detach commit' OR 'gluster volume tier detach force' directly. In either of the cases, it prompts for a confirmation message, if we would want to really go ahead with it. When we reply in negation, it does not go ahead and do the said operation. However, an event for the same is seen in the webhook listener, passing the wrong impression to any application that is consuming these events. Version-Release number of selected component (if applicable): ============================================================ 3.8.4-2 How reproducible: ================== Always Steps to Reproduce: ==================== 1. Have a 4 node cluster with eventing enabled. 2. Create 1*(4+2) disperse volume 'disp' and attach a 1*4 hot tier to the same 3. Execute the command 'gluster volume tier detach force', and when it asks for a confirmation message, reply 'n' to it 4. Execute the command 'gluster volume tier detach start'. 5. When the rebalance completes, execute the command 'gluster volume tier detach commit'. When probed for a confirmation, answer 'n' to it. Actual results: =============== Step3 generates an event TIER_DETACH_FORCE and step5 generates an event TIER_DETACH_COMMIT {u'message': {u'vol': u'disp'}, u'event': u'TIER_DETACH_FORCE', u'ts': 1476851551, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'} {u'message': {u'vol': u'disp'}, u'event': u'TIER_DETACH_COMMIT', u'ts': 1476850418, u'nodeid': u'ed362eb3-421c-4a25-ad0e-82ef157ea328'} Expected results: ================= Step3 and Step5 should not have generated any event as we answered negatively in the confirmation message. Additional info: ================ [root@dhcp46-239 ~]# gluster peer status Number of Peers: 3 Hostname: 10.70.46.240 Uuid: 72c4f894-61f7-433e-a546-4ad2d7f0a176 State: Peer in Cluster (Connected) Hostname: 10.70.46.242 Uuid: 1e8967ae-51b2-4c27-907e-a22a83107fd0 State: Peer in Cluster (Connected) Hostname: 10.70.46.218 Uuid: 0dea52e0-8c32-4616-8ef8-16db16120eaa State: Peer in Cluster (Connected) [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# rpm -qa | grep gluster nfs-ganesha-gluster-2.3.1-8.el7rhgs.x86_64 glusterfs-3.8.4-2.el7rhgs.x86_64 glusterfs-api-devel-3.8.4-2.el7rhgs.x86_64 glusterfs-debuginfo-3.8.4-1.el7rhgs.x86_64 glusterfs-libs-3.8.4-2.el7rhgs.x86_64 glusterfs-api-3.8.4-2.el7rhgs.x86_64 python-gluster-3.8.4-2.el7rhgs.noarch glusterfs-geo-replication-3.8.4-2.el7rhgs.x86_64 glusterfs-rdma-3.8.4-2.el7rhgs.x86_64 glusterfs-fuse-3.8.4-2.el7rhgs.x86_64 glusterfs-cli-3.8.4-2.el7rhgs.x86_64 glusterfs-server-3.8.4-2.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-2.el7rhgs.x86_64 glusterfs-client-xlators-3.8.4-2.el7rhgs.x86_64 glusterfs-devel-3.8.4-2.el7rhgs.x86_64 glusterfs-events-3.8.4-2.el7rhgs.x86_64 [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# gluster v info Volume Name: disp Type: Tier Volume ID: a9999464-b094-4213-a422-c11fed555674 Status: Started Snapshot Count: 0 Number of Bricks: 10 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 4 Brick1: 10.70.46.218:/bricks/brick2/disp_tier4 Brick2: 10.70.46.242:/bricks/brick2/disp_tier3 Brick3: 10.70.46.240:/bricks/brick2/disp_tier2 Brick4: 10.70.46.239:/bricks/brick2/disp_tier1 Cold Tier: Cold Tier Type : Disperse Number of Bricks: 1 x (4 + 2) = 6 Brick5: 10.70.46.239:/bricks/brick0/disp1 Brick6: 10.70.46.240:/bricks/brick0/disp2 Brick7: 10.70.46.242:/bricks/brick0/disp3 Brick8: 10.70.46.218:/bricks/brick0/disp4 Brick9: 10.70.46.239:/bricks/brick1/disp5 Brick10: 10.70.46.240:/bricks/brick1/disp6 Options Reconfigured: cluster.tier-mode: cache features.ctr-enabled: on transport.address-family: inet performance.readdir-ahead: on cluster.enable-shared-storage: enable [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# [root@dhcp46-239 ~]#
Will be fixed in the patch for https://bugzilla.redhat.com/show_bug.cgi?id=1387098
Fixed as part of: https://bugzilla.redhat.com/show_bug.cgi?id=1386185
Tested and verified this on the build 3.8.4-5 We no longer see an event when we answer 'no' to the confirmatory message displayed during 'tier detach force' and 'tier detach commit'. Tried out multiple times. The expected event is seen only when we answer in affirmation to the confirmation. Moving this to verified in 3.2. Detailed logs are pasted below. [root@dhcp46-239 ~]# rpm -qa | grep gluster nfs-ganesha-gluster-2.3.1-8.el7rhgs.x86_64 glusterfs-api-3.8.4-5.el7rhgs.x86_64 python-gluster-3.8.4-5.el7rhgs.noarch glusterfs-client-xlators-3.8.4-5.el7rhgs.x86_64 glusterfs-server-3.8.4-5.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-5.el7rhgs.x86_64 glusterfs-devel-3.8.4-5.el7rhgs.x86_64 glusterfs-libs-3.8.4-5.el7rhgs.x86_64 glusterfs-fuse-3.8.4-5.el7rhgs.x86_64 glusterfs-api-devel-3.8.4-5.el7rhgs.x86_64 glusterfs-rdma-3.8.4-5.el7rhgs.x86_64 glusterfs-3.8.4-5.el7rhgs.x86_64 glusterfs-cli-3.8.4-5.el7rhgs.x86_64 glusterfs-geo-replication-3.8.4-5.el7rhgs.x86_64 glusterfs-debuginfo-3.8.4-4.el7rhgs.x86_64 glusterfs-events-3.8.4-5.el7rhgs.x86_64 [root@dhcp46-239 ~]# gluster peer status Number of Peers: 3 Hostname: 10.70.46.240 Uuid: 72c4f894-61f7-433e-a546-4ad2d7f0a176 State: Peer in Cluster (Connected) Hostname: 10.70.46.242 Uuid: 1e8967ae-51b2-4c27-907e-a22a83107fd0 State: Peer in Cluster (Connected) Hostname: 10.70.46.218 Uuid: 0dea52e0-8c32-4616-8ef8-16db16120eaa State: Peer in Cluster (Connected) [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# gluster v info Volume Name: ozone Type: Tier Volume ID: 376cdde0-194f-460a-b273-3904a704a7dd Status: Started Snapshot Count: 0 Number of Bricks: 8 Transport-type: tcp Hot Tier : Hot Tier Type : Replicate Number of Bricks: 1 x 2 = 2 Brick1: 10.70.46.218:/bricks/brick1/ozone_tier1 Brick2: 10.70.46.218:/bricks/brick0/ozone_tier0 Cold Tier: Cold Tier Type : Disperse Number of Bricks: 1 x (4 + 2) = 6 Brick3: 10.70.46.239:/bricks/brick0/ozone0 Brick4: 10.70.46.240:/bricks/brick0/ozone2 Brick5: 10.70.46.242:/bricks/brick0/ozone2 Brick6: 10.70.46.239:/bricks/brick1/ozone3 Brick7: 10.70.46.240:/bricks/brick1/ozone4 Brick8: 10.70.46.242:/bricks/brick1/ozone5 Options Reconfigured: cluster.tier-mode: cache features.ctr-enabled: on transport.address-family: inet performance.readdir-ahead: on nfs.disable: on cluster.enable-shared-storage: disable [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# gluster v tier ozone detach start volume detach-tier start: success ID: 2cc1504c-773c-47ac-91a1-73fd58ef3cea [root@dhcp46-239 ~]# gluster v tier ozone detach status Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- 10.70.46.218 0 0Bytes 0 0 0 completed 0:0:1 [root@dhcp46-239 ~]# gluster v tier ozone detach Usage: volume tier <VOLNAME> status volume tier <VOLNAME> start [force] volume tier <VOLNAME> attach [<replica COUNT>] <NEW-BRICK>... [force] volume tier <VOLNAME> detach <start|stop|status|commit|force> Tier command failed [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# gluster v tier ozone detach commit Removing tier can result in data loss. Do you want to Continue? (y/n) n [root@dhcp46-239 ~]# gluster v tier ozone detach commit Removing tier can result in data loss. Do you want to Continue? (y/n) y volume detach-tier commit: success Check the detached bricks to ensure all files are migrated. If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick. [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# gluster v tier ozone attach replica 2 10.70.46.218:/bricks/brick0/ozone_tier0 10.70.46.218:/bricks/brick1/ozone_tier1 volume attach-tier: success Tiering Migration Functionality: ozone: success: Attach tier is successful on ozone. use tier status to check the status. ID: d3937d1a-5842-4c6e-8759-b6b565464ba7 [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# [root@dhcp46-239 ~]# gluster v tier ozone detach force Removing tier can result in data loss. Do you want to Continue? (y/n) n [root@dhcp46-239 ~]# gluster v tier ozone detach force Removing tier can result in data loss. Do you want to Continue? (y/n) y volume detach-tier commit force: success [root@dhcp46-239 ~]# gluster v tier ozone attach replica 2 10.70.46.218:/bricks/brick0/ozone_tier0 10.70.46.218:/bricks/brick1/ozone_tier1 volume attach-tier: success Tiering Migration Functionality: ozone: success: Attach tier is successful on ozone. use tier status to check the status. ID: 49d1c39d-0267-481b-ad00-b0ae42686641 [root@dhcp46-239 ~]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html