Bug 1223299
| Summary: | Data Tiering:Frequency counters not working | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Nag Pavan Chilakam <nchilaka> |
| Component: | tier | Assignee: | Bug Updates Notification Mailing List <rhs-bugs> |
| Status: | CLOSED ERRATA | QA Contact: | Nag Pavan Chilakam <nchilaka> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | rhgs-3.1 | CC: | josferna, rhs-bugs, storage-qa-internal, trao |
| Target Milestone: | --- | ||
| Target Release: | RHGS 3.1.0 | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | 3.7.1-1 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-07-29 04:44:12 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1202842, 1223636 | ||
|
Description
Nag Pavan Chilakam
2015-05-20 09:43:54 UTC
Please set cluster.tier-promote-frequency to 0 this bug is verified and found no issues: [root@rhsqa14-vm1 ~]# gluster v create moon replica 2 10.70.47.165:/rhs/brick1/m0 10.70.47.163:/rhs/brick1/m0 10.70.47.165:/rhs/brick2/m0 10.70.47.163:/rhs/brick2/m0 volume create: moon: success: please start the volume to access data [root@rhsqa14-vm1 ~]# gluster start moon unrecognized word: start (position 0) [root@rhsqa14-vm1 ~]# gluster v start moon volume start: moon: success [root@rhsqa14-vm1 ~]# gluster v info Volume Name: moon Type: Distributed-Replicate Volume ID: 1c6bd29b-2168-4b69-bedd-8b26a89a6eb9 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.70.47.165:/rhs/brick1/m0 Brick2: 10.70.47.163:/rhs/brick1/m0 Brick3: 10.70.47.165:/rhs/brick2/m0 Brick4: 10.70.47.163:/rhs/brick2/m0 Options Reconfigured: performance.readdir-ahead: on [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster v set moon cluster.tier-demote-frequency 20 volume set: success [root@rhsqa14-vm1 ~]# glustre v set moon features.record-counters on bash: glustre: command not found [root@rhsqa14-vm1 ~]# gluster v set moon features.record-counters on volume set: success [root@rhsqa14-vm1 ~]# gluster v set moon features.ctr-enabled on volume set: success [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster v info Volume Name: moon Type: Distributed-Replicate Volume ID: 1c6bd29b-2168-4b69-bedd-8b26a89a6eb9 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: 10.70.47.165:/rhs/brick1/m0 Brick2: 10.70.47.163:/rhs/brick1/m0 Brick3: 10.70.47.165:/rhs/brick2/m0 Brick4: 10.70.47.163:/rhs/brick2/m0 Options Reconfigured: features.ctr-enabled: on features.record-counters: on cluster.tier-demote-frequency: 20 performance.readdir-ahead: on [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster v attach-tier moon replica 2 10.70.47.165:/rhs/brick3/m0 10.70.47.163:/rhs/brick3/m0 Attach tier is recommended only for testing purposes in this release. Do you want to continue? (y/n) y volume attach-tier: success volume rebalance: moon: success: Rebalance on moon has been started successfully. Use rebalance status command to check status of the rebalance process. ID: 07b230dd-3bf4-4c18-b21e-f50cb2254d29 [root@rhsqa14-vm1 ~]# gluster v info Volume Name: moon Type: Tier Volume ID: 1c6bd29b-2168-4b69-bedd-8b26a89a6eb9 Status: Started Number of Bricks: 6 Transport-type: tcp Hot Tier : Hot Tier Type : Replicate Number of Bricks: 1 x 2 = 2 Brick1: 10.70.47.163:/rhs/brick3/m0 Brick2: 10.70.47.165:/rhs/brick3/m0 Cold Tier: Cold Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick3: 10.70.47.165:/rhs/brick1/m0 Brick4: 10.70.47.163:/rhs/brick1/m0 Brick5: 10.70.47.165:/rhs/brick2/m0 Brick6: 10.70.47.163:/rhs/brick2/m0 Options Reconfigured: features.ctr-enabled: on features.record-counters: on cluster.tier-demote-frequency: 20 performance.readdir-ahead: on [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# ls -la /rhs/brick1 total 4 drwxr-xr-x. 3 root root 15 Jun 11 23:55 . drwxr-xr-x. 8 root root 4096 Jun 10 06:37 .. drwxr-xr-x. 4 root root 78 Jun 12 00:00 m0 [root@rhsqa14-vm1 ~]# ls -la /rhs/brick1/m0 total 20 drwxr-xr-x. 4 root root 78 Jun 12 00:00 . drwxr-xr-x. 3 root root 15 Jun 11 23:55 .. -rwxr-xr-x. 2 root root 244 Jun 12 00:00 fw.stop drw-------. 8 root root 110 Jun 12 00:00 .glusterfs -rwxr-xr-x. 2 root root 13496 Jun 12 00:00 rhs-system-init.sh drwxr-xr-x. 3 root root 24 Jun 11 23:58 .trashcan [root@rhsqa14-vm1 ~]# ls -la /rhs/brick2/m0 total 4 drwxr-xr-x. 4 root root 56 Jun 12 00:00 . drwxr-xr-x. 3 root root 15 Jun 11 23:55 .. drw-------. 7 root root 101 Jun 12 00:00 .glusterfs -rwxr-xr-x. 2 root root 187 Jun 12 00:00 options.sh drwxr-xr-x. 3 root root 24 Jun 11 23:58 .trashcan [root@rhsqa14-vm1 ~]# ls -la /rhs/brick3/m0 total 0 drwxr-xr-x. 4 root root 39 Jun 12 00:00 . drwxr-xr-x. 3 root root 15 Jun 11 23:58 .. drw-------. 9 root root 119 Jun 12 00:00 .glusterfs drwxr-xr-x. 3 root root 24 Jun 11 23:58 .trashcan [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# ls -la /rhs/brick1/m0 total 20 drwxr-xr-x. 4 root root 78 Jun 12 00:00 . drwxr-xr-x. 3 root root 15 Jun 11 23:55 .. -rwxr-xr-x. 2 root root 244 Jun 12 00:00 fw.stop drw-------. 8 root root 110 Jun 12 00:00 .glusterfs -rwxr-xr-x. 2 root root 13496 Jun 12 00:00 rhs-system-init.sh drwxr-xr-x. 3 root root 24 Jun 11 23:58 .trashcan [root@rhsqa14-vm1 ~]# ls -la /rhs/brick3/m0 total 0 drwxr-xr-x. 4 root root 53 Jun 12 00:03 . drwxr-xr-x. 3 root root 15 Jun 11 23:58 .. drw-------. 10 root root 128 Jun 12 00:03 .glusterfs drwxr-xr-x. 3 root root 24 Jun 11 23:58 .trashcan -rw-r--r--. 2 root root 0 Jun 12 00:03 triveni [root@rhsqa14-vm1 ~]# ls -la /rhs/brick2/m0 total 4 drwxr-xr-x. 4 root root 56 Jun 12 00:00 . drwxr-xr-x. 3 root root 15 Jun 11 23:55 .. drw-------. 7 root root 101 Jun 12 00:00 .glusterfs -rwxr-xr-x. 2 root root 187 Jun 12 00:00 options.sh drwxr-xr-x. 3 root root 24 Jun 11 23:58 .trashcan [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# date Fri Jun 12 00:04:12 EDT 2015 [root@rhsqa14-vm1 ~]# ls -la /rhs/brick1/m0 total 20 drwxr-xr-x. 4 root root 78 Jun 12 00:00 . drwxr-xr-x. 3 root root 15 Jun 11 23:55 .. -rwxr-xr-x. 2 root root 244 Jun 12 00:00 fw.stop drw-------. 8 root root 110 Jun 12 00:00 .glusterfs -rwxr-xr-x. 2 root root 13496 Jun 12 00:00 rhs-system-init.sh drwxr-xr-x. 3 root root 24 Jun 11 23:58 .trashcan [root@rhsqa14-vm1 ~]# ls -la /rhs/brick3/m0 total 0 drwxr-xr-x. 4 root root 39 Jun 12 00:04 . drwxr-xr-x. 3 root root 15 Jun 11 23:58 .. drw-------. 10 root root 128 Jun 12 00:04 .glusterfs drwxr-xr-x. 3 root root 24 Jun 11 23:58 .trashcan [root@rhsqa14-vm1 ~]# ls -la /rhs/brick2/m0 total 4 drwxr-xr-x. 4 root root 70 Jun 12 00:04 . drwxr-xr-x. 3 root root 15 Jun 11 23:55 .. drw-------. 8 root root 110 Jun 12 00:04 .glusterfs -rwxr-xr-x. 2 root root 187 Jun 12 00:00 options.sh drwxr-xr-x. 3 root root 24 Jun 11 23:58 .trashcan -rw-r--r--. 2 root root 0 Jun 12 00:03 triveni [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# glusterfs --version glusterfs 3.7.1 built on Jun 9 2015 02:31:54 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. [root@rhsqa14-vm1 ~]# rpm -qa | grep gluster glusterfs-3.7.1-1.el6rhs.x86_64 glusterfs-cli-3.7.1-1.el6rhs.x86_64 glusterfs-libs-3.7.1-1.el6rhs.x86_64 glusterfs-client-xlators-3.7.1-1.el6rhs.x86_64 glusterfs-fuse-3.7.1-1.el6rhs.x86_64 glusterfs-server-3.7.1-1.el6rhs.x86_64 glusterfs-api-3.7.1-1.el6rhs.x86_64 [root@rhsqa14-vm1 ~]# The heat counters are not working as expected. The files are only getting hashed (-----T). When promote/demote happens, the file as such is not getting moved. Also, the tier rebalance status doesnt show the file getting moved at all. 1)Create a regular 4+2 EC vol and started it and mounted it both on nfs and cifs 2)pumping IOs 3)attached a 3x replica tier 4)Now set freq counters as below: [root@rhsqa14-vm4 ~]# gluster v info ecvol Volume Name: ecvol Type: Tier Volume ID: f25eecf7-a83e-478c-b73d-1d954d7a78fe Status: Started Number of Bricks: 12 Transport-type: tcp Hot Tier : Hot Tier Type : Distributed-Replicate Number of Bricks: 2 x 3 = 6 Brick1: 10.70.46.2:/rhs/brick6/ecvol Brick2: 10.70.47.159:/rhs/brick6/ecvol Brick3: 10.70.46.2:/rhs/brick5/ecvol Brick4: 10.70.47.159:/rhs/brick5/ecvol Brick5: 10.70.46.2:/rhs/brick4/ecvol Brick6: 10.70.47.159:/rhs/brick4/ecvol Cold Tier: Cold Tier Type : Disperse Number of Bricks: 1 x (4 + 2) = 6 Brick7: 10.70.47.159:/rhs/brick1/ecvol Brick8: 10.70.46.2:/rhs/brick1/ecvol Brick9: 10.70.47.159:/rhs/brick2/ecvol Brick10: 10.70.46.2:/rhs/brick2/ecvol Brick11: 10.70.46.2:/rhs/brick3/ecvol Brick12: 10.70.47.159:/rhs/brick3/ecvol Options Reconfigured: cluster.tier-promote-frequency: 6 cluster.tier-demote-frequency: 5 features.record-counters: on features.ctr-enabled: on performance.readdir-ahead: on Now. i created a file "newfile", which went to hot tier. But after about 6 sec, the file got hashed to cold tier(as part of demote) but didn't get moved to cold tier completely. The tieing rebalance status too shows as zero files rebalanced [root@rhsqa14-vm4 ~]# ls -l /rhs/brick*/ecvol/*file* ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick1/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick2/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick3/ecvol/newfile -rw-r-Sr-T. 2 root root 0 Jun 15 06:38 /rhs/brick4/ecvol/newfile -rw-r--r--. 2 root root 0 Jun 15 2015 /rhs/brick5/ecvol/xfs_file -rw-r--r--. 2 root root 0 Jun 15 2015 /rhs/brick6/ecvol/xfs_file [root@rhsqa14-vm4 ~]# ls -l /rhs/brick*/ecvol/*file* ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick1/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick2/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick3/ecvol/newfile -rw-r-Sr-T. 2 root root 0 Jun 15 06:41 /rhs/brick4/ecvol/newfile -rw-r--r--. 2 root root 0 Jun 15 2015 /rhs/brick5/ecvol/xfs_file -rw-r--r--. 2 root root 0 Jun 15 2015 /rhs/brick6/ecvol/xfs_file [root@rhsqa14-vm4 ~]# ls -l /rhs/brick*/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick1/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick2/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick3/ecvol/newfile -rw-r-Sr-T. 2 root root 0 Jun 15 06:41 /rhs/brick4/ecvol/newfile [root@rhsqa14-vm4 ~]# ls -l /rhs/brick*/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick1/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick2/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick3/ecvol/newfile -rw-r-Sr-T. 2 root root 0 Jun 15 06:41 /rhs/brick4/ecvol/newfile [root@rhsqa14-vm4 ~]# ls -l /rhs/brick*/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick1/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick2/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick3/ecvol/newfile -rw-r-Sr-T. 2 root root 0 Jun 15 06:41 /rhs/brick4/ecvol/newfile [root@rhsqa14-vm4 ~]# ls -l /rhs/brick*/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick1/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick2/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick3/ecvol/newfile -rw-r-Sr-T. 2 root root 0 Jun 15 06:41 /rhs/brick4/ecvol/newfile [root@rhsqa14-vm4 ~]# ls -l /rhs/brick*/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick1/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick2/ecvol/newfile ---------T. 2 root root 0 Jun 15 06:39 /rhs/brick3/ecvol/newfile -rw-r-Sr-T. 2 root root 0 Jun 15 06:41 /rhs/brick4/ecvol/newfile Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1495.html |