Bug 1540759
| Summary: | Failure to demote tiered volume file that is continuously modified by client during hot tier detachment. | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | vnosov <vnosov> |
| Component: | tiering | Assignee: | bugs <bugs> |
| Status: | CLOSED WONTFIX | QA Contact: | bugs <bugs> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | mainline | CC: | bugs, jbyers, vnosov |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-11-02 08:13:25 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Release 3.12 has been EOLd and this bug was still found to be in the NEW state, hence moving the version to mainline, to triage the same and take appropriate actions. Patch https://review.gluster.org/#/c/glusterfs/+/21331/ removes tier functionality from GlusterFS. https://bugzilla.redhat.com/show_bug.cgi?id=1642807 is used as the tracking bug for this. Recommendation is to convert your tier volume to regular volume (either replicate, ec, or plain distribute) with "tier detach" command before upgrade, and use backend features like dm-cache etc to utilize the caching from backend to provide better performance and functionality. |
Description of problem: Enable CIFS export for tiered volume. Map the volume to drive on Windows. Run IOMeter with writes and reads to the volume continuously during the test. Check that file created by the IOMeter is on hot tier. Start "detach" hot tier for the volume. Verify that status of the detachment is "completed". Issue "commit" the detachment. Check file used for the IOMeter testing. The file still has bit "T" at the permissions. Version-Release number of selected component (if applicable): glusterfs 3.12.3 How reproducible: 100% Steps to Reproduce: System: Centos 6. SC182, glusterFS 3.12.3 Test volume: tiered "nas-volume-0003", with enabled CIFS export. Size of cold tier is 1GB, size of hot tier is 1GB. Client: Windows host. Application: IOmeter, 64KB sequential, 50% reads and 50% writes. IOmeter creates file with name "iobw.tst". Size of the file is 512000000 bytes. [root@SC-182 log]# gluster volume info nas-volume-0003 Volume Name: nas-volume-0003 Type: Tier Volume ID: 6ea89454-985f-4ad6-925d-4f82560961e9 Status: Started Snapshot Count: 0 Number of Bricks: 2 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 1 Brick1: 10.10.60.182:/exports/nas-segment-0003/nas-volume-0003 Cold Tier: Cold Tier Type : Distribute Number of Bricks: 1 Brick2: 10.10.60.182:/exports/nas-segment-0002/nas-volume-0003 Options Reconfigured: cluster.tier-mode: cache features.ctr-enabled: on nfs.disable: on transport.address-family: inet nfs.addr-namelookup: off performance.stat-prefetch: off performance.quick-read: off server.allow-insecure: on [root@SC-182 log]# gluster volume status nas-volume-0003 Status of volume: nas-volume-0003 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick 10.10.60.182:/exports/nas-segment-0003/nas-volume-0003 62000 0 Y 10130 Cold Bricks: Brick 10.10.60.182:/exports/nas-segment-0002/nas-volume-0003 62001 0 Y 23421 Tier Daemon on localhost N/A N/A Y 10151 Task Status of Volume nas-volume-0003 ------------------------------------------------------------------------------ There are no active volume tasks Start detach: 13145 01/30/2018 14:33:18.715250327 1517351598 command: /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier nas-volume-0003 detach start 2>&1 13159 01/30/2018 14:33:18.780335608 1517351598 command: /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "start" 13159 01/30/2018 14:33:24.062642838 1517351604 status=0 /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "start" 13145 01/30/2018 14:33:24.078741347 1517351604 status=0 /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier nas-volume-0003 detach start 2>&1 [root@SC-182 log]# ls -l /exports/nas-segment-0003/nas-volume-0003/cifs_share/ total 500000 drwxr-xr-x 2 root root 6 Jan 29 16:29 dir1 -rw-rw---- 2 root nas_users 512000000 Jan 30 14:37 iobw.tst [root@SC-182 log]# ls -l /exports/nas-segment-0002/nas-volume-0003/cifs_share/ total 904 drwxr-xr-x 2 root root 8192 Jan 29 16:29 dir1 ---------T 2 root nas_users 453705728 Jan 30 14:34 iobw.tst [root@SC-182 log]# /usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "status" volume detach tier status: success Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 1 488.3MB 50 0 0 completed 0:00:08 Hot segment: [root@SC-182 log]# ls -l /exports/nas-segment-0003/nas-volume-0003/cifs_share/ total 500000 drwxr-xr-x 2 root root 6 Jan 29 16:29 dir1 -rw-rw---- 2 root nas_users 512000000 Jan 30 14:41 iobw.tst Cold segment: [root@SC-182 log]# [root@SC-182 log]# ls -l /exports/nas-segment-0002/nas-volume-0003/cifs_share/ total 904 drwxr-xr-x 2 root root 8192 Jan 29 16:29 dir1 ---------T 2 root nas_users 453705728 Jan 30 14:34 iobw.tst [root@SC-182 log]# gluster volume profile nas-volume-0003 start Starting volume profile on nas-volume-0003 has been successful [root@SC-182 log]# gluster volume profile nas-volume-0003 info clear Brick: 10.10.60.182:/exports/nas-segment-0002/nas-volume-0003 ------------------------------------------------------------- Cleared stats. Brick: 10.10.60.182:/exports/nas-segment-0003/nas-volume-0003 ------------------------------------------------------------- Cleared stats. [root@SC-182 log]# gluster volume profile nas-volume-0003 info Brick: 10.10.60.182:/exports/nas-segment-0003/nas-volume-0003 =============> Hot segment ------------------------------------------------------------- Cumulative Stats: Block Size: 65536b+ No. of Reads: 0 No. of Writes: 365 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 13 RELEASEDIR 0.71 70.43 us 32.00 us 140.00 us 14 OPENDIR 1.58 57.18 us 23.00 us 138.00 us 38 STAT 4.83 80.22 us 41.00 us 152.00 us 83 GETXATTR 5.53 61.54 us 28.00 us 152.00 us 124 STATFS 5.79 70.73 us 43.00 us 151.00 us 113 LOOKUP 81.56 308.24 us 227.00 us 479.00 us 365 WRITE Duration: 29 seconds Data Read: 0 bytes Data Written: 23920640 bytes Interval 0 Stats: Block Size: 65536b+ No. of Reads: 0 No. of Writes: 365 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 13 RELEASEDIR 0.71 70.43 us 32.00 us 140.00 us 14 OPENDIR 1.58 57.18 us 23.00 us 138.00 us 38 STAT 4.83 80.22 us 41.00 us 152.00 us 83 GETXATTR 5.53 61.52 us 28.00 us 152.00 us 124 STATFS 5.79 70.73 us 43.00 us 151.00 us 113 LOOKUP 81.56 308.24 us 227.00 us 479.00 us 365 WRITE Duration: 29 seconds Data Read: 0 bytes Data Written: 23920640 bytes Brick: 10.10.60.182:/exports/nas-segment-0002/nas-volume-0003 ===================> Cold segment ------------------------------------------------------------- Cumulative Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 13 RELEASEDIR 1.65 50.62 us 34.00 us 71.00 us 8 STATFS 4.01 70.50 us 36.00 us 126.00 us 14 OPENDIR 7.43 48.13 us 24.00 us 197.00 us 38 STAT 12.21 214.71 us 161.00 us 255.00 us 14 READDIRP 21.04 77.27 us 40.00 us 210.00 us 67 GETXATTR 53.66 62.89 us 41.00 us 162.00 us 210 LOOKUP Duration: 29 seconds Data Read: 0 bytes Data Written: 0 bytes Interval 0 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 13 RELEASEDIR 1.65 50.62 us 34.00 us 71.00 us 8 STATFS 4.01 70.50 us 36.00 us 126.00 us 14 OPENDIR 7.43 48.13 us 24.00 us 197.00 us 38 STAT 12.21 214.71 us 161.00 us 255.00 us 14 READDIRP 21.04 77.27 us 40.00 us 210.00 us 67 GETXATTR 53.66 62.89 us 41.00 us 162.00 us 210 LOOKUP Duration: 29 seconds Data Read: 0 bytes Data Written: 0 bytes [root@SC-182 log]# gluster volume profile nas-volume-0003 stop Stopping volume profile on nas-volume-0003 has been successful Do commit: 22219 01/30/2018 14:48:51.919431953 1517352531 command: /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier nas-volume-0003 detach commit 2>&1 22233 01/30/2018 14:48:51.984308571 1517352531 command: /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "commit" 22233 01/30/2018 14:48:53.212079189 1517352533 status=0 /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "commit" 22219 01/30/2018 14:48:53.228757277 1517352533 status=0 /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier nas-volume-0003 detach commit 2>&1 [root@SC-182 log]# ls -l /exports/nas-segment-0002/nas-volume-0003/cifs_share/ total 455440 drwxr-xr-x 2 root root 8192 Jan 29 16:29 dir1 ---------T 2 root nas_users 511967232 Jan 30 14:49 iobw.tst [root@SC-182 log]# ls -l /exports/nas-segment-0003/nas-volume-0003/cifs_share/ ls: cannot access /exports/nas-segment-0003/nas-volume-0003/cifs_share/: No such file or directory [root@SC-182 log]# ls -l /samba/nas-volume-0003/cifs_share/ total 499972 drwxr-xr-x 2 root root 4096 Jan 29 16:29 dir1 ---------T 1 root nas_users 511967232 Jan 30 14:49 iobw.tst CIFS client continue to do IO's! [root@SC-182 log]# ls -l /samba/nas-volume-0003/cifs_share/ total 499972 drwxr-xr-x 2 root root 4096 Jan 29 16:29 dir1 ---------T 1 root nas_users 511967232 Jan 30 14:51 iobw.tst Actual results: Check file "iobw.tst" on the volume: [root@SC-182 log]# ls -l /samba/nas-volume-0003/cifs_share/ total 499972 drwxr-xr-x 2 root root 4096 Jan 29 16:29 dir1 ---------T 1 root nas_users 511967232 Jan 30 14:51 iobw.tst Size and file permissions are wrong. Expected results: The test was executed successfully on glusterfs 3.7.18. The correct results have to be like the next: [root@SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/ total 500004 drwxr-xr-x 2 root root 4096 Jan 30 15:11 dir1 -rw-rw---- 1 root nas_users 512000000 Jan 30 15:31 iobw.tst Additional info: Results of successfull test on glusterFS 3.7.18: Do the same on SC183 that runs glusterfs 3.7.18. [root@SC-183 log]# gluster volume info tiered-volume-0015 Volume Name: tiered-volume-0015 Type: Distribute Volume ID: aefc0a6e-2515-40f1-a888-2f3168e0fa07 Status: Started Number of Bricks: 1 Transport-type: tcp Bricks: Brick1: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015 Options Reconfigured: server.allow-insecure: on performance.quick-read: off performance.stat-prefetch: off nfs.disable: on nfs.addr-namelookup: off performance.readdir-ahead: on snap-activate-on-create: enable cluster.enable-shared-storage: disable [root@SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/ total 500004 drwxr-xr-x 2 root root 4096 Jan 30 15:06 dir1 -rw-rw---- 1 root nas_users 512000000 Jan 30 15:06 iobw.tst [root@SC-183 log]# ls -l /exports/cold-segment-0004/tiered-volume-0015/cifs_share/ total 500000 drwxr-xr-x 2 root root 141 Jan 30 15:06 dir1 -rw-rw---- 2 root nas_users 512000000 Jan 30 15:07 iobw.tst [root@SC-183 log]# gluster volume status tiered-volume-0015 Status of volume: tiered-volume-0015 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.10.60.183:/exports/cold-segment-00 04/tiered-volume-0015 62078 0 Y 22474 Task Status of Volume tiered-volume-0015 ------------------------------------------------------------------------------ There are no active volume tasks [root@SC-183 log]# gluster volume profile tiered-volume-0015 start Starting volume profile on tiered-volume-0015 has been successful [root@SC-183 log]# gluster volume profile tiered-volume-0015 info clear Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015 ----------------------------------------------------------------- Cleared stats. [root@SC-183 log]# gluster volume profile tiered-volume-0015 info Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015 ----------------------------------------------------------------- Cumulative Stats: Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 9 0 37835 No. of Writes: 0 20334 9 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 30 RELEASEDIR 0.00 44.15 us 31.00 us 88.00 us 20 STATFS 0.01 53.50 us 39.00 us 150.00 us 30 OPENDIR 0.01 40.54 us 23.00 us 171.00 us 50 STAT 0.03 146.25 us 40.00 us 323.00 us 60 READDIRP 0.05 74.78 us 47.00 us 223.00 us 180 GETXATTR 0.08 74.15 us 56.00 us 186.00 us 280 LOOKUP 32.54 422.47 us 47.00 us 827002.00 us 20343 WRITE 67.28 469.44 us 31.00 us 826207.00 us 37851 READ Duration: 21 seconds Data Read: 4959404032 bytes Data Written: 1333788672 bytes Interval 0 Stats: Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 9 0 37835 No. of Writes: 0 20334 9 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 30 RELEASEDIR 0.00 44.15 us 31.00 us 88.00 us 20 STATFS 0.01 53.50 us 39.00 us 150.00 us 30 OPENDIR 0.01 40.54 us 23.00 us 171.00 us 50 STAT 0.03 146.25 us 40.00 us 323.00 us 60 READDIRP 0.05 74.78 us 47.00 us 223.00 us 180 GETXATTR 0.08 74.15 us 56.00 us 186.00 us 280 LOOKUP 32.54 422.47 us 47.00 us 827002.00 us 20343 WRITE 67.28 469.44 us 31.00 us 826207.00 us 37851 READ Duration: 21 seconds Data Read: 4959404032 bytes Data Written: 1333788672 bytes [root@SC-183 log]# gluster volume profile tiered-volume-0015 stop Stopping volume profile on tiered-volume-0015 has been successful Attach hot tier: 4780 01/30/2018 15:11:13.505967116 1517353873 command: /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015 attach 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015 2>&1 | /bin/grep "Tiering Migration Functionality: tiered-volume-0015: success" 4795 01/30/2018 15:11:13.604148675 1517353873 command: /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "attach" "10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015" 4795 01/30/2018 15:11:19.087525009 1517353879 status=0 /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "attach" "10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015" 4780 01/30/2018 15:11:19.105676210 1517353879 status=0 /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015 attach 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015 2>&1 | /bin/grep "Tiering Migration Functionality: tiered-volume-0015: success" [root@SC-183 log]# gluster volume info tiered-volume-0015 Volume Name: tiered-volume-0015 Type: Tier Volume ID: aefc0a6e-2515-40f1-a888-2f3168e0fa07 Status: Started Number of Bricks: 2 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 1 Brick1: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015 Cold Tier: Cold Tier Type : Distribute Number of Bricks: 1 Brick2: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015 Options Reconfigured: features.quota-deem-statfs: on features.inode-quota: on features.quota: on cluster.tier-mode: cache features.ctr-enabled: on server.allow-insecure: on performance.quick-read: off performance.stat-prefetch: off nfs.disable: on nfs.addr-namelookup: off performance.readdir-ahead: on snap-activate-on-create: enable cluster.enable-shared-storage: disable [root@SC-183 log]# gluster volume status tiered-volume-0015 Status of volume: tiered-volume-0015 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick 10.10.60.183:/exports/hot-segment-000 5/tiered-volume-0015 62079 0 Y 4824 Cold Bricks: Brick 10.10.60.183:/exports/cold-segment-00 04/tiered-volume-0015 62078 0 Y 22474 Quota Daemon on localhost N/A N/A Y 4980 Task Status of Volume tiered-volume-0015 ------------------------------------------------------------------------------ Task : Tier migration ID : cf15d19a-8ba6-4cfd-9794-fc7a77050f45 Status : in progress [root@SC-183 log]# ls -l /exports/cold-segment-0004/tiered-volume-0015/cifs_share/ total 448 drwxr-xr-x 2 root root 141 Jan 30 15:11 dir1 ---------T 2 root nas_users 236519424 Jan 30 15:12 iobw.tst [root@SC-183 log]# ls -l /exports/hot-segment-0005/tiered-volume-0015/cifs_share/ total 500080 drwxr-xr-x 2 root root 6 Jan 30 15:11 dir1 -rw-rw---- 2 root nas_users 512000000 Jan 30 15:17 iobw.tst [root@SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/ total 500004 drwxr-xr-x 2 root root 4096 Jan 30 15:11 dir1 -rw-rw---- 1 root nas_users 512000000 Jan 30 15:17 iobw.tst [root@SC-183 log]# gluster volume profile tiered-volume-0015 start Starting volume profile on tiered-volume-0015 has been successful [root@SC-183 log]# gluster volume profile tiered-volume-0015 info clear Brick: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015 ---------------------------------------------------------------- Cleared stats. Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015 ----------------------------------------------------------------- Cleared stats. [root@SC-183 log]# gluster volume profile tiered-volume-0015 info Brick: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015 ---------------------------------------------------------------- Cumulative Stats: Block Size: 65536b+ 131072b+ No. of Reads: 0 0 No. of Writes: 20702 364 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 30 RELEASEDIR 0.00 44.37 us 36.00 us 68.00 us 30 OPENDIR 0.00 33.45 us 26.00 us 65.00 us 49 STAT 0.03 648.21 us 36.00 us 1160.00 us 24 STATFS 0.04 77.75 us 45.00 us 160.00 us 218 GETXATTR 0.08 117.96 us 83.00 us 386.00 us 317 LOOKUP 99.85 2287.28 us 212.00 us 827377.00 us 21067 WRITE Duration: 20 seconds Data Read: 0 bytes Data Written: 1404436480 bytes Interval 0 Stats: Block Size: 65536b+ 131072b+ No. of Reads: 0 0 No. of Writes: 20702 364 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 30 RELEASEDIR 0.00 44.37 us 36.00 us 68.00 us 30 OPENDIR 0.00 33.45 us 26.00 us 65.00 us 49 STAT 0.03 648.21 us 36.00 us 1160.00 us 24 STATFS 0.04 77.75 us 45.00 us 160.00 us 218 GETXATTR 0.08 117.96 us 83.00 us 386.00 us 317 LOOKUP 99.85 2287.28 us 212.00 us 827377.00 us 21067 WRITE Duration: 20 seconds Data Read: 0 bytes Data Written: 1404436480 bytes Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015 ----------------------------------------------------------------- Cumulative Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 30 RELEASEDIR 1.67 61.13 us 41.00 us 97.00 us 30 OPENDIR 1.69 37.76 us 26.00 us 75.00 us 49 STAT 13.00 80.09 us 46.00 us 165.00 us 178 GETXATTR 14.22 779.65 us 619.00 us 1047.00 us 20 STATFS 14.76 269.72 us 67.00 us 815.00 us 60 READDIRP 54.66 108.40 us 81.00 us 351.00 us 553 LOOKUP Duration: 20 seconds Data Read: 0 bytes Data Written: 0 bytes Interval 0 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 30 RELEASEDIR 1.67 61.13 us 41.00 us 97.00 us 30 OPENDIR 1.69 37.76 us 26.00 us 75.00 us 49 STAT 13.00 80.09 us 46.00 us 165.00 us 178 GETXATTR 14.22 779.65 us 619.00 us 1047.00 us 20 STATFS 14.76 269.72 us 67.00 us 815.00 us 60 READDIRP 54.66 108.40 us 81.00 us 351.00 us 553 LOOKUP Duration: 20 seconds Data Read: 0 bytes Data Written: 0 bytes [root@SC-183 log]# gluster volume profile tiered-volume-0015 stop Stopping volume profile on tiered-volume-0015 has been successful Detach hot tier: 15364 01/30/2018 15:24:09.119881869 1517354649 command: /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015 detach start 2>&1 15378 01/30/2018 15:24:09.196084465 1517354649 command: /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "start" 15378 01/30/2018 15:24:14.514959281 1517354654 status=0 /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "start" 15364 01/30/2018 15:24:14.533662479 1517354654 status=0 /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015 detach start 2>&1 [root@SC-183 log]# /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "status" Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 1 488.3MB 7 0 0 completed 0:0:19 [root@SC-183 log]# ls -l /exports/hot-segment-0005/tiered-volume-0015/cifs_share/ total 0 drwxr-xr-x 2 root root 6 Jan 30 15:11 dir1 [root@SC-183 log]# ls -l /exports/cold-segment-0004/tiered-volume-0015/cifs_share/ total 500064 drwxr-xr-x 2 root root 141 Jan 30 15:11 dir1 -rw-rw---- 2 root nas_users 512000000 Jan 30 15:26 iobw.tst [root@SC-183 log]# gluster volume profile tiered-volume-0015 start Starting volume profile on tiered-volume-0015 has been successful [root@SC-183 log]# gluster volume profile tiered-volume-0015 info clear Brick: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015 ---------------------------------------------------------------- Cleared stats. Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015 ----------------------------------------------------------------- Cleared stats. [root@SC-183 log]# gluster volume profile tiered-volume-0015 info Brick: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015 ---------------------------------------------------------------- Cumulative Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 24 RELEASEDIR 2.34 57.12 us 38.00 us 84.00 us 24 OPENDIR 3.13 45.75 us 28.00 us 356.00 us 40 STAT 18.46 84.43 us 48.00 us 256.00 us 128 GETXATTR 21.07 770.94 us 556.00 us 1017.00 us 16 STATFS 55.00 118.36 us 84.00 us 261.00 us 272 LOOKUP Duration: 17 seconds Data Read: 0 bytes Data Written: 0 bytes Interval 0 Stats: %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 24 RELEASEDIR 2.34 57.12 us 38.00 us 84.00 us 24 OPENDIR 3.13 45.75 us 28.00 us 356.00 us 40 STAT 18.46 84.43 us 48.00 us 256.00 us 128 GETXATTR 21.07 770.94 us 556.00 us 1017.00 us 16 STATFS 55.00 118.36 us 84.00 us 261.00 us 272 LOOKUP Duration: 17 seconds Data Read: 0 bytes Data Written: 0 bytes Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015 ----------------------------------------------------------------- Cumulative Stats: Block Size: 65536b+ 131072b+ No. of Reads: 0 0 No. of Writes: 17104 275 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 24 RELEASEDIR 0.00 44.75 us 37.00 us 58.00 us 24 OPENDIR 0.00 33.20 us 26.00 us 71.00 us 40 STAT 0.04 767.94 us 603.00 us 958.00 us 16 STATFS 0.04 261.44 us 64.00 us 674.00 us 48 READDIRP 0.04 82.23 us 48.00 us 177.00 us 160 GETXATTR 0.17 112.10 us 80.00 us 554.00 us 448 LOOKUP 99.69 1678.12 us 214.00 us 629362.00 us 17378 WRITE Duration: 17 seconds Data Read: 0 bytes Data Written: 1156972544 bytes Interval 0 Stats: Block Size: 65536b+ 131072b+ No. of Reads: 0 0 No. of Writes: 17104 275 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 24 RELEASEDIR 0.00 44.75 us 37.00 us 58.00 us 24 OPENDIR 0.00 33.20 us 26.00 us 71.00 us 40 STAT 0.04 767.94 us 603.00 us 958.00 us 16 STATFS 0.04 261.44 us 64.00 us 674.00 us 48 READDIRP 0.04 82.23 us 48.00 us 177.00 us 160 GETXATTR 0.17 112.10 us 80.00 us 554.00 us 448 LOOKUP 99.69 1678.12 us 214.00 us 629362.00 us 17378 WRITE Duration: 17 seconds Data Read: 0 bytes Data Written: 1156972544 bytes [root@SC-183 log]# gluster volume profile tiered-volume-0015 stop Stopping volume profile on tiered-volume-0015 has been successful [root@SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/ total 500004 drwxr-xr-x 2 root root 4096 Jan 30 15:11 dir1 -rw-rw---- 1 root nas_users 512000000 Jan 30 15:30 iobw.tst Commit: 19949 01/30/2018 15:31:07.145654679 1517355067 command: /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015 detach commit 2>&1 19982 01/30/2018 15:31:08.636386875 1517355068 command: /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "commit" 19982 01/30/2018 15:31:10.000126293 1517355070 status=0 /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "commit" 19949 01/30/2018 15:31:10.047054376 1517355070 status=0 /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015 detach commit 2>&1 [root@SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/ total 500004 drwxr-xr-x 2 root root 4096 Jan 30 15:11 dir1 -rw-rw---- 1 root nas_users 512000000 Jan 30 15:31 iobw.tst [root@SC-183 log]# ls -l /exports/cold-segment-0004/tiered-volume-0015/cifs_share/ total 500064 drwxr-xr-x 2 root root 141 Jan 30 15:11 dir1 -rw-rw---- 2 root nas_users 512000000 Jan 30 15:32 iobw.tst [root@SC-183 log]# gluster volume profile tiered-volume-0015 start Starting volume profile on tiered-volume-0015 has been successful [root@SC-183 log]# gluster volume profile tiered-volume-0015 info clear Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015 ----------------------------------------------------------------- Cleared stats. [root@SC-183 log]# gluster volume profile tiered-volume-0015 info Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015 ----------------------------------------------------------------- Cumulative Stats: Block Size: 65536b+ 131072b+ No. of Reads: 0 0 No. of Writes: 16722 29 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 22 RELEASEDIR 0.00 47.50 us 37.00 us 69.00 us 22 OPENDIR 0.00 45.85 us 26.00 us 379.00 us 40 STAT 0.02 552.44 us 490.00 us 614.00 us 16 STATFS 0.02 239.14 us 63.00 us 491.00 us 44 READDIRP 0.03 74.09 us 44.00 us 157.00 us 204 GETXATTR 0.05 102.15 us 78.00 us 447.00 us 240 LOOKUP 99.87 2793.68 us 61.00 us 1026245.00 us 16751 WRITE Duration: 16 seconds Data Read: 0 bytes Data Written: 1099694080 bytes Interval 0 Stats: Block Size: 65536b+ 131072b+ No. of Reads: 0 0 No. of Writes: 16722 29 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 22 RELEASEDIR 0.00 47.50 us 37.00 us 69.00 us 22 OPENDIR 0.00 45.85 us 26.00 us 379.00 us 40 STAT 0.02 552.44 us 490.00 us 614.00 us 16 STATFS 0.02 239.14 us 63.00 us 491.00 us 44 READDIRP 0.03 74.09 us 44.00 us 157.00 us 204 GETXATTR 0.05 102.15 us 78.00 us 447.00 us 240 LOOKUP 99.87 2793.68 us 61.00 us 1026245.00 us 16751 WRITE Duration: 16 seconds Data Read: 0 bytes Data Written: 1099694080 bytes [root@SC-183 log]# gluster volume profile tiered-volume-0015 stop Stopping volume profile on tiered-volume-0015 has been successful