Bug 1540759 - Failure to demote tiered volume file that is continuously modified by client during hot tier detachment.
Summary: Failure to demote tiered volume file that is continuously modified by client ...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: GlusterFS
Classification: Community
Component: tiering
Version: mainline
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact: bugs@gluster.org
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-31 23:49 UTC by vnosov
Modified: 2018-11-02 08:13 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-02 08:13:25 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description vnosov 2018-01-31 23:49:30 UTC
Description of problem:

Enable CIFS export for tiered volume. Map the volume to drive on Windows. Run IOMeter with writes and reads to the volume continuously during the test. Check that file created by the IOMeter is on hot tier.
Start "detach" hot tier for the volume. Verify that status of the detachment is "completed". Issue "commit" the detachment. Check file used for the IOMeter testing. The file still has bit "T" at the permissions.


Version-Release number of selected component (if applicable):
glusterfs 3.12.3


How reproducible: 100%


Steps to Reproduce:

System: Centos 6.
SC182, glusterFS 3.12.3

Test volume: tiered "nas-volume-0003", with enabled CIFS export. Size of cold tier is 1GB, size of hot tier is 1GB.

Client: Windows host.
Application: IOmeter, 64KB sequential, 50% reads and 50% writes. IOmeter creates file with name "iobw.tst". Size of the file is 512000000 bytes.

[root@SC-182 log]# gluster volume info nas-volume-0003

Volume Name: nas-volume-0003
Type: Tier
Volume ID: 6ea89454-985f-4ad6-925d-4f82560961e9
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 1
Brick1: 10.10.60.182:/exports/nas-segment-0003/nas-volume-0003
Cold Tier:
Cold Tier Type : Distribute
Number of Bricks: 1
Brick2: 10.10.60.182:/exports/nas-segment-0002/nas-volume-0003
Options Reconfigured:
cluster.tier-mode: cache
features.ctr-enabled: on
nfs.disable: on
transport.address-family: inet
nfs.addr-namelookup: off
performance.stat-prefetch: off
performance.quick-read: off
server.allow-insecure: on

[root@SC-182 log]# gluster volume status nas-volume-0003
Status of volume: nas-volume-0003
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.10.60.182:/exports/nas-segment-0003/nas-volume-0003                           62000     0          Y       10130
Cold Bricks:
Brick 10.10.60.182:/exports/nas-segment-0002/nas-volume-0003                           62001     0          Y       23421
Tier Daemon on localhost                    N/A       N/A        Y       10151

Task Status of Volume nas-volume-0003
------------------------------------------------------------------------------
There are no active volume tasks

Start detach:

13145 01/30/2018 14:33:18.715250327 1517351598 command: /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier nas-volume-0003 detach start 2>&1
13159 01/30/2018 14:33:18.780335608 1517351598 command: /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "start"
13159 01/30/2018 14:33:24.062642838 1517351604 status=0 /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "start"
13145 01/30/2018 14:33:24.078741347 1517351604 status=0 /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier nas-volume-0003 detach start 2>&1


[root@SC-182 log]# ls -l /exports/nas-segment-0003/nas-volume-0003/cifs_share/
total 500000
drwxr-xr-x 2 root root              6 Jan 29 16:29 dir1
-rw-rw---- 2 root nas_users 512000000 Jan 30 14:37 iobw.tst

[root@SC-182 log]# ls -l /exports/nas-segment-0002/nas-volume-0003/cifs_share/
total 904
drwxr-xr-x 2 root root           8192 Jan 29 16:29 dir1
---------T 2 root nas_users 453705728 Jan 30 14:34 iobw.tst

[root@SC-182 log]# /usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "status"
volume detach tier status: success
                                    Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                1       488.3MB            50             0             0            completed        0:00:08

Hot segment:
[root@SC-182 log]# ls -l /exports/nas-segment-0003/nas-volume-0003/cifs_share/
total 500000
drwxr-xr-x 2 root root              6 Jan 29 16:29 dir1
-rw-rw---- 2 root nas_users 512000000 Jan 30 14:41 iobw.tst

Cold segment:
[root@SC-182 log]#
[root@SC-182 log]# ls -l /exports/nas-segment-0002/nas-volume-0003/cifs_share/
total 904
drwxr-xr-x 2 root root           8192 Jan 29 16:29 dir1
---------T 2 root nas_users 453705728 Jan 30 14:34 iobw.tst


[root@SC-182 log]# gluster volume profile nas-volume-0003 start
Starting volume profile on nas-volume-0003 has been successful

[root@SC-182 log]# gluster volume profile nas-volume-0003 info clear
Brick: 10.10.60.182:/exports/nas-segment-0002/nas-volume-0003
-------------------------------------------------------------
Cleared stats.
Brick: 10.10.60.182:/exports/nas-segment-0003/nas-volume-0003
-------------------------------------------------------------
Cleared stats.

[root@SC-182 log]# gluster volume profile nas-volume-0003 info
Brick: 10.10.60.182:/exports/nas-segment-0003/nas-volume-0003     =============> Hot segment
-------------------------------------------------------------
Cumulative Stats:
   Block Size:              65536b+
 No. of Reads:                    0
No. of Writes:                  365
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             13  RELEASEDIR
      0.71      70.43 us      32.00 us     140.00 us             14     OPENDIR
      1.58      57.18 us      23.00 us     138.00 us             38        STAT
      4.83      80.22 us      41.00 us     152.00 us             83    GETXATTR
      5.53      61.54 us      28.00 us     152.00 us            124      STATFS
      5.79      70.73 us      43.00 us     151.00 us            113      LOOKUP
     81.56     308.24 us     227.00 us     479.00 us            365       WRITE

    Duration: 29 seconds
   Data Read: 0 bytes
Data Written: 23920640 bytes

Interval 0 Stats:
   Block Size:              65536b+
 No. of Reads:                    0
No. of Writes:                  365
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             13  RELEASEDIR
      0.71      70.43 us      32.00 us     140.00 us             14     OPENDIR
      1.58      57.18 us      23.00 us     138.00 us             38        STAT
      4.83      80.22 us      41.00 us     152.00 us             83    GETXATTR
      5.53      61.52 us      28.00 us     152.00 us            124      STATFS
      5.79      70.73 us      43.00 us     151.00 us            113      LOOKUP
     81.56     308.24 us     227.00 us     479.00 us            365       WRITE

    Duration: 29 seconds
   Data Read: 0 bytes
Data Written: 23920640 bytes

Brick: 10.10.60.182:/exports/nas-segment-0002/nas-volume-0003  ===================> Cold segment
-------------------------------------------------------------
Cumulative Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             13  RELEASEDIR
      1.65      50.62 us      34.00 us      71.00 us              8      STATFS
      4.01      70.50 us      36.00 us     126.00 us             14     OPENDIR
      7.43      48.13 us      24.00 us     197.00 us             38        STAT
     12.21     214.71 us     161.00 us     255.00 us             14    READDIRP
     21.04      77.27 us      40.00 us     210.00 us             67    GETXATTR
     53.66      62.89 us      41.00 us     162.00 us            210      LOOKUP

    Duration: 29 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Interval 0 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             13  RELEASEDIR
      1.65      50.62 us      34.00 us      71.00 us              8      STATFS
      4.01      70.50 us      36.00 us     126.00 us             14     OPENDIR
      7.43      48.13 us      24.00 us     197.00 us             38        STAT
     12.21     214.71 us     161.00 us     255.00 us             14    READDIRP
     21.04      77.27 us      40.00 us     210.00 us             67    GETXATTR
     53.66      62.89 us      41.00 us     162.00 us            210      LOOKUP

    Duration: 29 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

[root@SC-182 log]# gluster volume profile nas-volume-0003 stop
Stopping volume profile on nas-volume-0003 has been successful

Do commit:

22219 01/30/2018 14:48:51.919431953 1517352531 command: /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier nas-volume-0003 detach commit 2>&1
22233 01/30/2018 14:48:51.984308571 1517352531 command: /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "commit"
22233 01/30/2018 14:48:53.212079189 1517352533 status=0 /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "nas-volume-0003" "detach" "commit"
22219 01/30/2018 14:48:53.228757277 1517352533 status=0 /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier nas-volume-0003 detach commit 2>&1


[root@SC-182 log]# ls -l /exports/nas-segment-0002/nas-volume-0003/cifs_share/
total 455440
drwxr-xr-x 2 root root           8192 Jan 29 16:29 dir1
---------T 2 root nas_users 511967232 Jan 30 14:49 iobw.tst

[root@SC-182 log]# ls -l /exports/nas-segment-0003/nas-volume-0003/cifs_share/
ls: cannot access /exports/nas-segment-0003/nas-volume-0003/cifs_share/: No such file or directory


[root@SC-182 log]# ls -l /samba/nas-volume-0003/cifs_share/
total 499972
drwxr-xr-x 2 root root           4096 Jan 29 16:29 dir1
---------T 1 root nas_users 511967232 Jan 30 14:49 iobw.tst

CIFS client continue to do IO's!

[root@SC-182 log]# ls -l /samba/nas-volume-0003/cifs_share/
total 499972
drwxr-xr-x 2 root root           4096 Jan 29 16:29 dir1
---------T 1 root nas_users 511967232 Jan 30 14:51 iobw.tst

Actual results:

Check file "iobw.tst" on the volume:

[root@SC-182 log]# ls -l /samba/nas-volume-0003/cifs_share/
total 499972
drwxr-xr-x 2 root root           4096 Jan 29 16:29 dir1
---------T 1 root nas_users 511967232 Jan 30 14:51 iobw.tst

Size and file permissions are wrong.

Expected results:

The test was executed successfully on glusterfs 3.7.18. The correct results have to be like the next:

[root@SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/
total 500004
drwxr-xr-x 2 root root           4096 Jan 30 15:11 dir1
-rw-rw---- 1 root nas_users 512000000 Jan 30 15:31 iobw.tst

Additional info:

Results of successfull test on glusterFS 3.7.18:

Do the same on SC183 that runs glusterfs 3.7.18.


[root@SC-183 log]# gluster volume info tiered-volume-0015

Volume Name: tiered-volume-0015
Type: Distribute
Volume ID: aefc0a6e-2515-40f1-a888-2f3168e0fa07
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
Options Reconfigured:
server.allow-insecure: on
performance.quick-read: off
performance.stat-prefetch: off
nfs.disable: on
nfs.addr-namelookup: off
performance.readdir-ahead: on
snap-activate-on-create: enable
cluster.enable-shared-storage: disable

[root@SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/
total 500004
drwxr-xr-x 2 root root           4096 Jan 30 15:06 dir1
-rw-rw---- 1 root nas_users 512000000 Jan 30 15:06 iobw.tst

[root@SC-183 log]# ls -l /exports/cold-segment-0004/tiered-volume-0015/cifs_share/
total 500000
drwxr-xr-x 2 root root            141 Jan 30 15:06 dir1
-rw-rw---- 2 root nas_users 512000000 Jan 30 15:07 iobw.tst

[root@SC-183 log]# gluster volume status tiered-volume-0015
Status of volume: tiered-volume-0015
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.10.60.183:/exports/cold-segment-00
04/tiered-volume-0015                       62078     0          Y       22474

Task Status of Volume tiered-volume-0015
------------------------------------------------------------------------------
There are no active volume tasks

[root@SC-183 log]# gluster volume profile tiered-volume-0015 start
Starting volume profile on tiered-volume-0015 has been successful
[root@SC-183 log]# gluster volume profile tiered-volume-0015 info clear
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cleared stats.

[root@SC-183 log]# gluster volume profile tiered-volume-0015 info
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cumulative Stats:
   Block Size:              32768b+               65536b+              131072b+
 No. of Reads:                    9                     0                 37835
No. of Writes:                    0                 20334                     9

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             30  RELEASEDIR
      0.00      44.15 us      31.00 us      88.00 us             20      STATFS
      0.01      53.50 us      39.00 us     150.00 us             30     OPENDIR
      0.01      40.54 us      23.00 us     171.00 us             50        STAT
      0.03     146.25 us      40.00 us     323.00 us             60    READDIRP
      0.05      74.78 us      47.00 us     223.00 us            180    GETXATTR
      0.08      74.15 us      56.00 us     186.00 us            280      LOOKUP
     32.54     422.47 us      47.00 us  827002.00 us          20343       WRITE
     67.28     469.44 us      31.00 us  826207.00 us          37851        READ

    Duration: 21 seconds
   Data Read: 4959404032 bytes
Data Written: 1333788672 bytes

Interval 0 Stats:
   Block Size:              32768b+               65536b+              131072b+
 No. of Reads:                    9                     0                 37835
No. of Writes:                    0                 20334                     9

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             30  RELEASEDIR
      0.00      44.15 us      31.00 us      88.00 us             20      STATFS
      0.01      53.50 us      39.00 us     150.00 us             30     OPENDIR
      0.01      40.54 us      23.00 us     171.00 us             50        STAT
      0.03     146.25 us      40.00 us     323.00 us             60    READDIRP
      0.05      74.78 us      47.00 us     223.00 us            180    GETXATTR
      0.08      74.15 us      56.00 us     186.00 us            280      LOOKUP
     32.54     422.47 us      47.00 us  827002.00 us          20343       WRITE
     67.28     469.44 us      31.00 us  826207.00 us          37851        READ

    Duration: 21 seconds
   Data Read: 4959404032 bytes
Data Written: 1333788672 bytes

[root@SC-183 log]# gluster volume profile tiered-volume-0015 stop
Stopping volume profile on tiered-volume-0015 has been successful

Attach hot tier:

4780 01/30/2018 15:11:13.505967116 1517353873 command: /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015 attach 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015 2>&1 | /bin/grep "Tiering Migration Functionality: tiered-volume-0015: success"
4795 01/30/2018 15:11:13.604148675 1517353873 command: /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "attach" "10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015"
4795 01/30/2018 15:11:19.087525009 1517353879 status=0 /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "attach" "10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015"
4780 01/30/2018 15:11:19.105676210 1517353879 status=0 /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015 attach 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015 2>&1 | /bin/grep "Tiering Migration Functionality: tiered-volume-0015: success"


[root@SC-183 log]# gluster volume info tiered-volume-0015

Volume Name: tiered-volume-0015
Type: Tier
Volume ID: aefc0a6e-2515-40f1-a888-2f3168e0fa07
Status: Started
Number of Bricks: 2
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 1
Brick1: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015
Cold Tier:
Cold Tier Type : Distribute
Number of Bricks: 1
Brick2: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
Options Reconfigured:
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
cluster.tier-mode: cache
features.ctr-enabled: on
server.allow-insecure: on
performance.quick-read: off
performance.stat-prefetch: off
nfs.disable: on
nfs.addr-namelookup: off
performance.readdir-ahead: on
snap-activate-on-create: enable
cluster.enable-shared-storage: disable


[root@SC-183 log]# gluster volume status tiered-volume-0015
Status of volume: tiered-volume-0015
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.10.60.183:/exports/hot-segment-000
5/tiered-volume-0015                        62079     0          Y       4824
Cold Bricks:
Brick 10.10.60.183:/exports/cold-segment-00
04/tiered-volume-0015                       62078     0          Y       22474
Quota Daemon on localhost                   N/A       N/A        Y       4980

Task Status of Volume tiered-volume-0015
------------------------------------------------------------------------------
Task                 : Tier migration
ID                   : cf15d19a-8ba6-4cfd-9794-fc7a77050f45
Status               : in progress

[root@SC-183 log]# ls -l /exports/cold-segment-0004/tiered-volume-0015/cifs_share/
total 448
drwxr-xr-x 2 root root            141 Jan 30 15:11 dir1
---------T 2 root nas_users 236519424 Jan 30 15:12 iobw.tst

[root@SC-183 log]# ls -l /exports/hot-segment-0005/tiered-volume-0015/cifs_share/
total 500080
drwxr-xr-x 2 root root              6 Jan 30 15:11 dir1
-rw-rw---- 2 root nas_users 512000000 Jan 30 15:17 iobw.tst

[root@SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/
total 500004
drwxr-xr-x 2 root root           4096 Jan 30 15:11 dir1
-rw-rw---- 1 root nas_users 512000000 Jan 30 15:17 iobw.tst


[root@SC-183 log]# gluster volume profile tiered-volume-0015 start
Starting volume profile on tiered-volume-0015 has been successful
[root@SC-183 log]# gluster volume profile tiered-volume-0015 info clear
Brick: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015
----------------------------------------------------------------
Cleared stats.
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cleared stats.

[root@SC-183 log]# gluster volume profile tiered-volume-0015 info
Brick: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015
----------------------------------------------------------------
Cumulative Stats:
   Block Size:              65536b+              131072b+
 No. of Reads:                    0                     0
No. of Writes:                20702                   364
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             30  RELEASEDIR
      0.00      44.37 us      36.00 us      68.00 us             30     OPENDIR
      0.00      33.45 us      26.00 us      65.00 us             49        STAT
      0.03     648.21 us      36.00 us    1160.00 us             24      STATFS
      0.04      77.75 us      45.00 us     160.00 us            218    GETXATTR
      0.08     117.96 us      83.00 us     386.00 us            317      LOOKUP
     99.85    2287.28 us     212.00 us  827377.00 us          21067       WRITE

    Duration: 20 seconds
   Data Read: 0 bytes
Data Written: 1404436480 bytes

Interval 0 Stats:
   Block Size:              65536b+              131072b+
 No. of Reads:                    0                     0
No. of Writes:                20702                   364
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             30  RELEASEDIR
      0.00      44.37 us      36.00 us      68.00 us             30     OPENDIR
      0.00      33.45 us      26.00 us      65.00 us             49        STAT
      0.03     648.21 us      36.00 us    1160.00 us             24      STATFS
      0.04      77.75 us      45.00 us     160.00 us            218    GETXATTR
      0.08     117.96 us      83.00 us     386.00 us            317      LOOKUP
     99.85    2287.28 us     212.00 us  827377.00 us          21067       WRITE

    Duration: 20 seconds
   Data Read: 0 bytes
Data Written: 1404436480 bytes

Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cumulative Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             30  RELEASEDIR
      1.67      61.13 us      41.00 us      97.00 us             30     OPENDIR
      1.69      37.76 us      26.00 us      75.00 us             49        STAT
     13.00      80.09 us      46.00 us     165.00 us            178    GETXATTR
     14.22     779.65 us     619.00 us    1047.00 us             20      STATFS
     14.76     269.72 us      67.00 us     815.00 us             60    READDIRP
     54.66     108.40 us      81.00 us     351.00 us            553      LOOKUP

    Duration: 20 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Interval 0 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             30  RELEASEDIR
      1.67      61.13 us      41.00 us      97.00 us             30     OPENDIR
      1.69      37.76 us      26.00 us      75.00 us             49        STAT
     13.00      80.09 us      46.00 us     165.00 us            178    GETXATTR
     14.22     779.65 us     619.00 us    1047.00 us             20      STATFS
     14.76     269.72 us      67.00 us     815.00 us             60    READDIRP
     54.66     108.40 us      81.00 us     351.00 us            553      LOOKUP

    Duration: 20 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

[root@SC-183 log]# gluster volume profile tiered-volume-0015 stop
Stopping volume profile on tiered-volume-0015 has been successful

Detach hot tier:

15364 01/30/2018 15:24:09.119881869 1517354649 command: /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015 detach start 2>&1
15378 01/30/2018 15:24:09.196084465 1517354649 command: /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "start"
15378 01/30/2018 15:24:14.514959281 1517354654 status=0 /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "start"
15364 01/30/2018 15:24:14.533662479 1517354654 status=0 /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015 detach start 2>&1

[root@SC-183 log]# /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "status"
                                    Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                1       488.3MB             7             0             0            completed        0:0:19

[root@SC-183 log]# ls -l /exports/hot-segment-0005/tiered-volume-0015/cifs_share/
total 0
drwxr-xr-x 2 root root 6 Jan 30 15:11 dir1

[root@SC-183 log]# ls -l /exports/cold-segment-0004/tiered-volume-0015/cifs_share/
total 500064
drwxr-xr-x 2 root root            141 Jan 30 15:11 dir1
-rw-rw---- 2 root nas_users 512000000 Jan 30 15:26 iobw.tst

[root@SC-183 log]# gluster volume profile tiered-volume-0015 start
Starting volume profile on tiered-volume-0015 has been successful
[root@SC-183 log]# gluster volume profile tiered-volume-0015 info clear
Brick: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015
----------------------------------------------------------------
Cleared stats.
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cleared stats.

[root@SC-183 log]# gluster volume profile tiered-volume-0015 info
Brick: 10.10.60.183:/exports/hot-segment-0005/tiered-volume-0015
----------------------------------------------------------------
Cumulative Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             24  RELEASEDIR
      2.34      57.12 us      38.00 us      84.00 us             24     OPENDIR
      3.13      45.75 us      28.00 us     356.00 us             40        STAT
     18.46      84.43 us      48.00 us     256.00 us            128    GETXATTR
     21.07     770.94 us     556.00 us    1017.00 us             16      STATFS
     55.00     118.36 us      84.00 us     261.00 us            272      LOOKUP

    Duration: 17 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Interval 0 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             24  RELEASEDIR
      2.34      57.12 us      38.00 us      84.00 us             24     OPENDIR
      3.13      45.75 us      28.00 us     356.00 us             40        STAT
     18.46      84.43 us      48.00 us     256.00 us            128    GETXATTR
     21.07     770.94 us     556.00 us    1017.00 us             16      STATFS
     55.00     118.36 us      84.00 us     261.00 us            272      LOOKUP

    Duration: 17 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cumulative Stats:
   Block Size:              65536b+              131072b+
 No. of Reads:                    0                     0
No. of Writes:                17104                   275
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             24  RELEASEDIR
      0.00      44.75 us      37.00 us      58.00 us             24     OPENDIR
      0.00      33.20 us      26.00 us      71.00 us             40        STAT
      0.04     767.94 us     603.00 us     958.00 us             16      STATFS
      0.04     261.44 us      64.00 us     674.00 us             48    READDIRP
      0.04      82.23 us      48.00 us     177.00 us            160    GETXATTR
      0.17     112.10 us      80.00 us     554.00 us            448      LOOKUP
     99.69    1678.12 us     214.00 us  629362.00 us          17378       WRITE

    Duration: 17 seconds
   Data Read: 0 bytes
Data Written: 1156972544 bytes

Interval 0 Stats:
   Block Size:              65536b+              131072b+
 No. of Reads:                    0                     0
No. of Writes:                17104                   275
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             24  RELEASEDIR
      0.00      44.75 us      37.00 us      58.00 us             24     OPENDIR
      0.00      33.20 us      26.00 us      71.00 us             40        STAT
      0.04     767.94 us     603.00 us     958.00 us             16      STATFS
      0.04     261.44 us      64.00 us     674.00 us             48    READDIRP
      0.04      82.23 us      48.00 us     177.00 us            160    GETXATTR
      0.17     112.10 us      80.00 us     554.00 us            448      LOOKUP
     99.69    1678.12 us     214.00 us  629362.00 us          17378       WRITE

    Duration: 17 seconds
   Data Read: 0 bytes
Data Written: 1156972544 bytes

[root@SC-183 log]# gluster volume profile tiered-volume-0015 stop
Stopping volume profile on tiered-volume-0015 has been successful

[root@SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/
total 500004
drwxr-xr-x 2 root root           4096 Jan 30 15:11 dir1
-rw-rw---- 1 root nas_users 512000000 Jan 30 15:30 iobw.tst

Commit:

19949 01/30/2018 15:31:07.145654679 1517355067 command: /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015 detach commit 2>&1
19982 01/30/2018 15:31:08.636386875 1517355068 command: /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "commit"
19982 01/30/2018 15:31:10.000126293 1517355070 status=0 /bin/echo "y" | /usr/sbin/gluster "volume" "tier" "tiered-volume-0015" "detach" "commit"
19949 01/30/2018 15:31:10.047054376 1517355070 status=0 /usr/local/StoneFly/bin/gluster_restart_retry.pl volume tier tiered-volume-0015 detach commit 2>&1

[root@SC-183 log]# ls -l /samba/tiered-volume-0015/cifs_share/
total 500004
drwxr-xr-x 2 root root           4096 Jan 30 15:11 dir1
-rw-rw---- 1 root nas_users 512000000 Jan 30 15:31 iobw.tst

[root@SC-183 log]# ls -l /exports/cold-segment-0004/tiered-volume-0015/cifs_share/
total 500064
drwxr-xr-x 2 root root            141 Jan 30 15:11 dir1
-rw-rw---- 2 root nas_users 512000000 Jan 30 15:32 iobw.tst

[root@SC-183 log]# gluster volume profile tiered-volume-0015 start
Starting volume profile on tiered-volume-0015 has been successful
[root@SC-183 log]# gluster volume profile tiered-volume-0015 info clear
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cleared stats.

[root@SC-183 log]# gluster volume profile tiered-volume-0015 info
Brick: 10.10.60.183:/exports/cold-segment-0004/tiered-volume-0015
-----------------------------------------------------------------
Cumulative Stats:
   Block Size:              65536b+              131072b+
 No. of Reads:                    0                     0
No. of Writes:                16722                    29
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             22  RELEASEDIR
      0.00      47.50 us      37.00 us      69.00 us             22     OPENDIR
      0.00      45.85 us      26.00 us     379.00 us             40        STAT
      0.02     552.44 us     490.00 us     614.00 us             16      STATFS
      0.02     239.14 us      63.00 us     491.00 us             44    READDIRP
      0.03      74.09 us      44.00 us     157.00 us            204    GETXATTR
      0.05     102.15 us      78.00 us     447.00 us            240      LOOKUP
     99.87    2793.68 us      61.00 us 1026245.00 us          16751       WRITE

    Duration: 16 seconds
   Data Read: 0 bytes
Data Written: 1099694080 bytes

Interval 0 Stats:
   Block Size:              65536b+              131072b+
 No. of Reads:                    0                     0
No. of Writes:                16722                    29
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             22  RELEASEDIR
      0.00      47.50 us      37.00 us      69.00 us             22     OPENDIR
      0.00      45.85 us      26.00 us     379.00 us             40        STAT
      0.02     552.44 us     490.00 us     614.00 us             16      STATFS
      0.02     239.14 us      63.00 us     491.00 us             44    READDIRP
      0.03      74.09 us      44.00 us     157.00 us            204    GETXATTR
      0.05     102.15 us      78.00 us     447.00 us            240      LOOKUP
     99.87    2793.68 us      61.00 us 1026245.00 us          16751       WRITE

    Duration: 16 seconds
   Data Read: 0 bytes
Data Written: 1099694080 bytes

[root@SC-183 log]# gluster volume profile tiered-volume-0015 stop
Stopping volume profile on tiered-volume-0015 has been successful

Comment 1 Shyamsundar 2018-10-23 14:54:07 UTC
Release 3.12 has been EOLd and this bug was still found to be in the NEW state, hence moving the version to mainline, to triage the same and take appropriate actions.

Comment 2 Amar Tumballi 2018-11-02 08:13:25 UTC
Patch https://review.gluster.org/#/c/glusterfs/+/21331/ removes tier functionality from GlusterFS. 

https://bugzilla.redhat.com/show_bug.cgi?id=1642807 is used as the tracking bug for this. Recommendation is to convert your tier volume to regular volume (either replicate, ec, or plain distribute) with "tier detach" command before upgrade, and use backend features like dm-cache etc to utilize the caching from backend to provide better performance and functionality.


Note You need to log in before you can comment on or make changes to this bug.