Bug 1300380 - Detach tier start and commit completes sucessfully even when there is no disk space in cold tier(hence some files remain in hot)
Detach tier start and commit completes sucessfully even when there is no disk...
Status: CLOSED NOTABUG
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
3.1
Unspecified Unspecified
unspecified Severity medium
: ---
: RHGS 3.1.3
Assigned To: Bug Updates Notification Mailing List
nchilaka
tier-attach-detach
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-20 10:47 EST by nchilaka
Modified: 2017-03-08 03:58 EST (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-07-01 01:30:04 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description nchilaka 2016-01-20 10:47:38 EST
When we do a detach tier and there is not sufficient space in cold tier, then the detach tier must fail.
but the detach tier pass , both start and commit and jsut mentions the number of files failed to rebalance in detach status.
I feel we must fail detach tier itself




files failed to migrate:
[root@dhcp37-59 ~]# ll /rhs/brick*/tomato_hot/
total 3135952
-rwxr-----. 2 root root 1605601323 Jan 20 20:34 bb2.mkv
-rwxr-----. 2 root root 1605601323 Jan 20 20:35 bb4.mkv


[root@dhcp37-59 glusterfs]# cat /var/log/glusterfs/tomato-rebalance.log|grep " E "
[2016-01-20 15:07:04.488897] E [MSGID: 109023] [dht-rebalance.c:1358:dht_migrate_file] 0-tomato-tier-dht: Migrate file failed: /bb2.mkv: failed to migrate data
[2016-01-20 15:07:08.304143] E [MSGID: 109023] [dht-rebalance.c:1358:dht_migrate_file] 0-tomato-tier-dht: Migrate file failed: /bb4.mkv: failed to migrate data
[root@dhcp37-59 glusterfs]# 


root@rhs-client21 ~]# ll /smallbrick/
total 430824
---------T. 2 root root 401400832 Jan 20 21:14 Baahubali.The.Beginning.2015.Hindi.720p.HDRiP.x264.ShAaNiG.mkv
---------T. 2 root root         0 Jan 20 21:12 bb2.mkv
---------T. 2 root root 288423936 Jan 20 21:14 bb3.mkv
---------T. 2 root root         0 Jan 20 21:12 bb4.mkv
-rw-r--r--. 2 root root         0 Jan 20 19:38 f1
-rw-r--r--. 2 root root         0 Jan 20 19:38 f10
-rw-r--r--. 2 root root         0 Jan 20 19:38 f11
-rw-r--r--. 2 root root         0 Jan 20 19:38 f12
-rw-r--r--. 2 root root         0 Jan 20 19:38 f13
-rw-r--r--. 2 root root         0 Jan 20 19:38 f14
-rw-r--r--. 2 root root         0 Jan 20 19:38 f15
-rw-r--r--. 2 root root         0 Jan 20 19:38 f16
-rw-r--r--. 2 root root         0 Jan 20 19:38 f17
-rw-r--r--. 2 root root         0 Jan 20 19:38 f18
-rw-r--r--. 2 root root         0 Jan 20 19:38 f19
-rw-r--r--. 2 root root         0 Jan 20 19:38 f2
-rw-r--r--. 2 root root         0 Jan 20 19:38 f20
-rw-r--r--. 2 root root         0 Jan 20 19:38 f21
-rw-r--r--. 2 root root         0 Jan 20 19:38 f22
-rw-r--r--. 2 root root         0 Jan 20 19:38 f23
-rw-r--r--. 2 root root         0 Jan 20 19:38 f24
-rw-r--r--. 2 root root         0 Jan 20 19:38 f25
-rw-r--r--. 2 root root         0 Jan 20 19:38 f26
-rw-r--r--. 2 root root         0 Jan 20 19:38 f27
-rw-r--r--. 2 root root         0 Jan 20 19:38 f28
-rw-r--r--. 2 root root         0 Jan 20 19:38 f29
-rw-r--r--. 2 root root         0 Jan 20 19:38 f3
-rw-r--r--. 2 root root         0 Jan 20 19:38 f30
-rw-r--r--. 2 root root         0 Jan 20 19:38 f4
-rw-r--r--. 2 root root         0 Jan 20 19:38 f5
-rw-r--r--. 2 root root         0 Jan 20 19:38 f6
-rw-r--r--. 2 root root         0 Jan 20 19:38 f7
-rw-r--r--. 2 root root         0 Jan 20 19:38 f8
-rw-r--r--. 2 root root         0 Jan 20 19:38 f9
[root@rhs-client21 ~]# ll /smallbrick/
total 784132
-rwxr-----. 2 root root 401400832 Jan 20 20:14 Baahubali.The.Beginning.2015.Hindi.720p.HDRiP.x264.ShAaNiG.mkv
---------T. 2 root root         0 Jan 20 21:12 bb2.mkv
-rwxr-----. 2 root root 401400832 Jan 20 20:33 bb3.mkv
---------T. 2 root root         0 Jan 20 21:12 bb4.mkv
-rw-r--r--. 2 root root         0 Jan 20 19:38 f1
-rw-r--r--. 2 root root         0 Jan 20 19:38 f10
-rw-r--r--. 2 root root         0 Jan 20 19:38 f11
-rw-r--r--. 2 root root         0 Jan 20 19:38 f12
-rw-r--r--. 2 root root         0 Jan 20 19:38 f13
-rw-r--r--. 2 root root         0 Jan 20 19:38 f14
-rw-r--r--. 2 root root         0 Jan 20 19:38 f15
-rw-r--r--. 2 root root         0 Jan 20 19:38 f16
-rw-r--r--. 2 root root         0 Jan 20 19:38 f17
-rw-r--r--. 2 root root         0 Jan 20 19:38 f18
-rw-r--r--. 2 root root         0 Jan 20 19:38 f19
-rw-r--r--. 2 root root         0 Jan 20 19:38 f2
-rw-r--r--. 2 root root         0 Jan 20 19:38 f20
-rw-r--r--. 2 root root         0 Jan 20 19:38 f21
-rw-r--r--. 2 root root         0 Jan 20 19:38 f22
-rw-r--r--. 2 root root         0 Jan 20 19:38 f23
-rw-r--r--. 2 root root         0 Jan 20 19:38 f24
-rw-r--r--. 2 root root         0 Jan 20 19:38 f25
-rw-r--r--. 2 root root         0 Jan 20 19:38 f26
-rw-r--r--. 2 root root         0 Jan 20 19:38 f27
-rw-r--r--. 2 root root         0 Jan 20 19:38 f28
-rw-r--r--. 2 root root         0 Jan 20 19:38 f29
-rw-r--r--. 2 root root         0 Jan 20 19:38 f3
-rw-r--r--. 2 root root         0 Jan 20 19:38 f30
-rw-r--r--. 2 root root         0 Jan 20 19:38 f4
-rw-r--r--. 2 root root         0 Jan 20 19:38 f5
-rw-r--r--. 2 root root         0 Jan 20 19:38 f6
-rw-r--r--. 2 root root         0 Jan 20 19:38 f7
-rw-r--r--. 2 root root         0 Jan 20 19:38 f8
-rw-r--r--. 2 root root         0 Jan 20 19:38 f9
[root@rhs-client21 ~]# gluster v tier tomato detach status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                             10.70.37.59                2         3.0GB             4             2             0            completed             656.00
                            10.70.37.150                0        0Bytes             0             0             0            completed               1.00
[root@rhs-client21 ~]# gluster v tier tomato status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            0                    0                    not started         
rhs-client4.lab.eng.blr.redhat.com 0                    0                    not started         
rhs-client20         0                    0                    not started         
10.70.37.59          0                    0                    completed           
10.70.37.150         0                    0                    completed           
rhs-client30         0                    0                    not started         
rhs-client9          0                    0                    not started         
rhs-client1          0                    0                    not started         
Tiering Migration Functionality: tomato: success
[root@rhs-client21 ~]# gluster v status tomato
Status of volume: tomato
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.70.37.59:/rhs/brick5/tomato_hot    49171     0          Y       22201
Brick 10.70.37.150:/rhs/brick5/tomato_hot   49171     0          Y       22711
Cold Bricks:
Brick rhs-client20:/smallbrick              49174     0          Y       17342
Brick rhs-client21:/smallbrick              49181     0          Y       15308
Brick 10.70.37.59:/smallbrick               49170     0          Y       22035
Brick 10.70.37.150:/smallbrick              49170     0          Y       22551
Brick rhs-client30:/smallbrick              49171     0          Y       13666
Brick rhs-client4:/smallbrick               49169     0          Y       2708 
NFS Server on localhost                     2049      0          Y       16530
Self-heal Daemon on localhost               N/A       N/A        Y       16538
NFS Server on rhs-client20                  2049      0          Y       17680
Self-heal Daemon on rhs-client20            N/A       N/A        Y       17688
NFS Server on rhs-client9                   2049      0          Y       31655
Self-heal Daemon on rhs-client9             N/A       N/A        Y       31663
NFS Server on 10.70.37.59                   2049      0          Y       22221
Self-heal Daemon on 10.70.37.59             N/A       N/A        Y       22229
NFS Server on rhs-client1                   2049      0          Y       31749
Self-heal Daemon on rhs-client1             N/A       N/A        Y       31757
NFS Server on rhs-client4.lab.eng.blr.redha
t.com                                       2049      0          Y       11419
Self-heal Daemon on rhs-client4.lab.eng.blr
.redhat.com                                 N/A       N/A        Y       11428
NFS Server on rhs-client30                  2049      0          Y       14434
Self-heal Daemon on rhs-client30            N/A       N/A        Y       14442
NFS Server on 10.70.37.150                  2049      0          Y       22731
Self-heal Daemon on 10.70.37.150            N/A       N/A        Y       22739
 
Task Status of Volume tomato
------------------------------------------------------------------------------
Task                 : Detach tier         
ID                   : 6d41a863-830c-4650-baf4-c7d83e15a743
Status               : completed
Comment 2 Mohammed Rafi KC 2016-07-01 01:30:04 EDT
When we do the commit for detach tier, it checks whether the status of rebalance process is completed or not. If it is completed then we will allow commit to pass through. But we won't look for the failures of files.

There are chances to fail a file, that is why the commit comment throws a warning to look for status.

So I think this is an expected behavior by design. So closing this bug, if you have any different opinion, please feel free to reopen it.
Comment 4 nchilaka 2017-03-08 03:58:40 EST
fine with the explanation

Note You need to log in before you can comment on or make changes to this bug.