Bug 1277154 - Data Tiering:new file creates fail after deleting files due to disk full scenario
Data Tiering:new file creates fail after deleting files due to disk full scen...
Status: ON_QA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
unspecified
Unspecified Unspecified
urgent Severity urgent
: ---
: ---
Assigned To: sankarshan
Sweta Anandpara
: ZStream
Depends On: 1291566
Blocks: 1260923
  Show dependency treegraph
 
Reported: 2015-11-02 08:55 EST by nchilaka
Modified: 2018-01-29 15:11 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description nchilaka 2015-11-02 08:55:26 EST
Description of problem:
=======================
I have created a tiered volume, where the hot tier is full and hence new writes were failing.
I then deleted all the files from the mount using rm -rf.
Now when I try to create files (zerobyte), it fails saying no space on disk
[root@mia diskfull]# touch f{1..10}
touch: cannot touch ‘f1’: No space left on device
touch: cannot touch ‘f2’: No space left on device




fuse client logs:
===============
[2015-11-02 06:34:28.438578] W [fuse-bridge.c:1978:fuse_create_cbk] 0-glusterfs-fuse: 9922003: /f8 => -1 (No space left on device)
[2015-11-02 06:34:28.452211] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-diskfull-client-7: remote operation failed. Path: /f9 [No space left on device]
[2015-11-02 06:34:28.452347] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-diskfull-client-6: remote operation failed. Path: /f9 [No space left on device]
[2015-11-02 06:34:28.452745] W [fuse-bridge.c:1978:fuse_create_cbk] 0-glusterfs-fuse: 9922006: /f9 => -1 (No space left on device)
[2015-11-02 06:34:28.468526] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-diskfull-client-5: remote operation failed. Path: /f10 [No space left on device]
[2015-11-02 06:34:28.468536] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-diskfull-client-4: remote operation failed. Path: /f10 [No space left on device]
[2015-11-02 06:34:28.469070] W [fuse-bridge.c:1978:fuse_create_cbk] 0-glusterfs-fuse: 9922009: /f10 => -1 (No space left on device)
[2015-11-02 06:34:28.331038] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-diskfull-client-4: remote operation failed. Path: /f1 [No space left on device]
[2015-11-02 06:34:28.331057] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-diskfull-client-5: remote operation failed. Path: /f1 [No space left on device]
[2015-11-02 06:34:28.349900] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-diskfull-client-5: remote operation failed. Path: /f2 [No space left on device]

TIer logs:
==========
################################################
[2015-11-02 13:46:00.962757] E [MSGID: 109037] [tier.c:463:tier_migrate_using_query_file] 0-diskfull-tier-dht: ERROR in current lookup

[2015-11-02 13:46:00.962932] E [MSGID: 109037] [tier.c:1488:tier_start] 0-diskfull-tier-dht: Demotion failed
[2015-11-02 13:48:00.981438] E [MSGID: 109037] [tier.c:463:tier_migrate_using_query_file] 0-diskfull-tier-dht: ERROR in current lookup

[2015-11-02 13:48:00.981573] E [MSGID: 109037] [tier.c:1488:tier_start] 0-diskfull-tier-dht: Demotion failed
[2015-11-02 13:50:01.000564] E [MSGID: 109037] [tier.c:463:tier_migrate_using_query_file] 0-diskfull-tier-dht: ERROR in current lookup

[2015-11-02 13:50:01.000704] E [MSGID: 109037] [tier.c:1488:tier_start] 0-diskfull-tier-dht: Demotion failed


Version-Release number of selected component (if applicable):
===========================================================
glusterfs-server-3.7.5-5.el7rhgs.x86_64


Steps to Reproduce:
=================
1.create tier volume with limited hot brick sizes
2.create files such that they exceed hot brick sizes and fail to create after sometime.
3.now create some other random files
4.Now create more files such that it fails saying no space
5. Now delete files using rm -rf and retry creating zero byte files
6. It says no space




Additional info:
Comment 2 nchilaka 2015-11-02 08:56:49 EST
backend bricks
[root@zod ~]# ll /*/bri*/dis*/
/dummy/brick106/diskfull_hot/:
total 4
---------T. 2 root root 0 Nov  2 18:08 iceman

/dummy/brick107/diskfull_hot/:
total 0

/rhs/brick1/diskfull/:
total 0

/rhs/brick2/diskfull/:
total 0
---------T. 2 root root 0 Nov  2 17:19 FnF7.mkv
[root@zod ~]# 
[root@zod ~]# 
[root@zod ~]# 
[root@zod ~]# 
[root@zod ~]# ls /
Comment 3 nchilaka 2015-11-03 00:53:59 EST
sosreports@ below location. Refer volume "diskfull"
[nchilaka@rhsqe-repo bug.1277088]$ pwd
/home/repo/sosreports/nchilaka/bug.1277088
Comment 6 Nithya Balachandran 2015-12-16 00:30:36 EST
Assigning this to Joseph as the patch he posted for BZ#1291566 should fix this as well.

http://review.gluster.org/12969
Comment 9 Joseph Elwin Fernandes 2016-01-19 07:18:17 EST
https://code.engineering.redhat.com/gerrit/#/c/64284/

Note You need to log in before you can comment on or make changes to this bug.