Bug 1278391 - Data Tiering:Files not getting migrated from hot to cold when the hot brick is full
Data Tiering:Files not getting migrated from hot to cold when the hot brick i...
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
Unspecified Unspecified
urgent Severity urgent
: ---
: ---
Assigned To: hari gowtham
: ZStream
: 1278392 (view as bug list)
Depends On: 1278392
Blocks: 1260923 1268895
  Show dependency treegraph
Reported: 2015-11-05 06:39 EST by nchilaka
Modified: 2018-02-06 12:43 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Known Issue
Doc Text:
Migration from the hot tier fails when the hot tier is completely full because there is no space left to set the extended attribute that triggers migration.
Story Points: ---
Clone Of: 1271151
Last Closed: 2018-02-06 12:43:24 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description nchilaka 2015-11-05 06:39:56 EST
+++ This bug was initially created as a clone of Bug #1271151 +++

Description of problem:
When a brick is full, files are not getting migrated at all.
I created a hot tier with 2x2 with each brick of 1GB size. That means effectively the hot tier is of 2GB size.
I started to create a file of 1.5GB using dd command, which was getting created on 1st brick. But as the 1GB gets filled, the new file stops to grow with below error:
dd: error writing ‘zerobyte.50’: Input/output error
dd: closing output file ‘zerobyte.50’: Input/output error

Now the file is not getting demoted at all and hence no more writes are going to the hot tier.

Similarly, if i fill the other brick too, then it essentially means that the whole hot tier is full.

In this case, no more new creates are being allowed even to the cold tier (also refer bz#1259312)

Version-Release number of selected component (if applicable):

Steps to Reproduce:
1.create a 2x2 vol and start it
2.attach a tier of 2x2 with limited size, say each brick of 1GB.
Also turn on ctr (note in this case quota was not turned on)
3.Now create a file of 1.5GB such that it goes to one of the hot bricks
It will fail after it creates 1GB
4. Now this file doesnt get demoted at all 
5.Now again create a 1.5GB file such that it goes to the other hot brick(to make hot layer full)
5. Now file doenst get demoted at all

Expected results:
File must get demoted once it is old and idle

--- Additional comment from nchilaka on 2015-10-13 05:39:20 EDT ---

vol name:estonia


--- Additional comment from nchilaka on 2015-10-13 06:43 EDT ---

--- Additional comment from Dan Lambright on 2015-10-29 17:01:33 EDT ---

DHT does not accept creation of files beyond the brick size. So I am not sure if we can fix the test the way it is structured, but nithya can confirm.

In practice the hot brick would be 10s or 100s of GB. A file created would be a fraction of that size. As files are created on the hot tier it would become more full, until eventually a watermarked was crossed. At that time files would be created on the cold tier.
Comment 2 Vivek Agarwal 2015-11-05 06:43:02 EST
*** Bug 1278392 has been marked as a duplicate of this bug. ***
Comment 6 Joseph Elwin Fernandes 2016-01-19 07:22:45 EST
Well when the hot tier is full 100% no space to set migration xattr then migration fails. The min free disk and water marks should guard us from this situation. This problem is also true for dht rebalance or afr self heal.
Comment 8 Dan Lambright 2016-01-22 09:28:25 EST

Yes, the doc field looks correct.
Comment 16 Shyamsundar 2018-02-06 12:43:24 EST
Thank you for your bug report.

We are no longer working on any improvements for Tier. This bug will be set to CLOSED WONTFIX to reflect this. Please reopen if the rfe is deemed critical.

Note You need to log in before you can comment on or make changes to this bug.