Bug 1028347 - Even after successful completion of rebalance after add-brick , One of the brick contains a file with "-rw-r-Sr-T" attributes
Even after successful completion of rebalance after add-brick , One of the br...
Status: CLOSED DEFERRED
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute (Show other bugs)
2.1
Unspecified Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Nithya Balachandran
storage-qa-internal@redhat.com
:
Depends On:
Blocks: 1286201
  Show dependency treegraph
 
Reported: 2013-11-08 04:26 EST by spandura
Modified: 2015-11-27 07:28 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1286201 (view as bug list)
Environment:
Last Closed: 2015-11-27 07:27:03 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description spandura 2013-11-08 04:26:19 EST
Description of problem:
=======================
On a 1 x 3 replicate volume added 3 more bricks to the volume to make it 2 x 3 distribute-replicate volume. Performed rebalance on the volume. When rebalance was in progress removed 2 bricks to change the replica count to 2 (remove-brick replica 2 <brick3> <brick6> force). This changed the volume type to 2 x 2 distribute-replicate volume. 

Rebalance got successfully completed. Check the attributes of the files on the bricks. One of the file still had the "-rw-r-Sr-T" atributes even after successful completion of rebalance. 


Version-Release number of selected component (if applicable):
=============================================================
glusterfs 3.4.0.35.1u2rhs built on Oct 21 2013 14:00:58

How reproducible:
================
Tried only once. 

Steps to Reproduce:
=====================
1. Create a replicate volume with 3 bricks (1 x 3). Start the volume. 

2. Create few files. open the files and keep reading , modifying the file. 

3. While the IO from mount point is in progress, add a new sub-volume with 3 bricks changing the volume type to distribute-replicate. 

4. Start rebalance on the volume. 

5. While rebalance is in progress, remove one replica from each sub volume.(remove-brick replica 2 <brick3> <brick6> force) 

6. Once the rebalance is complete check the attributes of all the files. 

Actual results:
===============
root@rhs-client11 [Nov-08-2013- 8:30:16] >gluster v rebalance vol_rep status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost               12         7.0GB            23             0             0            completed            2011.00
                            rhs-client13                0        0Bytes             0             0             0          not started               0.00
                            rhs-client12                0        0Bytes             0             0             0          not started               0.00
volume rebalance: vol_rep: success: 

root@rhs-client11 [Nov-08-2013- 8:30:39] >gluster v info vol_rep
 
Volume Name: vol_rep
Type: Distributed-Replicate
Volume ID: 7f996c6a-dafc-475a-8539-f488595034c5
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: rhs-client11:/rhs/bricks/b1
Brick2: rhs-client12:/rhs/bricks/b1-rep1
Brick3: rhs-client11:/rhs/bricks/b2
Brick4: rhs-client12:/rhs/bricks/b2-rep1

Brick1:-
+++++++++++++++++++
root@rhs-client11 [Nov-08-2013- 8:57:49] >ls -lh /rhs/bricks/b1/file_9
---------T 2 root root 1.1G Nov  7 11:54 /rhs/bricks/b1/file_9

Brick2:-
+++++++++++++++++++
root@rhs-client12 [Nov-08-2013- 8:59:41] >ls -lh /rhs/bricks/b1-rep1/file_9
---------T 2 root root 1.1G Nov  7 11:54 /rhs/bricks/b1-rep1/file_9

Brick3:-
+++++++++++++++++++
root@rhs-client11 [Nov-08-2013- 8:57:55] >ls -lh /rhs/bricks/b2/file_9
-rw-r-Sr-T 2 root root 1.1G Nov  7 11:51 /rhs/bricks/b2/file_9

Brick4:-
+++++++++++++++++++
root@rhs-client12 [Nov-08-2013- 8:59:50] >ls -lh /rhs/bricks/b2-rep1/file_9
-rw-r-Sr-T 2 root root 1.1G Nov  7 11:51 /rhs/bricks/b2-rep1/file_9

Attributes from removed bricks :-
++++++++++++++++++++++++++++++++
root@rhs-client13 [Nov-08-2013- 8:57:49] >ls -lh /rhs/bricks/b1-rep2/file_9
-rw-r--r-- 2 root root 1020M Nov  7 11:20 /rhs/bricks/b1-rep2/file_9

root@rhs-client13 [Nov-08-2013- 9:01:37] >ls -lh /rhs/bricks/b2-rep2/file_9
---------T 2 root root 0 Nov  7 11:17 /rhs/bricks/b2-rep2/file_9

Rebalance log messages:-
++++++++++++++++++++++++++++++++
2013-11-07 11:51:25.834846] I [dht-rebalance.c:672:dht_migrate_file] 1-vol_rep-dht: /file_9: attempting to move from vol_rep-replicate-0 to vol_rep-replicate-1
[2013-11-07 11:51:27.244412] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 1836.00 secs

[2013-11-07 11:51:27.987991] I [dht-rebalance.c:672:dht_migrate_file] 0-vol_rep-dht: /file_2: attempting to move from vol_rep-replicate-0 to vol_rep-replicate-1
[2013-11-07 11:51:57.345315] I [dht-rebalance.c:881:dht_migrate_file] 1-vol_rep-dht: completed migration of /file_9 from subvolume vol_rep-replicate-0 to vol_rep-replicate-1

[2013-11-07 11:53:55.221264] W [client-rpc-fops.c:1103:client3_3_getxattr_cbk] 0-vol_rep-client-2: remote operation failed: Transport endpoint is not connected. Path: /

file_9 (3438567a-d518-4cc3-9a7a-72871ef9ceb0). Key: trusted.glusterfs.pathinfo
[2013-11-07 11:53:55.224440] I [dht-rebalance.c:672:dht_migrate_file] 0-vol_rep-dht: /file_9: attempting to move from vol_rep-replicate-1 to vol_rep-replicate-0

Some of the files got migrated from replicate-subvolume-1 to replicate-subvolume-0 .i.e in reverse direction.

[2013-11-07 11:54:22.863915] I [dht-rebalance.c:881:dht_migrate_file] 1-vol_rep-dht: completed migration of /file_6 from subvolume vol_rep-replicate-1 to vol_rep-replicate-0

Expected results:
=================
Once the rebalance is successfully complete , the file should reside in any one of  the replicate-subvolume and the other subvolume can have the link-to file.

Note You need to log in before you can comment on or make changes to this bug.