| Summary: | Even after successful completion of rebalance after add-brick , One of the brick contains a file with "-rw-r-Sr-T" attributes | |||
|---|---|---|---|---|
| Product: | Red Hat Gluster Storage | Reporter: | spandura | |
| Component: | distribute | Assignee: | Nithya Balachandran <nbalacha> | |
| Status: | CLOSED DEFERRED | QA Contact: | storage-qa-internal <storage-qa-internal> | |
| Severity: | medium | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 2.1 | CC: | spalai, vbellur | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1286201 (view as bug list) | Environment: | ||
| Last Closed: | 2015-11-27 12:27:03 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Bug Depends On: | ||||
| Bug Blocks: | 1286201 | |||
Description of problem: ======================= On a 1 x 3 replicate volume added 3 more bricks to the volume to make it 2 x 3 distribute-replicate volume. Performed rebalance on the volume. When rebalance was in progress removed 2 bricks to change the replica count to 2 (remove-brick replica 2 <brick3> <brick6> force). This changed the volume type to 2 x 2 distribute-replicate volume. Rebalance got successfully completed. Check the attributes of the files on the bricks. One of the file still had the "-rw-r-Sr-T" atributes even after successful completion of rebalance. Version-Release number of selected component (if applicable): ============================================================= glusterfs 3.4.0.35.1u2rhs built on Oct 21 2013 14:00:58 How reproducible: ================ Tried only once. Steps to Reproduce: ===================== 1. Create a replicate volume with 3 bricks (1 x 3). Start the volume. 2. Create few files. open the files and keep reading , modifying the file. 3. While the IO from mount point is in progress, add a new sub-volume with 3 bricks changing the volume type to distribute-replicate. 4. Start rebalance on the volume. 5. While rebalance is in progress, remove one replica from each sub volume.(remove-brick replica 2 <brick3> <brick6> force) 6. Once the rebalance is complete check the attributes of all the files. Actual results: =============== root@rhs-client11 [Nov-08-2013- 8:30:16] >gluster v rebalance vol_rep status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 12 7.0GB 23 0 0 completed 2011.00 rhs-client13 0 0Bytes 0 0 0 not started 0.00 rhs-client12 0 0Bytes 0 0 0 not started 0.00 volume rebalance: vol_rep: success: root@rhs-client11 [Nov-08-2013- 8:30:39] >gluster v info vol_rep Volume Name: vol_rep Type: Distributed-Replicate Volume ID: 7f996c6a-dafc-475a-8539-f488595034c5 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: rhs-client11:/rhs/bricks/b1 Brick2: rhs-client12:/rhs/bricks/b1-rep1 Brick3: rhs-client11:/rhs/bricks/b2 Brick4: rhs-client12:/rhs/bricks/b2-rep1 Brick1:- +++++++++++++++++++ root@rhs-client11 [Nov-08-2013- 8:57:49] >ls -lh /rhs/bricks/b1/file_9 ---------T 2 root root 1.1G Nov 7 11:54 /rhs/bricks/b1/file_9 Brick2:- +++++++++++++++++++ root@rhs-client12 [Nov-08-2013- 8:59:41] >ls -lh /rhs/bricks/b1-rep1/file_9 ---------T 2 root root 1.1G Nov 7 11:54 /rhs/bricks/b1-rep1/file_9 Brick3:- +++++++++++++++++++ root@rhs-client11 [Nov-08-2013- 8:57:55] >ls -lh /rhs/bricks/b2/file_9 -rw-r-Sr-T 2 root root 1.1G Nov 7 11:51 /rhs/bricks/b2/file_9 Brick4:- +++++++++++++++++++ root@rhs-client12 [Nov-08-2013- 8:59:50] >ls -lh /rhs/bricks/b2-rep1/file_9 -rw-r-Sr-T 2 root root 1.1G Nov 7 11:51 /rhs/bricks/b2-rep1/file_9 Attributes from removed bricks :- ++++++++++++++++++++++++++++++++ root@rhs-client13 [Nov-08-2013- 8:57:49] >ls -lh /rhs/bricks/b1-rep2/file_9 -rw-r--r-- 2 root root 1020M Nov 7 11:20 /rhs/bricks/b1-rep2/file_9 root@rhs-client13 [Nov-08-2013- 9:01:37] >ls -lh /rhs/bricks/b2-rep2/file_9 ---------T 2 root root 0 Nov 7 11:17 /rhs/bricks/b2-rep2/file_9 Rebalance log messages:- ++++++++++++++++++++++++++++++++ 2013-11-07 11:51:25.834846] I [dht-rebalance.c:672:dht_migrate_file] 1-vol_rep-dht: /file_9: attempting to move from vol_rep-replicate-0 to vol_rep-replicate-1 [2013-11-07 11:51:27.244412] I [dht-rebalance.c:1768:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 1836.00 secs [2013-11-07 11:51:27.987991] I [dht-rebalance.c:672:dht_migrate_file] 0-vol_rep-dht: /file_2: attempting to move from vol_rep-replicate-0 to vol_rep-replicate-1 [2013-11-07 11:51:57.345315] I [dht-rebalance.c:881:dht_migrate_file] 1-vol_rep-dht: completed migration of /file_9 from subvolume vol_rep-replicate-0 to vol_rep-replicate-1 [2013-11-07 11:53:55.221264] W [client-rpc-fops.c:1103:client3_3_getxattr_cbk] 0-vol_rep-client-2: remote operation failed: Transport endpoint is not connected. Path: / file_9 (3438567a-d518-4cc3-9a7a-72871ef9ceb0). Key: trusted.glusterfs.pathinfo [2013-11-07 11:53:55.224440] I [dht-rebalance.c:672:dht_migrate_file] 0-vol_rep-dht: /file_9: attempting to move from vol_rep-replicate-1 to vol_rep-replicate-0 Some of the files got migrated from replicate-subvolume-1 to replicate-subvolume-0 .i.e in reverse direction. [2013-11-07 11:54:22.863915] I [dht-rebalance.c:881:dht_migrate_file] 1-vol_rep-dht: completed migration of /file_6 from subvolume vol_rep-replicate-1 to vol_rep-replicate-0 Expected results: ================= Once the rebalance is successfully complete , the file should reside in any one of the replicate-subvolume and the other subvolume can have the link-to file.