Bug 1698861
| Summary: | Renaming a directory when 2 bricks of multiple disperse subvols are down leaves both old and new dirs on the bricks. | |||
|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Nithya Balachandran <nbalacha> | |
| Component: | disperse | Assignee: | Ashish Pandey <aspandey> | |
| Status: | CLOSED DUPLICATE | QA Contact: | ||
| Severity: | high | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | mainline | CC: | aspandey, atumball, bugs, g.amedick, jahernan, pasik, pkarampu | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1701983 (view as bug list) | Environment: | ||
| Last Closed: | 2019-07-02 04:37:46 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1701983 | |||
Steps -
1 - Create 4+2 volume and mount it on /mnt/vol
Volume Name: vol
Type: Disperse
Volume ID: 742b8e08-1f16-4bad-aa94-5e36dd10fe91
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: apandey:/home/apandey/bricks/gluster/vol-1
Brick2: apandey:/home/apandey/bricks/gluster/vol-2
Brick3: apandey:/home/apandey/bricks/gluster/vol-3
Brick4: apandey:/home/apandey/bricks/gluster/vol-4
Brick5: apandey:/home/apandey/bricks/gluster/vol-5
Brick6: apandey:/home/apandey/bricks/gluster/vol-6
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
Status of volume: vol
Gluster process
2 - mkdir /mnt/vol/dir/old -p
3 -for i in {1..200}; do touch dir/old/file-$i ; done
[root@apandey glusterfs]# gluster v status
Status of volume: vol
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick apandey:/home/apandey/bricks/gluster/
vol-1 49152 0 Y 13401
Brick apandey:/home/apandey/bricks/gluster/
vol-2 49153 0 Y 11682
Brick apandey:/home/apandey/bricks/gluster/
vol-3 49154 0 Y 11702
Brick apandey:/home/apandey/bricks/gluster/
vol-4 49155 0 Y 11722
Brick apandey:/home/apandey/bricks/gluster/
vol-5 49156 0 Y 11742
Brick apandey:/home/apandey/bricks/gluster/
vol-6 49157 0 Y 11762
Self-heal Daemon on localhost N/A N/A Y 13427
Task Status of Volume vol
------------------------------------------------------------------------------
There are no active volume tasks
4 - Kill brick 1
[root@apandey glusterfs]# kill 13401
[root@apandey glusterfs]# gluster v status
Status of volume: vol
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick apandey:/home/apandey/bricks/gluster/
vol-1 N/A N/A N N/A
Brick apandey:/home/apandey/bricks/gluster/
vol-2 49153 0 Y 11682
Brick apandey:/home/apandey/bricks/gluster/
vol-3 49154 0 Y 11702
Brick apandey:/home/apandey/bricks/gluster/
vol-4 49155 0 Y 11722
Brick apandey:/home/apandey/bricks/gluster/
vol-5 49156 0 Y 11742
Brick apandey:/home/apandey/bricks/gluster/
vol-6 49157 0 Y 11762
Self-heal Daemon on localhost N/A N/A Y 13427
Task Status of Volume vol
------------------------------------------------------------------------------
There are no active volume tasks
5 - mv dir/old/ dir/new
6 - [root@apandey vol]# ll dir/new | wc -l
201
7 - gluster v start vol force
8 -ll dir/new | wc -l
1
9 - ll dir/old | wc -l
1
10-
[root@apandey glusterfs]# getfattr -m. -d -e hex /home/apandey/bricks/gluster/vol-*/dir/old
getfattr: Removing leading '/' from absolute path names
# file: home/apandey/bricks/gluster/vol-1/dir/old
trusted.ec.dirty=0x00000000000000010000000000000001
trusted.ec.version=0x00000000000000c800000000000000c8
trusted.gfid=0x098b8bf8e0ba406283f26334a0b83e23
trusted.glusterfs.dht=0x000000000000000000000000ffffffff
# file: home/apandey/bricks/gluster/vol-2/dir/old
trusted.ec.dirty=0x00000000000000000000000000000000
trusted.gfid=0x098b8bf8e0ba406283f26334a0b83e23
# file: home/apandey/bricks/gluster/vol-3/dir/old
trusted.ec.dirty=0x00000000000000000000000000000000
trusted.gfid=0x098b8bf8e0ba406283f26334a0b83e23
# file: home/apandey/bricks/gluster/vol-4/dir/old
trusted.ec.dirty=0x00000000000000000000000000000000
trusted.gfid=0x098b8bf8e0ba406283f26334a0b83e23
# file: home/apandey/bricks/gluster/vol-5/dir/old
trusted.ec.dirty=0x00000000000000000000000000000000
trusted.gfid=0x098b8bf8e0ba406283f26334a0b83e23
# file: home/apandey/bricks/gluster/vol-6/dir/old
trusted.ec.dirty=0x00000000000000000000000000000000
trusted.gfid=0x098b8bf8e0ba406283f26334a0b83e23
11-
[root@apandey glusterfs]# getfattr -m. -d -e hex /home/apandey/bricks/gluster/vol-*/dir/new
getfattr: Removing leading '/' from absolute path names
# file: home/apandey/bricks/gluster/vol-1/dir/new
trusted.gfid=0x098b8bf8e0ba406283f26334a0b83e23
# file: home/apandey/bricks/gluster/vol-2/dir/new
trusted.ec.version=0x00000000000000c800000000000000c8
trusted.gfid=0x098b8bf8e0ba406283f26334a0b83e23
trusted.glusterfs.dht=0x000000000000000000000000ffffffff
# file: home/apandey/bricks/gluster/vol-3/dir/new
trusted.ec.version=0x00000000000000c800000000000000c8
trusted.gfid=0x098b8bf8e0ba406283f26334a0b83e23
trusted.glusterfs.dht=0x000000000000000000000000ffffffff
# file: home/apandey/bricks/gluster/vol-4/dir/new
trusted.ec.version=0x00000000000000c800000000000000c8
trusted.gfid=0x098b8bf8e0ba406283f26334a0b83e23
trusted.glusterfs.dht=0x000000000000000000000000ffffffff
# file: home/apandey/bricks/gluster/vol-5/dir/new
trusted.ec.version=0x00000000000000c800000000000000c8
trusted.gfid=0x098b8bf8e0ba406283f26334a0b83e23
trusted.glusterfs.dht=0x000000000000000000000000ffffffff
# file: home/apandey/bricks/gluster/vol-6/dir/new
trusted.ec.version=0x00000000000000c800000000000000c8
trusted.gfid=0x098b8bf8e0ba406283f26334a0b83e23
trusted.glusterfs.dht=0x000000000000000000000000ffffffff
12 - [root@apandey glusterfs]# gluster v heal vol info
Brick apandey:/home/apandey/bricks/gluster/vol-1
Status: Connected
Number of entries: 0
Brick apandey:/home/apandey/bricks/gluster/vol-2
Status: Connected
Number of entries: 0
Brick apandey:/home/apandey/bricks/gluster/vol-3
Status: Connected
Number of entries: 0
Brick apandey:/home/apandey/bricks/gluster/vol-4
Status: Connected
Number of entries: 0
Brick apandey:/home/apandey/bricks/gluster/vol-5
Status: Connected
Number of entries: 0
Brick apandey:/home/apandey/bricks/gluster/vol-6
Status: Connected
Number of entries: 0
13 - As we can see that the trusted.glusterfs.dht=0x000000000000000000000000ffffffff is missing for "old" directory on all the 5 bricks, I set this xattr manually.
setfattr -n trusted.glusterfs.dht -v 0x000000000000000000000000ffffffff /home/apandey/bricks/gluster/vol-{2..6}/dir/old
14 - I copied the data from new dir to old dir on respective bricks -
15 - for i in {2..6} ; do yes | cp -rf /home/apandey/bricks/gluster/vol-$i/dir/new/* /home/apandey/bricks/gluster/vol-$i/dir/old/; done
16 - After this files were visible on both the old and new dir
[root@apandey vol]# ll dir/new | wc -l
201
[root@apandey vol]# ll dir/old | wc -l
201
[root@apandey vol]#
17 - Although this will have both the directories, if we have all the data back and all the bricks are UP, we can safely move the data in new directory.
This is working for the issue which we created using our set of steps. I am not sure if this case is exactly similar to what user is experiencing or not.
*** This bug has been marked as a duplicate of bug 1359681 *** |
Description of problem: Running the following .t results in both olddir and newdir visible from the mount point and listing them shows no files. Steps to Reproduce: #!/bin/bash . $(dirname $0)/../../include.rc . $(dirname $0)/../../volume.rc . $(dirname $0)/../../common-utils.rc cleanup TEST glusterd TEST pidof glusterd TEST $CLI volume create $V0 disperse 6 disperse-data 4 $H0:$B0/$V0-{1..24} force TEST $CLI volume start $V0 TEST glusterfs -s $H0 --volfile-id $V0 $M0 ls $M0/ mkdir $M0/olddir mkdir $M0/olddir/subdir touch $M0/olddir/file-{1..10} ls -lR TEST kill_brick $V0 $H0 $B0/$V0-1 TEST kill_brick $V0 $H0 $B0/$V0-2 TEST kill_brick $V0 $H0 $B0/$V0-7 TEST kill_brick $V0 $H0 $B0/$V0-8 TEST mv $M0/olddir $M0/newdir # Start all bricks TEST $CLI volume start $V0 force $CLI volume status # It takes a while for the client to reconnect to the brick sleep 5 ls -l $M0 # Cleanup #cleanup Version-Release number of selected component (if applicable): How reproducible: Consistently Actual results: [root@rhgs313-6 tests]# ls -lR /mnt/glusterfs/0/ /mnt/glusterfs/0/: total 8 drwxr-xr-x. 2 root root 4096 Apr 11 17:12 newdir drwxr-xr-x. 2 root root 4096 Apr 11 17:12 olddir /mnt/glusterfs/0/newdir: total 0 /mnt/glusterfs/0/olddir: total 0 [root@rhgs313-6 tests]# Expected results: Additional info: