Previously, upon a directory lookup, error logs were generated if trusted.glusterfs.dht extended attribute was missing. With this fix, error logs are not generated for directories with no layouts.
Description of problem:
DHT : directory selfheal is not creatting extended attribute trusted.glusterfs.dht after creating directory ,because of that on lookup 'disk layout missing',' mismatching layouts' messages are coming in log
Version-Release number of selected component (if applicable):
3.4.0.32rhs-1.el6_4.x86_64
How reproducible:
always
Steps to Reproduce:
1.creatd DHT volume and FUSE mount it
2.one brick is down ; create dir and files
[root@DHT1 1]# gluster v status
Status of volume: testdht
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick 10.70.37.195:/rhs/brick1/1 N/A N 435
Brick 10.70.37.195:/rhs/brick1/2 49167 Y 2048
Brick 10.70.37.98:/rhs/brick1/1 49154 Y 29098
NFS Server on localhost 2049 Y 2327
NFS Server on 10.70.37.98 2049 Y 29111
There are no active volume tasks
mount:-
[root@rhs-client22 dhttest]# cd up1
[root@rhs-client22 up1]# mkdir down{1..10}
mkdir: cannot create directory `down1': Transport endpoint is not connected
mkdir: cannot create directory `down2': Transport endpoint is not connected
mkdir: cannot create directory `down4': Transport endpoint is not connected
mkdir: cannot create directory `down5': Transport endpoint is not connected
mkdir: cannot create directory `down10': Transport endpoint is not connected
[root@rhs-client22 up1]# cd down1
-bash: cd: down1: No such file or directory
[root@rhs-client22 up1]# cd down3
[root@rhs-client22 down3]# touch down{1..10}
[root@rhs-client22 down3]# cd ..
[root@rhs-client22 up1]# touch fdown{1..10}
touch: cannot touch `fdown6': Transport endpoint is not connected
touch: cannot touch `fdown9': Transport endpoint is not connected
3. make that brick up and from mount point issue lookup on that dir.
self heal should create directory on sub-vol which was down previously and also set xattr.
[root@DHT1 1]# gluster v status
Status of volume: testdht
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick 10.70.37.195:/rhs/brick1/1 49166 Y 435
Brick 10.70.37.195:/rhs/brick1/2 49167 Y 2048
Brick 10.70.37.98:/rhs/brick1/1 49154 Y 29098
NFS Server on localhost 2049 Y 2327
NFS Server on 10.70.37.98 2049 Y 29111
There are no active volume tasks
mount:-
[root@rhs-client22 up1]# ls -l down3
log:-
[2013-09-09 03:54:19.773681] I [dht-layout.c:720:dht_layout_dir_mismatch] 0-testdht-dht: /up1/down3 - disk layout missing
[2013-09-09 03:54:19.773766] I [dht-common.c:638:dht_revalidate_cbk] 0-testdht-dht: mismatching layouts for /up1/down3
backend:-
previously down <-----------------------------------------
[root@DHT1 1]# getfattr -d -m . -e hex /rhs/brick1/1/up1/down3
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/1/up1/down3
trusted.gfid=0xc052b59c4d644d0f9002136f83975e10
[root@DHT1 1]# getfattr -d -m . -e hex /rhs/brick1/2/up1/down3
# file: rhs/brick1/2/up1/down3
trusted.gfid=0xc052b59c4d644d0f9002136f83975e10
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
[root@DHT3 1]# getfattr -d -m . -e hex /rhs/brick1/*/up1/down3
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick1/1/up1/down3
trusted.gfid=0xc052b59c4d644d0f9002136f83975e10
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
Actual results:
self heal is creating dir on previously down brick but it is not setting xattr for the same
Expected results:
self heal should create dir and set xattr for the same
Additional info:
In Anshi release it was creating dir on down sub-vol and was setting all xattr for the same
setting corbett and other flags as this is being tracked for corbett.
Comment 6Vijaikumar Mallikarjuna
2013-12-26 14:56:23 UTC
xattr are not set because there is no layout created for a directory.
Current workaround is to run fix-layout and rebalance from the CLI.
Comment 7Vijaikumar Mallikarjuna
2013-12-30 12:50:59 UTC
In Anshi release, xattrs was written with 0-0% hash range when there is no layout for the directory.
Current behavior is, we ignore the directories when the layout is absent and do not write xattrs.
This change is introduced by the patch: http://review.gluster.org/4668/
Logging for missed layout need to be fixed when err == 0, and start == stop.
Comment 8Vijaikumar Mallikarjuna
2013-12-30 13:22:37 UTC
verified with 3.4.0.53rhs-1.el6rhs.x86_64
self heal is creating dir on previously down brick and also set xattr for the same.
Hence moving it to verified
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
http://rhn.redhat.com/errata/RHEA-2014-0208.html
Comment 15Red Hat Bugzilla
2023-09-14 01:50:17 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days
Description of problem: DHT : directory selfheal is not creatting extended attribute trusted.glusterfs.dht after creating directory ,because of that on lookup 'disk layout missing',' mismatching layouts' messages are coming in log Version-Release number of selected component (if applicable): 3.4.0.32rhs-1.el6_4.x86_64 How reproducible: always Steps to Reproduce: 1.creatd DHT volume and FUSE mount it 2.one brick is down ; create dir and files [root@DHT1 1]# gluster v status Status of volume: testdht Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.37.195:/rhs/brick1/1 N/A N 435 Brick 10.70.37.195:/rhs/brick1/2 49167 Y 2048 Brick 10.70.37.98:/rhs/brick1/1 49154 Y 29098 NFS Server on localhost 2049 Y 2327 NFS Server on 10.70.37.98 2049 Y 29111 There are no active volume tasks mount:- [root@rhs-client22 dhttest]# cd up1 [root@rhs-client22 up1]# mkdir down{1..10} mkdir: cannot create directory `down1': Transport endpoint is not connected mkdir: cannot create directory `down2': Transport endpoint is not connected mkdir: cannot create directory `down4': Transport endpoint is not connected mkdir: cannot create directory `down5': Transport endpoint is not connected mkdir: cannot create directory `down10': Transport endpoint is not connected [root@rhs-client22 up1]# cd down1 -bash: cd: down1: No such file or directory [root@rhs-client22 up1]# cd down3 [root@rhs-client22 down3]# touch down{1..10} [root@rhs-client22 down3]# cd .. [root@rhs-client22 up1]# touch fdown{1..10} touch: cannot touch `fdown6': Transport endpoint is not connected touch: cannot touch `fdown9': Transport endpoint is not connected 3. make that brick up and from mount point issue lookup on that dir. self heal should create directory on sub-vol which was down previously and also set xattr. [root@DHT1 1]# gluster v status Status of volume: testdht Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.37.195:/rhs/brick1/1 49166 Y 435 Brick 10.70.37.195:/rhs/brick1/2 49167 Y 2048 Brick 10.70.37.98:/rhs/brick1/1 49154 Y 29098 NFS Server on localhost 2049 Y 2327 NFS Server on 10.70.37.98 2049 Y 29111 There are no active volume tasks mount:- [root@rhs-client22 up1]# ls -l down3 log:- [2013-09-09 03:54:19.773681] I [dht-layout.c:720:dht_layout_dir_mismatch] 0-testdht-dht: /up1/down3 - disk layout missing [2013-09-09 03:54:19.773766] I [dht-common.c:638:dht_revalidate_cbk] 0-testdht-dht: mismatching layouts for /up1/down3 backend:- previously down <----------------------------------------- [root@DHT1 1]# getfattr -d -m . -e hex /rhs/brick1/1/up1/down3 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/1/up1/down3 trusted.gfid=0xc052b59c4d644d0f9002136f83975e10 [root@DHT1 1]# getfattr -d -m . -e hex /rhs/brick1/2/up1/down3 # file: rhs/brick1/2/up1/down3 trusted.gfid=0xc052b59c4d644d0f9002136f83975e10 trusted.glusterfs.dht=0x00000001000000007fffffffffffffff [root@DHT3 1]# getfattr -d -m . -e hex /rhs/brick1/*/up1/down3 getfattr: Removing leading '/' from absolute path names # file: rhs/brick1/1/up1/down3 trusted.gfid=0xc052b59c4d644d0f9002136f83975e10 trusted.glusterfs.dht=0x0000000100000000000000007ffffffe Actual results: self heal is creating dir on previously down brick but it is not setting xattr for the same Expected results: self heal should create dir and set xattr for the same Additional info: In Anshi release it was creating dir on down sub-vol and was setting all xattr for the same