Hide Forgot
Description of problem: ======================= For many files, chmod command failed to change permission with error "cannot operate on dangling symlink" Version-Release number of selected component (if applicable): 3.8.4-2.26.git0a405a4.el7rhgs.x86_64 Steps to Reproduce: =================== 1) Create a distributed replica volume and start it. 2) Enable md-cache required settings on the volume ( Please see gluster volume info output for enabled md-cache settings) 3) Fuse mount the volume on multiple clients. 4) Perform below tasks simultaneosuly from multiple clients, a) From client-1, touch --> for i in {1..20000};do touch f$i;done b) From client-2, create hardlinks for the created files , for i in {1..20000};do ln f$i fl$i;done c) From client-3, change the permissions for the created files, for i in {1..20000};do ln f$i fl$i;done d) From client-4, do a continuous lookup. 5) While the tasks in step-4 are in progress, add few bricks to the volume and start rebalance. Wait till step-4 and step-5 completes. We can see for many files, chmod command fails with "cannot operate on dangling symlink". Checked the files on which the chmod command failed and the permission is not changed. [root@dhcp41-243 fuse]# for i in {1..20000};do chmod 660 f$i;done chmod: changing permissions of ‘f93’: No such file or directory chmod: changing permissions of ‘f96’: No such file or directory chmod: changing permissions of ‘f99’: No such file or directory chmod: changing permissions of ‘f104’: No such file or directory chmod: changing permissions of ‘f105’: No such file or directory chmod: changing permissions of ‘f109’: No such file or directory chmod: changing permissions of ‘f110’: No such file or directory chmod: changing permissions of ‘f114’: No such file or directory chmod: changing permissions of ‘f122’: No such file or directory chmod: changing permissions of ‘f125’: No such file or directory chmod: changing permissions of ‘f126’: No such file or directory chmod: changing permissions of ‘f130’: No such file or directory chmod: cannot operate on dangling symlink ‘f132’ chmod: changing permissions of ‘f135’: No such file or directory chmod: changing permissions of ‘f136’: No such file or directory chmod: changing permissions of ‘f140’: No such file or directory chmod: changing permissions of ‘f141’: No such file or directory chmod: changing permissions of ‘f143’: No such file or directory chmod: changing permissions of ‘f148’: No such file or directory chmod: changing permissions of ‘f149’: No such file or directory chmod: changing permissions of ‘f152’: No such file or directory chmod: changing permissions of ‘f154’: No such file or directory chmod: changing permissions of ‘f161’: No such file or directory chmod: changing permissions of ‘f166’: No such file or directory chmod: changing permissions of ‘f172’: No such file or directory chmod: changing permissions of ‘f175’: No such file or directory chmod: changing permissions of ‘f176’: No such file or directory chmod: changing permissions of ‘f177’: No such file or directory chmod: cannot access ‘f178’: No such file or directory chmod: cannot operate on dangling symlink ‘f179’ chmod: changing permissions of ‘f180’: No such file or directory chmod: changing permissions of ‘f181’: No such file or directory chmod: changing permissions of ‘f182’: No such file or directory chmod: changing permissions of ‘f183’: No such file or directory chmod: changing permissions of ‘f184’: No such file or directory chmod: changing permissions of ‘f185’: No such file or directory chmod: changing permissions of ‘f186’: No such file or directory chmod: changing permissions of ‘f187’: No such file or directory chmod: changing permissions of ‘f188’: No such file or directory chmod: changing permissions of ‘f189’: No such file or directory chmod: changing permissions of ‘f190’: No such file or directory chmod: cannot access ‘f191’: No such file or directory chmod: changing permissions of ‘f192’: No such file or directory chmod: changing permissions of ‘f193’: No such file or directory chmod: cannot access ‘f194’: No such file or directory chmod: changing permissions of ‘f195’: No such file or directory chmod: cannot access ‘f196’: No such file or directory chmod: cannot access ‘f197’: No such file or directory chmod: cannot access ‘f198’: No such file or directory chmod: changing permissions of ‘f199’: No such file or directory chmod: cannot access ‘f200’: No such file or directory chmod: cannot access ‘f201’: No such file or directory chmod: changing permissions of ‘f203’: No such file or directory chmod: changing permissions of ‘f204’: No such file or directory chmod: cannot operate on dangling symlink ‘f206’ chmod: cannot operate on dangling symlink ‘f208’ chmod: changing permissions of ‘f213’: No such file or directory chmod: changing permissions of ‘f216’: No such file or directory chmod: changing permissions of ‘f217’: No such file or directory chmod: changing permissions of ‘f218’: No such file or directory chmod: changing permissions of ‘f219’: No such file or directory chmod: cannot operate on dangling symlink ‘f220’ chmod: changing permissions of ‘f221’: No such file or directory chmod: cannot operate on dangling symlink ‘f222’ chmod: changing permissions of ‘f225’: No such file or directory chmod: changing permissions of ‘f226’: No such file or directory chmod: changing permissions of ‘f227’: No such file or directory chmod: changing permissions of ‘f228’: No such file or directory chmod: changing permissions of ‘f229’: No such file or directory chmod: changing permissions of ‘f230’: No such file or directory chmod: cannot access ‘f231’: No such file or directory chmod: cannot access ‘f232’: No such file or directory chmod: cannot access ‘f233’: No such file or directory chmod: cannot access ‘f234’: No such file or directory chmod: changing permissions of ‘f236’: No such file or directory chmod: cannot operate on dangling symlink ‘f243’ chmod: changing permissions of ‘f245’: No such file or directory chmod: changing permissions of ‘f248’: No such file or directory chmod: changing permissions of ‘f249’: No such file or directory chmod: changing permissions of ‘f250’: No such file or directory chmod: changing permissions of ‘f251’: No such file or directory chmod: changing permissions of ‘f252’: No such file or directory chmod: changing permissions of ‘f253’: No such file or directory chmod: cannot access ‘f254’: No such file or directory chmod: changing permissions of ‘f255’: No such file or directory chmod: cannot access ‘f256’: No such file or directory chmod: changing permissions of ‘f257’: No such file or directory chmod: changing permissions of ‘f258’: No such file or directory chmod: cannot access ‘f259’: No such file or directory chmod: cannot access ‘f260’: No such file or directory chmod: cannot access ‘f261’: No such file or directory chmod: cannot operate on dangling symlink ‘f263’ chmod: cannot operate on dangling symlink ‘f266’ chmod: changing permissions of ‘f267’: No such file or directory chmod: changing permissions of ‘f277’: No such file or directory chmod: changing permissions of ‘f278’: No such file or directory chmod: changing permissions of ‘f279’: No such file or directory chmod: changing permissions of ‘f280’: No such file or directory chmod: changing permissions of ‘f281’: No such file or directory chmod: changing permissions of ‘f282’: No such file or directory chmod: cannot access ‘f283’: No such file or directory chmod: changing permissions of ‘f284’: No such file or directory chmod: cannot access ‘f285’: No such file or directory chmod: cannot access ‘f286’: No such file or directory chmod: changing permissions of ‘f287’: No such file or directory chmod: cannot access ‘f288’: No such file or directory chmod: changing permissions of ‘f289’: No such file or directory chmod: changing permissions of ‘f290’: No such file or directory chmod: cannot operate on dangling symlink ‘f291’ chmod: cannot operate on dangling symlink ‘f292’ chmod: changing permissions of ‘f293’: No such file or directory chmod: changing permissions of ‘f294’: No such file or directory chmod: cannot operate on dangling symlink ‘f297’ chmod: changing permissions of ‘f303’: No such file or directory chmod: changing permissions of ‘f304’: No such file or directory chmod: changing permissions of ‘f306’: No such file or directory chmod: changing permissions of ‘f307’: No such file or directory chmod: changing permissions of ‘f308’: No such file or directory chmod: changing permissions of ‘f309’: No such file or directory chmod: changing permissions of ‘f310’: No such file or directory chmod: cannot access ‘f311’: No such file or directory chmod: changing permissions of ‘f312’: No such file or directory chmod: changing permissions of ‘f314’: No such file or directory chmod: changing permissions of ‘f316’: No such file or directory chmod: changing permissions of ‘f317’: No such file or directory chmod: changing permissions of ‘f318’: No such file or directory chmod: cannot operate on dangling symlink ‘f319’ chmod: changing permissions of ‘f322’: No such file or directory chmod: cannot operate on dangling symlink ‘f325’ chmod: changing permissions of ‘f334’: No such file or directory chmod: changing permissions of ‘f336’: No such file or directory chmod: changing permissions of ‘f337’: No such file or directory chmod: changing permissions of ‘f338’: No such file or directory chmod: changing permissions of ‘f339’: No such file or directory chmod: cannot operate on dangling symlink ‘f340’ chmod: changing permissions of ‘f341’: No such file or directory chmod: changing permissions of ‘f342’: No such file or directory chmod: changing permissions of ‘f343’: No such file or directory chmod: cannot access ‘f344’: No such file or directory chmod: changing permissions of ‘f345’: No such file or directory chmod: changing permissions of ‘f348’: No such file or directory chmod: changing permissions of ‘f350’: No such file or directory chmod: changing permissions of ‘f353’: No such file or directory chmod: changing permissions of ‘f360’: No such file or directory chmod: cannot operate on dangling symlink ‘f362’ chmod: changing permissions of ‘f363’: No such file or directory chmod: changing permissions of ‘f364’: No such file or directory chmod: changing permissions of ‘f365’: No such file or directory chmod: changing permissions of ‘f366’: No such file or directory chmod: cannot operate on dangling symlink ‘f367’ chmod: changing permissions of ‘f368’: No such file or directory chmod: changing permissions of ‘f369’: No such file or directory chmod: changing permissions of ‘f370’: No such file or directory chmod: changing permissions of ‘f371’: No such file or directory chmod: cannot access ‘f372’: No such file or directory chmod: changing permissions of ‘f373’: No such file or directory chmod: cannot access ‘f374’: No such file or directory chmod: changing permissions of ‘f375’: No such file or directory chmod: changing permissions of ‘f376’: No such file or directory chmod: changing permissions of ‘f377’: No such file or directory chmod: cannot access ‘f378’: No such file or directory chmod: cannot access ‘f379’: No such file or directory chmod: changing permissions of ‘f381’: No such file or directory chmod: cannot access ‘f382’: No such file or directory chmod: cannot operate on dangling symlink ‘f384’ chmod: changing permissions of ‘f391’: No such file or directory chmod: changing permissions of ‘f393’: No such file or directory chmod: changing permissions of ‘f394’: No such file or directory chmod: changing permissions of ‘f395’: No such file or directory chmod: changing permissions of ‘f397’: No such file or directory chmod: cannot operate on dangling symlink ‘f399’ chmod: changing permissions of ‘f400’: No such file or directory chmod: cannot operate on dangling symlink ‘f403’ chmod: changing permissions of ‘f415’: No such file or directory chmod: changing permissions of ‘f416’: No such file or directory chmod: changing permissions of ‘f417’: No such file or directory chmod: changing permissions of ‘f418’: No such file or directory chmod: changing permissions of ‘f419’: No such file or directory chmod: changing permissions of ‘f420’: No such file or directory chmod: changing permissions of ‘f421’: No such file or directory chmod: changing permissions of ‘f422’: No such file or directory chmod: changing permissions of ‘f423’: No such file or directory chmod: cannot access ‘f424’: No such file or directory chmod: changing permissions of ‘f425’: No such file or directory chmod: cannot access ‘f426’: No such file or directory chmod: cannot access ‘f427’: No such file or directory chmod: changing permissions of ‘f428’: No such file or directory chmod: changing permissions of ‘f429’: No such file or directory chmod: changing permissions of ‘f430’: No such file or directory chmod: changing permissions of ‘f431’: No such file or directory chmod: cannot access ‘f432’: No such file or directory chmod: cannot access ‘f433’: No such file or directory chmod: cannot access ‘f434’: No such file or directory chmod: cannot access ‘f435’: No such file or directory chmod: cannot access ‘f436’: No such file or directory chmod: cannot access ‘f437’: No such file or directory chmod: cannot access ‘f438’: No such file or directory chmod: changing permissions of ‘f440’: No such file or directory chmod: changing permissions of ‘f441’: No such file or directory chmod: changing permissions of ‘f442’: No such file or directory chmod: changing permissions of ‘f443’: No such file or directory chmod: changing permissions of ‘f444’: No such file or directory chmod: changing permissions of ‘f445’: No such file or directory chmod: cannot access ‘f446’: No such file or directory chmod: changing permissions of ‘f447’: No such file or directory chmod: changing permissions of ‘f448’: No such file or directory chmod: changing permissions of ‘f449’: No such file or directory chmod: changing permissions of ‘f450’: No such file or directory chmod: changing permissions of ‘f451’: No such file or directory chmod: cannot access ‘f452’: No such file or directory chmod: cannot access ‘f453’: No such file or directory chmod: changing permissions of ‘f454’: No such file or directory chmod: cannot access ‘f455’: No such file or directory chmod: cannot access ‘f456’: No such file or directory chmod: cannot operate on dangling symlink ‘f457’ chmod: cannot operate on dangling symlink ‘f458’ chmod: cannot operate on dangling symlink ‘f459’ chmod: changing permissions of ‘f464’: No such file or directory chmod: changing permissions of ‘f465’: No such file or directory chmod: changing permissions of ‘f466’: No such file or directory chmod: changing permissions of ‘f467’: No such file or directory chmod: changing permissions of ‘f468’: No such file or directory chmod: changing permissions of ‘f469’: No such file or directory chmod: changing permissions of ‘f470’: No such file or directory chmod: changing permissions of ‘f471’: No such file or directory chmod: cannot access ‘f472’: No such file or directory chmod: cannot access ‘f473’: No such file or directory chmod: cannot access ‘f474’: No such file or directory chmod: changing permissions of ‘f475’: No such file or directory chmod: changing permissions of ‘f476’: No such file or directory chmod: changing permissions of ‘f477’: No such file or directory chmod: changing permissions of ‘f478’: No such file or directory chmod: cannot access ‘f479’: No such file or directory chmod: changing permissions of ‘f480’: No such file or directory chmod: cannot operate on dangling symlink ‘f481’ chmod: cannot operate on dangling symlink ‘f484’ chmod: cannot operate on dangling symlink ‘f487’ chmod: cannot operate on dangling symlink ‘f488’ chmod: cannot operate on dangling symlink ‘f489’ chmod: changing permissions of ‘f490’: No such file or directory chmod: changing permissions of ‘f491’: No such file or directory chmod: changing permissions of ‘f492’: No such file or directory chmod: changing permissions of ‘f493’: No such file or directory chmod: changing permissions of ‘f494’: No such file or directory chmod: changing permissions of ‘f495’: No such file or directory chmod: cannot access ‘f496’: No such file or directory chmod: changing permissions of ‘f499’: No such file or directory chmod: changing permissions of ‘f503’: No such file or directory chmod: changing permissions of ‘f513’: No such file or directory chmod: changing permissions of ‘f514’: No such file or directory chmod: cannot operate on dangling symlink ‘f515’ chmod: changing permissions of ‘f516’: No such file or directory chmod: changing permissions of ‘f517’: No such file or directory chmod: changing permissions of ‘f518’: No such file or directory chmod: changing permissions of ‘f519’: No such file or directory chmod: cannot operate on dangling symlink ‘f520’ chmod: changing permissions of ‘f528’: No such file or directory chmod: changing permissions of ‘f532’: No such file or directory chmod: changing permissions of ‘f535’: No such file or directory chmod: changing permissions of ‘f536’: No such file or directory chmod: changing permissions of ‘f537’: No such file or directory chmod: changing permissions of ‘f538’: No such file or directory chmod: changing permissions of ‘f539’: No such file or directory chmod: changing permissions of ‘f540’: No such file or directory chmod: changing permissions of ‘f554’: No such file or directory chmod: changing permissions of ‘f555’: No such file or directory chmod: changing permissions of ‘f556’: No such file or directory chmod: changing permissions of ‘f557’: No such file or directory chmod: cannot access ‘f558’: No such file or directory chmod: changing permissions of ‘f559’: No such file or directory chmod: changing permissions of ‘f560’: No such file or directory chmod: changing permissions of ‘f561’: No such file or directory chmod: changing permissions of ‘f562’: No such file or directory chmod: changing permissions of ‘f563’: No such file or directory chmod: cannot access ‘f564’: No such file or directory chmod: changing permissions of ‘f565’: No such file or directory chmod: cannot access ‘f566’: No such file or directory chmod: cannot operate on dangling symlink ‘f568’ chmod: cannot operate on dangling symlink ‘f570’ chmod: cannot operate on dangling symlink ‘f574’ chmod: changing permissions of ‘f575’: No such file or directory chmod: changing permissions of ‘f576’: No such file or directory chmod: changing permissions of ‘f578’: No such file or directory chmod: changing permissions of ‘f579’: No such file or directory chmod: changing permissions of ‘f580’: No such file or directory chmod: changing permissions of ‘f581’: No such file or directory chmod: changing permissions of ‘f582’: No such file or directory chmod: cannot access ‘f583’: No such file or directory chmod: changing permissions of ‘f585’: No such file or directory chmod: changing permissions of ‘f586’: No such file or directory chmod: changing permissions of ‘f595’: No such file or directory chmod: changing permissions of ‘f596’: No such file or directory chmod: changing permissions of ‘f597’: No such file or directory chmod: changing permissions of ‘f598’: No such file or directory chmod: changing permissions of ‘f599’: No such file or directory chmod: changing permissions of ‘f600’: No such file or directory chmod: changing permissions of ‘f601’: No such file or directory chmod: changing permissions of ‘f602’: No such file or directory chmod: cannot access ‘f603’: No such file or directory chmod: cannot access ‘f604’: No such file or directory chmod: cannot operate on dangling symlink ‘f605’ chmod: changing permissions of ‘f611’: No such file or directory chmod: changing permissions of ‘f612’: No such file or directory chmod: changing permissions of ‘f613’: No such file or directory chmod: changing permissions of ‘f614’: No such file or directory chmod: cannot access ‘f615’: No such file or directory chmod: changing permissions of ‘f616’: No such file or directory chmod: changing permissions of ‘f617’: No such file or directory chmod: changing permissions of ‘f618’: No such file or directory chmod: changing permissions of ‘f619’: No such file or directory chmod: cannot access ‘f620’: No such file or directory chmod: cannot operate on dangling symlink ‘f621’ chmod: changing permissions of ‘f622’: No such file or directory chmod: cannot operate on dangling symlink ‘f623’ chmod: cannot operate on dangling symlink ‘f627’ chmod: changing permissions of ‘f628’: No such file or directory chmod: changing permissions of ‘f629’: No such file or directory chmod: cannot access ‘f630’: No such file or directory chmod: changing permissions of ‘f631’: No such file or directory chmod: changing permissions of ‘f632’: No such file or directory chmod: changing permissions of ‘f633’: No such file or directory chmod: changing permissions of ‘f634’: No such file or directory chmod: changing permissions of ‘f635’: No such file or directory chmod: cannot access ‘f636’: No such file or directory chmod: cannot operate on dangling symlink ‘f637’ chmod: cannot operate on dangling symlink ‘f638’ chmod: changing permissions of ‘f639’: No such file or directory chmod: cannot operate on dangling symlink ‘f640’ chmod: cannot operate on dangling symlink ‘f641’ chmod: cannot operate on dangling symlink ‘f642’ chmod: changing permissions of ‘f644’: No such file or directory chmod: changing permissions of ‘f647’: No such file or directory chmod: changing permissions of ‘f648’: No such file or directory chmod: changing permissions of ‘f649’: No such file or directory chmod: cannot access ‘f650’: No such file or directory chmod: cannot operate on dangling symlink ‘f651’ chmod: changing permissions of ‘f652’: No such file or directory chmod: cannot access ‘f653’: No such file or directory chmod: cannot access ‘f654’: No such file or directory chmod: changing permissions of ‘f655’: No such file or directory chmod: cannot operate on dangling symlink ‘f656’ chmod: changing permissions of ‘f662’: No such file or directory chmod: changing permissions of ‘f663’: No such file or directory chmod: changing permissions of ‘f664’: No such file or directory chmod: cannot operate on dangling symlink ‘f665’ chmod: changing permissions of ‘f666’: No such file or directory chmod: cannot operate on dangling symlink ‘f735’ chmod: cannot operate on dangling symlink ‘f746’ chmod: cannot operate on dangling symlink ‘f747’ chmod: cannot operate on dangling symlink ‘f759’ chmod: cannot operate on dangling symlink ‘f763’ chmod: cannot operate on dangling symlink ‘f797’ chmod: cannot operate on dangling symlink ‘f800’ chmod: cannot operate on dangling symlink ‘f810’ chmod: cannot operate on dangling symlink ‘f812’ chmod: cannot operate on dangling symlink ‘f814’ chmod: cannot operate on dangling symlink ‘f830’ chmod: cannot operate on dangling symlink ‘f834’ chmod: cannot operate on dangling symlink ‘f882’ chmod: cannot operate on dangling symlink ‘f884’ chmod: cannot operate on dangling symlink ‘f887’ chmod: cannot operate on dangling symlink ‘f899’ chmod: cannot operate on dangling symlink ‘f901’ chmod: cannot operate on dangling symlink ‘f902’ chmod: cannot operate on dangling symlink ‘f903’ chmod: cannot operate on dangling symlink ‘f905’ chmod: cannot operate on dangling symlink ‘f908’ chmod: cannot operate on dangling symlink ‘f921’ chmod: cannot operate on dangling symlink ‘f942’ chmod: cannot operate on dangling symlink ‘f958’ chmod: cannot operate on dangling symlink ‘f961’ chmod: cannot operate on dangling symlink ‘f978’ chmod: cannot operate on dangling symlink ‘f980’ chmod: cannot operate on dangling symlink ‘f981’ chmod: cannot operate on dangling symlink ‘f984’ chmod: cannot operate on dangling symlink ‘f986’ chmod: cannot operate on dangling symlink ‘f1001’ chmod: cannot operate on dangling symlink ‘f1007’ chmod: cannot operate on dangling symlink ‘f1024’ chmod: cannot access ‘f1026’: No such file or directory chmod: cannot operate on dangling symlink ‘f1029’ chmod: cannot operate on dangling symlink ‘f1037’ chmod: cannot operate on dangling symlink ‘f1046’ chmod: cannot operate on dangling symlink ‘f1048’ chmod: cannot operate on dangling symlink ‘f1052’ chmod: cannot operate on dangling symlink ‘f1053’ chmod: cannot access ‘f1085’: No such file or directory chmod: cannot operate on dangling symlink ‘f1087’ chmod: cannot operate on dangling symlink ‘f1088’ chmod: cannot operate on dangling symlink ‘f1096’ chmod: cannot operate on dangling symlink ‘f1103’ chmod: cannot operate on dangling symlink ‘f1109’ chmod: cannot operate on dangling symlink ‘f1116’ chmod: cannot operate on dangling symlink ‘f1117’ chmod: cannot operate on dangling symlink ‘f1127’ chmod: cannot operate on dangling symlink ‘f1128’ chmod: cannot operate on dangling symlink ‘f1129’ chmod: cannot operate on dangling symlink ‘f1130’ chmod: cannot operate on dangling symlink ‘f1143’ chmod: cannot operate on dangling symlink ‘f1154’ chmod: cannot operate on dangling symlink ‘f1158’ chmod: cannot operate on dangling symlink ‘f1160’ chmod: cannot operate on dangling symlink ‘f1161’ chmod: cannot operate on dangling symlink ‘f1169’ chmod: cannot operate on dangling symlink ‘f1170’ chmod: cannot operate on dangling symlink ‘f1171’ chmod: cannot operate on dangling symlink ‘f1180’ chmod: cannot operate on dangling symlink ‘f1184’ chmod: cannot operate on dangling symlink ‘f1185’ chmod: cannot operate on dangling symlink ‘f1196’ chmod: cannot operate on dangling symlink ‘f1197’ chmod: cannot operate on dangling symlink ‘f1199’ chmod: cannot operate on dangling symlink ‘f1211’ chmod: cannot operate on dangling symlink ‘f1213’ chmod: cannot operate on dangling symlink ‘f1259’ chmod: cannot operate on dangling symlink ‘f1262’ chmod: cannot operate on dangling symlink ‘f1269’ chmod: cannot operate on dangling symlink ‘f1278’ chmod: cannot operate on dangling symlink ‘f1279’ chmod: cannot operate on dangling symlink ‘f1289’ chmod: cannot operate on dangling symlink ‘f1299’ chmod: cannot operate on dangling symlink ‘f1301’ chmod: cannot operate on dangling symlink ‘f1302’ chmod: cannot operate on dangling symlink ‘f1307’ chmod: cannot operate on dangling symlink ‘f1309’ chmod: cannot operate on dangling symlink ‘f1313’ chmod: cannot operate on dangling symlink ‘f1315’ chmod: cannot access ‘f1316’: No such file or directory chmod: cannot operate on dangling symlink ‘f1317’ chmod: cannot operate on dangling symlink ‘f1325’ chmod: cannot operate on dangling symlink ‘f1328’ chmod: cannot operate on dangling symlink ‘f1332’ chmod: cannot operate on dangling symlink ‘f1340’ chmod: cannot operate on dangling symlink ‘f1363’ chmod: cannot operate on dangling symlink ‘f1365’ chmod: cannot operate on dangling symlink ‘f1405’ chmod: cannot operate on dangling symlink ‘f1409’ chmod: cannot operate on dangling symlink ‘f1440’ chmod: cannot operate on dangling symlink ‘f1446’ chmod: cannot operate on dangling symlink ‘f1452’ chmod: cannot access ‘f1453’: No such file or directory chmod: cannot operate on dangling symlink ‘f1473’ chmod: cannot operate on dangling symlink ‘f1482’ chmod: cannot operate on dangling symlink ‘f1484’ chmod: cannot operate on dangling symlink ‘f1492’ chmod: cannot operate on dangling symlink ‘f1499’ chmod: cannot operate on dangling symlink ‘f1505’ chmod: cannot operate on dangling symlink ‘f1543’ chmod: cannot operate on dangling symlink ‘f1544’ chmod: cannot operate on dangling symlink ‘f1589’ chmod: cannot operate on dangling symlink ‘f1591’ chmod: cannot operate on dangling symlink ‘f1600’ chmod: cannot operate on dangling symlink ‘f1607’ chmod: cannot operate on dangling symlink ‘f1620’ chmod: cannot operate on dangling symlink ‘f1632’ chmod: cannot operate on dangling symlink ‘f1638’ chmod: cannot operate on dangling symlink ‘f1640’ chmod: cannot operate on dangling symlink ‘f1641’ chmod: cannot operate on dangling symlink ‘f1663’ chmod: cannot operate on dangling symlink ‘f1665’ chmod: cannot operate on dangling symlink ‘f1685’ chmod: cannot operate on dangling symlink ‘f1691’ chmod: cannot operate on dangling symlink ‘f1694’ chmod: cannot operate on dangling symlink ‘f1703’ chmod: cannot operate on dangling symlink ‘f1728’ chmod: cannot operate on dangling symlink ‘f1773’ chmod: cannot operate on dangling symlink ‘f1775’ chmod: cannot access ‘f1776’: No such file or directory chmod: cannot operate on dangling symlink ‘f18026’ chmod: cannot operate on dangling symlink ‘f18033’ chmod: cannot operate on dangling symlink ‘f18050’ chmod: cannot operate on dangling symlink ‘f18058’ chmod: cannot operate on dangling symlink ‘f18059’ chmod: cannot access ‘f18060’: No such file or directory chmod: cannot operate on dangling symlink ‘f18135’ chmod: cannot operate on dangling symlink ‘f18136’ chmod: cannot operate on dangling symlink ‘f18137’ chmod: cannot operate on dangling symlink ‘f18138’ chmod: cannot operate on dangling symlink ‘f18140’ chmod: cannot access ‘f18175’: No such file or directory chmod: cannot operate on dangling symlink ‘f18177’ chmod: cannot operate on dangling symlink ‘f18235’ chmod: cannot operate on dangling symlink ‘f18325’ chmod: cannot operate on dangling symlink ‘f18364’ chmod: cannot operate on dangling symlink ‘f18371’ chmod: cannot operate on dangling symlink ‘f18417’ chmod: cannot operate on dangling symlink ‘f18590’ chmod: cannot operate on dangling symlink ‘f18598’ chmod: cannot operate on dangling symlink ‘f18606’ chmod: cannot operate on dangling symlink ‘f18607’ chmod: cannot operate on dangling symlink ‘f18608’ chmod: cannot operate on dangling symlink ‘f18609’ chmod: cannot operate on dangling symlink ‘f18610’ chmod: cannot access ‘f19092’: No such file or directory chmod: cannot operate on dangling symlink ‘f19093’ chmod: cannot operate on dangling symlink ‘f19094’ chmod: cannot operate on dangling symlink ‘f19586’ chmod: cannot access ‘f19587’: No such file or directory chmod: cannot operate on dangling symlink ‘f19589’ chmod: cannot operate on dangling symlink ‘f19591’ Actual results: For many files, chmod command failed to change permission. Expected results: Permission for a file should be changed successfully.
In Description -> Steps to reproduce -> 4-> c, command that was used for changing the file permission is for i in {1..20000};do chmod 660 f$i;done This issue is reproduced with glusterfs version 3.8.4-2.el7.x86_64 on an another setup which does not have md-cache settings enabled on the volume. So, this issue is not related to md-cache.
(In reply to Prasad Desala from comment #5) > In Description -> Steps to reproduce -> 4-> c, command that was used for > changing the file permission is > for i in {1..20000};do chmod 660 f$i;done > > This issue is reproduced with glusterfs version 3.8.4-2.el7.x86_64 on an > another setup which does not have md-cache settings enabled on the volume. > So, this issue is not related to md-cache. Can you provide access to this setup or load the sosreports?
Ignore comment#6 - I was able to reproduce this issue. It seems unrelated to add-brick/rebalance operations. At first glance it looks like a race between the create and the chmod but this needs further investigation. I was able to reproduce this on a single brick pure distribute volume with 2 Fuse clients. client 1: for i in {1..20000};do touch f$i;done client 2: for i in {1..20000};do chmod 660 f$i;done The same behaviour is not reproducible with 2 NFS mounts
IMO this can be worked on post 3.2.0 as this is not a common test case. Users will make sure files are created before attempting to change file permissions.
This is also seen with 3.1.3 so not a regression.
Csaba and I think we had figured this out. I was able to reproduce this with the steps listed in comment#7. The "cannot operate on dangling symlink" is returned by chmod: From coreutils -> chmod.c: static bool process_file (FTS *fts, FTSENT *ent) { ... case FTS_SLNONE: if (! force_silent) error (0, 0, _("cannot operate on dangling symlink %s"), quote (file_full_name)); ok = false; break; So fts_read returned FTS_SLNONE. Csaba ran an strace and figured out that chmod first tries a stat. If that fails it tries an lstat. If lstat succeeds, it assumes the entry is a dangling symlink. Adding debug messages to the chmod.c file, the info is different for those files for which chmod returns a "No such file or directory" error vs "cannot operate on dangling symlink". test f489: info = 10, stat ino = 0 test f489: info = 10, stat ino = 0 chmod: cannot access `f489': No such file or directory test f506: info = 10, stat ino = 0 test f506: info = 13, stat ino = 54287172 chmod: cannot operate on dangling symlink `f506' So the theory is that when the stat is performed, the file does not exist but has been created by the time the lstat is performed. Because of this, the chmod treats it like a dangling symlink and reports the error. The chmod behaviour is not something we can control and this is not a valid test case anyway. I am therefor closing this BZ with resolution NotABug.
Additional information: In the debug messages in comment#13, info refers to ent->fts_info. From fts.h: #define FTS_F 8 /* regular file */ ... #define FTS_NS 10 /* stat(2) failed */ ... #define FTS_SLNONE 13 /* symbolic link without target */