Description of problem: I was trying to create a directory, and it failed to create one with that same name quite a few times, may be there are other names also of similar kind which DHT considers not to be entertained. Volume_type is 6x2, volume status, [root@nfs2 ~]# gluster volume status Status of volume: dist-rep Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.37.62:/bricks/d1r1 49152 Y 9605 Brick 10.70.37.44:/bricks/d1r2 49155 Y 23293 Brick 10.70.37.201:/bricks/d2r1 49155 Y 17945 Brick 10.70.37.215:/bricks/d2r2 49152 Y 23031 Brick 10.70.37.62:/bricks/d3r1 49153 Y 9616 Brick 10.70.37.44:/bricks/d3r2 49156 Y 23300 Brick 10.70.37.201:/bricks/d4r1 49156 Y 17952 Brick 10.70.37.215:/bricks/d4r2 49153 Y 23042 Brick 10.70.37.62:/bricks/d5r1 49154 Y 9627 Brick 10.70.37.44:/bricks/d5r2 49157 Y 23305 Brick 10.70.37.201:/bricks/d6r1 49157 Y 17957 Brick 10.70.37.215:/bricks/d6r2 49154 Y 23053 NFS Server on localhost 2049 Y 23065 NFS Server on 10.70.37.44 2049 Y 23319 NFS Server on 10.70.37.62 2049 Y 9639 NFS Server on 10.70.37.201 2049 Y 17964 Task Status of Volume dist-rep ------------------------------------------------------------------------------ There are no active volume tasks Version-Release number of selected component (if applicable): glusterfs-3.6.0.1-1.el6rhs.x86_64 How reproducible: presently seen for one "string" used as name for creating directory Steps to Reproduce: 1. mkdir "run5959" on nfs mount type. Actual results: [root@rhsauto054 nfs-test]# mkdir run5959 mkdir: cannot create directory `run5959': Invalid argument result of logs from nfs.log, ============================= [2014-05-15 05:41:48.140416] W [dht-layout.c:178:dht_layout_search] 0-dist-rep-dht: no subvolume for hash (value) = 2672484811 [2014-05-15 05:41:48.144727] W [dht-layout.c:178:dht_layout_search] 0-dist-rep-dht: no subvolume for hash (value) = 2672484811 [2014-05-15 05:41:48.144838] W [nfs3.c:2722:nfs3svc_mkdir_cbk] 0-nfs: 6b260899: /run5959 => -1 (Invalid argument) [2014-05-15 05:41:48.144911] W [nfs3-helpers.c:3470:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: 6b260899, MKDIR: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 [2014-05-15 05:47:58.629002] W [dht-layout.c:178:dht_layout_search] 0-dist-rep-dht: no subvolume for hash (value) = 2672484811 [2014-05-15 05:47:58.633094] W [dht-layout.c:178:dht_layout_search] 0-dist-rep-dht: no subvolume for hash (value) = 2672484811 [2014-05-15 05:47:58.633291] W [nfs3.c:2722:nfs3svc_mkdir_cbk] 0-nfs: 6d260899: /run5959 => -1 (Invalid argument) [2014-05-15 05:47:58.633471] W [nfs3-helpers.c:3470:nfs3_log_newfh_res] 0-nfs-nfsv3: XID: 6d260899, MKDIR: NFS: 22(Invalid argument for operation), POSIX: 22(Invalid argument), FH: exportid 00000000-0000-0000-0000-000000000000, gfid 00000000-0000-0000-0000-000000000000 Expected results: directory is a basic operation for a filesystem, it should be pass like the one below on the same mount-point, [root@rhsauto054 nfs-test]# mkdir run8777 [root@rhsauto054 nfs-test]# ls newdir run8777 Additional info:
Again, seen for this below directory [root@rhsauto054 ~]# ls /mnt/nfs-test [root@rhsauto054 ~]# mkdir /mnt/nfs-test//run6582/ mkdir: cannot create directory `/mnt/nfs-test//run6582/': Invalid argument
Problem seen in this BZ is described bit more here, if you check the following lines of execution, then [root@rhsauto054 ~]# mkdir /mnt/nfs-test//run7777 mkdir: cannot create directory `/mnt/nfs-test//run7777': Invalid argument [root@rhsauto054 ~]# mkdir /mnt/nfs-test/run7777 mkdir: cannot create directory `/mnt/nfs-test/run7777': Invalid argument [root@rhsauto054 ~]# cd /mnt/nfs-test [root@rhsauto054 nfs-test]# mkdir run7777 mkdir: cannot create directory `run7777': Invalid argument if the mkdir is executed using "//" , bascially two slashes it fails ---> /mnt/nfs-test//run7777 ---- fails and even if you cd in the mount -point /mnt/nfs-test and try to create run7777, this attempt also fails. So, altogether the behaviour is wrong, needs correction.
some more observation, presently highly confusing, [root@rhsauto054 ~]# mkdir /mnt/nfs-test//lllll mkdir: cannot create directory `/mnt/nfs-test//lllll': Invalid argument [root@rhsauto054 ~]# mkdir /mnt/nfs-test/llll mkdir: cannot create directory `/mnt/nfs-test/llll': Invalid argument [root@rhsauto054 ~]# mkdir /mnt/nfs-test/ll mkdir: cannot create directory `/mnt/nfs-test/ll': Invalid argument [root@rhsauto054 ~]# mkdir /mnt/nfs-test/rrr [root@rhsauto054 ~]# mkdir /mnt/nfs-test//rrrr [root@rhsauto054 ~]# mkdir /mnt/nfs-test//nnnnn [root@rhsauto054 ~]# mkdir /mnt/nfs-test//mmmmmmm mkdir: cannot create directory `/mnt/nfs-test//mmmmmmm': Invalid argument [root@rhsauto054 ~]# mkdir /mnt/nfs-test/mmmm
Hi Saurabh, Tested the same on master branch upstream. Was not able reproduce the issue. From the log it seems layout had anomaly. [2014-05-15 05:41:48.140416] W [dht-layout.c:178:dht_layout_search] 0-dist-rep-dht: no subvolume for hash (value) = 2672484811 ---> Check Here [2014-05-15 05:41:48.144727] W [dht-layout.c:178:dht_layout_search] 0-dist-rep-dht: no subvolume for hash (value) = 2672484811 [2014-05-15 05:41:48.144838] W [nfs3.c:2722:nfs3svc_mkdir_cbk] 0-nfs: 6b260899: /run5959 => -1 (Invalid argument) Would you be able to provide sos reports so that I can find what might have caused anomaly in layout. * And dht does not blocks file creation with specific names.
Created attachment 898315 [details] nfs.log
Seems similar to bz 1110457. EINVAL is currently seen when a subvolume of dht is offline.
This needs to be tested against the latest build.
There is a WIP patch on upstream at: http://review.gluster.org/#/c/12414/ and this bug is most likely a duplicate of bz 1286208
https://code.engineering.redhat.com/gerrit/74435 should've fixed this. Hence marking this bug to be verified by QE.
Please suggest the steps/scenarios to verify the fix.
Tested this BZ with glusterfs version: 3.8.4-3.el7rhgs.x86_64. Raghavendra G has suggested the below scenario to verify the fix. 1) Create a distributed replicated volume and start it. 2) Enable nfs on the volume and NFS mount the volume on a client. 3) From mount point, create a directory. 4) Induce holes on the layout by setting the trusted.glusterfs.dht attribute to empty. 5) Bring down few bricks which are having correct layout. 6) Bring up the killed bricks. 7) Perform lookup on the directory and the directory layout should get healed without any holes or overlaps. No issues were seen. Hence, moving this BZ to Verified.