Description of problem: `ls` of the the export's root does not list the export's contents. Version-Release number of selected component (if applicable): NFS-Ganesha V2.5-dev-7 Commit 4ee0fb12872 Ceph and libcephfs 10.2.5-5648-g5b402f8 Commit 5b402f8a7b5a76 How reproducible: Always Steps to Reproduce: 1. Setup Ganesha with CephFS with a config file like, EXPORT { Export_ID=100; Path = /; Pseudo = /; Protocols = 3,4; Transports = TCP; FSAL { Name = CEPH; } CLIENT { Clients = 10.0.2.15; Access_Type = RW; } } 2. Start the ganesha service 3. Mount the root of the export, here '/' of the CephFS. $ sudo mount.nfs4 -o rw,noatime 10.0.2.15:/ /mnt/nfs4/ 4. Listing the mount root's contents $ ls /mnt/nfs4/ yields no result, but listing contents of a directory within the mount root does. $ ls /mnt/nfs4/dir0 file0 Faced the same issue even when a CephFS subdir was exported, i.e. Export's PATH and PSEUDO were `/subdir`. Actual results: `ls' of mount root does not display contents of the export. Below are the relevant parts of strace and ganesha log. Strace snippet open(".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFDIR|0755, st_size=104858184, ...}) = 0 getdents(3, 0x18f7cd0, 32768) = -1 ENOENT (No such file or directory) Ganesha log snippet mdcache_dirent_populate :NFS READDIR :DEBUG :FSAL readdir status=No such file or directory nfs4_Compound :NFS4 :DEBUG :Status of OP_READDIR in position 1 = NFS4ERR_NOENT nfs4_Compound :NFS4 :DEBUG :End status = NFS4ERR_NOENT lastindex = 1 Expected results: Should be able to list the contents of the export's root. Additional info:
Hmm, works fine for me with more recent (kraken) libcephfs client libs. I'll look at downgrading when I get a chance and see if it's reproducible.
Ok, I can reproduce when I downgrade the client-side libs to: libcephfs1-10.2.5-5648.g5b402f8a7b5a.el7.centos.x86_64
I think I might see what's going on: 2017-01-02 07:18:53.057461 7ff8a5ff3700 3 client.4115 ll_opendir 1.head = 0 (0x7ff800001130) 2017-01-02 07:18:53.057467 7ff8a5ff3700 3 client.4115 seekdir(0x7ff800001130, 0) 2017-01-02 07:18:53.057474 7ff8a5ff3700 10 client.4115 readdir_r_cb 1.head(faked_ino=0 ref=3 ll_ref=2 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-01-02 07:00:39.470046 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x7ff8cc0062a0) offset 0 at_end=0 hash_order=0 2017-01-02 07:18:53.057490 7ff8a5ff3700 15 client.4115 including . 2017-01-02 07:18:53.057492 7ff8a5ff3700 10 client.4115 fill_stat on 1 snap/devhead mode 040755 mtime 2017-01-02 07:00:39.470046 ctime 2017-01-02 07:00:39.470046 2017-01-02 07:18:53.057502 7ff8a5ff3700 10 client.4115 fill_dirent '.' -> 1 type 4 w/ next_off 1 2017-01-02 07:18:53.057506 7ff8a5ff3700 3 client.4115 ll_lookup 0x7ff8cc0062a0 . 2017-01-02 07:18:53.057509 7ff8a5ff3700 15 inode.get on 0x7ff8cc0062a0 1.head now 4 2017-01-02 07:18:53.057511 7ff8a5ff3700 10 client.4115 _lookup 1.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-01-02 07:00:39.470046 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x7ff8cc0062a0) . = 1.head(faked_ino=0 ref=4 ll_ref=2 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-01-02 07:00:39.470046 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x7ff8cc0062a0) 2017-01-02 07:18:53.057535 7ff8a5ff3700 10 client.4115 fill_stat on 1 snap/devhead mode 040755 mtime 2017-01-02 07:00:39.470046 ctime 2017-01-02 07:00:39.470046 2017-01-02 07:18:53.057544 7ff8a5ff3700 20 client.4115 _ll_get 0x7ff8cc0062a0 1 -> 3 2017-01-02 07:18:53.057547 7ff8a5ff3700 3 client.4115 ll_lookup 0x7ff8cc0062a0 . -> 0 (1) 2017-01-02 07:18:53.057549 7ff8a5ff3700 10 client.4115 put_inode on 1.head(faked_ino=0 ref=4 ll_ref=3 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-01-02 07:00:39.470046 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x7ff8cc0062a0) 2017-01-02 07:18:53.057558 7ff8a5ff3700 15 inode.put on 0x7ff8cc0062a0 1.head now 3 2017-01-02 07:18:53.057562 7ff8a5ff3700 10 client.4115 readdir_r_cb 1.head(faked_ino=0 ref=3 ll_ref=3 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-01-02 07:00:39.470046 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x7ff8cc0062a0) offset 1 at_end=0 hash_order=0 2017-01-02 07:18:53.057572 7ff8a5ff3700 15 client.4115 including .. 2017-01-02 07:18:53.057574 7ff8a5ff3700 10 client.4115 fill_dirent '..' -> 3 type 4 w/ next_off 2 2017-01-02 07:18:53.057577 7ff8a5ff3700 3 client.4115 ll_lookup 0x7ff8cc0062a0 .. 2017-01-02 07:18:53.057579 7ff8a5ff3700 10 client.4115 _lookup 1.head(faked_ino=0 ref=3 ll_ref=3 cap_refs={} open={} mode=40755 size=0/0 mtime=2017-01-02 07:00:39.470046 caps=pAsLsXsFs(0=pAsLsXsFs) has_dir_layout 0x7ff8cc0062a0) .. = -2 2017-01-02 07:18:53.057588 7ff8a5ff3700 3 client.4115 ll_lookup 0x7ff8cc0062a0 .. -> -2 (0) 2017-01-02 07:18:53.057591 7ff8a5ff3700 3 client.4115 ll_releasedir 0x7ff800001130 2017-01-02 07:18:53.057593 7ff8a5ff3700 10 client.4115 _closedir(0x7ff800001130) 2017-01-02 07:18:53.057595 7ff8a5ff3700 10 client.4115 _closedir detaching inode 0x7ff8cc0062a0 We end up issuing a lookup of ".." at the root which is still broken in jewel: if (dname == "..") { if (dir->dn_set.empty()) r = -ENOENT; else *target = dir->get_first_parent()->dir->parent_inode; //dirs can't be hard-linked goto done; } We should probably just take commit 30d4ca01db0de9a1e12658793ba9bf9faf0331dd into jewel, which would likely fix this. We could also try to work around this in ganesha, but that means special casing to handle these at the root of the mount.
Yes, that commit fixes it. Opened http://tracker.ceph.com/issues/18408 for the backport.
(In reply to Jeff Layton from comment #4) > Yes, that commit fixes it. Opened http://tracker.ceph.com/issues/18408 for > the backport. The referred Ceph tracker was resolved. The commit was backported to Ceph Jewel branch, on which the bug was filed. https://github.com/ceph/ceph/commit/7819adb3b5f9af813d4df05d3483175ee54e10df I tested using latest Jewel branch, version 10.2.5-6122-g420a9a0 and NFS-Ganesha V2.5-dev-16, and couldn't hit this bug. The issue is resolved. Can we close this BZ?