Bug 2118263

Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.

Ganesha Configuration:

# Ansible managed


NFS_Core_Param
{
       Bind_Addr=172.17.5.111;
}

EXPORT_DEFAULTS {
        Attr_Expiration_Time = 0;
}

CACHEINODE {
        Dir_Chunk = 0;

        NParts = 1;
        Cache_Size = 1;
}

RADOS_URLS {
   ceph_conf = '/etc/ceph/ceph.conf';
   userid = "manila";
}
%url rados://manila_data/ganesha-export-index

NFSv4 {
        RecoveryBackend = 'rados_kv';
        IdmapConf = "/etc/ganesha/idmap.conf";
}
RADOS_KV {
        ceph_conf = '/etc/ceph/ceph.conf';
        userid = "manila";
        pool = "manila_data";
}


LOG {
        Facility {
                name = FILE;
                destination = "/var/log/ganesha/ganesha.log";
                enable = active;
        }
}

The same scenario test passed in a manila-native-ceph environment.
Compose tested with RHOS-17.0-RHEL-9-20220811.

I have a few questions on https://bugzilla.redhat.com/show_bug.cgi?id=2118263#c0


Were there contents in the snapshot before a share was created from it? The output of the first `ll` within the share is empty. 


Are you able to see the snapshotted contents from within the share that was created from the snapshot?


When using a regularCephFS NFS share, do you observe the same issue of files just created not being listed sometimes by a NFS client?  It's not obvious to me why a share created from a snapshot (a new CephFS subvolume) would behave differently from a regular share (another CephFS subvolume).


Is this a regression? Was this test performed on earlier releases?

Created tracker ticket https://tracker.ceph.com/issues/57210 with reproducer steps on a Ceph dev vstart cluster . I can observe this issue in ceph main branch as well.

Goutham, did this test "test_write_data_to_share_created_from_snapshot"  ever pass in the upstream CI for the CephFS NFS driver? 

In my testing, I found that directly listing the newly created file in a cloned subvolume (share created from snapshot) works. See the description in https://tracker.ceph.com/issues/57210

To unblock your scenario testing upstream, you can disable the test  "test_write_data_to_share_created_from_snapshot" . And also copy over the test, modify it to check for the existence/absence of file in a subvolume clone by listing the file directly, instead of checking the contents of parent directory's listing. What do you think?

Once we root cause and fix the issue, we can re-enable the failed test.

(In reply to Ram Raja from comment #10)
> Goutham, did this test "test_write_data_to_share_created_from_snapshot" 
> ever pass in the upstream CI for the CephFS NFS driver? 
> 
> In my testing, I found that directly listing the newly created file in a
> cloned subvolume (share created from snapshot) works. See the description in
> https://tracker.ceph.com/issues/57210
> 
> To unblock your scenario testing upstream, you can disable the test 
> "test_write_data_to_share_created_from_snapshot" . And also copy over the
> test, modify it to check for the existence/absence of file in a subvolume
> clone by listing the file directly, instead of checking the contents of
> parent directory's listing. What do you think?
> 
> Once we root cause and fix the issue, we can re-enable the failed test.

Hey Ramana, 

Yes it did. We first enabled this test with CephFS-NFS and Ceph Nautilus with OpenStack Wallaby: https://review.opendev.org/c/openstack/manila-tempest-plugin/+/778188
We had this passing until very recent versions including with Ceph Octopus and Pacific. It feels like the breakage was recent. We can sure skip the test for now while we wait for this issue to be root-caused. Thanks Ramana!

The fix is merged in Ceph main branch.

For now, I've created a upstream pacific (16.2.x) backport PR, https://github.com/ceph/ceph/pull/48521 , for the pacific backport tracker ticket, https://tracker.ceph.com/issues/57880

Hemanth, I'm copying over the steps  from https://tracker.ceph.com/issues/57210 I used to reproduce this issue in a Ceph cluster without needing OpenStack manila . I used a vstart cluster, but the steps should be the same in a QE test cluster
```
$ ./bin/ceph fs volume create a
$ ./bin/ceph fs subvolume create a subvol01
$ ./bin/ceph fs subvolume getpath a subvol01

$ ./bin/ceph nfs cluster create nfs-ganesha
$ ./bin/ceph nfs export create cephfs nfs-ganesha /cephfs3 a `./bin/ceph fs subvolume getpath a subvol01`
$ sudo mount.nfs4 127.0.0.1:/cephfs3 /mnt/nfs1/
$ pushd /mnt/nfs1/
$ sudo touch file00
$ # can see newly created file when listing directory contents
$ ls
file00
$ popd

$ ./bin/ceph fs subvolume snapshot create a subvol01 snap01
$ ./bin/ceph fs subvolume snapshot clone a subvol01 snap01 clone01
$ ./bin/ceph nfs export create cephfs nfs-ganesha /cephfs4 a `./bin/ceph fs subvolume getpath a clone01`
$ sudo mount.nfs4 127.0.0.1:/cephfs4 /mnt/nfs2/
$ pushd /mnt/nfs2/
$ ls
file00
$ sudo touch file01
$ # can see cloned 'file00' but cannot see the newly created file 'file01' when reading the directory contents within the clone
$ ls
file00
```

With this fix, you should be able to see the newly created 'file01' too in the FS subvolume clone when listing using the NFS client.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 5.3 Bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:0980