Assigning to Ali.
Ali, you implemented this, so please have a look at fixing it.

Thanks in advance.

Vasistha, can you send me a copy of the entire ganesha.conf file on the nfs node? Also did you change any of the nfs related variables in the ceph-ansible repo used to generate the cluster?

-Ali

Hi Ali

Copy of ganesha conf file -

$ cat /etc/ganesha/ganesha.conf 
# Please do not change this file directly since it is managed by Ansible and will be overwritten

EXPORT
{
	Export_id=20134;

	Path = "/";

	Pseudo = /ceph;

	Access_Type = RW;

	Protocols = 3,4;

	Transports = TCP;

	SecType = sys,krb5,krb5i,krb5p;

	FSAL {
		Name = RGW;
		User_Id = "cephnfs";
		Access_Key_Id ="N3L2FC7DUHWP59W5UBSX";
		Secret_Access_Key = "PP8zPyNFNhJr47bksSZQtSX56mOpcXU9YNuCZ8uc";
	}

        

}

RGW {
        ceph_conf = "/etc/ceph/2017_sub.conf";
        cluster = "2017_sub";
        name = "client.rgw.magna020";
        
}

LOG {
        Facility {
                name = FILE;
                destination = "/var/log/ganesha/ganesha.log";
                enable = active;
        }

        
}


I didn't change anything in group_vars/nfss.yml.

Regards,
Vasishta

Ok, I think this still needs some wordsmithing, but here's an attempt to meet the requirements we need:

Export Path and Pseudo Configuration and Use
 
EXPORT and mount examples with mount_path_pseudo = false
 
Path            Pseudo          Tag     Mechanism   Mount
/export/test1   /export/test1   test1   v3 Path     mount -o vers=3 server:/export/test1
/export/test1   /export/test1   test1   v3 Tag      mount -o vers=3 server:test1
/export/test1   /export/test1   test1   v4 Pseudo   mount -o vers=4 server:/export/test1
/               /export/ceph1   ceph1   v3 Path     mount -o vers=3 server:/
/               /export/ceph1   ceph1   v3 Tag      mount -o vers=3 server:ceph1
/               /export/ceph1   ceph1   v4 Pseudo   mount -o vers=4 server:/export/ceph1
/               /export/ceph2   ceph2   v3 Path     not accessible
/               /export/ceph2   ceph2   v3 Tag      mount -o vers=3 server:ceph2
/               /export/ceph2   ceph2   v4 Pseudo   mount -o vers=4 server:/export/ceph2
 
EXPORT and mount examples with mount_path_pseudo = true
 
Path            Pseudo          Tag     Mechanism   Mount
/export/test1   /export/test1   test1   v3 Pseudo   mount -o vers=3 server:/export/test1
/export/test1   /export/test1   test1   v3 Tag      mount -o vers=3 server:test1
/export/test1   /export/test1   test1   v4 Pseudo   mount -o vers=4 server:/export/test1
/               /export/ceph1   ceph1   v3 Pseudo   mount -o vers=3 server:/export/ceph1
/               /export/ceph1   ceph1   v3 Tag      mount -o vers=3 server:ceph1
/               /export/ceph1   ceph1   v4 Pseudo   mount -o vers=4 server:/export/ceph1
/               /export/ceph2   ceph2   v3 Pseudo   mount -o vers=3 server:/export/ceph2
/               /export/ceph2   ceph2   v3 Tag      mount -o vers=3 server:ceph2
/               /export/ceph2   ceph2   v4 Pseudo   mount -o vers=4 server:/export/ceph2
 
Ganesha exports are configured somewhat more flexibly than
other NFS servers. This document is intended to help understand
the namespace aspects of Ganesha export configuration.
 
First, it is highly recommended that the following option be set in
the NFS_CORE_PARAM block:
 
    mount_path_pseudo = true;
 
This option will make NFS v3 and NFS v4.x mounts both use the same
server side path to reach an export, for example:
 
    mount -o vers=3 192.168.0.1:/export /mnt
    mount -o vers=4 192.168.0.1:/export /mnt
 
Without this option, NFS v3 mounts use the Path option and NFS v4.x
mounts use the Pseudo option.
 
The Path option instructs Ganesha where to find the export. For
the VFS FSAL, this is the location within the server's namespace.
For other FSALs, it may be the location within the filesystem
managed by that FSAL's namespace. For example, if the CEPH FSAL
is used to export an entire CephFS volume, Path would be /.
 
The Pseudo option instructs Ganesha where to place the export within
NFS v4's pseudo file system namespace. NFS v4 specifies the server may
construct a pseudo namespace that may not correspond to any actual
locations of exports, and portions of that pseudo filesystem may
exist only within the realm of the NFS server and not correspond
to any physical directories. Further, an NFS v4 server places all
its exports within a single namespace. It is possible to have a
single export exported as the pseudo filesystem root, but it is much
more common to have multiple exports placed in the pseudo filesystem.
With a traditional VFS, often the Pseudo location is the same as the
Path location, but again, returning to the example CephFS export
with "/" as the Path, if multiple exports are desired, the export
would likely have something else (for example /ceph) as the Pseudo
option.
 
An NFS v4 client can mount the root of the pseudo file system (and
in fact, under the covers this is always what happens, NFS v4 has
no special MOUNT protocol, instead, there is an operation
PUTROOTFH which instructs the server to navigate to the root of
the pseudo filesystem, from whence LOOKUP operations are used to
navigate to the directory of interest. Thus the NFS v4 mount
command:
 
    mount -o vers=4 192.168.0.1:/export /mnt
 
results in a PUTROOTFH followed by LOOKUP export, while
 
    mount -o vers=4 192.168.0.1:/ceph /mnt
 
would result in PUTROOTFH followed by LOOKUP ceph
 
Without the
 
    mount_path_pseudo = true;
 
option, the NFS v3 mount command to access the CephFS export would be:
 
    mount -o vers=3 192.168.0.1:/ /mnt
 
Note also that if several CephFS volumes were exported, all the
exports might have the same Path (/). Without the mount_path_pseudo
option, only the first would be accessible to NFS v3 mounts (unless
the obscure Tag export option was used).

See https://gitlab.cee.redhat.com/red-hat-ceph-storage-documentation/doc-Red_Hat_Ceph_Storage_3-Object_Gateway_Guide/commit/ba11861520366498c859d32636a35a2e046bbf5d

and

https://access.qa.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html-single/object_gateway_guide_for_red_hat_enterprise_linux/#exporting_the_namespace_to_nfs_ganesha

Document looks fine to me. Moving it to verified.