Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 1567511 - [manila] Disable NFS-Ganesha caching when using cephfs-nfs driver
[manila] Disable NFS-Ganesha caching when using cephfs-nfs driver
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates (Show other bugs)
13.0 (Queens)
Unspecified Unspecified
medium Severity medium
: z2
: 13.0 (Queens)
Assigned To: Ram Raja
Dustin Schoenbrun
mmurray
: Triaged, ZStream
Depends On: 1570878
Blocks:
  Show dependency treegraph
 
Reported: 2018-04-14 11:30 EDT by Ram Raja
Modified: 2018-09-05 16:21 EDT (History)
10 users (show)

See Also:
Fixed In Version: openstack-tripleo-heat-templates-8.0.3-2.el7ost
Doc Type: Bug Fix
Doc Text:
In CephFS-NFS driver deployments, the NFS-Ganesha server, backed by CephFS, performs dentry, inode, and attribute caching that is also performed by the libcephfs clients. The NFS-Ganesha server's redundant caching led to a large memory footprint. It also affected cache coherency. Turn off NFS-Ganesha server's inode, dentry, and attribute caching. This reduces the memory footprint of the NFS-Ganesha server. Cache coherency issues are less probable.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-08-29 12:35:26 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Launchpad 1769431 None None None 2018-05-09 06:14 EDT
OpenStack gerrit 561575 None master: MERGED tripleo-heat-templates: [DNM] ceph-nfs: disable ganesha caching (Ibbd2c80931014cffa4580111070a97ea6df3a430) 2018-07-19 10:54 EDT
OpenStack gerrit 566458 None stable/queens: MERGED tripleo-heat-templates: ceph-nfs: disable ganesha caching (Ibbd2c80931014cffa4580111070a97ea6df3a430) 2018-07-19 10:54 EDT
Ceph Project Bug Tracker 23393 None None None 2018-04-14 11:30 EDT
Red Hat Product Errata RHBA-2018:2574 None None None 2018-08-29 12:36 EDT

  None (edit)
Description Ram Raja 2018-04-14 11:30:50 EDT
Description of problem:

When setting up CephFS-NFS Manila driver, we want to tune NFS-Ganesha such that it doesn't cache the same information that libcephfs client caches. This is important to reduce the memory footprint of NFS-Ganesha, which runs along with other services such as Ceph MONs, MDSs, and OpenStack services in the controller node.  Minor changes are needed (at least) in ceph-ansible and OSP-director to set up Ganesha config such as dentry and attribute caching done by Ganesha is turned off as much as possible.
Comment 15 Ram Raja 2018-07-31 10:52:27 EDT
For verification,

Check whether the following setting are included in the ganesha.conf file used by the ganesha server,

# To disable caching
CACHEINODE {
	Dir_Max = 1;
	Dir_Chunk = 0;
	Cache_FDs = false;
	NParts = 1;
	Cache_Size = 1;
}


EXPORT_DEFAULTS {
	Attr_Expiration_Time = 0;
}


Also, check ganesha's log file for any warnings about the above config setting not being recognized by the ganesha server. There shouldn't be any. This is because the ganesha server doesn't error out if it can't recognize a setting, instead it logs a warning in the log file saying so.
e.g.,

[main] config_errs_to_log :CONFIG :WARN :Config File (/etc/ganesha/ganesha.conf:10): Unknown parameter (Attr_Expiration_Time)
Comment 16 Joanne O'Flynn 2018-08-15 03:56:34 EDT
This bug is marked for inclusion in the errata but does not currently contain draft documentation text. To ensure the timely release of this advisory please provide draft documentation text for this bug as soon as possible.

If you do not think this bug requires errata documentation, set the requires_doc_text flag to "-".


To add draft documentation text:

* Select the documentation type from the "Doc Type" drop down field.

* A template will be provided in the "Doc Text" field based on the "Doc Type" value selected. Enter draft text in the "Doc Text" field.
Comment 18 errata-xmlrpc 2018-08-29 12:35:26 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2574
Comment 19 Tom Barron 2018-09-05 16:21:51 EDT
With puddle 2018-08-22.2 and 8.0.4-20.el7ost the verification criteria from https://bugzilla.redhat.com/show_bug.cgi?id=1567511#c15 are met:

[root@controller-0 ~]# docker exec ceph-nfs-pacemaker grep -B 1 -A 12 'Attr_Expiration' /etc/ganesha/ganesha.conf
EXPORT_DEFAULTS {
	Attr_Expiration_Time = 0;
}

CACHEINODE {
	Dir_Max = 1;
	Dir_Chunk = 0;

	Cache_FDs = false;

	NParts = 1;
	Cache_Size = 1;
}

[root@controller-0 ~]# docker exec ceph-nfs-pacemaker grep WARN /var/log/ganesha/ganesha.log
05/09/2018 16:54:44 : epoch 5b900a54 : controller-0 : ganesha.nfsd-137[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!!
05/09/2018 16:54:44 : epoch 5b900a54 : controller-0 : ganesha.nfsd-137[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:0)
[root@controller-0 ~]#

Note You need to log in before you can comment on or make changes to this bug.