Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2248812

Summary: Enabling debug logging in ganesha.conf file causing mount to fail on client.
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Manisha Saini <msaini>
Component: NFS-GaneshaAssignee: Frank Filz <ffilz>
Status: CLOSED WORKSFORME QA Contact: Manisha Saini <msaini>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.0CC: cephqe-warriors, ffilz, kkeithle, sostapov, tserlin
Target Milestone: ---   
Target Release: 7.0z2   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-03-05 19:54:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Manisha Saini 2023-11-09 06:36:54 UTC
Description of problem:

===============

Hitting this issue while enabling debug logs for BZ - https://bugzilla.redhat.com/show_bug.cgi?id=2239769#c15 


Adding Debug log block in ganesha.conf file causing mount to fail on client

LOG {
 COMPONENTS {
     ALL = FULL_DEBUG;
 }
}

Post applying the customised ganesha.conf file, I can see the nfs ganesha process running but mount is getting failed on client.


-----------
4c7a13 : cali013 : ganesha.nfsd-2[main] init_logging :LOG :NULL :LOG: Setting log level for all components to NIV_EVENT
4c7a13 : cali013 : ganesha.nfsd-2[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.6
-----------


[root@ceph-mani-fq46a9-node4 mnt]# mount -t nfs -o vers=4.1,port=2049 10.8.130.13:/ganesha1 /mnt/ganesha/
mount.nfs: Connection refused



Version-Release number of selected component (if applicable):
=======================

# rpm -qa | grep nfs
libnfsidmap-2.5.4-18.el9.x86_64
nfs-utils-2.5.4-18.el9.x86_64
nfs-ganesha-selinux-5.6-3.el9cp.noarch
nfs-ganesha-5.6-3.el9cp.x86_64
nfs-ganesha-rgw-5.6-3.el9cp.x86_64
nfs-ganesha-ceph-5.6-3.el9cp.x86_64
nfs-ganesha-rados-grace-5.6-3.el9cp.x86_64
nfs-ganesha-rados-urls-5.6-3.el9cp.x86_64

# ceph --version
ceph version 18.2.0-113.el9cp (32cbda69435c7145d09eeaf5b5016e5d46370a5d) reef (stable)


How reproducible:
============
2/2


Steps to Reproduce:
============
1. Create ganesha cluster
2. Create and mount the export on client
3. Edit the ganesha.conf file to enable Debug logging
4. Apply the customised ganesha.conf file inside the shell
5. Try to remount the volume on client

Actual results:
========
Mount failed but ganesha process by up and running 


Expected results:
======
Debug logging should be enable and mount should pass


Additional info:
=============

Detailed steps ----->




[ceph: root@cali013 /]# ceph nfs cluster info nfsganesha
{
  "nfsganesha": {
    "backend": [
      {
        "hostname": "cali013",
        "ip": "10.8.130.13",
        "port": 2049
      }
    ],
    "virtual_ip": null
  }
}


[ceph: root@cali013 /]# ceph nfs export create cephfs nfsganesha /ganesha1 cephfs --path=/
{
  "bind": "/ganesha1",
  "cluster": "nfsganesha",
  "fs": "cephfs",
  "mode": "RW",
  "path": "/"
}



[ceph: root@cali013 /]# ceph orch ls
NAME                       PORTS   RUNNING  REFRESHED  AGE  PLACEMENT                        
mds.cephfs                             3/3  10m ago    4w   label:mds                        
mgr                                    3/3  10m ago    2w   cali013;cali015;cali016;count:3  
mon                                    1/5  66s ago    4w   <unmanaged>                      
nfs.nfsganesha             ?:2049      1/1  66s ago    73s  cali013                          
osd.all-available-devices               35  10m ago    4w   *                                
rgw.rgw.1                  ?:80        2/2  10m ago    4w   label:rgw   

                     
[ceph: root@cali013 /]# ceph orch ps | grep nfs
nfs.nfsganesha.0.0.cali013.zjblln  cali013  *:2049       running (74s)    73s ago  74s    18.1M        -  5.6               d5840a5e85fb  88d70f7a2e37  


[root@ceph-mani-fq46a9-node4 mnt]# mount -t nfs -o vers=4.1,port=2049 10.8.130.13:/ganesha1 /mnt/ganesha/

[root@ceph-mani-fq46a9-node4 mnt]# cd /mnt/ganesha/

[root@ceph-mani-fq46a9-node4 ganesha]# ls
volumes


============
Edit the ganesha.conf to enable debug logs
============

[root@cali013 ~]# podman ps | grep nfs
88d70f7a2e37  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:c4f1cf9ac78d873050a567493eb3a4956c661a1fdb52cf58ba05c995936f5ba0  -F -L STDERR -N N...  4 minutes ago  Up 4 minutes              ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln


[root@cali013 ~]# sudo podman exec -it 88d70f7a2e37 bash


*****Default ganesha.conf file inside container***************

[root@cali013 /]# cat /etc/ganesha/ganesha.conf
# This file is generated by cephadm.
NFS_CORE_PARAM {
        Enable_NLM = false;
        Enable_RQUOTA = false;
        Protocols = 4;
        NFS_Port = 2049;
}

NFSv4 {
        Delegations = false;
        RecoveryBackend = 'rados_cluster';
        Minor_Versions = 1, 2;
}

RADOS_KV {
        UserId = "nfs.nfsganesha.0.0.cali013.zjblln";
        nodeid = "nfs.nfsganesha.0";
        pool = ".nfs";
        namespace = "nfsganesha";
}

RADOS_URLS {
        UserId = "nfs.nfsganesha.0.0.cali013.zjblln";
        watch_url = "rados://.nfs/nfsganesha/conf-nfs.nfsganesha";
}

RGW {
        cluster = "ceph";
        name = "client.nfs.nfsganesha.0.0.cali013.zjblln-rgw";
}

%url    rados://.nfs/nfsganesha/conf-nfs.nfsganesha


*************Add Debug log block in ganesha.conf file****************

[root@cali013 ~]# cephadm shell --mount ganesha.conf:/var/lib/ceph/ganesha.conf
Inferring fsid 4e687a60-638e-11ee-8772-b49691cee574
Inferring config /var/lib/ceph/4e687a60-638e-11ee-8772-b49691cee574/mon.cali013/config
Using ceph image with id 'd5840a5e85fb' and tag 'ceph-7.0-rhel-9-containers-candidate-61699-20231101222746' created on 2023-11-01 22:29:56 +0000 UTC
registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:f3cec7ec40e412e54592ca5c53d60e136f9ddf2f7bc9a84fb160387e132c4574

[ceph: root@cali013 /]# ceph nfs cluster config set nfsganesha -i /var/lib/ceph/ganesha.conf


[ceph: root@cali013 /]# ceph nfs cluster config get nfsganesha
# This file is generated by cephadm.
NFS_CORE_PARAM {
        Enable_NLM = false;
        Enable_RQUOTA = false;
        Protocols = 4;
        NFS_Port = 2049;
}
NFSv4 {
        Delegations = false;
        RecoveryBackend = 'rados_cluster';
        Minor_Versions = 1, 2;
}
RADOS_KV {
        UserId = "nfs.nfsganesha.0.0.cali013.zjblln";
        nodeid = "nfs.nfsganesha.0";
        pool = ".nfs";
        namespace = "nfsganesha";
}
RADOS_URLS {
        UserId = "nfs.nfsganesha.0.0.cali013.zjblln";
        watch_url = "rados://.nfs/nfsganesha/conf-nfs.nfsganesha";
}
RGW {
        cluster = "ceph";
        name = "client.nfs.nfsganesha.0.0.cali013.zjblln-rgw";
}
LOG {
 COMPONENTS {
     ALL = FULL_DEBUG;
 }
}
%url    rados://.nfs/nfsganesha/conf-nfs.nfsganesha

# ceph orch ls
NAME                       PORTS   RUNNING  REFRESHED  AGE  PLACEMENT                        
mds.cephfs                             3/3  9m ago     4w   label:mds                        
mgr                                    3/3  9m ago     2w   cali013;cali015;cali016;count:3  
mon                                    1/5  44s ago    4w   <unmanaged>                      
nfs.nfsganesha             ?:2049      1/1  44s ago    9m   cali013                          
osd.all-available-devices               35  9m ago     4w   *                                
rgw.rgw.1                  ?:80        2/2  7m ago     4w   label:rgw                


[ceph: root@cali013 /]# ceph orch ps | grep nfs
nfs.nfsganesha.0.0.cali013.zjblln  cali013  *:2049       running (62s)    61s ago   9m    17.2M        -  5.6               d5840a5e85fb  488ca35640b6  


=============
Logs
=============

Nov 09 06:11:49 cali013 ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln[1355941]: 09/11/2023 06:11:49 : epoch 654c7822 : cali013 : ganesha.nfsd-2[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
Nov 09 06:11:49 cali013 ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln[1355941]: 09/11/2023 06:11:49 : epoch 654c7822 : cali013 : ganesha.nfsd-2[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
Nov 09 06:11:49 cali013 ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln[1355941]: 09/11/2023 06:11:49 : epoch 654c7822 : cali013 : ganesha.nfsd-2[main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
Nov 09 06:11:49 cali013 ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln[1355941]: 09/11/2023 06:11:49 : epoch 654c7822 : cali013 : ganesha.nfsd-2[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
Nov 09 06:12:38 cali013 ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln[1355941]: 09/11/2023 06:12:38 : epoch 654c7822 : cali013 : ganesha.nfsd-2[sigmgr] sigmgr_thread :MAIN :EVENT :SIGHUP_HANDLER: Received SIGHUP.... initiating export list reload
Nov 09 06:12:38 cali013 ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln[1355941]: 09/11/2023 06:12:38 : epoch 654c7822 : cali013 : ganesha.nfsd-2[sigmgr] reclaim_reset :FSAL :EVENT :start_reclaim failed: -2
Nov 09 06:12:38 cali013 ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln[1355941]: 09/11/2023 06:12:38 : epoch 654c7822 : cali013 : ganesha.nfsd-2[sigmgr] reread_exports :CONFIG :EVENT :Reread exports complete
Nov 09 06:19:45 cali013 ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln[1355941]: 09/11/2023 06:19:45 : epoch 654c7822 : cali013 : ganesha.nfsd-2[sigmgr] sigmgr_thread :MAIN :EVENT :SIGHUP_HANDLER: Received SIGHUP.... initiating export list reload
Nov 09 06:19:52 cali013 systemd[1]: Stopping Ceph nfs.nfsganesha.0.0.cali013.zjblln for 4e687a60-638e-11ee-8772-b49691cee574...
Nov 09 06:20:02 cali013 bash[1359483]: time="2023-11-09T06:20:02Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln in 10 seconds, resorting to SIGKILL"
Nov 09 06:20:02 cali013 podman[1359483]: 2023-11-09 06:20:02.976212835 +0000 UTC m=+10.045440204 container died 88d70f7a2e3701962639d0005599fcafe997165ce8149eeb9ce5fd936b2d947e (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:c4f1cf9ac78d873050a567493eb3a4956c661a1fdb52cf58ba05c995936f5ba0, name=ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux <gabrioux>, build-date=2023-11-01T22:27:47, release=120, GIT_BRANCH=main, distribution-scope=public, CEPH_POINT_RELEASE=, vcs-ref=6a3109234de1e767361375a550322ef998fe07ed, RELEASE=main, GIT_CLEAN=True, vcs-type=git, version=7, com.redhat.license_terms=https://www.redhat.com/agreements, GIT_COMMIT=54fe819971d3d2dbde321203c5644c08d10742d5, description=Red Hat Ceph Storage 7, name=rhceph, io.k8s.description=Red Hat Ceph Storage 7, summary=Provides the latest Red Hat Ceph Storage 7 on RHEL 9 in a fully featured and supported base image., io.buildah.version=1.29.0, io.openshift.expose-services=, architecture=x86_64, io.openshift.tags=rhceph ceph, ceph=True, com.redhat.component=rhceph-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/7-120, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=Red Hat Ceph Storage 7 on RHEL 9)
Nov 09 06:20:02 cali013 podman[1359483]: 2023-11-09 06:20:02.985691945 +0000 UTC m=+10.054919307 container cleanup 88d70f7a2e3701962639d0005599fcafe997165ce8149eeb9ce5fd936b2d947e (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:c4f1cf9ac78d873050a567493eb3a4956c661a1fdb52cf58ba05c995936f5ba0, name=ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln, release=120, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/7-120, io.buildah.version=1.29.0, vcs-ref=6a3109234de1e767361375a550322ef998fe07ed, vendor=Red Hat, Inc., GIT_CLEAN=True, summary=Provides the latest Red Hat Ceph Storage 7 on RHEL 9 in a fully featured and supported base image., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=rhceph-container, build-date=2023-11-01T22:27:47, RELEASE=main, version=7, ceph=True, CEPH_POINT_RELEASE=, io.k8s.display-name=Red Hat Ceph Storage 7 on RHEL 9, GIT_BRANCH=main, name=rhceph, com.redhat.license_terms=https://www.redhat.com/agreements, description=Red Hat Ceph Storage 7, vcs-type=git, io.openshift.expose-services=, GIT_COMMIT=54fe819971d3d2dbde321203c5644c08d10742d5, distribution-scope=public, maintainer=Guillaume Abrioux <gabrioux>, io.k8s.description=Red Hat Ceph Storage 7, io.openshift.tags=rhceph ceph)
Nov 09 06:20:02 cali013 bash[1359483]: ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln
Nov 09 06:20:03 cali013 podman[1359537]: 2023-11-09 06:20:03.009631263 +0000 UTC m=+0.031597064 container remove 88d70f7a2e3701962639d0005599fcafe997165ce8149eeb9ce5fd936b2d947e (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:c4f1cf9ac78d873050a567493eb3a4956c661a1fdb52cf58ba05c995936f5ba0, name=ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln, summary=Provides the latest Red Hat Ceph Storage 7 on RHEL 9 in a fully featured and supported base image., io.k8s.description=Red Hat Ceph Storage 7, CEPH_POINT_RELEASE=, version=7, maintainer=Guillaume Abrioux <gabrioux>, distribution-scope=public, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.openshift.tags=rhceph ceph, architecture=x86_64, release=120, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/7-120, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.component=rhceph-container, com.redhat.license_terms=https://www.redhat.com/agreements, io.k8s.display-name=Red Hat Ceph Storage 7 on RHEL 9, description=Red Hat Ceph Storage 7, build-date=2023-11-01T22:27:47, vcs-type=git, GIT_COMMIT=54fe819971d3d2dbde321203c5644c08d10742d5, RELEASE=main, vcs-ref=6a3109234de1e767361375a550322ef998fe07ed, GIT_BRANCH=main, name=rhceph, io.buildah.version=1.29.0)
Nov 09 06:20:03 cali013 systemd[1]: ceph-4e687a60-638e-11ee-8772-b49691cee574.0.0.cali013.zjblln.service: Main process exited, code=exited, status=137/n/a
Nov 09 06:20:03 cali013 systemd[1]: ceph-4e687a60-638e-11ee-8772-b49691cee574.0.0.cali013.zjblln.service: Failed with result 'exit-code'.
Nov 09 06:20:03 cali013 systemd[1]: Stopped Ceph nfs.nfsganesha.0.0.cali013.zjblln for 4e687a60-638e-11ee-8772-b49691cee574.
Nov 09 06:20:03 cali013 systemd[1]: ceph-4e687a60-638e-11ee-8772-b49691cee574.0.0.cali013.zjblln.service: Consumed 18.395s CPU time.
Nov 09 06:20:03 cali013 systemd[1]: Starting Ceph nfs.nfsganesha.0.0.cali013.zjblln for 4e687a60-638e-11ee-8772-b49691cee574...
Nov 09 06:20:03 cali013 podman[1359708]: 
Nov 09 06:20:03 cali013 podman[1359708]: 2023-11-09 06:20:03.397914151 +0000 UTC m=+0.033967414 container create 488ca35640b6cc5c6112b943efb6fdee2cbb3acbd4ed118c302f08d58940a685 (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:c4f1cf9ac78d873050a567493eb3a4956c661a1fdb52cf58ba05c995936f5ba0, name=ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln, com.redhat.license_terms=https://www.redhat.com/agreements, vendor=Red Hat, Inc., architecture=x86_64, maintainer=Guillaume Abrioux <gabrioux>, description=Red Hat Ceph Storage 7, GIT_COMMIT=54fe819971d3d2dbde321203c5644c08d10742d5, ceph=True, io.k8s.display-name=Red Hat Ceph Storage 7 on RHEL 9, distribution-scope=public, RELEASE=main, io.k8s.description=Red Hat Ceph Storage 7, vcs-ref=6a3109234de1e767361375a550322ef998fe07ed, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2023-11-01T22:27:47, GIT_CLEAN=True, name=rhceph, summary=Provides the latest Red Hat Ceph Storage 7 on RHEL 9 in a fully featured and supported base image., url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/7-120, CEPH_POINT_RELEASE=, GIT_BRANCH=main, com.redhat.component=rhceph-container, io.buildah.version=1.29.0, version=7, io.openshift.tags=rhceph ceph, release=120, io.openshift.expose-services=)
Nov 09 06:20:03 cali013 podman[1359708]: 2023-11-09 06:20:03.429345321 +0000 UTC m=+0.065398591 container init 488ca35640b6cc5c6112b943efb6fdee2cbb3acbd4ed118c302f08d58940a685 (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:c4f1cf9ac78d873050a567493eb3a4956c661a1fdb52cf58ba05c995936f5ba0, name=ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln, io.openshift.tags=rhceph ceph, io.openshift.expose-services=, io.buildah.version=1.29.0, version=7, GIT_BRANCH=main, name=rhceph, vcs-ref=6a3109234de1e767361375a550322ef998fe07ed, vcs-type=git, build-date=2023-11-01T22:27:47, com.redhat.component=rhceph-container, RELEASE=main, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/7-120, GIT_COMMIT=54fe819971d3d2dbde321203c5644c08d10742d5, ceph=True, io.k8s.display-name=Red Hat Ceph Storage 7 on RHEL 9, release=120, summary=Provides the latest Red Hat Ceph Storage 7 on RHEL 9 in a fully featured and supported base image., vendor=Red Hat, Inc., io.k8s.description=Red Hat Ceph Storage 7, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=, com.redhat.license_terms=https://www.redhat.com/agreements, architecture=x86_64, GIT_CLEAN=True, distribution-scope=public, maintainer=Guillaume Abrioux <gabrioux>, description=Red Hat Ceph Storage 7)
Nov 09 06:20:03 cali013 podman[1359708]: 2023-11-09 06:20:03.432412847 +0000 UTC m=+0.068466109 container start 488ca35640b6cc5c6112b943efb6fdee2cbb3acbd4ed118c302f08d58940a685 (image=registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:c4f1cf9ac78d873050a567493eb3a4956c661a1fdb52cf58ba05c995936f5ba0, name=ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln, io.buildah.version=1.29.0, io.k8s.description=Red Hat Ceph Storage 7, com.redhat.component=rhceph-container, GIT_BRANCH=main, io.openshift.tags=rhceph ceph, GIT_CLEAN=True, release=120, GIT_REPO=https://github.com/ceph/ceph-container.git, name=rhceph, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhceph/images/7-120, GIT_COMMIT=54fe819971d3d2dbde321203c5644c08d10742d5, build-date=2023-11-01T22:27:47, RELEASE=main, ceph=True, summary=Provides the latest Red Hat Ceph Storage 7 on RHEL 9 in a fully featured and supported base image., version=7, vcs-ref=6a3109234de1e767361375a550322ef998fe07ed, com.redhat.license_terms=https://www.redhat.com/agreements, CEPH_POINT_RELEASE=, io.openshift.expose-services=, distribution-scope=public, architecture=x86_64, maintainer=Guillaume Abrioux <gabrioux>, description=Red Hat Ceph Storage 7, io.k8s.display-name=Red Hat Ceph Storage 7 on RHEL 9, vcs-type=git)
Nov 09 06:20:03 cali013 podman[1359708]: 2023-11-09 06:20:03.385196701 +0000 UTC m=+0.021249971 image pull  registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:c4f1cf9ac78d873050a567493eb3a4956c661a1fdb52cf58ba05c995936f5ba0
Nov 09 06:20:03 cali013 bash[1359708]: 488ca35640b6cc5c6112b943efb6fdee2cbb3acbd4ed118c302f08d58940a685
Nov 09 06:20:03 cali013 systemd[1]: Started Ceph nfs.nfsganesha.0.0.cali013.zjblln for 4e687a60-638e-11ee-8772-b49691cee574.
Nov 09 06:20:03 cali013 ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln[1359740]: 09/11/2023 06:20:03 : epoch 654c7a13 : cali013 : ganesha.nfsd-2[main] init_logging :LOG :NULL :LOG: Setting log level for all components to NIV_EVENT
Nov 09 06:20:03 cali013 ceph-4e687a60-638e-11ee-8772-b49691cee574-nfs-nfsganesha-0-0-cali013-zjblln[1359740]: 09/11/2023 06:20:03 : epoch 654c7a13 : cali013 : ganesha.nfsd-2[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 5.6



[ceph: root@cali013 /]# ceph -s
  cluster:
    id:     4e687a60-638e-11ee-8772-b49691cee574
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum cali013 (age 6d)
    mgr: cali015.hvvbwh(active, since 15h), standbys: cali013.qakwdk, cali016.rhribl
    mds: 1/1 daemons up, 2 standby
    osd: 35 osds: 35 up (since 6d), 35 in (since 4w); 7 remapped pgs
    rgw: 2 daemons active (2 hosts, 1 zones)
 
  data:
    volumes: 1/1 healthy
    pools:   9 pools, 295 pgs
    objects: 20.87M objects, 22 GiB
    usage:   242 GiB used, 86 TiB / 86 TiB avail
    pgs:     2387810/62597658 objects misplaced (3.815%)
             269 active+clean
             13  active+clean+scrubbing
             6   active+clean+scrubbing+deep
             4   active+remapped+backfill_wait
             3   active+remapped+backfilling
 
  io:
    client:   11 KiB/s rd, 21 op/s rd, 0 op/s wr
    recovery: 83 KiB/s, 70 objects/s

Comment 1 Frank Filz 2023-11-09 19:18:52 UTC
Do you have the setup in a state I could check on it? Enabling debug logging should not have any effect on client ability to connect. I suspect some other issue has come up, perhaps due to Ganesha restart.

Comment 2 Manisha Saini 2023-11-14 05:48:16 UTC
(In reply to Frank Filz from comment #1)
> Do you have the setup in a state I could check on it? Enabling debug logging
> should not have any effect on client ability to connect. I suspect some
> other issue has come up, perhaps due to Ganesha restart.

Hi Frank, I don't have setup in same state currently. Will recreate the issue on new machines and will share the cluster.Thanks

Comment 3 Scott Ostapovicz 2023-11-16 16:13:37 UTC
This is not currently on the list of blocking issues for 7.0... retargeting to 7.0 z1.

Comment 4 Manisha Saini 2024-01-22 19:24:48 UTC
Unable to reproduce the issue when tested with latest RHCS 7.0 build. Debug logs were enabled successfully.

[ceph: root@ceph-nfs-debug-dqjc7q-node1-installer /]# ceph --version
ceph version 18.2.0-140.el9cp (17a3310c0d0f01e5c1de938eb8cc817e41f58979) reef (stable)


[ceph: root@ceph-nfs-debug-dqjc7q-node1-installer /]# rpm -qa | grep nfs
libnfsidmap-2.5.4-20.el9.x86_64
nfs-utils-2.5.4-20.el9.x86_64
nfs-ganesha-selinux-5.6-4.el9cp.noarch
nfs-ganesha-5.6-4.el9cp.x86_64
nfs-ganesha-rgw-5.6-4.el9cp.x86_64
nfs-ganesha-ceph-5.6-4.el9cp.x86_64
nfs-ganesha-rados-grace-5.6-4.el9cp.x86_64
nfs-ganesha-rados-urls-5.6-4.el9cp.x86_64

======================
[root@ceph-nfs-debug-dqjc7q-node1-installer ~]# sudo podman exec -it e4bfed0c72e8 bash


[root@ceph-nfs-debug-dqjc7q-node1-installer /]# cat /etc/ganesha/ganesha.conf 
# This file is generated by cephadm.
NFS_CORE_PARAM {
        Enable_NLM = false;
        Enable_RQUOTA = false;
        Protocols = 4;
        NFS_Port = 2049;
}

NFSv4 {
        Delegations = false;
        RecoveryBackend = 'rados_cluster';
        Minor_Versions = 1, 2;
}

RADOS_KV {
        UserId = "nfs.nfs.0.0.ceph-nfs-debug-dqjc7q-node1-installer.lzhmoo";
        nodeid = "nfs.nfs.0";
        pool = ".nfs";
        namespace = "nfs";
}

RADOS_URLS {
        UserId = "nfs.nfs.0.0.ceph-nfs-debug-dqjc7q-node1-installer.lzhmoo";
        watch_url = "rados://.nfs/nfs/conf-nfs.nfs";
}

RGW {
        cluster = "ceph";
        name = "client.nfs.nfs.0.0.ceph-nfs-debug-dqjc7q-node1-installer.lzhmoo-rgw";
}


%url    rados://.nfs/nfs/conf-nfs.nfs


[root@ceph-nfs-debug-dqjc7q-node1-installer ~]# cephadm shell --mount nfs.conf:/var/lib/ceph/ganesha.conf
Inferring fsid 1094ee42-b949-11ee-bce8-fa163e1c22f3
Inferring config /var/lib/ceph/1094ee42-b949-11ee-bce8-fa163e1c22f3/mon.ceph-nfs-debug-dqjc7q-node1-installer/config
Using ceph image with id '07c9c7bf170e' and tag 'ceph-7.0-rhel-9-containers-candidate-24715-20240118074145' created on 2024-01-18 07:44:56 +0000 UTC
registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:5154a1d743a366272184bb8c8749ca697a4060cc6a6a0878ff9cfe374974333d


[ceph: root@ceph-nfs-debug-dqjc7q-node1-installer /]#  ceph config-key set mgr/cephadm/services/nfs/ganesha.conf -i /var/lib/ceph/ganesha.conf 
set mgr/cephadm/services/nfs/ganesha.conf


[ceph: root@ceph-nfs-debug-dqjc7q-node1-installer /]#  ceph config-key get mgr/cephadm/services/nfs/ganesha.conf 
# This file is generated by cephadm.
NFS_CORE_PARAM {
        Enable_NLM = false;
        Enable_RQUOTA = false;
        Protocols = 4;
        NFS_Port = 2049;
}

NFSv4 {
        Delegations = false;
        RecoveryBackend = 'rados_cluster';
        Minor_Versions = 1, 2;
}

RADOS_KV {
        UserId = "nfs.nfs.0.0.ceph-nfs-debug-dqjc7q-node1-installer.lzhmoo";
        nodeid = "nfs.nfs.0";
        pool = ".nfs";
        namespace = "nfs";
}

RADOS_URLS {
        UserId = "nfs.nfs.0.0.ceph-nfs-debug-dqjc7q-node1-installer.lzhmoo";
        watch_url = "rados://.nfs/nfs/conf-nfs.nfs";
}

RGW {
        cluster = "ceph";
        name = "client.nfs.nfs.0.0.ceph-nfs-debug-dqjc7q-node1-installer.lzhmoo-rgw";
}

LOG {
 COMPONENTS {
     ALL = FULL_DEBUG;
 }
}

%url    rados://.nfs/nfs/conf-nfs.nfs

[ceph: root@ceph-nfs-debug-dqjc7q-node1-installer /]# ceph orch ls
NAME                       PORTS        RUNNING  REFRESHED  AGE  PLACEMENT                              
alertmanager               ?:9093,9094      1/1  6m ago     2h   count:1                                
ceph-exporter                               3/3  7m ago     2h   *                                      
crash                                       3/3  7m ago     2h   *                                      
grafana                    ?:3000           1/1  6m ago     2h   count:1                                
mds.cephfs                                  2/2  7m ago     2h   label:mds                              
mgr                                         2/2  7m ago     2h   count:2                                
mon                                         3/5  7m ago     2h   count:5                                
nfs.nfs                    ?:2049           1/1  6m ago     6m   ceph-nfs-debug-dqjc7q-node1-installer  
node-exporter              ?:9100           3/3  7m ago     2h   *                                      
osd.all-available-devices                    18  7m ago     2h   *                                      
prometheus                 ?:9095           1/1  6m ago     2h   count:1                                
rgw.rgw.1                  ?:80             2/2  7m ago     2h   label:rgw   

                           
[ceph: root@ceph-nfs-debug-dqjc7q-node1-installer /]# ceph orch redeploy nfs.nfs       
Scheduled to redeploy nfs.nfs.0.0.ceph-nfs-debug-dqjc7q-node1-installer.lzhmoo on host 'ceph-nfs-debug-dqjc7q-node1-installer'

[ceph: root@ceph-nfs-debug-dqjc7q-node1-installer /]# ceph orch ps | grep nfs.nfs
nfs.nfs.0.0.ceph-nfs-debug-dqjc7q-node1-installer.lzhmoo  ceph-nfs-debug-dqjc7q-node1-installer  *:2049            running (43s)    41s ago   7m    19.6M        -  5.6               07c9c7bf170e  9e69a785395b 


[root@ceph-nfs-debug-dqjc7q-node5 mnt]# mount -t nfs -o vers=4.1,port=2049 ceph-nfs-debug-dqjc7q-node1-installer:/export_1 /mnt/ganesha/


[root@ceph-nfs-debug-dqjc7q-node5 mnt]# cd /mnt/ganesha/

[root@ceph-nfs-debug-dqjc7q-node5 ganesha]# ls
f1

Comment 5 Frank Filz 2024-01-24 22:44:10 UTC
Can we close this then?

Comment 7 Frank Filz 2024-03-05 17:47:22 UTC
Can we close instead of kicking the can down the road? If this is not recreating, let's close it.

Comment 8 tserlin 2024-03-05 19:54:18 UTC
(In reply to Frank Filz from comment #7)
> Can we close instead of kicking the can down the road? If this is not
> recreating, let's close it.

I'll go ahead and close it, Frank. Especially since in comment #4 a later 7.0 build (nfs-ganesha-5.6-4.el9cp instead of the prior nfs-ganesha-5.6-3.el9cp build) worked OK for QE.

Thanks,

Thomas