Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 1962245

Summary: [CephFS-NFS] - Regression - command for cluster info is returning empty dictionary
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Amarnath <amk>
Component: CephadmAssignee: Sebastian Wagner <sewagner>
Status: CLOSED DUPLICATE QA Contact: Vasishta <vashastr>
Severity: high Docs Contact: Karen Norteman <knortema>
Priority: unspecified    
Version: 5.0CC: ceph-eng-bugs, hyelloji, sewagner, sweil, vereddy
Target Milestone: ---Keywords: Regression, Reopened
Target Release: 5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-08 11:15:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1964951    
Bug Blocks:    
Attachments:
Description Flags
Manager logs
none
Logs after Executing the script none

Description Amarnath 2021-05-19 15:08:15 UTC 
Description of problem:
command for cluster-info is returning empty dictionary

Version-Release number of selected component (if applicable):
ceph version 16.2.0-38.el8cp (54fb2271e5015808565bc05b6877deb6bf3f5da9) pacific (stable)

How reproducible:

Create nfs cluster 
check the info

[root@ceph-amar-bg-1621428120609-node1-mon-mgr-client-installer ~]# ceph nfs cluster create cephfs testnfs
NFS Cluster Created Successfully
[root@ceph-amar-bg-1621428120609-node1-mon-mgr-client-installer ~]# ceph nfs cluster info testnfs
{}
[root@ceph-amar-bg-1621428120609-node1-mon-mgr-client-installer ~]# ceph version
ceph version 16.2.0-38.el8cp (54fb2271e5015808565bc05b6877deb6bf3f5da9) pacific (stable)



Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Patrick Donnelly 2021-05-21 03:15:27 UTC 
(In reply to Amarnath from comment #0)
> Description of problem:
> command for cluster-info is returning empty dictionary
> 
> Version-Release number of selected component (if applicable):
> ceph version 16.2.0-38.el8cp (54fb2271e5015808565bc05b6877deb6bf3f5da9)
> pacific (stable)
> 
> How reproducible:
> 
> Create nfs cluster 
> check the info
> 
> [root@ceph-amar-bg-1621428120609-node1-mon-mgr-client-installer ~]# ceph nfs
> cluster create cephfs testnfs
> NFS Cluster Created Successfully
> [root@ceph-amar-bg-1621428120609-node1-mon-mgr-client-installer ~]# ceph nfs
> cluster info testnfs
> {}
> [root@ceph-amar-bg-1621428120609-node1-mon-mgr-client-installer ~]# ceph
> version
> ceph version 16.2.0-38.el8cp (54fb2271e5015808565bc05b6877deb6bf3f5da9)
> pacific (stable)

Perhaps the orchestrator has not deployed the nfs cluster yet. You may try polling `nfs cluster info` until it shows up?
Comment 2 Amarnath 2021-05-24 16:38:20 UTC 
Hi Patrick,

I tried almost after ~5 hrs even then I not getting the details.
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ sudo ceph nfs cluster create cephfs testnfs
NFS Cluster Created Successfully
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ ceph nfs cluster info testnfs
[errno 13] error connecting to the cluster
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ sudo ceph nfs cluster info testnfs 
{}
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ sudo ceph nfs cluster info testnfs 
{}
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ sudo ceph nfs cluster info testnfs 
{}
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ sudo ceph nfs cluster info testnfs 
^[[A{}
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ sudo ceph nfs cluster info testnfs 
{}
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ sudo ceph nfs cluster info testnfs 
{}
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ sudo ceph nfs cluster info testnfs 
{}
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ date
Mon May 24 07:08:46 EDT 2021
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ timed out waiting for input: auto-logout
Connection to 10.0.211.230 closed.
[amk@amk ~]$ ssh cephuser.211.230
cephuser.211.230's password: 
Activate the web console with: systemctl enable --now cockpit.socket

This system is not registered to Red Hat Insights. See https://cloud.redhat.com/
To register this system, run: insights-client --register

Last login: Mon May 24 07:06:49 2021 from 10.74.9.87
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ sudo ceph nfs cluster info testnfs
{}
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ date
Mon May 24 07:22:30 EDT 2021
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ timed out waiting for input: auto-logout
Connection to 10.0.211.230 closed.
[amk@amk ~]$ ssh cephuser.211.230
cephuser.211.230's password: 
Activate the web console with: systemctl enable --now cockpit.socket

This system is not registered to Red Hat Insights. See https://cloud.redhat.com/
To register this system, run: insights-client --register

Last login: Mon May 24 07:22:09 2021 from 10.74.9.87
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ sudo ceph nfs cluster info testnfs
{}
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$ date
Mon May 24 12:34:44 EDT 2021
[cephuser@ceph-rbd2-amarrbd-1621824120963-node1-mon-mgr-installer ~]$
Comment 3 Varsha 2021-05-25 05:07:57 UTC 
Can you please share `ceph orch ps` output ? Maybe the nfs-ganesha daemon was not deployed successfully.
Comment 4 Amarnath 2021-05-25 11:19:47 UTC 
This is new system.
IP : 10.74.8.53

[root@ceph-amar-bg-1621928213315-node1-mon-mgr-client-installer ~]# ceph orch ps
NAME                                                                   HOST                                                       STATUS        REFRESHED  AGE  PORTS   VERSION          IMAGE ID      CONTAINER ID  
mds.cephfs.ceph-amar-bg-1621928213315-node2-mon-mgr-mds-client.zeenhp  ceph-amar-bg-1621928213315-node2-mon-mgr-mds-client        running (2h)  76s ago    2h   -       16.2.0-38.el8cp  46f26486881c  bce942775917  
mds.cephfs.ceph-amar-bg-1621928213315-node3-osd-mon-mds-client.zrbqxc  ceph-amar-bg-1621928213315-node3-osd-mon-mds-client        running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  04e6b80e1771  
mds.cephfs.ceph-amar-bg-1621928213315-node4-osd-mds.mnysgy             ceph-amar-bg-1621928213315-node4-osd-mds                   running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  333563b94fda  
mds.cephfs.ceph-amar-bg-1621928213315-node5-osd-mds-client.clpnrj      ceph-amar-bg-1621928213315-node5-osd-mds-client            running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  fb49c5de6e9d  
mgr.ceph-amar-bg-1621928213315-node1-mon-mgr-client-installer.pqnjgf   ceph-amar-bg-1621928213315-node1-mon-mgr-client-installer  running (3h)  77s ago    3h   *:9283  16.2.0-38.el8cp  46f26486881c  dd6b5b0a0959  
mgr.ceph-amar-bg-1621928213315-node2-mon-mgr-mds-client.nctvpa         ceph-amar-bg-1621928213315-node2-mon-mgr-mds-client        running (2h)  76s ago    2h   *:8443  16.2.0-38.el8cp  46f26486881c  cc16b876b013  
mon.ceph-amar-bg-1621928213315-node1-mon-mgr-client-installer          ceph-amar-bg-1621928213315-node1-mon-mgr-client-installer  running (3h)  77s ago    3h   -       16.2.0-38.el8cp  46f26486881c  943cd4e076d3  
mon.ceph-amar-bg-1621928213315-node2-mon-mgr-mds-client                ceph-amar-bg-1621928213315-node2-mon-mgr-mds-client        running (2h)  76s ago    2h   -       16.2.0-38.el8cp  46f26486881c  9a93f3f20c63  
mon.ceph-amar-bg-1621928213315-node3-osd-mon-mds-client                ceph-amar-bg-1621928213315-node3-osd-mon-mds-client        running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  a6e8b57efbf1  
osd.0                                                                  ceph-amar-bg-1621928213315-node7-osd-nfs                   running (2h)  77s ago    2h   -       16.2.0-38.el8cp  46f26486881c  3b760c78ad25  
osd.1                                                                  ceph-amar-bg-1621928213315-node3-osd-mon-mds-client        running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  729587f1461c  
osd.10                                                                 ceph-amar-bg-1621928213315-node3-osd-mon-mds-client        running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  c139cb72cdb8  
osd.11                                                                 ceph-amar-bg-1621928213315-node7-osd-nfs                   running (2h)  77s ago    2h   -       16.2.0-38.el8cp  46f26486881c  fe95328f93e6  
osd.12                                                                 ceph-amar-bg-1621928213315-node5-osd-mds-client            running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  7fd4aa776abb  
osd.13                                                                 ceph-amar-bg-1621928213315-node4-osd-mds                   running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  884d2d36a621  
osd.14                                                                 ceph-amar-bg-1621928213315-node6-osd-nfs                   running (2h)  76s ago    2h   -       16.2.0-38.el8cp  46f26486881c  3c63168b0a04  
osd.15                                                                 ceph-amar-bg-1621928213315-node3-osd-mon-mds-client        running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  a228e531f8bf  
osd.16                                                                 ceph-amar-bg-1621928213315-node7-osd-nfs                   running (2h)  77s ago    2h   -       16.2.0-38.el8cp  46f26486881c  1fa2bfc59c06  
osd.17                                                                 ceph-amar-bg-1621928213315-node5-osd-mds-client            running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  f93409e34156  
osd.18                                                                 ceph-amar-bg-1621928213315-node4-osd-mds                   running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  5afefd4ce5c2  
osd.19                                                                 ceph-amar-bg-1621928213315-node6-osd-nfs                   running (2h)  76s ago    2h   -       16.2.0-38.el8cp  46f26486881c  c2570c005438  
osd.2                                                                  ceph-amar-bg-1621928213315-node5-osd-mds-client            running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  a8b23924b60b  
osd.3                                                                  ceph-amar-bg-1621928213315-node4-osd-mds                   running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  9556dd350ed6  
osd.4                                                                  ceph-amar-bg-1621928213315-node6-osd-nfs                   running (2h)  76s ago    2h   -       16.2.0-38.el8cp  46f26486881c  49b5bff7709d  
osd.5                                                                  ceph-amar-bg-1621928213315-node7-osd-nfs                   running (2h)  77s ago    2h   -       16.2.0-38.el8cp  46f26486881c  b3b0ac5a5b1f  
osd.6                                                                  ceph-amar-bg-1621928213315-node3-osd-mon-mds-client        running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  0a1da8ba5030  
osd.7                                                                  ceph-amar-bg-1621928213315-node5-osd-mds-client            running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  da2dad2e8870  
osd.8                                                                  ceph-amar-bg-1621928213315-node4-osd-mds                   running (2h)  13s ago    2h   -       16.2.0-38.el8cp  46f26486881c  7ecb584b6cd0  
osd.9                                                                  ceph-amar-bg-1621928213315-node6-osd-nfs                   running (2h)  76s ago    2h   -       16.2.0-38.el8cp  46f26486881c  4fe3c0985a19  


[root@ceph-amar-bg-1621928213315-node1-mon-mgr-client-installer ~]# ceph nfs cluster info testnfs
{}
[root@ceph-amar-bg-1621928213315-node1-mon-mgr-client-installer ~]#
Comment 5 Varsha 2021-05-25 11:41:08 UTC 
nfs daemon is not listed. Please share the mgr logs.
Comment 6 Amarnath 2021-05-25 15:05:15 UTC 
Hi Varsha,

I see below trace in manager log

Node Ip :10.0.208.240

debug 2021-05-25T14:41:37.382+0000 7f3815df3700  0 log_channel(cephadm) log [INF] : Checking pool "nfs-ganesha" exists for service nfs.testnfs
debug 2021-05-25T14:41:37.382+0000 7f3815df3700  0 log_channel(cephadm) log [INF] : Creating key for client.nfs.testnfs.ceph-amar-bg-1621928213315-node3-osd-mon-mds-client
debug 2021-05-25T14:41:37.399+0000 7f3815df3700  0 log_channel(cephadm) log [INF] : Rados config object exists: conf-nfs.testnfs
debug 2021-05-25T14:41:37.399+0000 7f3815df3700  0 log_channel(cephadm) log [INF] : Creating key for client.nfs.testnfs.ceph-amar-bg-1621928213315-node3-osd-mon-mds-client-rgw
debug 2021-05-25T14:41:37.417+0000 7f3815df3700  0 log_channel(cephadm) log [INF] : Deploying daemon nfs.testnfs.ceph-amar-bg-1621928213315-node3-osd-mon-mds-client on ceph-amar-bg-1621928213315-node3-osd-mon-mds-client
debug 2021-05-25T14:41:38.446+0000 7f3815df3700  0 [cephadm ERROR cephadm.serve] cephadm exited with an error code: 1, stderr:Deploy daemon nfs.testnfs.ceph-amar-bg-1621928213315-node3-osd-mon-mds-client ...
Verifying port 2049 ...
Cannot bind to IP 0.0.0.0 port 2049: [Errno 98] Address already in use
ERROR: TCP Port(s) '2049' required for nfs already in use
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 1173, in _remote_connection
yield (conn, connr)
File "/usr/share/ceph/mgr/cephadm/serve.py", line 1088, in _run_cephadm
code, '\n'.join(err)))
orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon nfs.testnfs.ceph-amar-bg-1621928213315-node3-osd-mon-mds-client ...
Verifying port 2049 ...
Cannot bind to IP 0.0.0.0 port 2049: [Errno 98] Address already in use
ERROR: TCP Port(s) '2049' required for nfs already in use
debug 2021-05-25T14:41:38.446+0000 7f3815df3700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr:Deploy daemon nfs.testnfs.ceph-amar-bg-1621928213315-node3-osd-mon-mds-client ...
Verifying port 2049 ...
Cannot bind to IP 0.0.0.0 port 2049: [Errno 98] Address already in use
ERROR: TCP Port(s) '2049' required for nfs already in use
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/serve.py", line 1173, in _remote_connection
yield (conn, connr)
File "/usr/share/ceph/mgr/cephadm/serve.py", line 1088, in _run_cephadm
code, '\n'.join(err)))
orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon nfs.testnfs.ceph-amar-bg-1621928213315-node3-osd-mon-mds-client ...
Verifying port 2049 ...
Cannot bind to IP 0.0.0.0 port 2049: [Errno 98] Address already in use
ERROR: TCP Port(s) '2049' required for nfs already in use
debug 2021-05-25T14:41:38.446+0000 7f3815df3700  0 log_channel(cephadm) log [INF] : Removing key for client.nfs.testnfs.ceph-amar-bg-1621928213315-node3-osd-mon-mds-client
debug 2021-05-25T14:41:38.464+0000 7f3815df3700  0 log_channel(cephadm) log [INF] : Removing key for client.nfs.testnfs.ceph-amar-bg-1621928213315-node3-osd-mon-mds-client-rgw
debug 2021-05-25T14:41:38.792+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11545: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail
debug 2021-05-25T14:41:40.794+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11546: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail; 0 B/s rd, 0 op/s
debug 2021-05-25T14:41:42.794+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11547: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail; 0 B/s rd, 0 op/s
debug 2021-05-25T14:41:44.795+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11548: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail; 0 B/s rd, 0 op/s
debug 2021-05-25T14:41:46.796+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11549: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail; 0 B/s rd, 0 op/s
debug 2021-05-25T14:41:48.796+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11550: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail; 0 B/s rd, 0 op/s
debug 2021-05-25T14:41:50.797+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11551: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail; 0 B/s rd, 0 op/s
debug 2021-05-25T14:41:52.798+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11552: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail
debug 2021-05-25T14:41:54.798+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11553: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail
debug 2021-05-25T14:41:56.799+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11554: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail
debug 2021-05-25T14:41:58.799+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11555: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail
debug 2021-05-25T14:42:00.800+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11556: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail
debug 2021-05-25T14:42:02.801+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11557: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail
debug 2021-05-25T14:42:04.801+0000 7f381e003700  0 log_channel(cluster) log [DBG] : pgmap v11558: 97 pgs: 97 active+clean; 482 MiB data, 12 GiB used, 288 GiB / 300 GiB avail
Comment 7 Varsha 2021-05-25 15:44:48 UTC 
Hi Amarnath,

NFS-Ganesha daemon listens on 2049 port. But some other process is using it. Kill that process and then deploy the cluster again. It should resolve the issue.
Comment 8 Amarnath 2021-05-27 11:19:44 UTC 
Hi Varsha,

I see nfs service running on the 2049 port

Nfs service node IP : 10.0.208.209

[cephuser@ceph-amar-bg-1621928213315-node3-osd-mon-mds-client ~]$ sudo netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 10.0.208.209:6811       0.0.0.0:*               LISTEN      26372/ceph-osd      
tcp        0      0 0.0.0.0:6812            0.0.0.0:*               LISTEN      27296/ceph-osd      
tcp        0      0 0.0.0.0:6813            0.0.0.0:*               LISTEN      27296/ceph-osd      
tcp        0      0 0.0.0.0:6814            0.0.0.0:*               LISTEN      27296/ceph-osd      
tcp        0      0 0.0.0.0:47775           0.0.0.0:*               LISTEN      37456/rpc.statd     
tcp        0      0 0.0.0.0:6815            0.0.0.0:*               LISTEN      27296/ceph-osd      
tcp        0      0 0.0.0.0:6816            0.0.0.0:*               LISTEN      28222/ceph-osd      
tcp        0      0 0.0.0.0:6817            0.0.0.0:*               LISTEN      28222/ceph-osd      
tcp        0      0 10.0.208.209:6818       0.0.0.0:*               LISTEN      27296/ceph-osd      
tcp        0      0 10.0.208.209:6819       0.0.0.0:*               LISTEN      27296/ceph-osd      
tcp        0      0 0.0.0.0:6820            0.0.0.0:*               LISTEN      28222/ceph-osd      
tcp        0      0 10.0.208.209:3300       0.0.0.0:*               LISTEN      20890/ceph-mon      
tcp        0      0 0.0.0.0:6821            0.0.0.0:*               LISTEN      28222/ceph-osd      
tcp        0      0 10.0.208.209:6789       0.0.0.0:*               LISTEN      20890/ceph-mon      
tcp        0      0 0.0.0.0:6822            0.0.0.0:*               LISTEN      28222/ceph-osd      
tcp        0      0 0.0.0.0:6823            0.0.0.0:*               LISTEN      28222/ceph-osd      
tcp        0      0 0.0.0.0:6824            0.0.0.0:*               LISTEN      29152/ceph-osd      
tcp        0      0 0.0.0.0:6825            0.0.0.0:*               LISTEN      29152/ceph-osd      
tcp        0      0 10.0.208.209:6826       0.0.0.0:*               LISTEN      28222/ceph-osd      
tcp        0      0 10.0.208.209:6827       0.0.0.0:*               LISTEN      28222/ceph-osd      
tcp        0      0 0.0.0.0:6828            0.0.0.0:*               LISTEN      29152/ceph-osd      
tcp        0      0 0.0.0.0:6829            0.0.0.0:*               LISTEN      29152/ceph-osd      
tcp        0      0 0.0.0.0:6830            0.0.0.0:*               LISTEN      29152/ceph-osd      
tcp        0      0 0.0.0.0:6831            0.0.0.0:*               LISTEN      29152/ceph-osd      
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 0.0.0.0:6800            0.0.0.0:*               LISTEN      26372/ceph-osd      
tcp        0      0 0.0.0.0:6801            0.0.0.0:*               LISTEN      26372/ceph-osd      
tcp        0      0 0.0.0.0:6802            0.0.0.0:*               LISTEN      31327/ceph-mds      
tcp        0      0 10.0.208.209:6834       0.0.0.0:*               LISTEN      29152/ceph-osd      
tcp        0      0 0.0.0.0:6803            0.0.0.0:*               LISTEN      31327/ceph-mds      
tcp        0      0 10.0.208.209:6835       0.0.0.0:*               LISTEN      29152/ceph-osd      
tcp        0      0 0.0.0.0:6804            0.0.0.0:*               LISTEN      26372/ceph-osd      
tcp        0      0 0.0.0.0:6805            0.0.0.0:*               LISTEN      26372/ceph-osd      
tcp        0      0 0.0.0.0:6806            0.0.0.0:*               LISTEN      26372/ceph-osd      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      4696/sshd           
tcp        0      0 0.0.0.0:6807            0.0.0.0:*               LISTEN      26372/ceph-osd      
tcp        0      0 0.0.0.0:6808            0.0.0.0:*               LISTEN      27296/ceph-osd      
tcp        0      0 0.0.0.0:6809            0.0.0.0:*               LISTEN      27296/ceph-osd      
tcp        0      0 10.0.208.209:6810       0.0.0.0:*               LISTEN      26372/ceph-osd      
tcp6       0      0 :::51259                :::*                    LISTEN      37456/rpc.statd     
tcp6       0      0 :::2049                 :::*                    LISTEN      37464/ganesha.nfsd  
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd           
tcp6       0      0 :::22                   :::*                    LISTEN      4696/sshd           
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1/systemd           
udp        0      0 0.0.0.0:59577           0.0.0.0:*                           37456/rpc.statd     
udp        0      0 127.0.0.1:323           0.0.0.0:*                           19388/chronyd       
udp        0      0 127.0.0.1:745           0.0.0.0:*                           37456/rpc.statd     
udp6       0      0 :::2049                 :::*                                37464/ganesha.nfsd  
udp6       0      0 :::111                  :::*                                1/systemd           
udp6       0      0 :::37042                :::*                                37456/rpc.statd     
udp6       0      0 ::1:323                 :::*                                19388/chronyd       
[cephuser@ceph-amar-bg-1621928213315-node3-osd-mon-mds-client ~]$ sudo netstat -tunlp | grep 2049
tcp6       0      0 :::2049                 :::*                    LISTEN      37464/ganesha.nfsd  
udp6       0      0 :::2049                 :::*                                37464/ganesha.nfsd
Comment 9 Amarnath 2021-05-28 03:26:42 UTC 
Hi Varsha,

I have created a new setup.
I still see the same issue.

[root@ceph-amar-bg-1622134789627-node1-mon-mgr-client-installer ~]# ceph orch ps
NAME                                                                   HOST                                                       STATUS         REFRESHED  AGE  PORTS   VERSION          IMAGE ID      CONTAINER ID  
mds.cephfs.ceph-amar-bg-1622134789627-node2-mon-mgr-mds-client.oermsa  ceph-amar-bg-1622134789627-node2-mon-mgr-mds-client        running (9h)   5m ago     9h   -       16.2.0-38.el8cp  46f26486881c  77031c9ca676  
mds.cephfs.ceph-amar-bg-1622134789627-node3-osd-mon-mds-client.ikxmdo  ceph-amar-bg-1622134789627-node3-osd-mon-mds-client        running (9h)   66s ago    9h   -       16.2.0-38.el8cp  46f26486881c  c43d40f78501  
mds.cephfs.ceph-amar-bg-1622134789627-node4-osd-mds.pqxjau             ceph-amar-bg-1622134789627-node4-osd-mds                   running (9h)   2m ago     9h   -       16.2.0-38.el8cp  46f26486881c  ca559ef77606  
mds.cephfs.ceph-amar-bg-1622134789627-node5-osd-mds-client.qkkjzq      ceph-amar-bg-1622134789627-node5-osd-mds-client            running (9h)   2m ago     9h   -       16.2.0-38.el8cp  46f26486881c  32bd951a35e3  
mgr.ceph-amar-bg-1622134789627-node1-mon-mgr-client-installer.gbioue   ceph-amar-bg-1622134789627-node1-mon-mgr-client-installer  running (9h)   5m ago     9h   *:9283  16.2.0-38.el8cp  46f26486881c  c153906bc470  
mgr.ceph-amar-bg-1622134789627-node2-mon-mgr-mds-client.omjbpv         ceph-amar-bg-1622134789627-node2-mon-mgr-mds-client        running (9h)   5m ago     9h   *:8443  16.2.0-38.el8cp  46f26486881c  2e8ec03095c0  
mon.ceph-amar-bg-1622134789627-node1-mon-mgr-client-installer          ceph-amar-bg-1622134789627-node1-mon-mgr-client-installer  running (9h)   5m ago     9h   -       16.2.0-38.el8cp  46f26486881c  a9906224f894  
mon.ceph-amar-bg-1622134789627-node2-mon-mgr-mds-client                ceph-amar-bg-1622134789627-node2-mon-mgr-mds-client        running (9h)   5m ago     9h   -       16.2.0-38.el8cp  46f26486881c  690f7550bdc5  
mon.ceph-amar-bg-1622134789627-node3-osd-mon-mds-client                ceph-amar-bg-1622134789627-node3-osd-mon-mds-client        running (9h)   66s ago    9h   -       16.2.0-38.el8cp  46f26486881c  3dcab685cce7  
nfs.cephfsnfs.ceph-amar-bg-1622134789627-node6-osd-nfs                 ceph-amar-bg-1622134789627-node6-osd-nfs                   running (54m)  3m ago     54m  *:2049  3.5              46f26486881c  260e17a88a22  
nfs.cephfsnfs.ceph-amar-bg-1622134789627-node7-osd-nfs                 ceph-amar-bg-1622134789627-node7-osd-nfs                   running (54m)  3m ago     54m  *:2049  3.5              46f26486881c  b342c86fd31b  
osd.0                                                                  ceph-amar-bg-1622134789627-node7-osd-nfs                   running (9h)   3m ago     9h   -       16.2.0-38.el8cp  46f26486881c  977b2b068bdf  
osd.1                                                                  ceph-amar-bg-1622134789627-node3-osd-mon-mds-client        running (9h)   66s ago    9h   -       16.2.0-38.el8cp  46f26486881c  e73412c27b1b  
osd.10                                                                 ceph-amar-bg-1622134789627-node3-osd-mon-mds-client        running (9h)   66s ago    9h   -       16.2.0-38.el8cp  46f26486881c  29bd6cbbdfb8  
osd.11                                                                 ceph-amar-bg-1622134789627-node7-osd-nfs                   running (9h)   3m ago     9h   -       16.2.0-38.el8cp  46f26486881c  5d705f531b3e  
osd.12                                                                 ceph-amar-bg-1622134789627-node5-osd-mds-client            running (9h)   2m ago     9h   -       16.2.0-38.el8cp  46f26486881c  b4b95f858ddf  
osd.13                                                                 ceph-amar-bg-1622134789627-node4-osd-mds                   running (9h)   2m ago     9h   -       16.2.0-38.el8cp  46f26486881c  dde797e5a2fa  
osd.14                                                                 ceph-amar-bg-1622134789627-node6-osd-nfs                   running (9h)   3m ago     9h   -       16.2.0-38.el8cp  46f26486881c  89be42615861  
osd.15                                                                 ceph-amar-bg-1622134789627-node3-osd-mon-mds-client        running (9h)   66s ago    9h   -       16.2.0-38.el8cp  46f26486881c  30d3bdcff509  
osd.16                                                                 ceph-amar-bg-1622134789627-node7-osd-nfs                   running (9h)   3m ago     9h   -       16.2.0-38.el8cp  46f26486881c  2a7b4018a74a  
osd.17                                                                 ceph-amar-bg-1622134789627-node5-osd-mds-client            running (9h)   2m ago     9h   -       16.2.0-38.el8cp  46f26486881c  57be79110ee9  
osd.18                                                                 ceph-amar-bg-1622134789627-node4-osd-mds                   running (9h)   2m ago     9h   -       16.2.0-38.el8cp  46f26486881c  56e23da8cafa  
osd.19                                                                 ceph-amar-bg-1622134789627-node6-osd-nfs                   running (9h)   3m ago     9h   -       16.2.0-38.el8cp  46f26486881c  18db74173d19  
osd.2                                                                  ceph-amar-bg-1622134789627-node5-osd-mds-client            running (9h)   2m ago     9h   -       16.2.0-38.el8cp  46f26486881c  98a10a4cf7b7  
osd.3                                                                  ceph-amar-bg-1622134789627-node4-osd-mds                   running (9h)   2m ago     9h   -       16.2.0-38.el8cp  46f26486881c  7d1bfb34eb75  
osd.4                                                                  ceph-amar-bg-1622134789627-node6-osd-nfs                   running (9h)   3m ago     9h   -       16.2.0-38.el8cp  46f26486881c  9f73e0d29a7e  
osd.5                                                                  ceph-amar-bg-1622134789627-node3-osd-mon-mds-client        running (9h)   66s ago    9h   -       16.2.0-38.el8cp  46f26486881c  012ad1c1c731  
osd.6                                                                  ceph-amar-bg-1622134789627-node7-osd-nfs                   running (9h)   3m ago     9h   -       16.2.0-38.el8cp  46f26486881c  a24e8a2be866  
osd.7                                                                  ceph-amar-bg-1622134789627-node5-osd-mds-client            running (9h)   2m ago     9h   -       16.2.0-38.el8cp  46f26486881c  f8af44670440  
osd.8                                                                  ceph-amar-bg-1622134789627-node4-osd-mds                   running (9h)   2m ago     9h   -       16.2.0-38.el8cp  46f26486881c  a16a33a577eb  
osd.9                                                                  ceph-amar-bg-1622134789627-node6-osd-nfs                   running (9h)   3m ago     9h   -       16.2.0-38.el8cp  46f26486881c  eff19e42a026  
[root@ceph-amar-bg-1622134789627-node1-mon-mgr-client-installer ~]# ceph nfs cluster info cephfsnfs
{}
[root@ceph-amar-bg-1622134789627-node1-mon-mgr-client-installer ~]# 
[root@ceph-amar-bg-1622134789627-node1-mon-mgr-client-installer ~]# 
[root@ceph-amar-bg-1622134789627-node1-mon-mgr-client-installer ~]# ceph nfs cluster info cephfsnfs
{}
[root@ceph-amar-bg-1622134789627-node1-mon-mgr-client-installer ~]# ceph nfs cluster info cephfsnfs
{}
[root@ceph-amar-bg-1622134789627-node1-mon-mgr-client-installer ~]#
Comment 10 Amarnath 2021-05-28 03:32:10 UTC 
Created attachment 1787744 [details]
Manager logs
Comment 11 Varsha 2021-05-28 05:23:27 UTC 
Something is wrong with orchestrator

[cephuser@ceph-amar-bg-1622134789627-node1-mon-mgr-client-installer ~]$ sudo ceph orch ls
NAME                       RUNNING  REFRESHED  AGE  PLACEMENT                                                                          
mds.cephfs                     4/4  7m ago     11h  label:mds                                                                          
mgr                            2/2  7m ago     11h  label:mgr                                                                          
mon                            3/3  7m ago     11h  label:mon                                                                          
nfs.cephfsnfs                  2/2  5m ago     2h   ceph-amar-bg-1622134789627-node6-osd-nfs;ceph-amar-bg-1622134789627-node7-osd-nfs  
osd.all-available-devices    20/27  5m ago     11h  *                                                                                  
[cephuser@ceph-amar-bg-1622134789627-node1-mon-mgr-client-installer ~]$ sudo ceph orch ls --service_type=nfs
NAME           RUNNING  REFRESHED  AGE  PLACEMENT                                                                          
nfs.cephfsnfs      0/2  -          2h   ceph-amar-bg-1622134789627-node6-osd-nfs;ceph-amar-bg-1622134789627-node7-osd-nfs
Comment 12 Sebastian Wagner 2021-05-31 16:38:30 UTC 
Could you execute https://gist.githubusercontent.com/sebastian-philipp/8e18f4815e90dc0f51fe3fbff8c8aae5/raw/e21a3b712ab04aa3440661c8a18748ea1635f364/cephadm_status.sh and
attach the result? 

(I don't want to ask the same information again and again.)
Comment 13 Amarnath 2021-06-01 13:30:31 UTC 
Created attachment 1788509 [details]
Logs after Executing the script

Please find the attached log after executing the script given
Comment 14 Sebastian Wagner 2021-06-03 10:21:53 UTC 
Indeed! The count is super wrong. This is already tracked in #1964951  

Mind if I close this as duplicated? Is there anything else that needs to be fixed, cause according to the output, The cluster looks healthy with active NFS daemons properly running.
Comment 15 Hemanth Kumar 2021-06-03 14:00:49 UTC 
(In reply to Sebastian Wagner from comment #14)
> Indeed! The count is super wrong. This is already tracked in #1964951  
> 
> Mind if I close this as duplicated? Is there anything else that needs to be
> fixed, cause according to the output, The cluster looks healthy with active
> NFS daemons properly running.

#1964951, Is this going to be fixed in 5.0 ?
Comment 16 Sebastian Wagner 2021-06-04 11:09:43 UTC 
5.0 is in blockers-only mode and 1964951 is not a blocker
Comment 17 Sebastian Wagner 2021-06-08 11:15:51 UTC 
Marking as duplicated

*** This bug has been marked as a duplicate of bug 1964951 ***