Created attachment 1990938 [details] Notification_regarding_daemon_operations_from_notification_bar Description of problem: ======================== On a IBM Ceph Storage 7.0 cluster with NFS Ganesha with HA configured, when the daemons for NFS Ganesha, HAproxy and Keepalive were redeployed using Ceph dashboard, the "Daemon Events" on the dashboard did not reflect the info/update of the operation. Version-Release number of selected component (if applicable): ============================================================= IBM 7.0 - 18.2.0-20.el9cp # dnf info cephadm Updating Subscription Management repositories. IBM-CEPH-7.0-20230911.ci.0 4.2 kB/s | 3.0 kB 00:00 Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) 31 MB/s | 25 MB 00:00 Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) 6.8 MB/s | 14 MB 00:02 Installed Packages Name : cephadm Epoch : 2 Version : 18.2.0 Release : 20.el9cp Architecture : noarch Size : 227 k Source : cephadm-18.2.0-20.el9cp.src.rpm Repository : @System From repo : IBM-CEPH-7.0-20230911.ci.0 Summary : Utility to bootstrap Ceph clusters URL : https://ceph.io License : LGPL-2.1 Description : Utility to bootstrap a Ceph cluster and manage Ceph daemons deployed : with systemd and podman. How reproducible: ======================== Always Steps to Reproduce: ======================== 1. Setup a NFS Ganesha cluster by following the steps in https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/6/html-single/operations_guide/index#creating-the-nfs-ganesha-cluster-using-the-ceph-orchestrator_ops 2. Deploy HA using steps in https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/6/html-single/operations_guide/index#deploying-ha-for-cephfs-nfs-using-a-specification-file_ops 3. Once the ingress service is up and running, check if the daemons for the service using #ceph orch ps command 4. Go to the Ceph Dashboard > Services and try to redeploy the daemons for NFS Ganesha service and Haproxy and Keepalive (under ingress service) Actual results: ======================== When the daemon were redeployed, there was a notification generated that daemon redeploy has been scheduled (screenshot attached), but the "Daemon events" did not reflect the same and no update was reflected either. (please check the attached screenshot) When same operation was performed for Grafana service daemon, the current update was reflected under "Daemon events" (please check attached screenshot) Expected results: ======================== Any daemon event for all the service daemons should be reflected correctly. Additional info: ================ # ceph orch ps | grep nfs haproxy.nfs.nfsganesha.clara004.qozngr clara004 *:2050,9000 running (16m) - 3h 8544k - <unknown> 2bb64a680c36 181f3f147397 haproxy.nfs.nfsganesha.clara006.kfkbih clara006 *:2050,9000 running (3h) - 3h 8854k - <unknown> 2bb64a680c36 39f5a4fd3bcd keepalived.nfs.nfsganesha.clara004.tcompt clara004 running (13m) - 4h 1765k - 2.2.4 5366a49ebd2b 5dbd17fb91dd keepalived.nfs.nfsganesha.clara006.ogmxmd clara006 running (3h) - 4h 1765k - 2.2.4 5366a49ebd2b 670d9d7cb617 nfs.nfsganesha.0.0.clara004.onkrxm clara004 *:2049 running (13m) - 4h 79.5M - 5.1 48c75ff3dd69 bac199ea4417 nfs.nfsganesha.1.0.clara005.bteggj clara005 *:2049 running (15m) - 4h 79.0M - 5.1 48c75ff3dd69 83fffe6a4a2d nfs.nfsganesha.2.0.clara006.pkmybx clara006 *:2049 running (22m) - 4h 78.3M - 5.1 48c75ff3dd69 1be1a5eb8aab [ceph: root@clara001 /]# ceph nfs cluster info nfsganesha { "nfsganesha": { "backend": [ { "hostname": "clara004", "ip": "10.8.129.4", "port": 2049 }, { "hostname": "clara005", "ip": "10.8.129.5", "port": 2049 }, { "hostname": "clara006", "ip": "10.8.129.6", "port": 2049 } ], "monitor_port": 9000, "port": 2050, "virtual_ip": "10.8.129.26" } } [ceph: root@clara001 /]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 3/3 - 6d count:3 ceph-exporter 5/5 - 2w * grafana ?:3000 1/1 - 2w count:1 ingress.nfs.nfsganesha 10.8.129.26:2050,9000 4/4 - 3h count:2 mds.cephfs 2/2 - 2w label:mds mds.cephfs_io_1 2/2 - 3d count:2 mds.cephfs_io_2 1/1 - 2d count:1 mgr 2/2 - 2w label:mgr mon 2/2 - 2w label:mon nfs.nfsganesha ?:2049 2/3 - 4h clara004;clara005;clara006 node-exporter ?:9100 3/3 - 6d count:3 osd.all-available-devices 15 - 6d * prometheus ?:9095 1/1 - 2w count:1 rgw.rgw.1 ?:80 3/3 - 2w label:rgw [ceph: root@clara001 /]# ceph -s cluster: id: 9fe64b64-5214-11ee-9edb-0cc47a6c6944 health: HEALTH_WARN 3 nearfull osd(s) 17 pool(s) nearfull 1 pool(s) have non-power-of-two pg_num services: mon: 2 daemons, quorum clara001,clara003 (age 6d) mgr: clara001.pzjuid(active, since 6d), standbys: clara003.bhyudz mds: 3/3 daemons up, 2 standby osd: 15 osds: 15 up (since 6d), 15 in (since 6d) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 3/3 healthy pools: 17 pools, 605 pgs objects: 6.26M objects, 830 GiB usage: 2.5 TiB used, 742 GiB / 3.2 TiB avail pgs: 605 active+clean io: client: 340 B/s rd, 0 op/s rd, 0 op/s wr Cephadm log snippet- 2023-09-28T08:57:53.374487+0000 mgr.clara001.pzjuid (mgr.74516) 300906 : cephadm [INF] Schedule stop daemon nfs.nfsganesha.2.0.clara006.pkmybx 2023-09-28T09:37:22.021441+0000 mgr.clara001.pzjuid (mgr.74516) 304094 : cephadm [INF] Schedule stop daemon nfs.nfsganesha.1.0.clara005.bteggj 2023-09-28T09:55:47.829774+0000 mgr.clara001.pzjuid (mgr.74516) 306013 : cephadm [INF] Schedule start daemon nfs.nfsganesha.1.0.clara005.bteggj 2023-09-28T09:56:59.952779+0000 mgr.clara001.pzjuid (mgr.74516) 306158 : cephadm [INF] Schedule start daemon nfs.nfsganesha.2.0.clara006.pkmybx 2023-09-28T09:57:58.103908+0000 mgr.clara001.pzjuid (mgr.74516) 306265 : cephadm [INF] Schedule redeploy daemon haproxy.nfs.nfsganesha.clara004.qozngr 2023-09-28T09:58:05.052203+0000 mgr.clara001.pzjuid (mgr.74516) 306278 : cephadm [INF] Deploying daemon haproxy.nfs.nfsganesha.clara004.qozngr on clara004 2023-09-28T10:00:00.265256+0000 mgr.clara001.pzjuid (mgr.74516) 306492 : cephadm [INF] Schedule redeploy daemon keepalived.nfs.nfsganesha.clara004.tcompt 2023-09-28T10:00:08.102178+0000 mgr.clara001.pzjuid (mgr.74516) 306501 : cephadm [INF] 10.8.129.26 is in 10.8.128.0/21 on clara004 interface eno1 2023-09-28T10:00:08.102383+0000 mgr.clara001.pzjuid (mgr.74516) 306502 : cephadm [INF] 10.8.129.26 is in 10.8.128.0/21 on clara006 interface eno1 2023-09-28T10:00:08.103049+0000 mgr.clara001.pzjuid (mgr.74516) 306503 : cephadm [INF] Deploying daemon keepalived.nfs.nfsganesha.clara004.tcompt on clara004 2023-09-28T10:02:14.165784+0000 mgr.clara001.pzjuid (mgr.74516) 306735 : cephadm [INF] Schedule restart daemon keepalived.nfs.nfsganesha.clara004.tcompt 2023-09-28T10:02:48.021856+0000 mgr.clara001.pzjuid (mgr.74516) 306793 : cephadm [INF] Schedule redeploy daemon haproxy.nfs.nfsganesha.clara004.qozngr 2023-09-28T10:02:55.901585+0000 mgr.clara001.pzjuid (mgr.74516) 306822 : cephadm [INF] Deploying daemon haproxy.nfs.nfsganesha.clara004.qozngr on clara004 2023-09-28T10:04:07.933816+0000 mgr.clara001.pzjuid (mgr.74516) 306947 : cephadm [INF] Schedule redeploy daemon nfs.nfsganesha.1.0.clara005.bteggj 2023-09-28T10:04:20.246228+0000 mgr.clara001.pzjuid (mgr.74516) 306974 : cephadm [INF] Creating key for client.nfs.nfsganesha.1.0.clara005.bteggj 2023-09-28T10:04:20.247629+0000 mgr.clara001.pzjuid (mgr.74516) 306975 : cephadm [INF] Ensuring nfs.nfsganesha.1 is in the ganesha grace table 2023-09-28T10:04:20.456148+0000 mgr.clara001.pzjuid (mgr.74516) 306976 : cephadm [INF] Rados config object exists: conf-nfs.nfsganesha 2023-09-28T10:04:20.456366+0000 mgr.clara001.pzjuid (mgr.74516) 306977 : cephadm [INF] Creating key for client.nfs.nfsganesha.1.0.clara005.bteggj-rgw 2023-09-28T10:04:20.458069+0000 mgr.clara001.pzjuid (mgr.74516) 306978 : cephadm [WRN] Bind address in nfs.nfsganesha.1.0.clara005.bteggj's ganesha conf is defaulting to empty 2023-09-28T10:04:20.461215+0000 mgr.clara001.pzjuid (mgr.74516) 306979 : cephadm [INF] Deploying daemon nfs.nfsganesha.1.0.clara005.bteggj on clara005 2023-09-28T10:04:51.117432+0000 mgr.clara001.pzjuid (mgr.74516) 307041 : cephadm [INF] Schedule redeploy daemon grafana.clara001 2023-09-28T10:04:58.715028+0000 mgr.clara001.pzjuid (mgr.74516) 307052 : cephadm [INF] Deploying daemon grafana.clara001 on clara001 2023-09-28T10:05:40.310482+0000 mgr.clara001.pzjuid (mgr.74516) 307127 : cephadm [INF] Schedule redeploy daemon keepalived.nfs.nfsganesha.clara004.tcompt 2023-09-28T10:05:48.429046+0000 mgr.clara001.pzjuid (mgr.74516) 307136 : cephadm [INF] 10.8.129.26 is in 10.8.128.0/21 on clara004 interface eno1 2023-09-28T10:05:48.429235+0000 mgr.clara001.pzjuid (mgr.74516) 307137 : cephadm [INF] 10.8.129.26 is in 10.8.128.0/21 on clara006 interface eno1 2023-09-28T10:05:48.429914+0000 mgr.clara001.pzjuid (mgr.74516) 307138 : cephadm [INF] Deploying daemon keepalived.nfs.nfsganesha.clara004.tcompt on clara004 2023-09-28T10:06:33.432102+0000 mgr.clara001.pzjuid (mgr.74516) 307230 : cephadm [INF] Schedule redeploy daemon nfs.nfsganesha.0.0.clara004.onkrxm 2023-09-28T10:06:40.362459+0000 mgr.clara001.pzjuid (mgr.74516) 307242 : cephadm [INF] Creating key for client.nfs.nfsganesha.0.0.clara004.onkrxm 2023-09-28T10:06:40.363877+0000 mgr.clara001.pzjuid (mgr.74516) 307243 : cephadm [INF] Ensuring nfs.nfsganesha.0 is in the ganesha grace table 2023-09-28T10:06:40.572102+0000 mgr.clara001.pzjuid (mgr.74516) 307244 : cephadm [INF] Rados config object exists: conf-nfs.nfsganesha 2023-09-28T10:06:40.572297+0000 mgr.clara001.pzjuid (mgr.74516) 307245 : cephadm [INF] Creating key for client.nfs.nfsganesha.0.0.clara004.onkrxm-rgw 2023-09-28T10:06:40.573811+0000 mgr.clara001.pzjuid (mgr.74516) 307246 : cephadm [WRN] Bind address in nfs.nfsganesha.0.0.clara004.onkrxm's ganesha conf is defaulting to empty 2023-09-28T10:06:40.576748+0000 mgr.clara001.pzjuid (mgr.74516) 307247 : cephadm [INF] Deploying daemon nfs.nfsganesha.0.0.clara004.onkrxm on clara004
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Critical: Red Hat Ceph Storage 7.1 security, enhancements, and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:3925