Bug 2371418
| Summary: | [Hotfix Build] Ceph Manager Daemon Crashes During Parallel NFS Deployment and Node Reboot in Hotfix Build | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Manisha Saini <msaini> | |
| Component: | Cephadm | Assignee: | Adam King <adking> | |
| Status: | CLOSED ERRATA | QA Contact: | Manisha Saini <msaini> | |
| Severity: | high | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 8.0 | CC: | akane, bkunal, cephqe-warriors, jcaratza, tserlin | |
| Target Milestone: | --- | |||
| Target Release: | 9.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | ceph-20.1.0-14 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2372433 (view as bug list) | Environment: | ||
| Last Closed: | 2026-01-29 06:50:05 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 2394541 | |||
| Bug Blocks: | 2372433 | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Ceph Storage 9.0 Security and Enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2026:1536 |
Description of problem: ============ While deploying multiple NFS servers in parallel on a Ceph cluster and simultaneously rebooting a node, the Ceph Manager daemon crashes. This issue was observed in a hotfix build. As a result, the deployment of NFS daemons either failed or became stuck, and the ceph status reported a HEALTH_ERR state. ------------------ # ceph -s cluster: id: 5064a9a2-4252-11f0-aa51-b49691cee574 health: HEALTH_ERR Failed to apply 1 service(s): nfs.ganesha_nfs2 Module 'cephadm' has failed: local variable 'failures' referenced before assignment 1 mgr modules have recently crashed services: mon: 3 daemons, quorum cali013,cali016,cali015 (age 46m) mgr: cali013.oymilt(active, since 12h), standbys: cali020.xbroho mds: 1/1 daemons up, 1 standby osd: 28 osds: 28 up (since 46m), 28 in (since 3d) data: volumes: 1/1 healthy pools: 4 pools, 1089 pgs objects: 504 objects, 708 KiB usage: 1.6 GiB used, 69 TiB / 69 TiB avail pgs: 1089 active+clean io: client: 32 KiB/s rd, 31 op/s rd, 0 op/s wr --------------- [ceph: root@cali013 /]# ceph crash info 2025-06-10T13:06:48.512886Z_30d3b636-e625-4f38-8fa0-cfba4e2a7e23 { "backtrace": [ " File \"/usr/share/ceph/mgr/cephadm/module.py\", line 858, in serve\n serve.serve()", " File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 107, in serve\n if self._apply_all_services():", " File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 703, in _apply_all_services\n results = self.mgr.wait_async(_deploy_and_remove_all(all_conflicting_daemons, all_daemons_to_deploy, all_daemons_to_remove))", " File \"/usr/share/ceph/mgr/cephadm/module.py\", line 870, in wait_async\n return self.event_loop.get_result(coro, timeout)", " File \"/usr/share/ceph/mgr/cephadm/ssh.py\", line 136, in get_result\n return future.result(timeout)", " File \"/lib64/python3.9/concurrent/futures/_base.py\", line 446, in result\n return self.__get_result()", " File \"/lib64/python3.9/concurrent/futures/_base.py\", line 391, in __get_result\n raise self._exception", " File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 698, in _deploy_and_remove_all\n return await gather(*futures)", " File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 675, in _parallel_deploy_and_remove\n daemons_placed, daemon_deployed_hosts, daemon_place_fails = await self.deploy_given_daemons(to_deploy)", " File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 1147, in deploy_given_daemons\n for daemon_name, msg in failures.items():", "UnboundLocalError: local variable 'failures' referenced before assignment" ], "ceph_version": "19.2.0-137.5.hotfix.bz2364414.el9cp", "crash_id": "2025-06-10T13:06:48.512886Z_30d3b636-e625-4f38-8fa0-cfba4e2a7e23", "entity_name": "mgr.cali013.oymilt", "mgr_module": "cephadm", "mgr_module_caller": "PyModuleRunner::serve", "mgr_python_exception": "UnboundLocalError", "os_id": "rhel", "os_name": "Red Hat Enterprise Linux", "os_version": "9.6 (Plow)", "os_version_id": "9.6", "process_name": "ceph-mgr", "stack_sig": "f82bb7ac7866eebaa5004057f877cc3028e70349986f20918a920e5e224b5a7c", "timestamp": "2025-06-10T13:06:48.512886Z", "utsname_hostname": "cali013", "utsname_machine": "x86_64", "utsname_release": "5.14.0-503.40.1.el9_5.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Thu Apr 24 08:27:29 EDT 2025" } ---------------- [ceph: root@cali013 /]# ceph health detail HEALTH_ERR Failed to apply 1 service(s): nfs.ganesha_nfs2; Module 'cephadm' has failed: local variable 'failures' referenced before assignment; 1 mgr modules have recently crashed [WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): nfs.ganesha_nfs2 nfs.ganesha_nfs2: Command '['rados', '-n', 'mgr.cali013.oymilt', '-k', '/var/lib/ceph/mgr/ceph-cali013.oymilt/keyring', '-p', '.nfs', '--namespace', 'ganesha_nfs2', 'get', 'conf-nfs.ganesha_nfs2', '-']' timed out after 10 seconds [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: local variable 'failures' referenced before assignment Module 'cephadm' has failed: local variable 'failures' referenced before assignment [WRN] RECENT_MGR_MODULE_CRASH: 1 mgr modules have recently crashed mgr module cephadm crashed in daemon mgr.cali013.oymilt on host cali013 at 2025-06-10T13:06:48.512886Z Version-Release number of selected component (if applicable): ============= # ceph --version ceph version 19.2.0-137.5.hotfix.bz2364414.el9cp (fce75b6455600409709f669a96dfd26e932d616e) squid (stable) How reproducible: ============ 1/1 Steps to Reproduce: ================== 1. Create ceph cluster 2. Deploy 80 Ganesha clusters in parallel using spec file 3. While deployment is ongoing, reboot one of the ceph node Actual results: ========= Ceph mgr daemon crashes during parallel NFS server deployments and node reboot scenario. Deployment of NFS Cluster fail/Stuck Expected results: ========= NFS servers should be deployed successfully Node reboots should not crash the mgr Additional info: