Bug 2370257
| Summary: | [Test Build] Multiple NFS daemon deployment is failing on the Testbuild created above 8.0z5 | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Manisha Saini <msaini> |
| Component: | Cephadm | Assignee: | Adam King <adking> |
| Status: | CLOSED ERRATA | QA Contact: | Arunkumar Ravindranath <aravindr> |
| Severity: | high | Docs Contact: | Rivka Pollack <rpollack> |
| Priority: | unspecified | ||
| Version: | 8.0 | CC: | cephqe-warriors, jcaratza, mobisht, rpollack |
| Target Milestone: | --- | ||
| Target Release: | 8.1z1 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ceph-19.2.1-223 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2025-08-18 14:02:07 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Ceph Storage 8.1 security and bug fix updates), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2025:14015 |
Description of problem: ============= Build Details --> Testbuild created above 8.0z5 builds Issue --> NFS Daemon deployment is failing while deploying using spec file. # cat nfs1.yaml service_type: nfs service_id: nfs1 placement: host_pattern: '*' spec: port: 2049 --- service_type: nfs service_id: nfs2 placement: host_pattern: '*' spec: port: 2050 --- service_type: nfs service_id: nfs3 placement: host_pattern: '*' spec: port: 2051 Version-Release number of selected component (if applicable): =========== # ceph --version ceph version 19.2.0-139.el9cp (d625bfed07c649268e36061ca769a5ffc77e797a) squid (stable) # rpm -qa | grep nfs libnfsidmap-2.5.4-34.el9.x86_64 nfs-utils-2.5.4-34.el9.x86_64 nfs-ganesha-selinux-6.5-12.5.el9cp.noarch nfs-ganesha-6.5-12.5.el9cp.x86_64 nfs-ganesha-ceph-6.5-12.5.el9cp.x86_64 nfs-ganesha-rados-grace-6.5-12.5.el9cp.x86_64 nfs-ganesha-rados-urls-6.5-12.5.el9cp.x86_64 nfs-ganesha-rgw-6.5-12.5.el9cp.x86_64 nfs-ganesha-utils-6.5-12.5.el9cp.x86_64 How reproducible: ============== 1/1 Steps to Reproduce: ================= 1.Deploy multiple NFS daemon using the spec file Actual results: ============ NFS cluster deployment failed Expected results: ======== NFS cluster deployment should pass Additional info: =============== ceph mgr logs Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-mgr[77345]: [cephadm ERROR root] Hit an exception preparing daemons for creation/removal: 1 Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: 2025-06-04T14:42:43.130+0000 7fea2c5b1640 -1 log_channel(cephadm) log [ERR] : Hit an exception preparing daemons for creation/removal: 1 Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-mgr[77345]: log_channel(cephadm) log [ERR] : Hit an exception preparing daemons for creation/removal: 1 Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-mgr[77345]: [cephadm ERROR cephadm.serve] Failed to apply nfs.nfs3 spec NFSServiceSpec.from_json(yaml.safe_load('''service_type: nfs service_id: nfs3 service_name: nfs.nfs3 placement: host_pattern: '*' spec: port: 2051 ''')): 1 Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 639, in _apply_all_services conflicting_daemons, daemons_to_deploy, daemons_to_remove = self.prepare_daemons_to_add_and_remove_by_service(spec) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in prepare_daemons_to_add_and_remove_by_service slots_to_add = self.handle_slot_names_and_rank_map_for_service(spec) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1005, in handle_slot_names_and_rank_map_for_service assert rank_map[slot.rank][slot.rank_generation] is None KeyError: 1 Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: 2025-06-04T14:42:43.131+0000 7fea2c5b1640 -1 log_channel(cephadm) log [ERR] : Failed to apply nfs.nfs3 spec NFSServiceSpec.from_json(yaml.safe_load('''service_type: nfs Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: service_id: nfs3 Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: service_name: nfs.nfs3 Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: placement: Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: host_pattern: '*' Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: spec: Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: port: 2051 Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: ''')): 1 Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: Traceback (most recent call last): Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 639, in _apply_all_services Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: conflicting_daemons, daemons_to_deploy, daemons_to_remove = self.prepare_daemons_to_add_and_remove_by_service(spec) Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in prepare_daemons_to_add_and_remove_by_service Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: slots_to_add = self.handle_slot_names_and_rank_map_for_service(spec) Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1005, in handle_slot_names_and_rank_map_for_service Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: assert rank_map[slot.rank][slot.rank_generation] is None Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-b61d89de-412c-11f0-943d-fa163eea55b5-mgr-ceph-manisaini-y08wfj-node1-installer-bngtsx[77341]: KeyError: 1 Jun 04 14:42:43 ceph-manisaini-y08wfj-node1-installer ceph-mgr[77345]: log_channel(cephadm) log [ERR] : Failed to apply nfs.nfs3 spec NFSServiceSpec.from_json(yaml.safe_load('''service_type: nfs service_id: nfs3 service_name: nfs.nfs3 placement: host_pattern: '*' spec: port: 2051 ''')): 1 Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 639, in _apply_all_services conflicting_daemons, daemons_to_deploy, daemons_to_remove = self.prepare_daemons_to_add_and_remove_by_service(spec) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1168, in prepare_daemons_to_add_and_remove_by_service slots_to_add = self.handle_slot_names_and_rank_map_for_service(spec) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1005, in handle_slot_names_and_rank_map_for_service assert rank_map[slot.rank][slot.rank_generation] is None KeyError: 1