Bug 2365926
| Summary: | [cephadm][rgw-ssl]: "ceph orch ps --refresh" is triggering a daemon reconfig multiple times | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Hemanth Sai <hmaheswa> |
| Component: | Cephadm | Assignee: | Adam King <adking> |
| Status: | CLOSED ERRATA | QA Contact: | Hemanth Sai <hmaheswa> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.1 | CC: | cephqe-warriors, mobisht, tserlin |
| Target Milestone: | --- | ||
| Target Release: | 8.1 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ceph-19.2.1-193.el9cp | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2025-06-26 12:31:50 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2025:9775 |
Description of problem: "ceph orch ps --refresh" is triggering a daemon reconfig multiple times this is causing restart not to happen and taking longer time to restart ######################################################################## screeen1: [cephuser@ceph-hsm-ssl-81-4yge7h-node6 ~]$ [cephuser@ceph-hsm-ssl-81-4yge7h-node6 ~]$ ceph orch ps --refresh NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID alertmanager.ceph-hsm-ssl-81-4yge7h-node5 ceph-hsm-ssl-81-4yge7h-node5 *:9093,9094 running (4d) 7m ago 4d 26.9M - 0.26.0 9ad09e138a1c e5e5b201d163 crash.ceph-hsm-ssl-81-4yge7h-node1-installer ceph-hsm-ssl-81-4yge7h-node1-installer running (5d) 84s ago 5d 6612k - 19.2.1-184.el9cp 9682b3d45b47 98ea045628d0 crash.ceph-hsm-ssl-81-4yge7h-node2 ceph-hsm-ssl-81-4yge7h-node2 running (5d) 3m ago 5d 6580k - 19.2.1-184.el9cp 9682b3d45b47 957fbccc21d1 crash.ceph-hsm-ssl-81-4yge7h-node3 ceph-hsm-ssl-81-4yge7h-node3 running (5d) 3m ago 5d 6576k - 19.2.1-184.el9cp 9682b3d45b47 f8d189da450d crash.ceph-hsm-ssl-81-4yge7h-node4 ceph-hsm-ssl-81-4yge7h-node4 running (5d) 3m ago 5d 6576k - 19.2.1-184.el9cp 9682b3d45b47 8ce46a288ef3 crash.ceph-hsm-ssl-81-4yge7h-node5 ceph-hsm-ssl-81-4yge7h-node5 running (4d) 7m ago 5d 6572k - 19.2.1-184.el9cp 9682b3d45b47 9148e99822e5 grafana.ceph-hsm-ssl-81-4yge7h-node1-installer ceph-hsm-ssl-81-4yge7h-node1-installer *:3000 running (4d) 84s ago 4d 88.3M - 10.4.16 9f56f82decdb e63e52e3442e mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo ceph-hsm-ssl-81-4yge7h-node1-installer *:9283,8765 running (5d) 84s ago 5d 673M - 19.2.1-184.el9cp 9682b3d45b47 39f5b656270d mgr.ceph-hsm-ssl-81-4yge7h-node2.qqikyi ceph-hsm-ssl-81-4yge7h-node2 *:8443,8765 running (5d) 3m ago 5d 443M - 19.2.1-184.el9cp 9682b3d45b47 26629c03fb8e mon.ceph-hsm-ssl-81-4yge7h-node1-installer ceph-hsm-ssl-81-4yge7h-node1-installer running (5d) 84s ago 5d 432M 2048M 19.2.1-184.el9cp 9682b3d45b47 4ccd4d5f4022 mon.ceph-hsm-ssl-81-4yge7h-node2 ceph-hsm-ssl-81-4yge7h-node2 running (5d) 3m ago 5d 420M 2048M 19.2.1-184.el9cp 9682b3d45b47 1f3458dd9e36 mon.ceph-hsm-ssl-81-4yge7h-node3 ceph-hsm-ssl-81-4yge7h-node3 running (5d) 3m ago 5d 422M 2048M 19.2.1-184.el9cp 9682b3d45b47 48626df7803a node-exporter.ceph-hsm-ssl-81-4yge7h-node1-installer ceph-hsm-ssl-81-4yge7h-node1-installer *:9100 running (4d) 84s ago 5d 18.4M - 1.7.0 f22df3572881 49fc82a804ca node-exporter.ceph-hsm-ssl-81-4yge7h-node2 ceph-hsm-ssl-81-4yge7h-node2 *:9100 running (4d) 3m ago 5d 16.3M - 1.7.0 f22df3572881 21b280337eef node-exporter.ceph-hsm-ssl-81-4yge7h-node3 ceph-hsm-ssl-81-4yge7h-node3 *:9100 running (4d) 3m ago 5d 15.6M - 1.7.0 f22df3572881 9334047bda74 node-exporter.ceph-hsm-ssl-81-4yge7h-node4 ceph-hsm-ssl-81-4yge7h-node4 *:9100 running (4d) 3m ago 5d 15.8M - 1.7.0 f22df3572881 10c21a9ba8ad node-exporter.ceph-hsm-ssl-81-4yge7h-node5 ceph-hsm-ssl-81-4yge7h-node5 *:9100 running (4d) 7m ago 5d 15.8M - 1.7.0 f22df3572881 9ce5c4378888 osd.0 ceph-hsm-ssl-81-4yge7h-node4 running (5d) 3m ago 5d 118M 4096M 19.2.1-184.el9cp 9682b3d45b47 e0cef315241a osd.1 ceph-hsm-ssl-81-4yge7h-node3 running (5d) 3m ago 5d 123M 4096M 19.2.1-184.el9cp 9682b3d45b47 07c19125b63e osd.2 ceph-hsm-ssl-81-4yge7h-node2 running (5d) 3m ago 5d 98.6M 4096M 19.2.1-184.el9cp 9682b3d45b47 2643ac4bedfe osd.3 ceph-hsm-ssl-81-4yge7h-node1-installer running (5d) 84s ago 5d 117M 4096M 19.2.1-184.el9cp 9682b3d45b47 d8584bdb4772 osd.4 ceph-hsm-ssl-81-4yge7h-node4 running (5d) 3m ago 5d 134M 4096M 19.2.1-184.el9cp 9682b3d45b47 40111ebf96b7 osd.5 ceph-hsm-ssl-81-4yge7h-node2 running (5d) 3m ago 5d 120M 4096M 19.2.1-184.el9cp 9682b3d45b47 d975932fc028 osd.6 ceph-hsm-ssl-81-4yge7h-node3 running (5d) 3m ago 5d 138M 4096M 19.2.1-184.el9cp 9682b3d45b47 0408cf679ac0 osd.7 ceph-hsm-ssl-81-4yge7h-node1-installer running (5d) 84s ago 5d 122M 4096M 19.2.1-184.el9cp 9682b3d45b47 e3dceab7199d osd.8 ceph-hsm-ssl-81-4yge7h-node1-installer running (5d) 84s ago 5d 124M 4096M 19.2.1-184.el9cp 9682b3d45b47 c9772169ce08 osd.9 ceph-hsm-ssl-81-4yge7h-node3 running (5d) 3m ago 5d 115M 4096M 19.2.1-184.el9cp 9682b3d45b47 ca5accff19dd osd.10 ceph-hsm-ssl-81-4yge7h-node2 running (5d) 3m ago 5d 115M 4096M 19.2.1-184.el9cp 9682b3d45b47 fe469b121be1 osd.11 ceph-hsm-ssl-81-4yge7h-node4 running (5d) 3m ago 5d 145M 4096M 19.2.1-184.el9cp 9682b3d45b47 ab769ce63b8f osd.12 ceph-hsm-ssl-81-4yge7h-node1-installer running (5d) 84s ago 5d 102M 4096M 19.2.1-184.el9cp 9682b3d45b47 acb32e8e1760 osd.13 ceph-hsm-ssl-81-4yge7h-node2 running (5d) 3m ago 5d 150M 4096M 19.2.1-184.el9cp 9682b3d45b47 fdef3286356d osd.14 ceph-hsm-ssl-81-4yge7h-node3 running (5d) 3m ago 5d 116M 4096M 19.2.1-184.el9cp 9682b3d45b47 680bbbc494ca osd.15 ceph-hsm-ssl-81-4yge7h-node4 running (5d) 3m ago 5d 117M 4096M 19.2.1-184.el9cp 9682b3d45b47 eb9d7de88427 prometheus.ceph-hsm-ssl-81-4yge7h-node1-installer ceph-hsm-ssl-81-4yge7h-node1-installer *:9095 running (4d) 84s ago 4d 88.5M - 2.48.0 27bb8acd48b0 0e419edc108a rgw.rgw.2.ceph-hsm-ssl-81-4yge7h-node5.qaiwjb ceph-hsm-ssl-81-4yge7h-node5 *:80 running (7h) 7m ago 7h 87.3M - 19.2.1-184.el9cp 9682b3d45b47 f06ee9bf9cd8 rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj ceph-hsm-ssl-81-4yge7h-node5 *:443 running (4d) 7m ago 4d 5500M - 19.2.1-184.el9cp 9682b3d45b47 92d4927c4e00 rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne ceph-hsm-ssl-81-4yge7h-node5 *:8081 running (8h) 7m ago 8h 89.0M - 19.2.1-184.el9cp 9682b3d45b47 2345b70bfa06 [cephuser@ceph-hsm-ssl-81-4yge7h-node6 ~]$ [cephuser@ceph-hsm-ssl-81-4yge7h-node6 ~]$ [cephuser@ceph-hsm-ssl-81-4yge7h-node6 ~]$ [cephuser@ceph-hsm-ssl-81-4yge7h-node6 ~]$ ceph orch ps --refresh NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID alertmanager.ceph-hsm-ssl-81-4yge7h-node5 ceph-hsm-ssl-81-4yge7h-node5 *:9093,9094 running (4d) 23s ago 4d 26.9M - 0.26.0 9ad09e138a1c e5e5b201d163 crash.ceph-hsm-ssl-81-4yge7h-node1-installer ceph-hsm-ssl-81-4yge7h-node1-installer running (5d) 23s ago 5d 6612k - 19.2.1-184.el9cp 9682b3d45b47 98ea045628d0 crash.ceph-hsm-ssl-81-4yge7h-node2 ceph-hsm-ssl-81-4yge7h-node2 running (5d) 23s ago 5d 6580k - 19.2.1-184.el9cp 9682b3d45b47 957fbccc21d1 crash.ceph-hsm-ssl-81-4yge7h-node3 ceph-hsm-ssl-81-4yge7h-node3 running (5d) 23s ago 5d 6576k - 19.2.1-184.el9cp 9682b3d45b47 f8d189da450d crash.ceph-hsm-ssl-81-4yge7h-node4 ceph-hsm-ssl-81-4yge7h-node4 running (5d) 23s ago 5d 6576k - 19.2.1-184.el9cp 9682b3d45b47 8ce46a288ef3 crash.ceph-hsm-ssl-81-4yge7h-node5 ceph-hsm-ssl-81-4yge7h-node5 running (4d) 23s ago 5d 6572k - 19.2.1-184.el9cp 9682b3d45b47 9148e99822e5 grafana.ceph-hsm-ssl-81-4yge7h-node1-installer ceph-hsm-ssl-81-4yge7h-node1-installer *:3000 running (4d) 23s ago 4d 88.6M - 10.4.16 9f56f82decdb e63e52e3442e mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo ceph-hsm-ssl-81-4yge7h-node1-installer *:9283,8765 running (5d) 23s ago 5d 673M - 19.2.1-184.el9cp 9682b3d45b47 39f5b656270d mgr.ceph-hsm-ssl-81-4yge7h-node2.qqikyi ceph-hsm-ssl-81-4yge7h-node2 *:8443,8765 running (5d) 23s ago 5d 443M - 19.2.1-184.el9cp 9682b3d45b47 26629c03fb8e mon.ceph-hsm-ssl-81-4yge7h-node1-installer ceph-hsm-ssl-81-4yge7h-node1-installer running (5d) 23s ago 5d 434M 2048M 19.2.1-184.el9cp 9682b3d45b47 4ccd4d5f4022 mon.ceph-hsm-ssl-81-4yge7h-node2 ceph-hsm-ssl-81-4yge7h-node2 running (5d) 23s ago 5d 419M 2048M 19.2.1-184.el9cp 9682b3d45b47 1f3458dd9e36 mon.ceph-hsm-ssl-81-4yge7h-node3 ceph-hsm-ssl-81-4yge7h-node3 running (5d) 23s ago 5d 423M 2048M 19.2.1-184.el9cp 9682b3d45b47 48626df7803a node-exporter.ceph-hsm-ssl-81-4yge7h-node1-installer ceph-hsm-ssl-81-4yge7h-node1-installer *:9100 running (4d) 23s ago 5d 18.5M - 1.7.0 f22df3572881 49fc82a804ca node-exporter.ceph-hsm-ssl-81-4yge7h-node2 ceph-hsm-ssl-81-4yge7h-node2 *:9100 running (4d) 23s ago 5d 16.5M - 1.7.0 f22df3572881 21b280337eef node-exporter.ceph-hsm-ssl-81-4yge7h-node3 ceph-hsm-ssl-81-4yge7h-node3 *:9100 running (4d) 23s ago 5d 15.6M - 1.7.0 f22df3572881 9334047bda74 node-exporter.ceph-hsm-ssl-81-4yge7h-node4 ceph-hsm-ssl-81-4yge7h-node4 *:9100 running (4d) 23s ago 5d 15.5M - 1.7.0 f22df3572881 10c21a9ba8ad node-exporter.ceph-hsm-ssl-81-4yge7h-node5 ceph-hsm-ssl-81-4yge7h-node5 *:9100 running (4d) 23s ago 5d 15.8M - 1.7.0 f22df3572881 9ce5c4378888 osd.0 ceph-hsm-ssl-81-4yge7h-node4 running (5d) 23s ago 5d 118M 4096M 19.2.1-184.el9cp 9682b3d45b47 e0cef315241a osd.1 ceph-hsm-ssl-81-4yge7h-node3 running (5d) 23s ago 5d 123M 4096M 19.2.1-184.el9cp 9682b3d45b47 07c19125b63e osd.2 ceph-hsm-ssl-81-4yge7h-node2 running (5d) 23s ago 5d 98.6M 4096M 19.2.1-184.el9cp 9682b3d45b47 2643ac4bedfe osd.3 ceph-hsm-ssl-81-4yge7h-node1-installer running (5d) 23s ago 5d 117M 4096M 19.2.1-184.el9cp 9682b3d45b47 d8584bdb4772 osd.4 ceph-hsm-ssl-81-4yge7h-node4 running (5d) 23s ago 5d 134M 4096M 19.2.1-184.el9cp 9682b3d45b47 40111ebf96b7 osd.5 ceph-hsm-ssl-81-4yge7h-node2 running (5d) 23s ago 5d 120M 4096M 19.2.1-184.el9cp 9682b3d45b47 d975932fc028 osd.6 ceph-hsm-ssl-81-4yge7h-node3 running (5d) 23s ago 5d 138M 4096M 19.2.1-184.el9cp 9682b3d45b47 0408cf679ac0 osd.7 ceph-hsm-ssl-81-4yge7h-node1-installer running (5d) 23s ago 5d 122M 4096M 19.2.1-184.el9cp 9682b3d45b47 e3dceab7199d osd.8 ceph-hsm-ssl-81-4yge7h-node1-installer running (5d) 23s ago 5d 124M 4096M 19.2.1-184.el9cp 9682b3d45b47 c9772169ce08 osd.9 ceph-hsm-ssl-81-4yge7h-node3 running (5d) 23s ago 5d 115M 4096M 19.2.1-184.el9cp 9682b3d45b47 ca5accff19dd osd.10 ceph-hsm-ssl-81-4yge7h-node2 running (5d) 23s ago 5d 115M 4096M 19.2.1-184.el9cp 9682b3d45b47 fe469b121be1 osd.11 ceph-hsm-ssl-81-4yge7h-node4 running (5d) 23s ago 5d 145M 4096M 19.2.1-184.el9cp 9682b3d45b47 ab769ce63b8f osd.12 ceph-hsm-ssl-81-4yge7h-node1-installer running (5d) 23s ago 5d 102M 4096M 19.2.1-184.el9cp 9682b3d45b47 acb32e8e1760 osd.13 ceph-hsm-ssl-81-4yge7h-node2 running (5d) 23s ago 5d 150M 4096M 19.2.1-184.el9cp 9682b3d45b47 fdef3286356d osd.14 ceph-hsm-ssl-81-4yge7h-node3 running (5d) 23s ago 5d 116M 4096M 19.2.1-184.el9cp 9682b3d45b47 680bbbc494ca osd.15 ceph-hsm-ssl-81-4yge7h-node4 running (5d) 23s ago 5d 117M 4096M 19.2.1-184.el9cp 9682b3d45b47 eb9d7de88427 prometheus.ceph-hsm-ssl-81-4yge7h-node1-installer ceph-hsm-ssl-81-4yge7h-node1-installer *:9095 running (4d) 23s ago 4d 89.2M - 2.48.0 27bb8acd48b0 0e419edc108a rgw.rgw.2.ceph-hsm-ssl-81-4yge7h-node5.qaiwjb ceph-hsm-ssl-81-4yge7h-node5 *:80 running (7h) 23s ago 7h 87.3M - 19.2.1-184.el9cp 9682b3d45b47 f06ee9bf9cd8 rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj ceph-hsm-ssl-81-4yge7h-node5 *:443 running (4d) 23s ago 4d 5500M - 19.2.1-184.el9cp 9682b3d45b47 92d4927c4e00 rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne ceph-hsm-ssl-81-4yge7h-node5 *:8081 running (8h) 23s ago 8h 89.0M - 19.2.1-184.el9cp 9682b3d45b47 2345b70bfa06 [cephuser@ceph-hsm-ssl-81-4yge7h-node6 ~]$ [cephuser@ceph-hsm-ssl-81-4yge7h-node6 ~]$ ------------------------------------------------------------------ screen2: [root@ceph-hsm-ssl-81-4yge7h-node5 ~]# ceph -W cephadm cluster: id: dad04558-2c07-11f0-9f0e-fa163e4a0aa7 health: HEALTH_WARN Detected 2 cephadm certificate(s) issues: 2 expiring Module 'volumes' has failed dependency: unexpected indent (module.py, line 770) services: mon: 3 daemons, quorum ceph-hsm-ssl-81-4yge7h-node1-installer,ceph-hsm-ssl-81-4yge7h-node3,ceph-hsm-ssl-81-4yge7h-node2 (age 5d) mgr: ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo(active, since 5d), standbys: ceph-hsm-ssl-81-4yge7h-node2.qqikyi osd: 16 osds: 16 up (since 5d), 16 in (since 5d) rgw: 2 daemons active (1 hosts, 1 zones) data: pools: 8 pools, 705 pgs objects: 617 objects, 785 KiB usage: 1.3 GiB used, 239 GiB / 240 GiB avail pgs: 705 active+clean 2025-05-13T14:23:45.498053+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj deps ['680ce4fd8c18cbdcad2d55efeb61dc98'] -> ['ssl-cert:680ce4fd8c18cbdcad2d55efeb61dc98'] (diff {'680ce4fd8c18cbdcad2d55efeb61dc98', 'ssl-cert:680ce4fd8c18cbdcad2d55efeb61dc98'}) 2025-05-13T14:23:45.501202+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring daemon rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj on ceph-hsm-ssl-81-4yge7h-node5 2025-05-13T14:23:46.658968+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne deps ['05c7c06e6b57904eccbc4046a20ad349'] -> ['ssl-cert:05c7c06e6b57904eccbc4046a20ad349'] (diff {'ssl-cert:05c7c06e6b57904eccbc4046a20ad349', '05c7c06e6b57904eccbc4046a20ad349'}) 2025-05-13T14:23:46.662140+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring daemon rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne on ceph-hsm-ssl-81-4yge7h-node5 2025-05-13T14:23:48.346637+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Adjusting osd_memory_target on ceph-hsm-ssl-81-4yge7h-node3 to 98208k 2025-05-13T14:23:48.347528+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [WRN] Unable to set osd_memory_target on ceph-hsm-ssl-81-4yge7h-node3 to 100565401: error parsing value: Value '100565401' is below minimum 939524096 2025-05-13T14:23:48.347750+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Adjusting osd_memory_target on ceph-hsm-ssl-81-4yge7h-node4 to 351.9M 2025-05-13T14:23:48.348298+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [WRN] Unable to set osd_memory_target on ceph-hsm-ssl-81-4yge7h-node4 to 369000140: error parsing value: Value '369000140' is below minimum 939524096 2025-05-13T14:23:48.482764+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj deps ['680ce4fd8c18cbdcad2d55efeb61dc98'] -> ['ssl-cert:680ce4fd8c18cbdcad2d55efeb61dc98'] (diff {'680ce4fd8c18cbdcad2d55efeb61dc98', 'ssl-cert:680ce4fd8c18cbdcad2d55efeb61dc98'}) 2025-05-13T14:23:48.485559+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring daemon rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj on ceph-hsm-ssl-81-4yge7h-node5 2025-05-13T14:23:49.453686+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne deps ['05c7c06e6b57904eccbc4046a20ad349'] -> ['ssl-cert:05c7c06e6b57904eccbc4046a20ad349'] (diff {'ssl-cert:05c7c06e6b57904eccbc4046a20ad349', '05c7c06e6b57904eccbc4046a20ad349'}) 2025-05-13T14:23:49.456320+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring daemon rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne on ceph-hsm-ssl-81-4yge7h-node5 2025-05-13T14:24:10.631280+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj deps ['680ce4fd8c18cbdcad2d55efeb61dc98'] -> ['ssl-cert:680ce4fd8c18cbdcad2d55efeb61dc98'] (diff {'680ce4fd8c18cbdcad2d55efeb61dc98', 'ssl-cert:680ce4fd8c18cbdcad2d55efeb61dc98'}) 2025-05-13T14:24:10.634044+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring daemon rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj on ceph-hsm-ssl-81-4yge7h-node5 2025-05-13T14:24:11.579570+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne deps ['05c7c06e6b57904eccbc4046a20ad349'] -> ['ssl-cert:05c7c06e6b57904eccbc4046a20ad349'] (diff {'ssl-cert:05c7c06e6b57904eccbc4046a20ad349', '05c7c06e6b57904eccbc4046a20ad349'}) 2025-05-13T14:24:11.582479+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring daemon rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne on ceph-hsm-ssl-81-4yge7h-node5 2025-05-13T14:24:12.743953+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj deps ['680ce4fd8c18cbdcad2d55efeb61dc98'] -> ['ssl-cert:680ce4fd8c18cbdcad2d55efeb61dc98'] (diff {'680ce4fd8c18cbdcad2d55efeb61dc98', 'ssl-cert:680ce4fd8c18cbdcad2d55efeb61dc98'}) 2025-05-13T14:24:12.748606+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring daemon rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj on ceph-hsm-ssl-81-4yge7h-node5 2025-05-13T14:24:13.731579+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne deps ['05c7c06e6b57904eccbc4046a20ad349'] -> ['ssl-cert:05c7c06e6b57904eccbc4046a20ad349'] (diff {'ssl-cert:05c7c06e6b57904eccbc4046a20ad349', '05c7c06e6b57904eccbc4046a20ad349'}) 2025-05-13T14:24:13.735468+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring daemon rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne on ceph-hsm-ssl-81-4yge7h-node5 2025-05-13T14:24:27.345927+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj deps ['680ce4fd8c18cbdcad2d55efeb61dc98'] -> ['ssl-cert:680ce4fd8c18cbdcad2d55efeb61dc98'] (diff {'680ce4fd8c18cbdcad2d55efeb61dc98', 'ssl-cert:680ce4fd8c18cbdcad2d55efeb61dc98'}) 2025-05-13T14:24:27.351083+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring daemon rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj on ceph-hsm-ssl-81-4yge7h-node5 2025-05-13T14:24:28.309779+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne deps ['05c7c06e6b57904eccbc4046a20ad349'] -> ['ssl-cert:05c7c06e6b57904eccbc4046a20ad349'] (diff {'ssl-cert:05c7c06e6b57904eccbc4046a20ad349', '05c7c06e6b57904eccbc4046a20ad349'}) 2025-05-13T14:24:28.311928+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring daemon rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne on ceph-hsm-ssl-81-4yge7h-node5 2025-05-13T14:24:29.390035+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj deps ['680ce4fd8c18cbdcad2d55efeb61dc98'] -> ['ssl-cert:680ce4fd8c18cbdcad2d55efeb61dc98'] (diff {'680ce4fd8c18cbdcad2d55efeb61dc98', 'ssl-cert:680ce4fd8c18cbdcad2d55efeb61dc98'}) 2025-05-13T14:24:29.392519+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring daemon rgw.rgw.ssl3.ceph-hsm-ssl-81-4yge7h-node5.qxyyxj on ceph-hsm-ssl-81-4yge7h-node5 2025-05-13T14:24:30.373962+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne deps ['05c7c06e6b57904eccbc4046a20ad349'] -> ['ssl-cert:05c7c06e6b57904eccbc4046a20ad349'] (diff {'ssl-cert:05c7c06e6b57904eccbc4046a20ad349', '05c7c06e6b57904eccbc4046a20ad349'}) 2025-05-13T14:24:30.376697+0000 mgr.ceph-hsm-ssl-81-4yge7h-node1-installer.skgvjo [INF] Reconfiguring daemon rgw.rgw.ssl4.ceph-hsm-ssl-81-4yge7h-node5.jqtnne on ceph-hsm-ssl-81-4yge7h-node5 ######################################################################## rgw-ssl service: [cephuser@ceph-hsm-ssl-81-4yge7h-node6 ~]$ ceph orch ls --export --service-name rgw.rgw.ssl4 service_type: rgw service_id: rgw.ssl4 service_name: rgw.rgw.ssl4 placement: hosts: - ceph-hsm-ssl-81-4yge7h-node5 spec: rgw_exit_timeout_secs: 120 rgw_frontend_port: 8081 rgw_frontend_ssl_certificate: '-----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEApcd7vXKg7C8n4AtYAsHfZoCG6Hzls0ZgkXxyVwM92uH7aIgl JstfcKkNfFODYBwInc4yL6kOIEAZ+Zg3O8QFaZXCjSQg6CYCitDQGpU6AtELt0NR MByfgvpG+8L2aAS/qsYZAbj1qz4ny1Us1VlrG+eYPnBZYnS59hFC6W5wQ615NHHp bwqWUhWBHMKcLni3hH0S8wUSUG8Ng4dOQZkTD2R1HifDD1+xU9HwLUEpkF7tQc6j ZmHwNP9UCP8z65QPs+q3WaLEkwD8CQb6GDrji0avavLfut1NjQ6JeHek+DntGyGR 9c2HECnbhweRE4wB8Q+kyIIGEJT2j1bp1HvQ2QIDAQABAoIBAArduXRLItXNqCJy UTwqXaydTg/HvVKekU6CnAgCCWLjHxR/Sq1xSnna9YGprlnKsfHjWxIZVaNdhZZW Q4CNB6x1CYHbgAtXZSFDhfByyZD3siSQAAP+9vb3KqyZNId65YlEX3fnxLU1vMFX fY8lGPUvnUtdF9lxPI3ZccuqgSdJCNNMAR9xRmhQlTxqeh9RReiY57aSaw3I2CaX EFI7yir1Ql574XSh+KUpQRK3iaQbcn3TZeUXeEdRuyZXS+Pjal/2r4E25q1H1eJG nylqSHynn+kvcSXbmWIzzEZWDOZdr94g9O6oEc2BF3Nkg7z8dmh8oquWAHCRq4zx f8RJ5NUCgYEA37fgNJeaQs9FM6CrDrkRIekfd1t7aEjB6S1W06lM1NgRH+Asph0s TmhXQbrwnaGaPDgSyzQ1uKbUtmNSBTqFYLSeBxRbpbTeMjbQ+EJBNSOG3C4W24x0 Y1DdA/qsNqmgWTYgWijNWCxOOhBZFA9JZeHMp6wLLu4J2mlk1W1mLOMCgYEAvbNY 2QJIoAP/CDIcr3W4ARWL3+s9rt1t+aQMHIQ6QtwUgBsutL5Y+Wi2zHvuv/Xa2FWk +gleQTPR6Rwk/oIO90NpuR3SV0Dp7Mv2jV2B5rnne06JwWWwdEnoyYVdG0lM+/po 6Nc2EcDxDD+tcpPaJ+l+6XitWS1DkawwsQfkVBMCgYAZ3I+2hK/4eS1uwmwDz7ED UoeorJKfg4mnAu6+mRugnzmbjbabnVO53KX5mzqlfkDuJS1HAEoVhnOggerBX/p7 K2X5O4k8pqPo+h0+FaqkEe9NFKbfymTJ9lcWVqwBxaobpf4nbgAO6VBBthIWF45M kMWvvFuumSXaccSOR49AMQKBgQCsh7I15GrTBe9sr88BEaqhacxGgjgekwtTsgzF 3sjySCFpBc/r2PcpF0CyJ8FT2zIH8ZTTR7Puz0eJoOxQgU9FOY2tLF9CtipLEJIT rkjX5JqQvjRxrZ/Xk+Tha6brizY2MmA57IojiXxyLHP+slQGWsnvY68dXRqIIbhD Oi9C+wKBgQDZVT574r6EqObTlI/aOL+aKrmTFOhJ29DzCXfdALRcqU4Q35BpcWEh PUVUFR2635WnhODSTrwlU3NV674zSR2wQe4dpg8qUrSrB76+Zw2VAVenmSdJxxrE t5q3DFjtG/vx8rHSUDMZZxwwhRzPTdnvD0H9E9IGNJmPKF4WB67DSA== -----END RSA PRIVATE KEY----- -----BEGIN CERTIFICATE----- MIID+jCCAuKgAwIBAgIUWImsLR10AwmjCTjalxS9JHoTLcIwDQYJKoZIhvcNAQEL BQAwgZUxCzAJBgNVBAYTAklOMRIwEAYDVQQIDAlLYXJuYXRha2ExEjAQBgNVBAcM CUJhbmdhbG9yZTEUMBIGA1UECgwLUmVkIEhhdCBJbmMxFTATBgNVBAsMDERhdGEg U3RvcmFnZTEPMA0GA1UEAwwGY2VwaENJMSAwHgYJKoZIhvcNAQkBFhFjZXBoY2lA cmVkaGF0LmNvbTAeFw0yNTA1MTMwNTM0MTlaFw0yNTA2MTIwNTM0MTlaMIGEMQsw CQYDVQQGEwJJTjESMBAGA1UECAwJS2FybmF0YWthMRIwEAYDVQQHDAlCZW5nYWx1 cnUxFDASBgNVBAoMC1JlZCBIYXQgSW5jMRAwDgYDVQQLDAdTdG9yYWdlMSUwIwYD VQQDDBxjZXBoLWhzbS1zc2wtODEtNHlnZTdoLW5vZGU1MIIBIjANBgkqhkiG9w0B AQEFAAOCAQ8AMIIBCgKCAQEApcd7vXKg7C8n4AtYAsHfZoCG6Hzls0ZgkXxyVwM9 2uH7aIglJstfcKkNfFODYBwInc4yL6kOIEAZ+Zg3O8QFaZXCjSQg6CYCitDQGpU6 AtELt0NRMByfgvpG+8L2aAS/qsYZAbj1qz4ny1Us1VlrG+eYPnBZYnS59hFC6W5w Q615NHHpbwqWUhWBHMKcLni3hH0S8wUSUG8Ng4dOQZkTD2R1HifDD1+xU9HwLUEp kF7tQc6jZmHwNP9UCP8z65QPs+q3WaLEkwD8CQb6GDrji0avavLfut1NjQ6JeHek +DntGyGR9c2HECnbhweRE4wB8Q+kyIIGEJT2j1bp1HvQ2QIDAQABo1EwTzBNBgNV HREERjBEgh4qLmNlcGgtaHNtLXNzbC04MS00eWdlN2gtbm9kZTWCHGNlcGgtaHNt LXNzbC04MS00eWdlN2gtbm9kZTWHBAoAQmYwDQYJKoZIhvcNAQELBQADggEBABRt nZtkdwtleu/bOFBxYNUsU8XDElBrzL9VGQB4iG09g/QP1oCuCjY8JkGLbpMC60oG PLmQys7AYF/I50OHIv/+Tk5/PxJE5d/QJCba5SKZPBrFQaNgmdkP/cPLVR2lRzMz hKtLxksiXG8fPrnWnepr3cQmPNZa3pdhlG3AIgbmPLlhxIEraR6oh7sPtsBkKW8a 9b4EX4EqQsHc8Ng5UKmpaThxSOtpVXcAVex1tN1iDZboIJNz+T4v7BJScuwZTlPs HQ7ft201rM+k32SOIPCStJGu4ENpLnBdhi/Wwfok3afGl7luWi1yjEW0izaheUmm uwlXVwPsCLQ3+YsGiig= -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIID/zCCAuegAwIBAgIJAI0xMKJjzvKNMA0GCSqGSIb3DQEBCwUAMIGVMQswCQYD VQQGEwJJTjESMBAGA1UECAwJS2FybmF0YWthMRIwEAYDVQQHDAlCYW5nYWxvcmUx FDASBgNVBAoMC1JlZCBIYXQgSW5jMRUwEwYDVQQLDAxEYXRhIFN0b3JhZ2UxDzAN BgNVBAMMBmNlcGhDSTEgMB4GCSqGSIb3DQEJARYRY2VwaGNpQHJlZGhhdC5jb20w HhcNMjEwODIzMDkxNzQ2WhcNMzEwODIxMDkxNzQ2WjCBlTELMAkGA1UEBhMCSU4x EjAQBgNVBAgMCUthcm5hdGFrYTESMBAGA1UEBwwJQmFuZ2Fsb3JlMRQwEgYDVQQK DAtSZWQgSGF0IEluYzEVMBMGA1UECwwMRGF0YSBTdG9yYWdlMQ8wDQYDVQQDDAZj ZXBoQ0kxIDAeBgkqhkiG9w0BCQEWEWNlcGhjaUByZWRoYXQuY29tMIIBIjANBgkq hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwwtZlkbTvHsMjpA7WYf1e6c+XdaOeVN/ QBXCUss2KSDaFYKfX0keOfTSFuE3yYucG4hXAVyjfEHrNMC5wt5omggUlW3cLtXX PEXMUFTgBlerNVqvkO0L6JqaZhVaAhwMWSc/5gx4kA2vquoMSQisb7vwVuo2okSO L07oGXsOBhGjggwuBRb0muY23tIb6Z63kEy8baLaeQJti2to4VJ6btOMV4sYyOEO 6ppd170Gb8MOb43HJjgk7apSj24Aq4koMddKOOFCSCUre6PiiZirDuHsxZmEK+MF 3RKFrix3b/m80/a4aUmwOQezQgtq0pIFOb+rLITby9jr/+aNfP9MkQIDAQABo1Aw TjAdBgNVHQ4EFgQUMgp0QuoVzuVfmR1/nWdhOZtBayowHwYDVR0jBBgwFoAUMgp0 QuoVzuVfmR1/nWdhOZtBayowDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOC AQEAUZAaOZDp8ksEfds8w2wgrool8n1e/zds8t9dbJoGiAyttvdcq94fzH9VyKsu eoUYoC2fhsMYwxuYfCiKM5bpiuNGT+LnErKzzykRcnkiGFm8ocBSSkA5+1+8I73m JmhEmy6lPz6dvxswlnOmz6n6SM9GMte0OqkdmsJ/LDJFkjRZYWyI4gumZK1f1uzd H43OdiO1crszx79DIFPK0eucmep+xMSiTLgS3qQnHsOX3WHb4STbtPdxmreg87va vgU3PEe5pI3Kx0top1GQqHXyNPiZaQQZv9stDyn9GRccsqZ80zUN/k/oDlogh2At ZMlXvHmwYFEu/j1lMMWmvTsQ+A== -----END CERTIFICATE----- ' ssl: true [cephuser@ceph-hsm-ssl-81-4yge7h-node6 ~]$ ######################################################################## rgw-ssl restart log: https://docs.google.com/document/d/1ZB6tMxHJ59s0ddX_rRd401O6poRegCUZbn_1dtzEt80/edit?usp=sharing Version-Release number of selected component (if applicable): ceph version 19.2.1-184.el9cp How reproducible: always Steps to Reproduce: 1.deploy rgw-ssl with rgw_frontend_ssl_certificate given in the spec 2. execute "ceph -W cephadm" in one screen and execute "ceph orch ps -refresh" in the one screen. repeated daemon reconfigs are logged. 3.also restart rgw-ssl is taking longer time to restart Actual results: repeated damon reconfigs are causing rgw-ssl daemon delay in restart Expected results: expected daemon reconfigs should not happen repeatedly Additional info: