Bug 2092508
| Summary: | [iscsi] tcmu container crash when cluster is upgraded from 5.1Z1 to 5.2 | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Preethi <pnataraj> |
| Component: | Cephadm | Assignee: | Adam King <adking> |
| Status: | CLOSED ERRATA | QA Contact: | Preethi <pnataraj> |
| Severity: | high | Docs Contact: | Anjana Suparna Sriram <asriram> |
| Priority: | unspecified | ||
| Version: | 5.0 | CC: | adking, akraj, gabrioux, tonay, tserlin, vereddy |
| Target Milestone: | --- | ||
| Target Release: | 5.2 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ceph-16.2.8-45.el8cp | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-08-09 17:38:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2102272 | ||
Ceph version of 5.1Z1 - ceph version 16.2.7-112.el8cp @Adam, Issue still exist with latest 5.2 builds and update in the BZ https://bugzilla.redhat.com/show_bug.cgi?id=1976128. This could be verified once we have fix for the BZ dependent as per the discussions. However, I was able to see the issue when I upgraded to the latest builds. I guess we cannot verify until we have fix i.e pids.max is set for both tcmu and iscsi containers As per the discussions, moved the issue back to assigned state, Also,We cannot perform lun test if the fix of BZ 1976128 available with PIDs to MAX limit for both ISCSI and TCMU containers and the upgrade is again dependent on 255 luns and cannot be verified until we have the fix for BZ 1976128. issue is not seen with the latest 5.2 builds where fix is implemented.
Below snippet after upgrade
[ceph: root@magna021 /]# ceph status
cluster:
id: c8ce6d50-c0a1-11ec-a99b-002590fc2a2e
health: HEALTH_OK
services:
mon: 5 daemons, quorum magna021,magna022,magna024,magna025,magna026 (age 19m)
mgr: magna022.icxgsh(active, since 22m), standbys: magna021.syfuos
osd: 42 osds: 42 up (since 6m), 42 in (since 8w)
rbd-mirror: 1 daemon active (1 hosts)
tcmu-runner: 512 portals active (2 hosts)
data:
pools: 11 pools, 801 pgs
objects: 891.65k objects, 3.4 TiB
usage: 10 TiB used, 28 TiB / 38 TiB avail
pgs: 801 active+clean
io:
client: 218 KiB/s rd, 231 KiB/s wr, 59 op/s rd, 56 op/s wr
[ceph: root@magna021 /]# ceph version
ceph version 16.2.8-47.el8cp (48087358763c55c41f590e2beabc1fd341b89226) pacific (stable)
[ceph: root@magna021 /]# ceph versions
{
"mon": {
"ceph version 16.2.8-47.el8cp (48087358763c55c41f590e2beabc1fd341b89226) pacific (stable)": 5
},
"mgr": {
"ceph version 16.2.8-47.el8cp (48087358763c55c41f590e2beabc1fd341b89226) pacific (stable)": 2
},
"osd": {
"ceph version 16.2.8-47.el8cp (48087358763c55c41f590e2beabc1fd341b89226) pacific (stable)": 42
},
"mds": {},
"rbd-mirror": {
"ceph version 16.2.8-47.el8cp (48087358763c55c41f590e2beabc1fd341b89226) pacific (stable)": 1
},
"tcmu-runner": {
"ceph version 16.2.8-47.el8cp (48087358763c55c41f590e2beabc1fd341b89226) pacific (stable)": 512
},
"overall": {
"ceph version 16.2.8-47.el8cp (48087358763c55c41f590e2beabc1fd341b89226) pacific (stable)": 562
}
}
[ceph: root@magna021 /]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Ceph Storage Security, Bug Fix, and Enhancement Update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5997 |
Description of problem: tcmu container crash when cluster is upgraded from 5.1Z1 to 5.2 Version-Release number of selected component (if applicable): [root@magna021 /]# ceph version ceph version 16.2.8-22.el8cp (857c12016f00d5089a358b3bbc25abf43d84a7a2) pacific (stable) How reproducible: Steps to Reproduce: 1. Perform upgrade to 5.2 latest build from 5.1Z1 cluster which is having 255 luns via ceph orch upgrade command 2. post the upgrade successful check the ceph status and podman ps in iscsi gateway nodes configuration and cluster status before upgrade - [root@magna021 ubuntu]# ceph status cluster: id: c8ce6d50-c0a1-11ec-a99b-002590fc2a2e health: HEALTH_OK services: mon: 5 daemons, quorum magna021,magna022,magna024,magna025,magna026 (age 4d) mgr: magna021.syfuos(active, since 6d), standbys: magna022.icxgsh osd: 42 osds: 42 up (since 6d), 42 in (since 3w) rbd-mirror: 1 daemon active (1 hosts) tcmu-runner: 510 portals active (2 hosts) data: pools: 11 pools, 417 pgs objects: 579.62k objects, 2.2 TiB usage: 6.6 TiB used, 32 TiB / 38 TiB avail pgs: 417 active+clean io: client: 612 KiB/s rd, 41 KiB/s wr, 731 op/s rd, 77 op/s wr [root@magna021 ubuntu]# Actual results: Upgrade was successful but ceph status do not report about tcmu-runner status and when we check podman ps in iscsi nodes we see there is no tcmu container exist. [root@plena001 ubuntu]# podman ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9ad27ceb33e9 registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:691c0eb6ddd228e9ddf6999bbb3903c7312d67f909377d614e1fa27a279a2cce -n client.crash.p... 36 hours ago Up 36 hours ago ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-crash-plena001 df9d0c1e7ea8 registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:691c0eb6ddd228e9ddf6999bbb3903c7312d67f909377d614e1fa27a279a2cce -n osd.1 -f --set... 36 hours ago Up 36 hours ago ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-osd-1 f7ee016dc2f9 registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:691c0eb6ddd228e9ddf6999bbb3903c7312d67f909377d614e1fa27a279a2cce -n osd.0 -f --set... 36 hours ago Up 36 hours ago ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-osd-0 21415866b400 registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:691c0eb6ddd228e9ddf6999bbb3903c7312d67f909377d614e1fa27a279a2cce -n osd.3 -f --set... 36 hours ago Up 36 hours ago ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-osd-3 ab3011fdbb05 registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:691c0eb6ddd228e9ddf6999bbb3903c7312d67f909377d614e1fa27a279a2cce -n osd.4 -f --set... 36 hours ago Up 36 hours ago ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-osd-4 ccacf7be065a registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:691c0eb6ddd228e9ddf6999bbb3903c7312d67f909377d614e1fa27a279a2cce -n osd.2 -f --set... 36 hours ago Up 36 hours ago ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-osd-2 e4ed9adeb721 registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:691c0eb6ddd228e9ddf6999bbb3903c7312d67f909377d614e1fa27a279a2cce -n osd.6 -f --set... 36 hours ago Up 36 hours ago ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-osd-6 71417fc784d3 registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:691c0eb6ddd228e9ddf6999bbb3903c7312d67f909377d614e1fa27a279a2cce -n osd.5 -f --set... 36 hours ago Up 36 hours ago ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-osd-5 5e5536635ebd registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.10 --no-collector.ti... 11 hours ago Up 11 hours ago ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-node-exporter-plena001 d90c674c4d03 registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:691c0eb6ddd228e9ddf6999bbb3903c7312d67f909377d614e1fa27a279a2cce 5 hours ago Up 5 hours ago ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor [root@plena001 ubuntu]# Expected results: We should see the tcmu container and active portal should be listed in ceph status crash logs- [root@plena001 ubuntu]# journalctl -xeu ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e.plena001.adsjor.service Jun 01 12:47:42 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Adding image 'iscsipool/test204' to LIO backstore user:rbd Jun 01 12:47:42 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Successfully added iscsipool/test204 to LIO Jun 01 12:47:42 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug Setup group ano2 for iscsipool.test204 on tpg 2 (state 1, owner False, failover type 1) Jun 01 12:47:42 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Adding image 'iscsipool/test205' to LIO backstore user:rbd Jun 01 12:47:42 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Successfully added iscsipool/test205 to LIO Jun 01 12:47:42 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug Setup group ao for iscsipool.test205 on tpg 2 (state 0, owner True, failover type 1) Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Adding image 'iscsipool/test206' to LIO backstore user:rbd Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Successfully added iscsipool/test206 to LIO Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug Setup group ano2 for iscsipool.test206 on tpg 2 (state 1, owner False, failover type 1) Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Adding image 'iscsipool/test207' to LIO backstore user:rbd Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Successfully added iscsipool/test207 to LIO Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug Setup group ao for iscsipool.test207 on tpg 2 (state 0, owner True, failover type 1) Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Adding image 'iscsipool/test208' to LIO backstore user:rbd Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Successfully added iscsipool/test208 to LIO Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug Setup group ano2 for iscsipool.test208 on tpg 2 (state 1, owner False, failover type 1) Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Adding image 'iscsipool/test209' to LIO backstore user:rbd Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Successfully added iscsipool/test209 to LIO Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug Setup group ao for iscsipool.test209 on tpg 2 (state 0, owner True, failover type 1) Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Adding image 'iscsipool/test21' to LIO backstore user:rbd Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Successfully added iscsipool/test21 to LIO Jun 01 12:47:43 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug Setup group ao for iscsipool.test21 on tpg 2 (state 0, owner True, failover type 1) Jun 01 12:47:44 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Adding image 'iscsipool/test210' to LIO backstore user:rbd Jun 01 12:47:44 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Successfully added iscsipool/test210 to LIO Jun 01 12:47:44 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug Setup group ano2 for iscsipool.test210 on tpg 2 (state 1, owner False, failover type 1) Jun 01 12:47:44 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Adding image 'iscsipool/test211' to LIO backstore user:rbd Jun 01 12:47:44 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Successfully added iscsipool/test211 to LIO Jun 01 12:47:44 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug Setup group ao for iscsipool.test211 on tpg 2 (state 0, owner True, failover type 1) Jun 01 12:47:44 plena001 ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor[2851943]: debug (LUN.add_dev_to_lio) Adding image 'iscsipool/test2110' to LIO backstore user:rbd Jun 01 12:47:44 plena001 bash[2851481]: Thread::try_create(): pthread_create failed with error 11/builddir/build/BUILD/ceph-16.2.8/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' th> Jun 01 12:47:44 plena001 bash[2851481]: /builddir/build/BUILD/ceph-16.2.8/src/common/Thread.cc: 165: FAILED ceph_assert(ret == 0) Jun 01 12:47:44 plena001 bash[2851481]: ceph version 16.2.8-22.el8cp (857c12016f00d5089a358b3bbc25abf43d84a7a2) pacific (stable) Jun 01 12:47:44 plena001 bash[2851481]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f9c4252fd48] Jun 01 12:47:44 plena001 bash[2851481]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x276f62) [0x7f9c4252ff62] Jun 01 12:47:44 plena001 bash[2851481]: 3: /usr/lib64/ceph/libceph-common.so.2(+0x363127) [0x7f9c4261c127] Jun 01 12:47:44 plena001 bash[2851481]: 4: (CommonSafeTimer<std::mutex>::init()+0x1fe) [0x7f9c4262278e] Jun 01 12:47:44 plena001 bash[2851481]: 5: (MgrClient::init()+0x40) [0x7f9c428f2570] Jun 01 12:47:44 plena001 bash[2851481]: 6: /lib64/librados.so.2(+0xbbdb1) [0x7f9c4b769db1] Jun 01 12:47:44 plena001 bash[2851481]: 7: rados_connect() Jun 01 12:47:44 plena001 bash[2851481]: 8: /usr/lib64/tcmu-runner/handler_rbd.so(+0x3831) [0x7f9c4c530831] Jun 01 12:47:44 plena001 bash[2851481]: 9: /usr/lib64/tcmu-runner/handler_rbd.so(+0x48e8) [0x7f9c4c5318e8] Jun 01 12:47:44 plena001 bash[2851481]: 10: /usr/bin/tcmu-runner() [0x414ef1] Jun 01 12:47:44 plena001 bash[2851481]: 11: /lib64/libtcmu.so.2(+0xb576) [0x7f9c52d52576] Jun 01 12:47:44 plena001 bash[2851481]: 12: /lib64/libtcmu.so.2(+0xa7b3) [0x7f9c52d517b3] Jun 01 12:47:44 plena001 bash[2851481]: 13: /lib64/libnl-genl-3.so.200(+0x3fb5) [0x7f9c515d3fb5] Jun 01 12:47:44 plena001 bash[2851481]: 14: nl_recvmsgs_report() Jun 01 12:47:44 plena001 bash[2851481]: 15: nl_recvmsgs() Jun 01 12:47:44 plena001 bash[2851481]: 16: tcmulib_master_fd_ready() Jun 01 12:47:44 plena001 bash[2851481]: 17: /usr/bin/tcmu-runner() [0x4133de] Jun 01 12:47:44 plena001 bash[2851481]: 18: g_main_context_dispatch() Jun 01 12:47:44 plena001 bash[2851481]: 19: /lib64/libglib-2.0.so.0(+0x4dd18) [0x7f9c5247ed18] Jun 01 12:47:44 plena001 bash[2851481]: 20: g_main_loop_run() Jun 01 12:47:44 plena001 bash[2851481]: 21: /usr/bin/tcmu-runner() [0x415c5b] Jun 01 12:47:44 plena001 bash[2851481]: 22: __libc_start_main() Jun 01 12:47:44 plena001 bash[2851481]: 23: /usr/bin/tcmu-runner() [0x40817e] Additional info: magna021 - root/q plena001 - iscsi gateway node1 plena002 - iscsi gateway node2 ceph status post upgrade: [root@magna021 /]# ceph status cluster: id: c8ce6d50-c0a1-11ec-a99b-002590fc2a2e health: HEALTH_WARN Redeploying daemon node-exporter.magna021 on host magna021 failed. services: mon: 5 daemons, quorum magna021,magna022,magna024,magna025,magna026 (age 12h) mgr: magna021.syfuos(active, since 35h), standbys: magna022.icxgsh osd: 42 osds: 42 up (since 35h), 42 in (since 6w) rbd-mirror: 1 daemon active (1 hosts) data: pools: 14 pools, 897 pgs objects: 882.61k objects, 3.3 TiB usage: 10 TiB used, 28 TiB / 38 TiB avail pgs: 897 active+clean io: client: 654 KiB/s rd, 50 KiB/s wr, 779 op/s rd, 87 op/s wr progress: Upgrade to 16.2.8-22.el8cp (21m) [======================......] (remaining: 5m) [root@magna021 /]# ceph health detail HEALTH_WARN Redeploying daemon node-exporter.magna021 on host magna021 failed. [WRN] UPGRADE_REDEPLOY_DAEMON: Redeploying daemon node-exporter.magna021 on host magna021 failed. Upgrade daemon: node-exporter.magna021: cephadm exited with an error code: 1, stderr:Redeploy daemon node-exporter.magna021 ... Non-zero exit code 1 from systemctl start ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e systemctl: stderr Job for ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e.service failed because the control process exited with error code. systemctl: stderr See "systemctl status ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e.service" and "journalctl -xe" for details. Traceback (most recent call last): File "/var/lib/ceph/c8ce6d50-c0a1-11ec-a99b-002590fc2a2e/cephadm.259b6943093ee438dfe0837018f826fc5bc47fd2a4f5b27678a18e8e65a29026", line 9121, in <module> main() File "/var/lib/ceph/c8ce6d50-c0a1-11ec-a99b-002590fc2a2e/cephadm.259b6943093ee438dfe0837018f826fc5bc47fd2a4f5b27678a18e8e65a29026", line 9109, in main r = ctx.func(ctx) File "/var/lib/ceph/c8ce6d50-c0a1-11ec-a99b-002590fc2a2e/cephadm.259b6943093ee438dfe0837018f826fc5bc47fd2a4f5b27678a18e8e65a29026", line 1970, in _default_image return func(ctx) File "/var/lib/ceph/c8ce6d50-c0a1-11ec-a99b-002590fc2a2e/cephadm.259b6943093ee438dfe0837018f826fc5bc47fd2a4f5b27678a18e8e65a29026", line 5033, in command_deploy ports=daemon_ports) File "/var/lib/ceph/c8ce6d50-c0a1-11ec-a99b-002590fc2a2e/cephadm.259b6943093ee438dfe0837018f826fc5bc47fd2a4f5b27678a18e8e65a29026", line 2925, in deploy_daemon c, osd_fsid=osd_fsid, ports=ports) File "/var/lib/ceph/c8ce6d50-c0a1-11ec-a99b-002590fc2a2e/cephadm.259b6943093ee438dfe0837018f826fc5bc47fd2a4f5b27678a18e8e65a29026", line 3170, in deploy_daemon_units call_throws(ctx, ['systemctl', 'start', unit_name]) File "/var/lib/ceph/c8ce6d50-c0a1-11ec-a99b-002590fc2a2e/cephadm.259b6943093ee438dfe0837018f826fc5bc47fd2a4f5b27678a18e8e65a29026", line 1637, in call_throws raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') RuntimeError: Failed command: systemctl start ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e: Job for ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e.service failed because the control process exited with error code. See "systemctl status ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e.service" and "journalctl -xe" for details. [root@magna021 /]# ceph orch ls- [root@magna021 /]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 2m ago 6w count:1 crash 14/14 5m ago 6w * grafana ?:3000 1/1 2m ago 6w count:1 iscsi.iscsipool 2/2 2m ago 2w plena001;plena002 mgr 2/2 2m ago 6w count:2 mon 5/5 2m ago 3w magna021;magna022;magna024;magna025;magna026 node-exporter ?:9100 14/14 5m ago 6w * osd 42 2m ago - <unmanaged> prometheus ?:9095 1/1 2m ago 6w count:1 rbd-mirror 1/1 2m ago 3w magna026 [root@magna021 /]# d90c674c4d03 registry-proxy.engineering.redhat.com/rh-osbs/rhceph@sha256:691c0eb6ddd228e9ddf6999bbb3903c7312d67f909377d614e1fa27a279a2cce 5 hours ago Up 5 hours ago ceph-c8ce6d50-c0a1-11ec-a99b-002590fc2a2e-iscsi-iscsipool-plena001-adsjor [root@plena001 ubuntu]# podman exec -it d90c674c4d03 /bin/bash [root@plena001 /]# gwcli Warning: Could not load preferences file /root/.gwcli/prefs.bin. REST API failure, code : 500 Unable to access the configuration object AttributeError: 'Settings' object has no attribute 'api_endpoint' [root@plena001 /]# ls