Bug 2281800
| Summary: | [Longevity cluster]GW crashes eventually with _rbd_update_callback: *ERROR*: Failed getting size -108 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Rahul Lepakshi <rlepaksh> | ||||
| Component: | NVMeOF | Assignee: | Gil Bregman <gbregman> | ||||
| Status: | CLOSED COMPLETED | QA Contact: | Rahul Lepakshi <rlepaksh> | ||||
| Severity: | urgent | Docs Contact: | ceph-doc-bot <ceph-doc-bugzilla> | ||||
| Priority: | unspecified | ||||||
| Version: | 7.1 | CC: | acaro, cephqe-warriors, gbregman, jcaratza, mgowri, owasserm | ||||
| Target Milestone: | --- | Keywords: | BetaBlocker, External, TestBlocker | ||||
| Target Release: | 7.1 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | ceph-nvmeof-container-1.2.10-1 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2024-08-22 08:49:58 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 2034138 [details]
Logs
We cannot be 100% sure but this seems to be related to an unhandled exception in Prometheus that causes the GW and SPDK to exit unexpectedly. I managed to bring up the GW on this cluster just by disabling Prometheus, and this might also be an indication that this is the issue.
From the logs:
May 17 13:31:32 argo023 ceph-a44110aa-12ca-11ef-bd4c-b4835101e458-nvmeof-nvmeof_pool-argo023-iefwqr[1150507]: terminate called after throwing an instance of 'std::runtime_error'
May 17 13:31:32 argo023 ceph-a44110aa-12ca-11ef-bd4c-b4835101e458-nvmeof-nvmeof_pool-argo023-iefwqr[1150507]: what(): Lost connection to the monitor (gw map unavailable).
May 17 13:31:33 argo023 ceph-a44110aa-12ca-11ef-bd4c-b4835101e458-nvmeof-nvmeof_pool-argo023-iefwqr[1150507]: conn_info = self.connections[nqn]
May 17 13:31:33 argo023 ceph-a44110aa-12ca-11ef-bd4c-b4835101e458-nvmeof-nvmeof_pool-argo023-iefwqr[1150507]: KeyError: 'nqn.2016-06.io.spdk:cnode1'
May 17 13:31:33 argo023 ceph-a44110aa-12ca-11ef-bd4c-b4835101e458-nvmeof-nvmeof_pool-argo023-iefwqr[1150507]: ----------------------------------------
May 17 13:31:33 argo023 ceph-a44110aa-12ca-11ef-bd4c-b4835101e458-nvmeof-nvmeof_pool-argo023-iefwqr[1150507]: Exception occurred during processing of request from ('10.8.131.14', 37194)
The unhandled exception in Prometheus is fixed in GW 1.2.9. We need to try to see if it still happens with the latest build.
@aviv.caro @owasserm @leonidc.com
I am again seeing same issue with recent build as below -
ceph version 18.2.1-185.el9cp (610720d1f4016a4ac043901b452620472c2f3dcb) reef (stable)
cp.stg.icr.io/cp/ibm-ceph/nvmeof-rhel9:1.2.9-1
[22-May-2024 10:46:21] ERROR prometheus.py:217: Stats refresh time > interval time of 10 secs
[22-May-2024 10:46:21] INFO grpc.py:914: Received request to add a namespace using NSID 132 and UUID 02dee270-5a4d-4e20-a33d-cf5fd2206a1f to nqn.2016-06.io.spdk:cnode2, ana group 1 context: None
[22-May-2024 10:46:21] INFO grpc.py:274: Received request to create bdev bdev_02dee270-5a4d-4e20-a33d-cf5fd2206a1f from rbd/DZLH-image132 (size 0 bytes) with block size 512, will not create image if doesn't exist
[22-May-2024 10:46:21] INFO grpc.py:209: get_cluster cluster_name='cluster_context_1_2' number bdevs: 2
[2024-05-22 10:46:26.299199] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_02dee270-5a4d-4e20-a33d-cf5fd2206a1f rbd disk to lun
[2024-05-22 10:46:26.499881] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108
[2024-05-22 10:46:26.700302] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108
[2024-05-22 10:46:26.900821] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108
[2024-05-22 10:46:27.101453] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108
[112818]: [2024-05-22 10:46:36.928588] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108
[112818]: [2024-05-22 10:46:36.928951] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108
[112818]: [2024-05-22 10:46:36.929148] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108
[112818]: [2024-05-22 10:46:36.929377] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108
[112818]: [2024-05-22 10:46:37.129692] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108
[112818]: terminate called after throwing an instance of 'std::runtime_error'
[112818]: what(): Lost connection to the monitor (gw map unavailable).
[112818]: *** Caught signal (Aborted) **
[112818]: in thread 7f14bbe49640 thread_name:ms_dispatch
[112818]: ceph version 18.2.1-185.el9cp (610720d1f4016a4ac043901b452620472c2f3dcb) reef (stable)
[112818]: 1: /lib64/libc.so.6(+0x3e6f0) [0x7f14c111b6f0]
[112818]: 2: /lib64/libc.so.6(+0x8b94c) [0x7f14c116894c]
[112818]: 3: raise()
[112818]: 4: abort()
[112818]: 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f14c147fb21]
[112818]: 6: /lib64/libstdc++.so.6(+0xad52c) [0x7f14c148b52c]
[112818]: 7: /lib64/libstdc++.so.6(+0xad597) [0x7f14c148b597]
[112818]: 8: /lib64/libstdc++.so.6(+0xad7f9) [0x7f14c148b7f9]
[112818]: 9: /usr/bin/ceph-nvmeof-monitor-client(+0x8a758) [0x564ebf7c1758]
[112818]: 10: (NVMeofGwMonitorClient::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x185) [0x564ebf855e25]
[112818]: 11: (DispatchQueue::entry()+0x542) [0x7f14c1ed2f52]
[112818]: 12: /usr/lib64/ceph/libceph-common.so.2(+0x3fad81) [0x7f14c1f6ad81]
[112818]: 13: /lib64/libc.so.6(+0x89c02) [0x7f14c1166c02]
[112818]: 14: /lib64/libc.so.6(+0x10ec40) [0x7f14c11ebc40]
[112818]: [22-May-2024 10:46:37] INFO grpc.py:914: Received request to add a namespace using NSID 187 and UUID 547d767b-3585-4d92-927d-4f4990b8360f to nqn.2016-06.io.spdk:cnode2, ana group 2 context: None
[112818]: [22-May-2024 10:46:37] INFO grpc.py:274: Received request to create bdev bdev_547d767b-3585-4d92-927d-4f4990b8360f from rbd/DZLH-image187 (size 0 bytes) with block size 512, will not create image if doesn't exist
[112818]: [22-May-2024 10:46:37] INFO grpc.py:209: get_cluster cluster_name='cluster_context_2_1' number bdevs: 28
core.
[112818]: [22-May-2024 10:46:37] ERROR server.py:42: GatewayServer: SIGCHLD received signum=17
[112818]: [22-May-2024 10:46:37] ERROR server.py:46: PID of terminated child process is 19
[112818]: [22-May-2024 10:46:37] ERROR server.py:110: GatewayServer exception occurred:
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/__main__.py", line 45, in <module>
[112818]: gateway.keep_alive()
[112818]: File "/remote-source/ceph-nvmeof/app/control/server.py", line 491, in keep_alive
[112818]: time.sleep(self.spdk_ping_interval_in_seconds)
[112818]: File "/remote-source/ceph-nvmeof/app/control/server.py", line 55, in sigchld_handler
[112818]: raise SystemExit(f"Gateway subprocess terminated {pid=} {exit_code=}")
[112818]: SystemExit: Gateway subprocess terminated pid=19 exit_code=-6
[112818]: [22-May-2024 10:46:37] INFO server.py:411: Aborting (client.nvmeof.nvmeof_pool.argo024.iflydk) pid 19...
[112818]: [22-May-2024 10:46:37] INFO server.py:411: Aborting (client.nvmeof.nvmeof_pool.argo024.iflydk) pid 94...
[112818]: [22-May-2024 10:46:39] INFO grpc.py:227: put_cluster name='cluster_context_2_1' number bdevs: 27
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:333: bdev_rbd_create bdev_547d767b-3585-4d92-927d-4f4990b8360f failed
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 317, in create_bdev
[112818]: bdev_name = rpc_bdev.bdev_rbd_create(
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1252, in bdev_rbd_create
[112818]: return client.call('bdev_rbd_create', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 196, in call
[112818]: raise e
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 189, in call
[112818]: response = self.recv()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 170, in recv
[112818]: raise JSONRPCException("Connection closed with partial response:\n%s\n" % self._recv_buf)
[112818]: spdk.rpc.client.JSONRPCException: Connection closed with partial response:
[112818]:
[112818]:
[112818]:
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:958: Failure adding namespace using NSID 187 and UUID 547d767b-3585-4d92-927d-4f4990b8360f to nqn.2016-06.io.spdk:cnode2: Failure creating bdev bdev_547d767b-3585-4d92-927d-4f4990b8360f: Connection closed with partial response:
[112818]:
[112818]:
[112818]:
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:1214: Got exception while getting bdev bdev_547d767b-3585-4d92-927d-4f4990b8360f info
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1209, in get_bdev_info
[112818]: bdevs = rpc_bdev.bdev_get_bdevs(self.spdk_rpc_client, name=bdev_name)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
[112818]: return client.call('bdev_get_bdevs', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO grpc.py:914: Received request to add a namespace using NSID 133 and UUID f874c6c6-f445-4cb9-9680-5f02419deda8 to nqn.2016-06.io.spdk:cnode2, ana group 2 context: None
[112818]: [22-May-2024 10:46:39] INFO grpc.py:274: Received request to create bdev bdev_f874c6c6-f445-4cb9-9680-5f02419deda8 from rbd/DZLH-image133 (size 0 bytes) with block size 512, will not create image if doesn't exist
[112818]: [22-May-2024 10:46:39] INFO grpc.py:209: get_cluster cluster_name='cluster_context_2_1' number bdevs: 28
[112818]: [22-May-2024 10:46:39] INFO grpc.py:227: put_cluster name='cluster_context_2_1' number bdevs: 27
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:333: bdev_rbd_create bdev_f874c6c6-f445-4cb9-9680-5f02419deda8 failed
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 317, in create_bdev
[112818]: bdev_name = rpc_bdev.bdev_rbd_create(
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1252, in bdev_rbd_create
[112818]: return client.call('bdev_rbd_create', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:958: Failure adding namespace using NSID 133 and UUID f874c6c6-f445-4cb9-9680-5f02419deda8 to nqn.2016-06.io.spdk:cnode2: bdev_rbd_create bdev_f874c6c6-f445-4cb9-9680-5f02419deda8 failed with:
[112818]: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:1214: Got exception while getting bdev bdev_f874c6c6-f445-4cb9-9680-5f02419deda8 info
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1209, in get_bdev_info
[112818]: bdevs = rpc_bdev.bdev_get_bdevs(self.spdk_rpc_client, name=bdev_name)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
[112818]: return client.call('bdev_get_bdevs', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO grpc.py:914: Received request to add a namespace using NSID 53 and UUID b756353e-17c0-4c91-bfba-3e568f3e3268 to nqn.2016-06.io.spdk:cnode2, ana group 2 context: None
[112818]: [22-May-2024 10:46:39] INFO grpc.py:274: Received request to create bdev bdev_b756353e-17c0-4c91-bfba-3e568f3e3268 from rbd/DZLH-image53 (size 0 bytes) with block size 512, will not create image if doesn't exist
[112818]: Traceback (most recent call last):
[112818]: File "/usr/lib64/python3.9/wsgiref/handlers.py", line 137, in run
[112818]: self.result = application(self.environ, self.start_response)
[112818]: File "/usr/local/lib/python3.9/site-packages/prometheus_client/exposition.py", line 128, in prometheus_app
[112818]: status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression)
[112818]: File "/usr/local/lib/python3.9/site-packages/prometheus_client/exposition.py", line 104, in _bake_output
[112818]: output = encoder(registry)
[112818]: File "/usr/local/lib/python3.9/site-packages/prometheus_client/openmetrics/exposition.py", line 21, in generate_latest
[112818]: for metric in registry.collect():
[112818]: File "/usr/local/lib/python3.9/site-packages/prometheus_client/registry.py", line 97, in collect
[112818]: yield from collector.collect()
[112818]: File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 43, in wrapped
[112818]: for metric in method(self, *args, **kwargs):
[112818]: File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 213, in collect
[112818]: self._get_data()
[112818]: File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 197, in _get_data
[112818]: self.bdev_io_stats = self._get_bdev_io_stats()
[112818]: File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 57, in call
[112818]: result = method(self, *args, **kwargs)
[112818]: File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 159, in _get_bdev_io_stats
[112818]: return rpc.bdev.bdev_get_iostat(self.spdk_rpc_client)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1704, in bdev_get_iostat
[112818]: return client.call('bdev_get_iostat', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO server.py:124: Stopping the server...
[112818]: Traceback (most recent call last):
[112818]: File "/usr/lib64/python3.9/wsgiref/handlers.py", line 137, in run
[112818]: self.result = application(self.environ, self.start_response)
[112818]: File "/usr/local/lib/python3.9/site-packages/prometheus_client/exposition.py", line 128, in prometheus_app
[112818]: status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression)
[112818]: File "/usr/local/lib/python3.9/site-packages/prometheus_client/exposition.py", line 104, in _bake_output
[112818]: output = encoder(registry)
[112818]: File "/usr/local/lib/python3.9/site-packages/prometheus_client/openmetrics/exposition.py", line 21, in generate_latest
[112818]: for metric in registry.collect():
[112818]: File "/usr/local/lib/python3.9/site-packages/prometheus_client/registry.py", line 97, in collect
[112818]: yield from collector.collect()
[112818]: File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 43, in wrapped
[112818]: for metric in method(self, *args, **kwargs):
[112818]: File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 213, in collect
[112818]: self._get_data()
[112818]: [22-May-2024 10:46:39] INFO grpc.py:209: get_cluster cluster_name='cluster_context_2_1' number bdevs: 28
[112818]: File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 196, in _get_data
[112818]: self.bdev_info = self._get_bdev_info()
[112818]: File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 57, in call
[112818]: result = method(self, *args, **kwargs)
[112818]: File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 154, in _get_bdev_info
[112818]: return rpc.bdev.bdev_get_bdevs(self.spdk_rpc_client)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
[112818]: return client.call('bdev_get_bdevs', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO grpc.py:227: put_cluster name='cluster_context_2_1' number bdevs: 27
[112818]: [22-May-2024 10:46:39] INFO server.py:443: Terminating discovery service...
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:333: bdev_rbd_create bdev_b756353e-17c0-4c91-bfba-3e568f3e3268 failed
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 317, in create_bdev
[112818]: bdev_name = rpc_bdev.bdev_rbd_create(
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1252, in bdev_rbd_create
[112818]: return client.call('bdev_rbd_create', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:958: Failure adding namespace using NSID 53 and UUID b756353e-17c0-4c91-bfba-3e568f3e3268 to nqn.2016-06.io.spdk:cnode2: bdev_rbd_create bdev_b756353e-17c0-4c91-bfba-3e568f3e3268 failed with:
[112818]: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:1214: Got exception while getting bdev bdev_b756353e-17c0-4c91-bfba-3e568f3e3268 info
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1209, in get_bdev_info
[112818]: bdevs = rpc_bdev.bdev_get_bdevs(self.spdk_rpc_client, name=bdev_name)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
[112818]: return client.call('bdev_get_bdevs', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO grpc.py:914: Received request to add a namespace using NSID 120 and UUID c1ad066e-7ad3-4c81-9613-30a1f09ecd52 to nqn.2016-06.io.spdk:cnode1, ana group 1 context: None
[112818]: [22-May-2024 10:46:39] INFO grpc.py:274: Received request to create bdev bdev_c1ad066e-7ad3-4c81-9613-30a1f09ecd52 from rbd/5Y8B-image120 (size 0 bytes) with block size 512, will not create image if doesn't exist
[112818]: [22-May-2024 10:46:39] INFO grpc.py:209: get_cluster cluster_name='cluster_context_1_2' number bdevs: 3
[112818]: [22-May-2024 10:46:39] INFO grpc.py:227: put_cluster name='cluster_context_1_2' number bdevs: 2
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:333: bdev_rbd_create bdev_c1ad066e-7ad3-4c81-9613-30a1f09ecd52 failed
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 317, in create_bdev
[112818]: bdev_name = rpc_bdev.bdev_rbd_create(
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1252, in bdev_rbd_create
[112818]: return client.call('bdev_rbd_create', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:958: Failure adding namespace using NSID 120 and UUID c1ad066e-7ad3-4c81-9613-30a1f09ecd52 to nqn.2016-06.io.spdk:cnode1: bdev_rbd_create bdev_c1ad066e-7ad3-4c81-9613-30a1f09ecd52 failed with:
[112818]: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:1214: Got exception while getting bdev bdev_c1ad066e-7ad3-4c81-9613-30a1f09ecd52 info
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1209, in get_bdev_info
[112818]: bdevs = rpc_bdev.bdev_get_bdevs(self.spdk_rpc_client, name=bdev_name)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
[112818]: return client.call('bdev_get_bdevs', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO grpc.py:914: Received request to add a namespace using NSID 11 and UUID 084161c6-de9b-412c-a192-94287ae595c1 to nqn.2016-06.io.spdk:cnode1, ana group 2 context: None
[112818]: [22-May-2024 10:46:39] INFO grpc.py:274: Received request to create bdev bdev_084161c6-de9b-412c-a192-94287ae595c1 from rbd/5Y8B-image11 (size 0 bytes) with block size 512, will not create image if doesn't exist
[112818]: [22-May-2024 10:46:39] INFO grpc.py:209: get_cluster cluster_name='cluster_context_2_1' number bdevs: 28
[112818]: [22-May-2024 10:46:39] INFO grpc.py:227: put_cluster name='cluster_context_2_1' number bdevs: 27
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:333: bdev_rbd_create bdev_084161c6-de9b-412c-a192-94287ae595c1 failed
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 317, in create_bdev
[112818]: bdev_name = rpc_bdev.bdev_rbd_create(
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1252, in bdev_rbd_create
[112818]: return client.call('bdev_rbd_create', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:958: Failure adding namespace using NSID 11 and UUID 084161c6-de9b-412c-a192-94287ae595c1 to nqn.2016-06.io.spdk:cnode1: bdev_rbd_create bdev_084161c6-de9b-412c-a192-94287ae595c1 failed with:
[112818]: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:1214: Got exception while getting bdev bdev_084161c6-de9b-412c-a192-94287ae595c1 info
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1209, in get_bdev_info
[112818]: bdevs = rpc_bdev.bdev_get_bdevs(self.spdk_rpc_client, name=bdev_name)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
[112818]: return client.call('bdev_get_bdevs', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO grpc.py:914: Received request to add a namespace using NSID 119 and UUID 3d4c4b5c-524c-4c94-b384-e23bb049c145 to nqn.2016-06.io.spdk:cnode1, ana group 2 context: None
[112818]: [22-May-2024 10:46:39] INFO grpc.py:274: Received request to create bdev bdev_3d4c4b5c-524c-4c94-b384-e23bb049c145 from rbd/5Y8B-image119 (size 0 bytes) with block size 512, will not create image if doesn't exist
[112818]: [22-May-2024 10:46:39] INFO grpc.py:209: get_cluster cluster_name='cluster_context_2_1' number bdevs: 28
[112818]: [22-May-2024 10:46:39] INFO grpc.py:227: put_cluster name='cluster_context_2_1' number bdevs: 27
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:333: bdev_rbd_create bdev_3d4c4b5c-524c-4c94-b384-e23bb049c145 failed
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 317, in create_bdev
[112818]: bdev_name = rpc_bdev.bdev_rbd_create(
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1252, in bdev_rbd_create
[112818]: return client.call('bdev_rbd_create', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO server.py:450: Discovery service terminated
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:958: Failure adding namespace using NSID 119 and UUID 3d4c4b5c-524c-4c94-b384-e23bb049c145 to nqn.2016-06.io.spdk:cnode1: bdev_rbd_create bdev_3d4c4b5c-524c-4c94-b384-e23bb049c145 failed with:
[112818]: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO server.py:131: Exiting the gateway process.
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:1214: Got exception while getting bdev bdev_3d4c4b5c-524c-4c94-b384-e23bb049c145 info
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1209, in get_bdev_info
[112818]: bdevs = rpc_bdev.bdev_get_bdevs(self.spdk_rpc_client, name=bdev_name)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
[112818]: return client.call('bdev_get_bdevs', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO grpc.py:914: Received request to add a namespace using NSID 113 and UUID 0e5cdfb9-f911-476a-a15a-6cc48f7a56b9 to nqn.2016-06.io.spdk:cnode1, ana group 2 context: None
[112818]: [22-May-2024 10:46:39] INFO grpc.py:274: Received request to create bdev bdev_0e5cdfb9-f911-476a-a15a-6cc48f7a56b9 from rbd/5Y8B-image113 (size 0 bytes) with block size 512, will not create image if doesn't exist
[112818]: [22-May-2024 10:46:39] INFO grpc.py:209: get_cluster cluster_name='cluster_context_2_1' number bdevs: 28
[112818]: [22-May-2024 10:46:39] INFO grpc.py:227: put_cluster name='cluster_context_2_1' number bdevs: 27
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:333: bdev_rbd_create bdev_0e5cdfb9-f911-476a-a15a-6cc48f7a56b9 failed
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 317, in create_bdev
[112818]: bdev_name = rpc_bdev.bdev_rbd_create(
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1252, in bdev_rbd_create
[112818]: return client.call('bdev_rbd_create', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:958: Failure adding namespace using NSID 113 and UUID 0e5cdfb9-f911-476a-a15a-6cc48f7a56b9 to nqn.2016-06.io.spdk:cnode1: bdev_rbd_create bdev_0e5cdfb9-f911-476a-a15a-6cc48f7a56b9 failed with:
[112818]: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:1214: Got exception while getting bdev bdev_0e5cdfb9-f911-476a-a15a-6cc48f7a56b9 info
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1209, in get_bdev_info
[112818]: bdevs = rpc_bdev.bdev_get_bdevs(self.spdk_rpc_client, name=bdev_name)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
[112818]: return client.call('bdev_get_bdevs', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO grpc.py:914: Received request to add a namespace using NSID 21 and UUID 9df0239e-6b14-4739-b73c-729efab5914f to nqn.2016-06.io.spdk:cnode2, ana group 2 context: None
[112818]: [22-May-2024 10:46:39] INFO grpc.py:274: Received request to create bdev bdev_9df0239e-6b14-4739-b73c-729efab5914f from rbd/DZLH-image21 (size 0 bytes) with block size 512, will not create image if doesn't exist
[112818]: [22-May-2024 10:46:39] INFO grpc.py:209: get_cluster cluster_name='cluster_context_2_1' number bdevs: 28
[112818]: [22-May-2024 10:46:39] INFO grpc.py:227: put_cluster name='cluster_context_2_1' number bdevs: 27
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:333: bdev_rbd_create bdev_9df0239e-6b14-4739-b73c-729efab5914f failed
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 317, in create_bdev
[112818]: bdev_name = rpc_bdev.bdev_rbd_create(
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1252, in bdev_rbd_create
[112818]: return client.call('bdev_rbd_create', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:958: Failure adding namespace using NSID 21 and UUID 9df0239e-6b14-4739-b73c-729efab5914f to nqn.2016-06.io.spdk:cnode2: bdev_rbd_create bdev_9df0239e-6b14-4739-b73c-729efab5914f failed with:
[112818]: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:1214: Got exception while getting bdev bdev_9df0239e-6b14-4739-b73c-729efab5914f info
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1209, in get_bdev_info
[112818]: bdevs = rpc_bdev.bdev_get_bdevs(self.spdk_rpc_client, name=bdev_name)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
[112818]: return client.call('bdev_get_bdevs', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO grpc.py:914: Received request to add a namespace using NSID 47 and UUID 522daca1-a942-4218-b120-0e8d4c804ab4 to nqn.2016-06.io.spdk:cnode1, ana group 2 context: None
[112818]: [22-May-2024 10:46:39] INFO grpc.py:274: Received request to create bdev bdev_522daca1-a942-4218-b120-0e8d4c804ab4 from rbd/5Y8B-image47 (size 0 bytes) with block size 512, will not create image if doesn't exist
[112818]: [22-May-2024 10:46:39] INFO grpc.py:209: get_cluster cluster_name='cluster_context_2_1' number bdevs: 28
[112818]: [22-May-2024 10:46:39] INFO grpc.py:227: put_cluster name='cluster_context_2_1' number bdevs: 27
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:333: bdev_rbd_create bdev_522daca1-a942-4218-b120-0e8d4c804ab4 failed
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 317, in create_bdev
[112818]: bdev_name = rpc_bdev.bdev_rbd_create(
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1252, in bdev_rbd_create
[112818]: return client.call('bdev_rbd_create', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:958: Failure adding namespace using NSID 47 and UUID 522daca1-a942-4218-b120-0e8d4c804ab4 to nqn.2016-06.io.spdk:cnode1: bdev_rbd_create bdev_522daca1-a942-4218-b120-0e8d4c804ab4 failed with:
[112818]: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:1214: Got exception while getting bdev bdev_522daca1-a942-4218-b120-0e8d4c804ab4 info
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1209, in get_bdev_info
[112818]: bdevs = rpc_bdev.bdev_get_bdevs(self.spdk_rpc_client, name=bdev_name)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
[112818]: return client.call('bdev_get_bdevs', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO grpc.py:914: Received request to add a namespace using NSID 60 and UUID 7bec4431-4774-42a2-8722-b019517ecc22 to nqn.2016-06.io.spdk:cnode2, ana group 1 context: None
[112818]: [22-May-2024 10:46:39] INFO grpc.py:274: Received request to create bdev bdev_7bec4431-4774-42a2-8722-b019517ecc22 from rbd/DZLH-image60 (size 0 bytes) with block size 512, will not create image if doesn't exist
[112818]: [22-May-2024 10:46:39] INFO grpc.py:209: get_cluster cluster_name='cluster_context_1_2' number bdevs: 3
[112818]: [22-May-2024 10:46:39] INFO grpc.py:227: put_cluster name='cluster_context_1_2' number bdevs: 2
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:333: bdev_rbd_create bdev_7bec4431-4774-42a2-8722-b019517ecc22 failed
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 317, in create_bdev
[112818]: bdev_name = rpc_bdev.bdev_rbd_create(
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1252, in bdev_rbd_create
[112818]: return client.call('bdev_rbd_create', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:958: Failure adding namespace using NSID 60 and UUID 7bec4431-4774-42a2-8722-b019517ecc22 to nqn.2016-06.io.spdk:cnode2: bdev_rbd_create bdev_7bec4431-4774-42a2-8722-b019517ecc22 failed with:
[112818]: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:1214: Got exception while getting bdev bdev_7bec4431-4774-42a2-8722-b019517ecc22 info
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1209, in get_bdev_info
[112818]: bdevs = rpc_bdev.bdev_get_bdevs(self.spdk_rpc_client, name=bdev_name)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
[112818]: return client.call('bdev_get_bdevs', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO grpc.py:914: Received request to add a namespace using NSID 178 and UUID 2f9385ec-b972-4377-a848-fc07fd3d72eb to nqn.2016-06.io.spdk:cnode1, ana group 1 context: None
[112818]: [22-May-2024 10:46:39] INFO grpc.py:274: Received request to create bdev bdev_2f9385ec-b972-4377-a848-fc07fd3d72eb from rbd/5Y8B-image178 (size 0 bytes) with block size 512, will not create image if doesn't exist
[112818]: [22-May-2024 10:46:39] INFO grpc.py:209: get_cluster cluster_name='cluster_context_1_2' number bdevs: 3
[112818]: [22-May-2024 10:46:39] INFO grpc.py:227: put_cluster name='cluster_context_1_2' number bdevs: 2
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:333: bdev_rbd_create bdev_2f9385ec-b972-4377-a848-fc07fd3d72eb failed
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 317, in create_bdev
[112818]: bdev_name = rpc_bdev.bdev_rbd_create(
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1252, in bdev_rbd_create
[112818]: return client.call('bdev_rbd_create', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:958: Failure adding namespace using NSID 178 and UUID 2f9385ec-b972-4377-a848-fc07fd3d72eb to nqn.2016-06.io.spdk:cnode1: bdev_rbd_create bdev_2f9385ec-b972-4377-a848-fc07fd3d72eb failed with:
[112818]: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:1214: Got exception while getting bdev bdev_2f9385ec-b972-4377-a848-fc07fd3d72eb info
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1209, in get_bdev_info
[112818]: bdevs = rpc_bdev.bdev_get_bdevs(self.spdk_rpc_client, name=bdev_name)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
[112818]: return client.call('bdev_get_bdevs', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO grpc.py:914: Received request to add a namespace using NSID 119 and UUID a33b225d-e2c4-469e-b9aa-69eeafe06ff2 to nqn.2016-06.io.spdk:cnode2, ana group 2 context: None
[112818]: [22-May-2024 10:46:39] INFO grpc.py:274: Received request to create bdev bdev_a33b225d-e2c4-469e-b9aa-69eeafe06ff2 from rbd/DZLH-image119 (size 0 bytes) with block size 512, will not create image if doesn't exist
[112818]: [22-May-2024 10:46:39] INFO grpc.py:209: get_cluster cluster_name='cluster_context_2_1' number bdevs: 28
[112818]: [22-May-2024 10:46:39] INFO grpc.py:227: put_cluster name='cluster_context_2_1' number bdevs: 27
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:333: bdev_rbd_create bdev_a33b225d-e2c4-469e-b9aa-69eeafe06ff2 failed
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 317, in create_bdev
[112818]: bdev_name = rpc_bdev.bdev_rbd_create(
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1252, in bdev_rbd_create
[112818]: return client.call('bdev_rbd_create', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO utils.py:392: Will compress log file /var/log/ceph/nvmeof-client.nvmeof.nvmeof_pool.argo024.iflydk/nvmeof-log to /var/log/ceph/nvmeof-client.nvmeof.nvmeof_pool.argo024.iflydk/nvmeof-log.gz
[112818]: Gateway subprocess terminated pid=19 exit_code=-6
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:958: Failure adding namespace using NSID 119 and UUID a33b225d-e2c4-469e-b9aa-69eeafe06ff2 to nqn.2016-06.io.spdk:cnode2: bdev_rbd_create bdev_a33b225d-e2c4-469e-b9aa-69eeafe06ff2 failed with:
[112818]: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] ERROR grpc.py:1214: Got exception while getting bdev bdev_a33b225d-e2c4-469e-b9aa-69eeafe06ff2 info
[112818]: Traceback (most recent call last):
[112818]: File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1209, in get_bdev_info
[112818]: bdevs = rpc_bdev.bdev_get_bdevs(self.spdk_rpc_client, name=bdev_name)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
[112818]: return client.call('bdev_get_bdevs', params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
[112818]: req_id = self.send(method, params)
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
[112818]: self.flush()
[112818]: File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
[112818]: self.sock.sendall(reqstr.encode("utf-8"))
[112818]: AttributeError: 'NoneType' object has no attribute 'sendall'
[112818]: [22-May-2024 10:46:39] INFO grpc.py:914: Received request to add a namespace using NSID 170 and UUID 1ddb7d17-5eff-4f8d-82c0-b01c21f38e6f to nqn.2016-06.io.spdk:cnode2, ana group 1 context: None
[112818]: Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedWriter name='<stderr>'> at interpreter shutdown, possibly due to daemon threads
[112818]: Python runtime state: finalizing (tstate=0x55c63ec6e7b0)
[112818]:
[112818]: Current thread 0x00007f4d1e5c2740 (most recent call first):
[112818]: <no Python frame>
May 22 10:46:41 argo024 systemd-coredump[121229]: Process 112824 (python3) of user 0 dumped core.
May 22 10:46:41 argo024 podman[121234]: 2024-05-22 10:46:41.174372147 +0000 UTC m=+0.036660453 container died ba7cd20f1848d695d8b390dea044ec936a9a233721713bafa182b82796b089ea (image=cp.stg.icr.io/cp/ibm-ceph/nvmeof-rhel9:1.2.9-1, name=ceph-b0d69cb6-1803-11ef-b429-b4835101e458-nvmeof-nvmeof_pool-argo024-iflydk, distribution-scope=public, name=ceph-nvmeof, release=1, description=Ceph NVMe over Fabrics Gateway, build-date=2024-05-20T03:09:47, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 9 Minimal, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ceph-nvmeof/images/1.2.9-1, io.k8s.description=Ceph NVMe over Fabrics Gateway, vcs-ref=5603a160785be330e0885139ca02c8f5aa588f05, com.redhat.component=ceph-nvmeof-container, version=1.2.9, io.openshift.tags=minimal rhel9, architecture=x86_64, vcs-type=git, io.buildah.version=1.29.0, summary=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., maintainer=Aviv Caro <aviv.caro>, com.redhat.license_terms=https://www.redhat.com/agreements)
May 22 10:46:41 argo024 podman[121234]: 2024-05-22 10:46:41.194126994 +0000 UTC m=+0.056415296 container remove ba7cd20f1848d695d8b390dea044ec936a9a233721713bafa182b82796b089ea (image=cp.stg.icr.io/cp/ibm-ceph/nvmeof-rhel9:1.2.9-1, name=ceph-b0d69cb6-1803-11ef-b429-b4835101e458-nvmeof-nvmeof_pool-argo024-iflydk, summary=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ceph-nvmeof/images/1.2.9-1, version=1.2.9, distribution-scope=public, vcs-type=git, release=1, io.openshift.expose-services=, maintainer=Aviv Caro <aviv.caro>, vcs-ref=5603a160785be330e0885139ca02c8f5aa588f05, com.redhat.license_terms=https://www.redhat.com/agreements, io.k8s.description=Ceph NVMe over Fabrics Gateway, build-date=2024-05-20T03:09:47, name=ceph-nvmeof, io.buildah.version=1.29.0, io.openshift.tags=minimal rhel9, com.redhat.component=ceph-nvmeof-container, description=Ceph NVMe over Fabrics Gateway, io.k8s.display-name=Red Hat Universal Base Image 9 Minimal, vendor=Red Hat, Inc.)
May 22 10:46:41 argo024 systemd[1]: ceph-b0d69cb6-1803-11ef-b429-b4835101e458.argo024.iflydk.service: Main process exited, code=exited, status=134/n/a
May 22 10:46:41 argo024 systemd[1]: ceph-b0d69cb6-1803-11ef-b429-b4835101e458.argo024.iflydk.service: Failed with result 'exit-code'.
May 22 10:46:41 argo024 systemd[1]: ceph-b0d69cb6-1803-11ef-b429-b4835101e458.argo024.iflydk.service: Consumed 20min 33.268s CPU time.
May 22 10:46:51 argo024 systemd[1]: ceph-b0d69cb6-1803-11ef-b429-b4835101e458.argo024.iflydk.service: Scheduled restart job, restart counter is at 5.
May 22 10:46:51 argo024 systemd[1]: Stopped Ceph nvmeof.nvmeof_pool.argo024.iflydk for b0d69cb6-1803-11ef-b429-b4835101e458.
May 22 10:46:51 argo024 systemd[1]: ceph-b0d69cb6-1803-11ef-b429-b4835101e458.argo024.iflydk.service: Consumed 20min 33.268s CPU time.
May 22 10:46:51 argo024 systemd[1]: Starting Ceph nvmeof.nvmeof_pool.argo024.iflydk for b0d69cb6-1803-11ef-b429-b4835101e458...
A few days ago we added a try block in prometheus:
```
diff --git a/control/prometheus.py b/control/prometheus.py
index 4d3886b..d54676c 100644
--- a/control/prometheus.py
+++ b/control/prometheus.py
@@ -391,7 +391,11 @@ class NVMeOFCollector:
str(ns.anagrpid)
], 1)
- conn_info = self.connections[nqn]
+ try:
+ conn_info = self.connections[nqn]
+ except KeyError:
+ logger.debug(f"couldn't find {nqn} in connection list, skipping")
+ continue
for conn in conn_info.connections:
host_connection_state.add_metric([
self.gw_metadata.name,
@@ -426,7 +430,10 @@ class NVMeOFCollector:
if not device_name:
# listener defined to an unbound address!
continue
- nic = nics.adapters[device_name]
+ try:
+ nic = nics.adapters[device_name]
+ except KeyError:
+ continue
subsystem_listener_iface_info.add_metric([
device_name,
nic.operstate,
```
This change makes sure we won't crash in cake of KeyError exception from a missing subsystem NQN. But, there is another possibility of prometheus crashing the gateway on an exception. As prometheus calls SPDK function, it might get exceptions in case these functions fail. Right now we don't catch such exceptions. So, any SPDK failure will result in a gateway crash. We should enclose these calls in a try block and return an error code instead of crashing.
Opened https://github.com/ceph/ceph-nvmeof/issues/666 for this. @rlepaksh I committed a change to Prometheus which should eliminate the gateway crash on such occasions.You should get it with the next image. In the meantime you can try working with Prometheus disabled. @rlepaksh Can you send us the command line used to create the namespaces? Do you use the "--rbd-create-image" parameter? *** Bug 2282839 has been marked as a duplicate of this bug. *** *** Bug 2281609 has been marked as a duplicate of this bug. *** The error seen in this bug - https://bugzilla.redhat.com/show_bug.cgi?id=2282839 which is marked as duplicate of the current bug here, is again seen post upgrade to ceph version ceph-18.2.1-190.el9cp. Gateway came up to running state once upgrade completed. However gateway again went to failed state while I tried to list the listeners. Attached the nvme service log for the failed gateway. (In reply to Gil Bregman from comment #7) > @rlepaksh Can you send us the command line used to create the > namespaces? Do you use the "--rbd-create-image" parameter? Gil, its a mix and match actually. For once I create rbd image in the namespace add CLI and then for another set of namespaces I tag already existing images to namespaces through namespace add Verifying this BZ as this particular issue is not seen with recent builds ceph version 18.2.1-190.el9cp (5eee8f17de7cfe7a752abc74828d97473040534e) reef (stable) and cp.stg.icr.io/cp/ibm-ceph/nvmeof-rhel9:1.2.10-1 But we are seeing https://bugzilla.redhat.com/show_bug.cgi?id=2282839#c13 now |
Description of problem: On longevity cluster, I am observing frequent issues as below and GW tries to start but never succeeds. The service is down over the weekend. bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 bdev_rbd.c: 434:bdev_rbd_init_context: *ERROR*: Failed to open specif bdev_rbd.c: 475:bdev_rbd_init: *ERROR*: Cannot init rbd context for rbd=0x32bcc10 bdev_rbd.c:1414:bdev_rbd_create: *ERROR*: Failed to init rbd device [2024-05-17 13:24:47.435759] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 terminate called after throwing an instance of 'std::runtime_error' [2024-05-17 13:24:47.436200] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 [2024-05-17 13:24:47.436542] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 what(): Lost connection to the monitor (gw map unavailable). [2024-05-17 13:24:47.436988] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 [2024-05-17 13:24:47.437437] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 *** Caught signal (Aborted) ** in thread 7f5d81187640 thread_name:ms_dispatch [2024-05-17 13:24:47.437833] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 [2024-05-17 13:24:47.438220] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 [2024-05-17 13:24:47.438654] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 [2024-05-17 13:24:47.439203] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 [2024-05-17 13:24:47.439593] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 [2024-05-17 13:24:47.439928] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 [2024-05-17 13:24:47.440529] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 [2024-05-17 13:24:47.440837] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 Version-Release number of selected component (if applicable): [ceph: root@tala014 /]# ceph version ceph version 18.2.1-173.el9cp (6820b327efd7c9026508bce15c960bb8c75297d3) reef (stable) nvmeof - icr.io/ibm-ceph-beta/nvmeof-rhel9:1.2.7-1 How reproducible:1/1 Steps to Reproduce: 1.deploy nvmeof cluster 2.Do some failover Failback operations - they are successful 3.Leave the cluster for 1-2 days and eventually it GW crashes Actual results: On longevity cluster upon a failure as rbd init cannot be performed, GW tries to start several times but eventually fails Expected results: On longevity cluster, we expect no GW failure but this issue has to be root caused Additional info: [17-May-2024 13:01:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:01:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:01:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:01:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:02:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:02:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:02:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:03:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:03:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:03:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:03:52] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:04:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:04:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:04:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:05:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:05:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:05:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:05:52] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:06:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:06:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:06:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:07:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:07:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:07:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:07:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:08:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:08:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:08:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:09:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:09:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:09:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:09:52] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:10:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:10:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:10:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:11:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:11:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:11:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:11:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:11:52] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:12:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:12:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:12:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:12:52] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:13:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:13:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:13:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:14:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:14:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:14:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:14:52] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:15:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:15:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:15:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:16:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:16:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:16:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:16:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:16:52] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:17:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:17:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:17:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:18:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:18:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:18:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:18:52] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:19:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:19:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:19:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:19:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:19:52] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:20:02] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:20:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:20:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:20:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:20:52] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:21:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:21:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:21:42] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:21:52] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:22:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:22:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:22:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:22:52] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:23:12] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:23:22] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:23:32] INFO grpc.py:2414: Received request to list the subsystem, context: None : [17-May-2024 13:23:52] INFO grpc.py:2414: Received request to list the subsystem, context: None : [2024-05-17 13:24:47.350392] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.351295] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.351795] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.352200] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.352624] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.352941] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.353495] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.353944] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.355187] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.355627] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.355981] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.356460] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.356814] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.357131] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.357615] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.357991] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.358492] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.358865] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.359187] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.359486] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.359758] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.360086] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.360404] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.360623] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.361039] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.361308] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.361527] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.361747] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.361971] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.362197] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.362550] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.362923] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.363262] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.363482] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.363783] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.364218] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.364573] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.364982] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.365275] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.365537] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.365826] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.366191] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.366482] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.366854] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.367171] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.367461] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.367846] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.368527] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.369034] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.369551] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.369943] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.370658] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.371137] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.371675] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.372200] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.372683] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.373047] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.373400] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.373681] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.374040] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.374301] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.374599] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.374934] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.375226] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.375542] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.375884] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.376171] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.376425] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.376666] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.376919] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.377150] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.377390] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.377706] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.377945] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.378203] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.378437] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.379008] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.379477] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.379917] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.380270] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.380506] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.380848] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.381098] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.381398] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.381630] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.381901] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.382250] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.382638] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.383031] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.383336] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.383609] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.383886] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.384179] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.384472] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.384713] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.384915] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.385218] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.385454] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.385777] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.386041] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.386316] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.386740] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.387124] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.387434] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.387753] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.388069] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.388434] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.388806] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.389086] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.389472] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.389892] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.390157] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.390484] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.390734] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.391149] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.391444] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.391698] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.392145] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.392537] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.392876] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.393188] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.393494] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.393825] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.394107] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.394538] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.394846] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.395103] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.395399] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.395738] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.396003] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.396239] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.396454] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.396739] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.397013] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.397291] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.397634] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.397860] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.398151] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.398429] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.398769] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.399145] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.399496] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.399774] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.400041] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.400472] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.400824] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.401133] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.401367] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.401677] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.402160] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.402696] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.403174] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.403485] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.403845] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.404482] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.404812] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.405187] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.405457] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.405773] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.406037] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.406382] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.406633] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.424281] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.424924] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.425322] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.425588] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.425970] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.426341] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.426686] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.426870] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.427436] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.428031] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.428582] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.429170] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.429712] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.430205] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.433099] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.433356] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.433564] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.433923] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.434110] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.434270] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.434476] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.434627] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.434964] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.435193] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.435507] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.435759] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : terminate called after throwing an instance of 'std::runtime_error' : [2024-05-17 13:24:47.436200] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.436542] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : what(): Lost connection to the monitor (gw map unavailable). : [2024-05-17 13:24:47.436988] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.437437] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : *** Caught signal (Aborted) ** : in thread 7f5d81187640 thread_name:ms_dispatch : [2024-05-17 13:24:47.437833] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.438220] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.438654] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.439203] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.439593] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.439928] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.440529] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : [2024-05-17 13:24:47.440837] bdev_rbd.c: 105:_rbd_update_callback: *ERROR*: Failed getting size -108 : ceph version 18.2.1-173.el9cp (6820b327efd7c9026508bce15c960bb8c75297d3) reef (stable) : 1: /lib64/libc.so.6(+0x3e6f0) [0x7f5d864596f0] : 2: /lib64/libc.so.6(+0x8b94c) [0x7f5d864a694c] : 3: raise() : 4: abort() : 5: /lib64/libstdc++.so.6(+0xa1b21) [0x7f5d867bdb21] : 6: /lib64/libstdc++.so.6(+0xad52c) [0x7f5d867c952c] : 7: /lib64/libstdc++.so.6(+0xad597) [0x7f5d867c9597] : 8: /lib64/libstdc++.so.6(+0xad7f9) [0x7f5d867c97f9] : 9: /usr/bin/ceph-nvmeof-monitor-client(+0x8a758) [0x564123570758] : 10: (NVMeofGwMonitorClient::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x185) [0x564123604e25] : 11: (DispatchQueue::entry()+0x542) [0x7f5d87210f52] : 12: /usr/lib64/ceph/libceph-common.so.2(+0x3fad81) [0x7f5d872a8d81] : 13: /lib64/libc.so.6(+0x89c02) [0x7f5d864a4c02] : 14: /lib64/libc.so.6(+0x10ec40) [0x7f5d86529c40] : [17-May-2024 13:24:47] ERROR server.py:42: GatewayServer: SIGCHLD received signum=17 : [17-May-2024 13:24:47] ERROR server.py:109: GatewayServer exception occurred: : Traceback (most recent call last): : File "/remote-source/ceph-nvmeof/app/control/__main__.py", line 44, in <module> : gateway.keep_alive() : File "/remote-source/ceph-nvmeof/app/control/server.py", line 485, in keep_alive : timedout = self.server.wait_for_termination(timeout=1) : File "/usr/local/lib64/python3.9/site-packages/grpc/_server.py", line 1118, in wait_for_termination : return _common.wait(self._state.termination_event.wait, : File "/usr/local/lib64/python3.9/site-packages/grpc/_common.py", line 157, in wait : _wait_once(wait_fn, remaining, spin_cb) : File "/usr/local/lib64/python3.9/site-packages/grpc/_common.py", line 112, in _wait_once : wait_fn(timeout=timeout) : File "/usr/lib64/python3.9/threading.py", line 581, in wait : signaled = self._cond.wait(timeout) : File "/usr/lib64/python3.9/threading.py", line 316, in wait : gotit = waiter.acquire(True, timeout) : File "/remote-source/ceph-nvmeof/app/control/server.py", line 54, in sigchld_handler : raise SystemExit(f"Gateway subprocess terminated {pid=} {exit_code=}") : SystemExit: Gateway subprocess terminated pid=19 exit_code=-6 : [17-May-2024 13:24:47] INFO server.py:409: Aborting (client.nvmeof.nvmeof_pool.argo023.iefwqr) pid 19... : [17-May-2024 13:24:47] INFO server.py:409: Aborting (client.nvmeof.nvmeof_pool.argo023.iefwqr) pid 94... : Traceback (most recent call last): : Traceback (most recent call last): : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 137, in run : self.result = application(self.environ, self.start_response) : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 137, in run : self.result = application(self.environ, self.start_response) : File "/usr/local/lib/python3.9/site-packages/prometheus_client/exposition.py", line 128, in prometheus_app : status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression) : File "/usr/local/lib/python3.9/site-packages/prometheus_client/exposition.py", line 128, in prometheus_app : status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression) : File "/usr/local/lib/python3.9/site-packages/prometheus_client/exposition.py", line 104, in _bake_output : output = encoder(registry) : File "/usr/local/lib/python3.9/site-packages/prometheus_client/exposition.py", line 104, in _bake_output : output = encoder(registry) : File "/usr/local/lib/python3.9/site-packages/prometheus_client/openmetrics/exposition.py", line 21, in generate_latest : for metric in registry.collect(): : File "/usr/local/lib/python3.9/site-packages/prometheus_client/openmetrics/exposition.py", line 21, in generate_latest : for metric in registry.collect(): : File "/usr/local/lib/python3.9/site-packages/prometheus_client/registry.py", line 97, in collect : yield from collector.collect() : File "/usr/local/lib/python3.9/site-packages/prometheus_client/registry.py", line 97, in collect : yield from collector.collect() : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 43, in wrapped : for metric in method(self, *args, **kwargs): : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 43, in wrapped : for metric in method(self, *args, **kwargs): : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 213, in collect : self._get_data() : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 213, in collect : self._get_data() : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 196, in _get_data : self.bdev_info = self._get_bdev_info() : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 197, in _get_data : self.bdev_io_stats = self._get_bdev_io_stats() : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 57, in call : result = method(self, *args, **kwargs) : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 57, in call : result = method(self, *args, **kwargs) : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 154, in _get_bdev_info : return rpc.bdev.bdev_get_bdevs(self.spdk_rpc_client) : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 159, in _get_bdev_io_stats : return rpc.bdev.bdev_get_iostat(self.spdk_rpc_client) : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs : return client.call('bdev_get_bdevs', params) : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1704, in bdev_get_iostat : return client.call('bdev_get_iostat', params) : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call : req_id = self.send(method, params) : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 196, in call : raise e : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send : self.flush() : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 189, in call : response = self.recv() : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush : self.sock.sendall(reqstr.encode("utf-8")) : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 170, in recv : raise JSONRPCException("Connection closed with partial response:\n%s\n" % self._recv_buf) : AttributeError: 'NoneType' object has no attribute 'sendall' : spdk.rpc.client.JSONRPCException: Connection closed with partial response: : : : : ---------------------------------------- : ---------------------------------------- : Exception occurred during processing of request from ('10.8.131.14', 47686) : Exception occurred during processing of request from ('10.8.131.14', 59824) : Traceback (most recent call last): : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 137, in run : self.result = application(self.environ, self.start_response) : File "/usr/local/lib/python3.9/site-packages/prometheus_client/exposition.py", line 128, in prometheus_app : status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression) : File "/usr/local/lib/python3.9/site-packages/prometheus_client/exposition.py", line 104, in _bake_output : output = encoder(registry) : Traceback (most recent call last): : File "/usr/local/lib/python3.9/site-packages/prometheus_client/openmetrics/exposition.py", line 21, in generate_latest : for metric in registry.collect(): : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 137, in run : self.result = application(self.environ, self.start_response) : File "/usr/local/lib/python3.9/site-packages/prometheus_client/registry.py", line 97, in collect : yield from collector.collect() : File "/usr/local/lib/python3.9/site-packages/prometheus_client/exposition.py", line 128, in prometheus_app : status, headers, output = _bake_output(registry, accept_header, accept_encoding_header, params, disable_compression) : File "/usr/local/lib/python3.9/site-packages/prometheus_client/exposition.py", line 104, in _bake_output : output = encoder(registry) : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 43, in wrapped : for metric in method(self, *args, **kwargs): : File "/usr/local/lib/python3.9/site-packages/prometheus_client/openmetrics/exposition.py", line 21, in generate_latest : for metric in registry.collect(): : File "/usr/local/lib/python3.9/site-packages/prometheus_client/registry.py", line 97, in collect : yield from collector.collect() : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 43, in wrapped : for metric in method(self, *args, **kwargs): : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 213, in collect : self._get_data() : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 213, in collect : self._get_data() : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 196, in _get_data : self.bdev_info = self._get_bdev_info() : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 57, in call : result = method(self, *args, **kwargs) : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 154, in _get_bdev_info : return rpc.bdev.bdev_get_bdevs(self.spdk_rpc_client) : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs : return client.call('bdev_get_bdevs', params) : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call : req_id = self.send(method, params) : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send : self.flush() : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush : self.sock.sendall(reqstr.encode("utf-8")) : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 197, in _get_data : self.bdev_io_stats = self._get_bdev_io_stats() : AttributeError: 'NoneType' object has no attribute 'sendall' : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 57, in call : result = method(self, *args, **kwargs) : File "/remote-source/ceph-nvmeof/app/control/prometheus.py", line 159, in _get_bdev_io_stats : return rpc.bdev.bdev_get_iostat(self.spdk_rpc_client) : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1704, in bdev_get_iostat : return client.call('bdev_get_iostat', params) : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 196, in call : raise e : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 189, in call : response = self.recv() : File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 170, in recv : raise JSONRPCException("Connection closed with partial response:\n%s\n" % self._recv_buf) : : During handling of the above exception, another exception occurred: : : spdk.rpc.client.JSONRPCException: Connection closed with partial response: : : : : Traceback (most recent call last): : : During handling of the above exception, another exception occurred: : : File "/usr/lib64/python3.9/socketserver.py", line 683, in process_request_thread : self.finish_request(request, client_address) : Traceback (most recent call last): : File "/usr/lib64/python3.9/socketserver.py", line 360, in finish_request : self.RequestHandlerClass(request, client_address, self) : File "/usr/lib64/python3.9/socketserver.py", line 683, in process_request_thread : self.finish_request(request, client_address) : File "/usr/lib64/python3.9/socketserver.py", line 747, in __init__ : self.handle() : File "/usr/lib64/python3.9/socketserver.py", line 360, in finish_request : self.RequestHandlerClass(request, client_address, self) : File "/usr/lib64/python3.9/wsgiref/simple_server.py", line 134, in handle : handler.run(self.server.get_app()) : File "/usr/lib64/python3.9/socketserver.py", line 747, in __init__ : self.handle() : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 145, in run : self.handle_error() : File "/usr/lib64/python3.9/wsgiref/simple_server.py", line 134, in handle : handler.run(self.server.get_app()) : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 382, in handle_error : self.finish_response() : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 145, in run : self.handle_error() : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 382, in handle_error : self.finish_response() : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 184, in finish_response : self.write(data) : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 184, in finish_response : self.write(data) : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 288, in write : self.send_headers() : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 346, in send_headers : self.send_preamble() : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 268, in send_preamble : self._write( : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 467, in _write : result = self.stdout.write(data) : File "/usr/lib64/python3.9/socketserver.py", line 826, in write : self._sock.sendall(b) : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 288, in write : self.send_headers() : BrokenPipeError: [Errno 32] Broken pipe : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 346, in send_headers : self.send_preamble() : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 272, in send_preamble : self._write(('Server: %s\r\n' % self.server_software).encode('iso-8859-1')) : ---------------------------------------- : File "/usr/lib64/python3.9/wsgiref/handlers.py", line 467, in _write : result = self.stdout.write(data) : File "/usr/lib64/python3.9/socketserver.py", line 826, in write : self._sock.sendall(b) : BrokenPipeError: [Errno 32] Broken pipe : ---------------------------------------- : [17-May-2024 13:24:56] INFO server.py:123: Stopping the server... : [17-May-2024 13:24:56] INFO server.py:441: Terminating discovery service... : [17-May-2024 13:24:56] INFO server.py:448: Discovery service terminated : [17-May-2024 13:24:56] INFO server.py:130: Exiting the gateway process. : [17-May-2024 13:24:56] INFO utils.py:392: Will compress log file /var/log/ceph/nvmeof-client.nvmeof.nvmeof_pool.argo023.iefwqr/nvmeof-log to /var/log/ceph/nvmeof-client.nvmeof.nvmeof_pool.argo023.iefwqr/nvmeof-log.gz : Gateway subprocess terminated pid=19 exit_code=-6 May 17 13:24:56 argo023 ceph-a44110aa-12ca-11ef-bd4c-b4835101e458-nvmeof-nvmeof_pool-argo023-iefwqr[553660]: Gateway subprocess terminated pid=19 exit_code=-6 May 17 13:24:56 argo023 podman[1144773]: 2024-05-17 13:24:56.487517043 +0000 UTC m=+0.034208133 container died 4bf1694753b28e1c81fdcb23568fab322500fe857628fa0fdac911e110c7c4b6 (image=icr.io/ibm-ceph-beta/nvmeof-rhel9:1.2.7-1, name=ceph-a44110aa-12ca-11ef-bd4c-b4835101e458-nvmeof-nvmeof_pool-argo023-iefwqr, io.buildah.version=1.29.0, io.openshift.expose-services=, vcs-ref=d38cea6ec0f9d2375fee988966ed08ce445bdda7, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/agreements, summary=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 9 Minimal, maintainer=Aviv Caro <aviv.caro>, description=Ceph NVMe over Fabrics Gateway, io.k8s.description=Ceph NVMe over Fabrics Gateway, io.openshift.tags=minimal rhel9, name=ceph-nvmeof, release=1, vendor=Red Hat, Inc., version=1.2.7, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ceph-nvmeof/images/1.2.7-1, build-date=2024-05-13T13:32:55, com.redhat.component=ceph-nvmeof-container) May 17 13:24:56 argo023 podman[1144773]: 2024-05-17 13:24:56.514861001 +0000 UTC m=+0.061552076 container remove 4bf1694753b28e1c81fdcb23568fab322500fe857628fa0fdac911e110c7c4b6 (image=icr.io/ibm-ceph-beta/nvmeof-rhel9:1.2.7-1, name=ceph-a44110aa-12ca-11ef-bd4c-b4835101e458-nvmeof-nvmeof_pool-argo023-iefwqr, maintainer=Aviv Caro <aviv.caro>, version=1.2.7, architecture=x86_64, name=ceph-nvmeof, vcs-type=git, description=Ceph NVMe over Fabrics Gateway, distribution-scope=public, io.k8s.description=Ceph NVMe over Fabrics Gateway, io.openshift.tags=minimal rhel9, release=1, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ceph-nvmeof/images/1.2.7-1, build-date=2024-05-13T13:32:55, io.k8s.display-name=Red Hat Universal Base Image 9 Minimal, vendor=Red Hat, Inc., vcs-ref=d38cea6ec0f9d2375fee988966ed08ce445bdda7, io.buildah.version=1.29.0, summary=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., io.openshift.expose-services=, com.redhat.component=ceph-nvmeof-container, com.redhat.license_terms=https://www.redhat.com/agreements) May 17 13:24:56 argo023 systemd[1]: ceph-a44110aa-12ca-11ef-bd4c-b4835101e458.argo023.iefwqr.service: Main process exited, code=exited, status=1/FAILURE May 17 13:24:56 argo023 systemd[1]: ceph-a44110aa-12ca-11ef-bd4c-b4835101e458.argo023.iefwqr.service: Failed with result 'exit-code'. May 17 13:24:56 argo023 systemd[1]: ceph-a44110aa-12ca-11ef-bd4c-b4835101e458.argo023.iefwqr.service: Consumed 4d 21h 8min 35.522s CPU time. May 17 13:25:06 argo023 systemd[1]: ceph-a44110aa-12ca-11ef-bd4c-b4835101e458.argo023.iefwqr.service: Scheduled restart job, restart counter is at 1. May 17 13:25:06 argo023 systemd[1]: Stopped Ceph nvmeof.nvmeof_pool.argo023.iefwqr for a44110aa-12ca-11ef-bd4c-b4835101e458. May 17 13:25:06 argo023 systemd[1]: ceph-a44110aa-12ca-11ef-bd4c-b4835101e458.argo023.iefwqr.service: Consumed 4d 21h 8min 35.522s CPU time. May 17 13:25:06 argo023 systemd[1]: Starting Ceph nvmeof.nvmeof_pool.argo023.iefwqr for a44110aa-12ca-11ef-bd4c-b4835101e458...