Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2282364

Summary:	NVMeoF GW Upgrade from 18.2.1-173 to 18.2.1-188 failed
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Sunil Kumar Nagaraju <sunnagar>
Component:	Cephadm	Assignee:	Adam King <adking>
Status:	CLOSED ERRATA	QA Contact:	Sunil Kumar Nagaraju <sunnagar>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	7.1	CC:	cephqe-warriors, owasserm, sostapov, tserlin, vereddy
Target Milestone:	---	Keywords:	UpgradeBlocker
Target Release:	7.1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-18.2.1-190.el8cp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2024-06-13 14:32:55 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sunil Kumar Nagaraju 2024-05-22 09:30:18 UTC

Description of problem:

Build to Build Upgrade failed from 18.2.1-173 to 18.2.1-188 with below error,

Module 'cephadm' has failed: 'NvmeofService' object has no attribute 'pool'; 1 mgr modules have recently crashed

[ceph: root@ceph-sunilkumar-00-wu7f9b-node1-installer /]# ceph -s
  cluster:
    id:     2f6cdb28-1811-11ef-a525-fa163e45664f
    health: HEALTH_ERR
            1 failed cephadm daemon(s)
            Module 'cephadm' has failed: 'NvmeofService' object has no attribute 'pool'
            1 mgr modules have recently crashed

  services:
    mon: 3 daemons, quorum ceph-sunilkumar-00-wu7f9b-node1-installer,ceph-sunilkumar-00-wu7f9b-node2,ceph-sunilkumar-00-wu7f9b-node3 (age 20m)
    mgr: ceph-sunilkumar-00-wu7f9b-node1-installer.betcxx(active, since 20m), standbys: ceph-sunilkumar-00-wu7f9b-node2.weqlcd
    osd: 12 osds: 12 up (since 16m), 12 in (since 43m)

  data:
    pools:   2 pools, 65 pgs
    objects: 29 objects, 450 KiB
    usage:   346 MiB used, 240 GiB / 240 GiB avail
    pgs:     65 active+clean

  io:
    client:   6.7 KiB/s rd, 2 op/s rd, 0 op/s wr

  progress:
    Upgrade to 18.2.1-188.el9cp (4m)
      [===================.........] (remaining: 2m)

[ceph: root@ceph-sunilkumar-00-wu7f9b-node1-installer /]#
[ceph: root@ceph-sunilkumar-00-wu7f9b-node1-installer /]# ceph health detail
HEALTH_ERR 1 failed cephadm daemon(s); Module 'cephadm' has failed: 'NvmeofService' object has no attribute 'pool'; 1 mgr modules have recently crashed
[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)
    daemon nvmeof.rbd.ceph-sunilkumar-00-wu7f9b-node8.obpiyw on ceph-sunilkumar-00-wu7f9b-node8 is in unknown state
[ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: 'NvmeofService' object has no attribute 'pool'
    Module 'cephadm' has failed: 'NvmeofService' object has no attribute 'pool'
[WRN] RECENT_MGR_MODULE_CRASH: 1 mgr modules have recently crashed
    mgr module cephadm crashed in daemon mgr.ceph-sunilkumar-00-wu7f9b-node1-installer.betcxx on host ceph-sunilkumar-00-wu7f9b-node1-installer at 2024-05-22T08:39:41.326745Z
[ceph: root@ceph-sunilkumar-00-wu7f9b-node1-installer /]# ceph crash ls
ID                                                                ENTITY                                                NEW
2024-05-22T08:39:41.326745Z_15d4193e-5c39-4827-9241-6b5c9fa3733b  mgr.ceph-sunilkumar-00-wu7f9b-node1-installer.betcxx   *
[ceph: root@ceph-sunilkumar-00-wu7f9b-node1-installer /]# ceph crash info 2024-05-22T08:39:41.326745Z_15d4193e-5c39-4827-9241-6b5c9fa3733b
{
    "backtrace": [
        "  File \"/usr/share/ceph/mgr/cephadm/module.py\", line 693, in serve\n    serve.serve()",
        "  File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 104, in serve\n    self._check_daemons()",
        "  File \"/usr/share/ceph/mgr/cephadm/serve.py\", line 1132, in _check_daemons\n    self.mgr._get_cephadm_service(daemon_type_to_service(",
        "  File \"/usr/share/ceph/mgr/cephadm/services/nvmeof.py\", line 80, in daemon_check_post\n    if not self.pool or self.group is None:",
        "AttributeError: 'NvmeofService' object has no attribute 'pool'"
    ],
    "ceph_version": "18.2.1-188.el9cp",
    "crash_id": "2024-05-22T08:39:41.326745Z_15d4193e-5c39-4827-9241-6b5c9fa3733b",
    "entity_name": "mgr.ceph-sunilkumar-00-wu7f9b-node1-installer.betcxx",
    "mgr_module": "cephadm",
    "mgr_module_caller": "PyModuleRunner::serve",
    "mgr_python_exception": "AttributeError",
    "os_id": "rhel",
    "os_name": "Red Hat Enterprise Linux",
    "os_version": "9.4 (Plow)",
    "os_version_id": "9.4",
    "process_name": "ceph-mgr",
    "stack_sig": "c58c44668afb6184785fe6f5f59e05a1b392f7a28f9215c2049992767793317a",
    "timestamp": "2024-05-22T08:39:41.326745Z",
    "utsname_hostname": "ceph-sunilkumar-00-wu7f9b-node1-installer",
    "utsname_machine": "x86_64",
    "utsname_release": "5.14.0-427.16.1.el9_4.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Fri Apr 26 18:16:09 EDT 2024"
}


Version-Release number of selected component (if applicable):
Upgraded to 18.2.1-188 from 18.2.1-173



How reproducible: always


Steps to Reproduce:
1. Deploy Ceph with 18.2.1-173
2. Configure MON, MGR, OSDs and NVMeoF(1.2.7-1) services.
3. Upgrade teh cluster with 18.2.1-188 with NVMe 1.2.9-2


[ceph: root@ceph-sunilkumar-00-wu7f9b-node1-installer /]# ceph -s
  cluster:
    id:     2f6cdb28-1811-11ef-a525-fa163e45664f
    health: HEALTH_ERR
            1 failed cephadm daemon(s)
            Module 'cephadm' has failed: 'NvmeofService' object has no attribute 'pool'
            1 mgr modules have recently crashed

  services:
    mon: 3 daemons, quorum ceph-sunilkumar-00-wu7f9b-node1-installer,ceph-sunilkumar-00-wu7f9b-node2,ceph-sunilkumar-00-wu7f9b-node3 (age 55m)
    mgr: ceph-sunilkumar-00-wu7f9b-node1-installer.betcxx(active, since 56m), standbys: ceph-sunilkumar-00-wu7f9b-node2.weqlcd
    osd: 12 osds: 12 up (since 51m), 12 in (since 79m)

  data:
    pools:   2 pools, 65 pgs
    objects: 29 objects, 450 KiB
    usage:   346 MiB used, 240 GiB / 240 GiB avail
    pgs:     65 active+clean

  io:
    client:   6.7 KiB/s rd, 2 op/s rd, 0 op/s wr

  progress:
    Upgrade to 18.2.1-188.el9cp (4m)
      [===================.........] (remaining: 2m)

[ceph: root@ceph-sunilkumar-00-wu7f9b-node1-installer /]#
[ceph: root@ceph-sunilkumar-00-wu7f9b-node1-installer /]#
[ceph: root@ceph-sunilkumar-00-wu7f9b-node1-installer /]# ceph orch ps | grep nvme
nvmeof.rbd.ceph-sunilkumar-00-wu7f9b-node6.wkqysx        ceph-sunilkumar-00-wu7f9b-node6            *:5500,4420,8009  running (48m)    48m ago  76m     123M        -                    2a89845c0654  796bba245836
nvmeof.rbd.ceph-sunilkumar-00-wu7f9b-node7.ffmxwo        ceph-sunilkumar-00-wu7f9b-node7            *:5500,4420,8009  running (48m)    48m ago  75m     118M        -                    2a89845c0654  7e3edc4da01d
nvmeof.rbd.ceph-sunilkumar-00-wu7f9b-node8.obpiyw        ceph-sunilkumar-00-wu7f9b-node8            *:5500,4420,8009  unknown          48m ago  76m        -        -                    2a89845c0654  8e6a48e80c84
nvmeof.rbd.ceph-sunilkumar-00-wu7f9b-node9.amccwo        ceph-sunilkumar-00-wu7f9b-node9            *:5500,4420,8009  running (48m)    48m ago  75m    45.6M        -                    2a89845c0654  27d3d6be62f7
[ceph: root@ceph-sunilkumar-00-wu7f9b-node1-installer /]# ceph versions
{
    "mon": {
        "ceph version 18.2.1-188.el9cp (b1ae9c989e2f41dcfec0e680c11d1d9465b1db0e) reef (stable)": 3
    },
    "mgr": {
        "ceph version 18.2.1-188.el9cp (b1ae9c989e2f41dcfec0e680c11d1d9465b1db0e) reef (stable)": 2
    },
    "osd": {
        "ceph version 18.2.1-188.el9cp (b1ae9c989e2f41dcfec0e680c11d1d9465b1db0e) reef (stable)": 12
    },
    "overall": {
        "ceph version 18.2.1-188.el9cp (b1ae9c989e2f41dcfec0e680c11d1d9465b1db0e) reef (stable)": 17
    }
}
[ceph: root@ceph-sunilkumar-00-wu7f9b-node1-installer /]#

Additional info:

[ceph: root@ceph-sunilkumar-00-wu7f9b-node1-installer /]# ceph orch upgrade status
{
    "target_image": "registry-proxy.engineering.redhat.com/rh-osbs/ibm-ceph@sha256:6f6f42c84c5c39bc3150281754f96ef4f91b88b8f9246b3a4e109d3384bd42a1",
    "in_progress": true,
    "which": "Upgrading all daemon types on all hosts",
    "services_complete": [
        "mon",
        "mgr",
        "crash",
        "osd",
        "ceph-exporter"
    ],
    "progress": "35/51 daemons upgraded",
    "message": "Currently upgrading nvmeof daemons",
    "is_paused": false
}

Comment 3 Sunil Kumar Nagaraju 2024-05-22 13:31:28 UTC

After debugging more,
- Found one of the NVMe GW daemon is in unknown state, below is the crash inside the daemon.
- Orch PS status for the daemon is unknown, but container is still running

[root@ceph-sunilkumar-00-wu7f9b-node8 ~]# podman ps | grep nvme
8e6a48e80c84  registry-proxy.engineering.redhat.com/rh-osbs/ceph-nvmeof:1.2.9-2                                                               -c ceph-nvmeof.co...  5 hours ago  Up 5 hours              ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw


May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:209: get_cluster cluster_name='cluster_context_3_0' number bdevs: 1
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [2024-05-22 08:39:27.152760] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_34ad1f17-bd59-4681-aadd-7a976e6f28c4 rbd disk to lun
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:760: Received request to add bdev_34ad1f17-bd59-4681-aadd-7a976e6f28c4 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 7 and UUID 34ad1f17-bd59-4681-aadd-7a976e6f28c4
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:914: Received request to add a namespace using NSID 8 and UUID 04c45124-482f-467b-b2b5-5b8d18a02910 to nqn.2016-06.io.spdk:cnode1,  ana group 3  context: None
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:274: Received request to create bdev bdev_04c45124-482f-467b-b2b5-5b8d18a02910 from rbd/IRVL-image1 (size 2147483648 bytes) with block size 512, will not create image if doesn't exist
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:209: get_cluster cluster_name='cluster_context_3_0' number bdevs: 2
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [2024-05-22 08:39:27.187825] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_04c45124-482f-467b-b2b5-5b8d18a02910 rbd disk to lun
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:760: Received request to add bdev_04c45124-482f-467b-b2b5-5b8d18a02910 to nqn.2016-06.io.spdk:cnode1 with ANA group id 3 using NSID 8 and UUID 04c45124-482f-467b-b2b5-5b8d18a02910
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:914: Received request to add a namespace using NSID 5 and UUID 66fb6841-26f8-453a-a63e-15df97ec0821 to nqn.2016-06.io.spdk:cnode1,  ana group 1  context: None
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:274: Received request to create bdev bdev_66fb6841-26f8-453a-a63e-15df97ec0821 from rbd/QFOE-image0 (size 2147483648 bytes) with block size 512, will not create image if doesn't exist
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:209: get_cluster cluster_name='cluster_context_1_0' number bdevs: 2
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: terminate called after throwing an instance of 'std::runtime_error'
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   what():  Lost connection to the monitor (gw map unavailable).
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: *** Caught signal (Aborted) **
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  in thread 7ff0e787c640 thread_name:ms_dispatch
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  ceph version 18.2.1-188.el9cp (b1ae9c989e2f41dcfec0e680c11d1d9465b1db0e) reef (stable)
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  1: /lib64/libc.so.6(+0x3e6f0) [0x7ff0ecb4e6f0]
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  2: /lib64/libc.so.6(+0x8b94c) [0x7ff0ecb9b94c]
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  3: raise()
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  4: abort()
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  5: /lib64/libstdc++.so.6(+0xa1b21) [0x7ff0eceb2b21]
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  6: /lib64/libstdc++.so.6(+0xad52c) [0x7ff0ecebe52c]
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  7: /lib64/libstdc++.so.6(+0xad597) [0x7ff0ecebe597]
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  8: /lib64/libstdc++.so.6(+0xad7f9) [0x7ff0ecebe7f9]
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  9: /usr/bin/ceph-nvmeof-monitor-client(+0x8a758) [0x5582f0edb758]
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  10: (NVMeofGwMonitorClient::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x185) [0x5582f0f6fe25]
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  11: (DispatchQueue::entry()+0x542) [0x7ff0ed905f52]
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  12: /usr/lib64/ceph/libceph-common.so.2(+0x3fad81) [0x7ff0ed99dd81]
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  13: /lib64/libc.so.6(+0x89c02) [0x7ff0ecb99c02]
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:  14: /lib64/libc.so.6(+0x10ec40) [0x7ff0ecc1ec40]
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [2024-05-22 08:39:27.237646] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_66fb6841-26f8-453a-a63e-15df97ec0821 rbd disk to lun
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:760: Received request to add bdev_66fb6841-26f8-453a-a63e-15df97ec0821 to nqn.2016-06.io.spdk:cnode1 with ANA group id 1 using NSID 5 and UUID 66fb6841-26f8-453a-a63e-15df97ec0821
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:914: Received request to add a namespace using NSID 2 and UUID 2742961f-4f78-45d0-b770-bde282bc176d to nqn.2016-06.io.spdk:cnode1,  ana group 2  context: None
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:274: Received request to create bdev bdev_2742961f-4f78-45d0-b770-bde282bc176d from rbd/Z579-image1 (size 2147483648 bytes) with block size 512, will not create image if doesn't exist
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:209: get_cluster cluster_name='cluster_context_2_0' number bdevs: 2
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 systemd-coredump[27972]: Process 26953 (ceph-nvmeof-mon) of user 0 dumped core.
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [2024-05-22 08:39:27.299040] bdev_rbd.c:1438:bdev_rbd_create: *NOTICE*: Add bdev_2742961f-4f78-45d0-b770-bde282bc176d rbd disk to lun
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] ERROR server.py:42: GatewayServer: SIGCHLD received signum=17
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO grpc.py:760: Received request to add bdev_2742961f-4f78-45d0-b770-bde282bc176d to nqn.2016-06.io.spdk:cnode1 with ANA group id 2 using NSID 2 and UUID 2742961f-4f78-45d0-b770-bde282bc176d
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] ERROR server.py:46: PID of terminated child process is 19
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] ERROR server.py:110: GatewayServer exception occurred:
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: Traceback (most recent call last):
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/remote-source/ceph-nvmeof/app/control/__main__.py", line 45, in <module>
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     gateway.keep_alive()
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/remote-source/ceph-nvmeof/app/control/server.py", line 487, in keep_alive
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     timedout = self.server.wait_for_termination(timeout=1)
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib64/python3.9/site-packages/grpc/_server.py", line 1118, in wait_for_termination
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     return _common.wait(self._state.termination_event.wait,
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib64/python3.9/site-packages/grpc/_common.py", line 157, in wait
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     _wait_once(wait_fn, remaining, spin_cb)
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib64/python3.9/site-packages/grpc/_common.py", line 112, in _wait_once
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     wait_fn(timeout=timeout)
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/lib64/python3.9/threading.py", line 581, in wait
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     signaled = self._cond.wait(timeout)
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/lib64/python3.9/threading.py", line 316, in wait
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     gotit = waiter.acquire(True, timeout)
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/remote-source/ceph-nvmeof/app/control/server.py", line 55, in sigchld_handler
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     raise SystemExit(f"Gateway subprocess terminated {pid=} {exit_code=}")
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: SystemExit: Gateway subprocess terminated pid=19 exit_code=-6
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO server.py:411: Aborting (client.nvmeof.rbd.ceph-sunilkumar-00-wu7f9b-node8.obpiyw) pid 19...
May 22 04:39:27 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:27] INFO server.py:411: Aborting (client.nvmeof.rbd.ceph-sunilkumar-00-wu7f9b-node8.obpiyw) pid 70...
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 systemd-coredump[27977]: Process 27004 (reactor_0) of user 0 dumped core.

                                                                         Stack trace of thread 70:
                                                                         #0  0x00007fdfb39b76c5 n/a (/usr/local/lib/libspdk_thread.so.9.0 + 0x46c5)
                                                                         ELF object binary architecture: AMD x86-64
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] ERROR grpc.py:785: Failure adding namespace using NSID 2 and UUID 2742961f-4f78-45d0-b770-bde282bc176dto nqn.2016-06.io.spdk:cnode1
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: Traceback (most recent call last):
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 773, in create_namespace
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     nsid = rpc_nvmf.nvmf_subsystem_add_ns(
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/nvmf.py", line 501, in nvmf_subsystem_add_ns
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     return client.call('nvmf_subsystem_add_ns', params)
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 189, in call
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     response = self.recv()
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 166, in recv
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     newdata = self.sock.recv(4096)
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: ConnectionResetError: [Errno 104] Connection reset by peer
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO grpc.py:416: Received request to delete bdev bdev_2742961f-4f78-45d0-b770-bde282bc176d
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] ERROR grpc.py:429: Failure deleting bdev bdev_2742961f-4f78-45d0-b770-bde282bc176d
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: Traceback (most recent call last):
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 418, in delete_bdev
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     ret = rpc_bdev.bdev_rbd_delete(
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1262, in bdev_rbd_delete
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     return client.call('bdev_rbd_delete', params)
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     req_id = self.send(method, params)
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     self.flush()
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     self.sock.sendall(reqstr.encode("utf-8"))
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: BrokenPipeError: [Errno 32] Broken pipe
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] WARNING grpc.py:987: Failure 22 deleting bdev bdev_2742961f-4f78-45d0-b770-bde282bc176d: Failure deleting bdev bdev_2742961f-4f78-45d0-b770-bde282bc176d:
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [Errno 32] Broken pipe
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] ERROR grpc.py:994: Failure adding namespace using NSID 2 and UUID 2742961f-4f78-45d0-b770-bde282bc176d to nqn.2016-06.io.spdk:cnode1:Failure adding namespace using NSID 2 and UUID 2742961f-4f78-45d0-b770-bde282bc176dto nqn.2016-06.io.spdk:cnode1:
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [Errno 104] Connection reset by peer
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO grpc.py:914: Received request to add a namespace using NSID 3 and UUID f2abe28b-da30-4064-aec2-a5cbde7f5236 to nqn.2016-06.io.spdk:cnode1,  ana group 4  context: None
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO grpc.py:274: Received request to create bdev bdev_f2abe28b-da30-4064-aec2-a5cbde7f5236 from rbd/PZB9-image1 (size 2147483648 bytes) with block size 512, will not create image if doesn't exist
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO grpc.py:209: get_cluster cluster_name='cluster_context_4_0' number bdevs: 2
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO grpc.py:227: put_cluster name='cluster_context_4_0' number bdevs: 1
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] ERROR grpc.py:333: bdev_rbd_create bdev_f2abe28b-da30-4064-aec2-a5cbde7f5236 failed
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: Traceback (most recent call last):
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 317, in create_bdev
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     bdev_name = rpc_bdev.bdev_rbd_create(
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1252, in bdev_rbd_create
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     return client.call('bdev_rbd_create', params)
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     req_id = self.send(method, params)
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     self.flush()
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     self.sock.sendall(reqstr.encode("utf-8"))
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: BrokenPipeError: [Errno 32] Broken pipe
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] ERROR grpc.py:958: Failure adding namespace using NSID 3 and UUID f2abe28b-da30-4064-aec2-a5cbde7f5236 to nqn.2016-06.io.spdk:cnode1: bdev_rbd_create bdev_f2abe28b-da30-4064-aec2-a5cbde7f5236 failed with:
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [Errno 32] Broken pipe
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] ERROR grpc.py:1214: Got exception while getting bdev bdev_f2abe28b-da30-4064-aec2-a5cbde7f5236 info
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: Traceback (most recent call last):
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1209, in get_bdev_info
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     bdevs = rpc_bdev.bdev_get_bdevs(self.spdk_rpc_client, name=bdev_name)
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/bdev.py", line 1686, in bdev_get_bdevs
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     return client.call('bdev_get_bdevs', params)
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     req_id = self.send(method, params)
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     self.flush()
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     self.sock.sendall(reqstr.encode("utf-8"))
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: BrokenPipeError: [Errno 32] Broken pipe
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO grpc.py:1755: Received request to allow any host access for nqn.2016-06.io.spdk:cnode1, context: None
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] ERROR grpc.py:1773: Failure allowing open host access to nqn.2016-06.io.spdk:cnode1
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: Traceback (most recent call last):
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 1756, in add_host_safe
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     ret = rpc_nvmf.nvmf_subsystem_allow_any_host(
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/nvmf.py", line 583, in nvmf_subsystem_allow_any_host
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     return client.call('nvmf_subsystem_allow_any_host', params)
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     req_id = self.send(method, params)
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     self.flush()
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     self.sock.sendall(reqstr.encode("utf-8"))
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: BrokenPipeError: [Errno 32] Broken pipe
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO grpc.py:2099: Received request to create ceph-sunilkumar-00-wu7f9b-node9 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.0.210.186:4420, context: None
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO grpc.py:2099: Received request to create ceph-sunilkumar-00-wu7f9b-node8 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.0.208.167:4420, context: None
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] ERROR grpc.py:2140: Failure adding nqn.2016-06.io.spdk:cnode1 listener at 10.0.208.167:4420
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: Traceback (most recent call last):
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/remote-source/ceph-nvmeof/app/control/grpc.py", line 2120, in create_listener_safe
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     ret = rpc_nvmf.nvmf_subsystem_add_listener(
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/nvmf.py", line 272, in nvmf_subsystem_add_listener
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     return client.call('nvmf_subsystem_add_listener', params)
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 187, in call
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     req_id = self.send(method, params)
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 145, in send
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     self.flush()
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:   File "/usr/local/lib/python3.9/site-packages/spdk/rpc/client.py", line 141, in flush
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]:     self.sock.sendall(reqstr.encode("utf-8"))
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: BrokenPipeError: [Errno 32] Broken pipe
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO grpc.py:2099: Received request to create ceph-sunilkumar-00-wu7f9b-node7 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.0.208.224:4420, context: None
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO grpc.py:2099: Received request to create ceph-sunilkumar-00-wu7f9b-node6 TCP ipv4 listener for nqn.2016-06.io.spdk:cnode1 at 10.0.211.90:4420, context: None
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO server.py:124: Stopping the server...
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO server.py:443: Terminating discovery service...
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO server.py:450: Discovery service terminated
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO server.py:131: Exiting the gateway process.
May 22 04:39:28 ceph-sunilkumar-00-wu7f9b-node8 ceph-2f6cdb28-1811-11ef-a525-fa163e45664f-nvmeof-rbd-ceph-sunilkumar-00-wu7f9b-node8-obpiyw[26932]: [22-May-2024 08:39:28] INFO utils.py:392: Will compress log file /var/log/ceph/nvmeof-client.nvmeof.rbd.ceph-sunilkumar-00-wu7f9b-node8.obpiyw/nvmeof-log to /var/log/ceph/nvmeof-client.nvmeof.rbd.ceph-sunilkumar-00-wu7f9b-node8.obpiyw/nvmeof-log.gz

Comment 4 Orit Wasserman 2024-05-22 14:40:33 UTC

This looks like cephadm issue

Comment 5 Scott Ostapovicz 2024-05-22 14:56:01 UTC

Reassigned as per Orit

Comment 10 Sunil Kumar Nagaraju 2024-05-28 14:58:51 UTC

Upgrade is successful from 18.2.1-188 to 18.2.1-190 along NVMe 1.2.9-2 to 1.2.10-1.

# ceph versions
{
    "mon": {
        "ceph version 18.2.1-190.el9cp (5eee8f17de7cfe7a752abc74828d97473040534e) reef (stable)": 3
    },
    "mgr": {
        "ceph version 18.2.1-190.el9cp (5eee8f17de7cfe7a752abc74828d97473040534e) reef (stable)": 2
    },
    "osd": {
        "ceph version 18.2.1-190.el9cp (5eee8f17de7cfe7a752abc74828d97473040534e) reef (stable)": 12
    },
    "overall": {
        "ceph version 18.2.1-190.el9cp (5eee8f17de7cfe7a752abc74828d97473040534e) reef (stable)": 17
    }
}

[ceph: root@ceph-sunilkumar-01-huwm3f-node1-installer /]# ceph -s
  cluster:
    id:     50d4f036-1c0b-11ef-b9a1-fa163e3681e4
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph-sunilkumar-01-huwm3f-node1-installer,ceph-sunilkumar-01-huwm3f-node3,ceph-sunilkumar-01-huwm3f-node2 (age 47m)
    mgr: ceph-sunilkumar-01-huwm3f-node2.fzztlh(active, since 48m), standbys: ceph-sunilkumar-01-huwm3f-node1-installer.ukgshv
    osd: 12 osds: 12 up (since 43m), 12 in (since 29h)

  data:
    pools:   2 pools, 257 pgs
    objects: 10.79k objects, 42 GiB
    usage:   139 GiB used, 101 GiB / 240 GiB avail
    pgs:     257 active+clean

  io:
    client:   171 KiB/s rd, 195 op/s rd, 0 op/s wr

[ceph: root@ceph-sunilkumar-01-huwm3f-node1-installer /]# ceph orch ls
NAME                       PORTS             RUNNING  REFRESHED  AGE  PLACEMENT
alertmanager               ?:9093,9094           1/1  8m ago     29h  count:1
ceph-exporter                                    9/9  8m ago     29h  *
crash                                            9/9  8m ago     29h  *
grafana                    ?:3000                1/1  8m ago     29h  count:1
mgr                                              2/2  8m ago     29h  label:mgr
mon                                              3/3  8m ago     29h  label:mon
node-exporter              ?:9100                9/9  8m ago     29h  *
node-proxy                                       0/0  -          29h  *
nvmeof.rbd                 ?:4420,5500,8009      4/4  8m ago     29h  ceph-sunilkumar-01-huwm3f-node6;ceph-sunilkumar-01-huwm3f-node7;ceph-sunilkumar-01-huwm3f-node8;ceph-sunilkumar-01-huwm3f-node9
osd.all-available-devices                         12  8m ago     29h  *
prometheus                 ?:9095                1/1  8m ago     29h  count:1
[ceph: root@ceph-sunilkumar-01-huwm3f-node1-installer /]# ceph orch ps
NAME                                                     HOST                                       PORTS             STATUS         REFRESHED  AGE  MEM USE  MEM LIM  VERSION           IMAGE ID      CONTAINER ID
alertmanager.ceph-sunilkumar-01-huwm3f-node1-installer   ceph-sunilkumar-01-huwm3f-node1-installer  *:9093,9094       running (40m)     8m ago  29h    27.9M        -  0.26.0            2bdd88ba9d9f  6d339897c992
ceph-exporter.ceph-sunilkumar-01-huwm3f-node1-installer  ceph-sunilkumar-01-huwm3f-node1-installer                    running (44m)     8m ago  29h    6895k        -  18.2.1-190.el9cp  1d74135bc193  07837e23ed06
ceph-exporter.ceph-sunilkumar-01-huwm3f-node2            ceph-sunilkumar-01-huwm3f-node2                              running (44m)     9m ago  29h    6815k        -  18.2.1-190.el9cp  1d74135bc193  9b56b6f881e5
ceph-exporter.ceph-sunilkumar-01-huwm3f-node3            ceph-sunilkumar-01-huwm3f-node3                              running (44m)     9m ago  29h    9893k        -  18.2.1-190.el9cp  1d74135bc193  9ee709f88a84
ceph-exporter.ceph-sunilkumar-01-huwm3f-node4            ceph-sunilkumar-01-huwm3f-node4                              running (44m)     9m ago  29h    9457k        -  18.2.1-190.el9cp  1d74135bc193  c6d6bec966d1
ceph-exporter.ceph-sunilkumar-01-huwm3f-node5            ceph-sunilkumar-01-huwm3f-node5                              running (44m)     9m ago  29h    9416k        -  18.2.1-190.el9cp  1d74135bc193  8e5e2562f56f
ceph-exporter.ceph-sunilkumar-01-huwm3f-node6            ceph-sunilkumar-01-huwm3f-node6                              running (44m)     9m ago  29h    5330k        -  18.2.1-190.el9cp  1d74135bc193  1a11aa613ebd
ceph-exporter.ceph-sunilkumar-01-huwm3f-node7            ceph-sunilkumar-01-huwm3f-node7                              running (44m)     9m ago  29h    5322k        -  18.2.1-190.el9cp  1d74135bc193  c0f73cac1e8a
ceph-exporter.ceph-sunilkumar-01-huwm3f-node8            ceph-sunilkumar-01-huwm3f-node8                              running (44m)     9m ago  29h    5326k        -  18.2.1-190.el9cp  1d74135bc193  79bf8daacae9
ceph-exporter.ceph-sunilkumar-01-huwm3f-node9            ceph-sunilkumar-01-huwm3f-node9                              running (44m)     9m ago  29h    5318k        -  18.2.1-190.el9cp  1d74135bc193  b5add590fba0
crash.ceph-sunilkumar-01-huwm3f-node1-installer          ceph-sunilkumar-01-huwm3f-node1-installer                    running (48m)     8m ago  29h    6891k        -  18.2.1-190.el9cp  1d74135bc193  a52e06439132
crash.ceph-sunilkumar-01-huwm3f-node2                    ceph-sunilkumar-01-huwm3f-node2                              running (48m)     9m ago  29h    6899k        -  18.2.1-190.el9cp  1d74135bc193  cf798683e75f
crash.ceph-sunilkumar-01-huwm3f-node3                    ceph-sunilkumar-01-huwm3f-node3                              running (48m)     9m ago  29h    6895k        -  18.2.1-190.el9cp  1d74135bc193  3299d2905544
crash.ceph-sunilkumar-01-huwm3f-node4                    ceph-sunilkumar-01-huwm3f-node4                              running (48m)     9m ago  29h    6895k        -  18.2.1-190.el9cp  1d74135bc193  1edbcd6c4d12
crash.ceph-sunilkumar-01-huwm3f-node5                    ceph-sunilkumar-01-huwm3f-node5                              running (47m)     9m ago  29h    6899k        -  18.2.1-190.el9cp  1d74135bc193  21cbc1cf1005
crash.ceph-sunilkumar-01-huwm3f-node6                    ceph-sunilkumar-01-huwm3f-node6                              running (47m)     9m ago  29h    6899k        -  18.2.1-190.el9cp  1d74135bc193  7822d3b8767c
crash.ceph-sunilkumar-01-huwm3f-node7                    ceph-sunilkumar-01-huwm3f-node7                              running (47m)     9m ago  29h    6903k        -  18.2.1-190.el9cp  1d74135bc193  44438550c176
crash.ceph-sunilkumar-01-huwm3f-node8                    ceph-sunilkumar-01-huwm3f-node8                              running (46m)     9m ago  29h    6903k        -  18.2.1-190.el9cp  1d74135bc193  8fc117c3041b
crash.ceph-sunilkumar-01-huwm3f-node9                    ceph-sunilkumar-01-huwm3f-node9                              running (46m)     9m ago  29h    6895k        -  18.2.1-190.el9cp  1d74135bc193  37b6f4f6ee96
grafana.ceph-sunilkumar-01-huwm3f-node1-installer        ceph-sunilkumar-01-huwm3f-node1-installer  *:3000            running (39m)     8m ago  29h    87.2M        -  10.4.0-pre        f381132c8bda  401ee9f8beea
mgr.ceph-sunilkumar-01-huwm3f-node1-installer.ukgshv     ceph-sunilkumar-01-huwm3f-node1-installer  *:8443,9283,8765  running (49m)     8m ago  29h     466M        -  18.2.1-190.el9cp  1d74135bc193  b594969756ff
mgr.ceph-sunilkumar-01-huwm3f-node2.fzztlh               ceph-sunilkumar-01-huwm3f-node2            *:8443,9283,8765  running (49m)     9m ago  29h     554M        -  18.2.1-190.el9cp  1d74135bc193  016105a9131b
mon.ceph-sunilkumar-01-huwm3f-node1-installer            ceph-sunilkumar-01-huwm3f-node1-installer                    running (49m)     8m ago  29h     106M    2048M  18.2.1-190.el9cp  1d74135bc193  b30ed06b74c3
mon.ceph-sunilkumar-01-huwm3f-node2                      ceph-sunilkumar-01-huwm3f-node2                              running (49m)     9m ago  29h    97.2M    2048M  18.2.1-190.el9cp  1d74135bc193  71097f11089f
mon.ceph-sunilkumar-01-huwm3f-node3                      ceph-sunilkumar-01-huwm3f-node3                              running (48m)     9m ago  29h    97.6M    2048M  18.2.1-190.el9cp  1d74135bc193  b1dbf62ef411
node-exporter.ceph-sunilkumar-01-huwm3f-node1-installer  ceph-sunilkumar-01-huwm3f-node1-installer  *:9100            running (40m)     8m ago  29h    15.5M        -  1.7.0             cf2bcc5cf8d9  b6273c0bacb9
node-exporter.ceph-sunilkumar-01-huwm3f-node2            ceph-sunilkumar-01-huwm3f-node2            *:9100            running (40m)     9m ago  29h    15.0M        -  1.7.0             cf2bcc5cf8d9  23890c6ba2c9
node-exporter.ceph-sunilkumar-01-huwm3f-node3            ceph-sunilkumar-01-huwm3f-node3            *:9100            running (40m)     9m ago  29h    17.8M        -  1.7.0             cf2bcc5cf8d9  5fc034d871e1
node-exporter.ceph-sunilkumar-01-huwm3f-node4            ceph-sunilkumar-01-huwm3f-node4            *:9100            running (40m)     9m ago  29h    18.1M        -  1.7.0             cf2bcc5cf8d9  318bab0f88be
node-exporter.ceph-sunilkumar-01-huwm3f-node5            ceph-sunilkumar-01-huwm3f-node5            *:9100            running (40m)     9m ago  29h    17.2M        -  1.7.0             cf2bcc5cf8d9  1af9e31c4b15
node-exporter.ceph-sunilkumar-01-huwm3f-node6            ceph-sunilkumar-01-huwm3f-node6            *:9100            running (40m)     9m ago  29h    16.3M        -  1.7.0             cf2bcc5cf8d9  58f7536f65e0
node-exporter.ceph-sunilkumar-01-huwm3f-node7            ceph-sunilkumar-01-huwm3f-node7            *:9100            running (40m)     9m ago  29h    15.8M        -  1.7.0             cf2bcc5cf8d9  f2ae252a44b6
node-exporter.ceph-sunilkumar-01-huwm3f-node8            ceph-sunilkumar-01-huwm3f-node8            *:9100            running (40m)     9m ago  29h    15.9M        -  1.7.0             cf2bcc5cf8d9  100cced0f868
node-exporter.ceph-sunilkumar-01-huwm3f-node9            ceph-sunilkumar-01-huwm3f-node9            *:9100            running (40m)     9m ago  29h    15.5M        -  1.7.0             cf2bcc5cf8d9  49a8c4b4608e
nvmeof.rbd.ceph-sunilkumar-01-huwm3f-node6.iahvtk        ceph-sunilkumar-01-huwm3f-node6            *:5500,4420,8009  running (41m)     9m ago  29h     180M        -                    6307315c3a02  77969b9c400d
nvmeof.rbd.ceph-sunilkumar-01-huwm3f-node7.arhlfv        ceph-sunilkumar-01-huwm3f-node7            *:5500,4420,8009  running (41m)     9m ago  29h     157M        -                    6307315c3a02  9528d5707813
nvmeof.rbd.ceph-sunilkumar-01-huwm3f-node8.msxael        ceph-sunilkumar-01-huwm3f-node8            *:5500,4420,8009  running (40m)     9m ago  29h     203M        -                    6307315c3a02  3da7abb7f6dd
nvmeof.rbd.ceph-sunilkumar-01-huwm3f-node9.jqwqsd        ceph-sunilkumar-01-huwm3f-node9            *:5500,4420,8009  running (40m)     9m ago  29h     157M        -                    6307315c3a02  0cdf0ba519c1
osd.0                                                    ceph-sunilkumar-01-huwm3f-node3                              running (46m)     9m ago  29h     995M    1953M  18.2.1-190.el9cp  1d74135bc193  2b74e306f78d
osd.1                                                    ceph-sunilkumar-01-huwm3f-node5                              running (45m)     9m ago  29h    1043M    2209M  18.2.1-190.el9cp  1d74135bc193  5468b13adcc2
osd.2                                                    ceph-sunilkumar-01-huwm3f-node4                              running (45m)     9m ago  29h    1171M    2209M  18.2.1-190.el9cp  1d74135bc193  53aae75e78ff
osd.3                                                    ceph-sunilkumar-01-huwm3f-node3                              running (46m)     9m ago  29h    1079M    1953M  18.2.1-190.el9cp  1d74135bc193  dad4eba0b9c7
osd.4                                                    ceph-sunilkumar-01-huwm3f-node5                              running (44m)     9m ago  29h    1056M    2209M  18.2.1-190.el9cp  1d74135bc193  5e5dfe9ba794
osd.5                                                    ceph-sunilkumar-01-huwm3f-node4                              running (45m)     9m ago  29h     814M    2209M  18.2.1-190.el9cp  1d74135bc193  9ee83586d298
osd.6                                                    ceph-sunilkumar-01-huwm3f-node3                              running (46m)     9m ago  29h     965M    1953M  18.2.1-190.el9cp  1d74135bc193  cd4d0f4193c3
osd.7                                                    ceph-sunilkumar-01-huwm3f-node5                              running (44m)     9m ago  29h     989M    2209M  18.2.1-190.el9cp  1d74135bc193  8890bbec1a2b
osd.8                                                    ceph-sunilkumar-01-huwm3f-node4                              running (45m)     9m ago  29h    1035M    2209M  18.2.1-190.el9cp  1d74135bc193  f6ff6f377a2f
osd.9                                                    ceph-sunilkumar-01-huwm3f-node3                              running (46m)     9m ago  29h     984M    1953M  18.2.1-190.el9cp  1d74135bc193  473da6f6d148
osd.10                                                   ceph-sunilkumar-01-huwm3f-node5                              running (44m)     9m ago  29h     944M    2209M  18.2.1-190.el9cp  1d74135bc193  f342e773035f
osd.11                                                   ceph-sunilkumar-01-huwm3f-node4                              running (45m)     9m ago  29h    1040M    2209M  18.2.1-190.el9cp  1d74135bc193  6d387f3a9cf7
prometheus.ceph-sunilkumar-01-huwm3f-node1-installer     ceph-sunilkumar-01-huwm3f-node1-installer  *:9095            running (40m)     8m ago  29h     127M        -  2.48.0            d1ad5c044d2e  2f98ac4f5772

Comment 11 errata-xmlrpc 2024-06-13 14:32:55 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Critical: Red Hat Ceph Storage 7.1 security, enhancements, and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:3925