Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2398113

Summary: Gateway nodes are going down while creating 4096 namespaces
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: JAYA PRAKASH P <jprakash>
Component: NVMeOFAssignee: Aviv Caro <acaro>
Status: CLOSED UPSTREAM QA Contact: JAYA PRAKASH P <jprakash>
Severity: medium Docs Contact: Rivka Pollack <rpollack>
Priority: high    
Version: 9.0CC: aarsharm, acaro, aindenba, bdavidov, bkunal, cdommeti, cephqe-warriors, gbregman, kjosy, lchernin, linuxkidd, pdhange, rpollack, vumrao
Target Milestone: ---Keywords: External
Target Release: 9.0z1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
.Gateway nodes go down when creating 4096 namespaces When running NVMe-oF gateway nodes on VMs and creating 4096 namespaces in sequence, the gateways may go down unexpectedly after creating a few thousand namespaces. This is caused by latency in the code when gateways are busy creating a large number of namespaces. As a result, the gateway nodes stop during the namespace creation process. As a workaround, manually restart the gateway nodes and continue creating namespaces from where the process stopped. restarting the gateways allows namespace creation to resume.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2026-03-04 09:55:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2388233, 2425975    

Description JAYA PRAKASH P 2025-09-25 11:56:58 UTC 
Description of problem:
Gateway nodes are going down while creating 4096 namespaces

[ceph: root@ceph-jp-upstream-functional-j2lr20-node1-installer /]# ceph -s
  cluster:
    id:     9b4a7aa2-99df-11f0-841b-fa163efd1c3d
    health: HEALTH_WARN
            2 stray daemon(s) not managed by cephadm

  services:
    mon:    3 daemons, quorum ceph-jp-upstream-functional-j2lr20-node1-installer,ceph-jp-upstream-functional-j2lr20-node3,ceph-jp-upstream-functional-j2lr20-node2 (age 2h) [leader: ceph-jp-upstream-functional-j2lr20-node1-installer]
    mgr:    ceph-jp-upstream-functional-j2lr20-node1-installer.ephrdk(active, since 2h), standbys: ceph-jp-upstream-functional-j2lr20-node2.nulebq
    osd:    12 osds: 12 up (since 2h), 12 in (since 2h)
    nvmeof: 2 gateways active (2 hosts)

  data:
    pools:   2 pools, 33 pgs
    objects: 7.04k objects, 711 KiB
    usage:   1.3 GiB used, 239 GiB / 240 GiB avail
    pgs:     33 active+clean

  io:
    client:   635 KiB/s rd, 1.8 KiB/s wr, 159 op/s rd, 2 op/s wr

[ceph: root@ceph-jp-upstream-functional-j2lr20-node1-installer /]# ceph orch host ls
HOST                                                ADDR         LABELS                    STATUS
ceph-jp-upstream-functional-j2lr20-node1-installer  10.0.64.19   _admin,installer,mon,mgr
ceph-jp-upstream-functional-j2lr20-node2            10.0.64.115  mon,mgr
ceph-jp-upstream-functional-j2lr20-node3            10.0.64.234  osd,mon
ceph-jp-upstream-functional-j2lr20-node4            10.0.64.224  mds,osd
ceph-jp-upstream-functional-j2lr20-node5            10.0.66.16   mds,osd,rgw
ceph-jp-upstream-functional-j2lr20-node6            10.0.65.9    nvmeof-gw
ceph-jp-upstream-functional-j2lr20-node7            10.0.66.130  nvmeof-gw
ceph-jp-upstream-functional-j2lr20-node8            10.0.65.183  nvmeof-gw
ceph-jp-upstream-functional-j2lr20-node9            10.0.66.233  nvmeof-gw
9 hosts in cluster
[ceph: root@ceph-jp-upstream-functional-j2lr20-node1-installer /]# ceph nvme-gw show pool1 group1
{
    "epoch": 2914,
    "pool": "pool1",
    "group": "group1",
    "features": "LB",
    "rebalance_ana_group": 1,
    "num gws": 2,
    "GW-epoch": 18,
    "Anagrp list": "[ 1 2 ]",
    "num-namespaces": 2344,
    "Created Gateways:": [
        {
            "gw-id": "client.nvmeof.pool1.group1.ceph-jp-upstream-functional-j2lr20-node6.knvhjy",
            "anagrp-id": 1,
            "num-namespaces": 1171,
            "performed-full-startup": 1,
            "Availability": "CREATED",
            "ana states": " 1: STANDBY ,  2: STANDBY " 
        },
        {
            "gw-id": "client.nvmeof.pool1.group1.ceph-jp-upstream-functional-j2lr20-node7.lgduti",
            "anagrp-id": 2,
            "num-namespaces": 1173,
            "performed-full-startup": 1,
            "Availability": "AVAILABLE",
            "num-listeners": 2,
            "ana states": " 1: ACTIVE ,  2: ACTIVE " 
        }
    ]
}
[ceph: root@ceph-jp-upstream-functional-j2lr20-node1-installer /]# ceph -v
ceph version 20.3.0-3184-g84ac9dea (84ac9dea641de15ab9f089203847802a13cc5a91) tentacle (dev)
[ceph: root@ceph-jp-upstream-functional-j2lr20-node1-installer /]#
Tried creating images using

[ceph: root@ceph-jp-upstream-functional-j2lr20-node1-installer /]# for i in  {160..2048};do ceph nvmeof ns add  --nqn nqn.2016-06.io.spdk:cnode2 --rbd_image_name sub2image$i --rbd_pool pool1 --create-image --size 1G;done
Got errors as below

Success
Success
Success
Success
Error EINVAL: Failure adding namespace to nqn.2016-06.io.spdk:cnode2: Failure adding namespace to nqn.2016-06.io.spdk:cnode2: Unable to find a target.
Error EINVAL: Gateway is going down
Error EINVAL: Gateway is going down
Error EINVAL: Gateway is going down
Error EINVAL: Gateway is going down
Error EINVAL: Gateway is going down
Error EINVAL: Gateway is going down
From Journalctl logs

Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] INFO grpc.py:1970 (2): Received request to add bdev_3d37b50f-9781-4e96-b869-9e930e76b754 to nqn.2016-06.io.spdk:cnode1 with load balancing group id 2 using ID 205, auto_visible: True, RBD image pool1/image204, context: None
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [2025-09-25 09:45:46.776227] bdev_rbd.c:1658:bdev_rbd_ns_reservation_load_json: *ERROR*: Failed to get metadata  key = reservation_key rbd-name image204
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR server.py:906 (2): System exit message was set to Will abort gateway because of an error in namespace_add_safe()
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR server.py:189 (2): GatewayServer exception occurred
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: Traceback (most recent call last):
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]:   File "/src/control/__main__.py", line 33, in <module>
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]:     gateway.keep_alive()
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]:   File "/src/control/server.py", line 933, in keep_alive
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]:     self.exit_gateway_if_needed()
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]:   File "/src/control/server.py", line 908, in exit_gateway_if_needed
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]:     raise SystemExit(exit_msg)
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: SystemExit: Will abort gateway because of an error in namespace_add_safe()
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] INFO server.py:725 (2): Terminating sub process of (client.nvmeof.pool1.group1.ceph-jp-upstream-functional-j2lr20-node6.knvhjy) pid 19 args ['/usr/bin/ceph-nvmeof-monitor-client', '--gateway-name', 'client.nvmeof.pool1.group1.ceph-jp-upstream-functional-j2lr20-node6.knvhjy', '--gateway-address', '10.0.65.9:5500', '--gateway-pool', 'pool1', '--gateway-group', 'group1', '--monitor-group-address', '10.0.65.9:5499', '-c', '/etc/ceph/ceph.conf', '-n', 'client.nvmeof.pool1.group1.ceph-jp-upstream-functional-j2lr20-node6.knvhjy', '-k', '/etc/ceph/keyring'] ...
Sep 25 09:45:46 ceph-jp-upstream-functional-j2lr20-node6 ceph-9b4a7aa2-99df-11f0-841b-fa163efd1c3d-nvmeof-pool1-group1-ceph-jp-upstream-functional-j2lr20-node6-knvhjy[944790]: [25-Sep-2025 09:45:46] ERROR grpc.py:1172 (2): Gateway is going down
When I checked before gateway going down there are namespaces in subsystems

[ceph: root@ceph-jp-upstream-functional-j2lr20-node1-installer /]# ceph nvme-gw show pool1 group1
{
    "epoch": 2244,
    "pool": "pool1",
    "group": "group1",
    "features": "LB",
    "rebalance_ana_group": 2,
    "num gws": 2,
    "GW-epoch": 8,
    "Anagrp list": "[ 1 2 ]",
    "num-namespaces": 2200,
    "Created Gateways:": [
        {
            "gw-id": "client.nvmeof.pool1.group1.ceph-jp-upstream-functional-j2lr20-node6.knvhjy",
            "anagrp-id": 1,
            "num-namespaces": 1099,
            "performed-full-startup": 1,
            "Availability": "AVAILABLE",
            "num-listeners": 2,
            "ana states": " 1: ACTIVE ,  2: STANDBY " 
        },
        {
            "gw-id": "client.nvmeof.pool1.group1.ceph-jp-upstream-functional-j2lr20-node7.lgduti",
            "anagrp-id": 2,
            "num-namespaces": 1101,
            "performed-full-startup": 1,
            "Availability": "AVAILABLE",
            "num-listeners": 2,
            "ana states": " 1: STANDBY ,  2: ACTIVE " 
        }
    ]
}
But after going gateway down the namespaces are empty

[ceph: root@ceph-jp-upstream-functional-j2lr20-node1-installer /]# ceph nvme-gw show pool1 group1
{
    "epoch": 2806,
    "pool": "pool1",
    "group": "group1",
    "features": "LB",
    "rebalance_ana_group": 1,
    "num gws": 2,
    "GW-epoch": 16,
    "Anagrp list": "[ 1 2 ]",
    "num-namespaces": 0,
    "Created Gateways:": [
        {
            "gw-id": "client.nvmeof.pool1.group1.ceph-jp-upstream-functional-j2lr20-node6.knvhjy",
            "anagrp-id": 1,
            "num-namespaces": 0,
            "performed-full-startup": 1,
            "Availability": "CREATED",
            "ana states": " 1: STANDBY ,  2: STANDBY " 
        },
        {
            "gw-id": "client.nvmeof.pool1.group1.ceph-jp-upstream-functional-j2lr20-node7.lgduti",
            "anagrp-id": 2,
            "num-namespaces": 0,
            "performed-full-startup": 1,
            "Availability": "CREATED",
            "ana states": " 1: STANDBY ,  2: STANDBY " 
        }
    ]
}

After some time namespaces are back

[ceph: root@ceph-jp-upstream-functional-j2lr20-node1-installer /]# ceph nvme-gw show pool1 group1
{
    "epoch": 2977,
    "pool": "pool1",
    "group": "group1",
    "features": "LB",
    "rebalance_ana_group": 2,
    "num gws": 2,
    "GW-epoch": 18,
    "Anagrp list": "[ 1 2 ]",
    "num-namespaces": 2344,
    "Created Gateways:": [
        {
            "gw-id": "client.nvmeof.pool1.group1.ceph-jp-upstream-functional-j2lr20-node6.knvhjy",
            "anagrp-id": 1,
            "num-namespaces": 1171,
            "performed-full-startup": 1,
            "Availability": "CREATED",
            "ana states": " 1: STANDBY ,  2: STANDBY " 
        },
        {
            "gw-id": "client.nvmeof.pool1.group1.ceph-jp-upstream-functional-j2lr20-node7.lgduti",
            "anagrp-id": 2,
            "num-namespaces": 1173,
            "performed-full-startup": 1,
            "Availability": "AVAILABLE",
            "num-listeners": 2,
            "ana states": " 1: ACTIVE ,  2: ACTIVE " 
        }
    ]
}
[ceph: root@ceph-jp-upstream-functional-j2lr20-node1-installer /]#



Version-Release number of selected component (if applicable):
ceph version 20.3.0-3184-g84ac9dea (84ac9dea641de15ab9f089203847802a13cc5a91) tentacle (dev - RelWithDebInfo)

How reproducible:
-

Steps to Reproduce:
we have 9 node cluster with one nvme service deployed on two gateway nodes
we have created two subsystems, added listeners and initiators on both subsystems
we tried configured namespaces using below shell script.
for i in  {1..2048};do ceph nvmeof ns add  --nqn nqn.2016-06.io.spdk:cnode1 --rbd_image_name image$i --rbd_pool pool1 --create-image --size 1G;done
we will try to add 2048 namespaces in each subsystem using above command

Actual results:
Gateway nodes are going down while creating 4096 namespaces

Expected results:
We should able to create 4096 namespaces without any issues

Additional info:
Comment 1 Storage PM bot 2025-09-25 11:57:11 UTC 
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
Comment 10 Red Hat Bugzilla 2026-03-04 09:55:33 UTC 
This product has been discontinued or is no longer tracked in Red Hat Bugzilla.