Red Hat Bugzilla – 2420434 – Few namespaces are missing in initiator after setting "Disable Auto Resize is set to true" and manually refresh rbd image size and namespace size
.Disabling automatic resizing does not work as expected
When the automatic resize feature is disabled and you manually refresh the Ceph Block Device image size and namespace size, some namespaces might be missing on the NVMe-oF initiator. In addition, the `disable-auto-resize` option is not functioning correctly and can cause the initiator to crash.
Currently, there is no workaround. To avoid this issue, do not use the `disable-auto-resize` option.
Created attachment 2118114 [details] journalctl logs Description of problem: Few namespaces are missing in initiator after setting "Disable Auto Resize is set to true" and manually refresh rbd image size and namespace size Version-Release number of selected component (if applicable): ceph version 20.1.0-120.el9cp How reproducible: Easy Steps to Reproduce: 1. Deployed cluster with ceph version 20.1.0-120.el9cp (810391a4e0bcc3734d005b357fbe501331ed1c8d) tentacle 2. Deployed nvmeof service with two gateways 3. Created two subsystems, added listeners, added host access, added two namespaces per subsystems 4. Discovered namespaces in initiator side, total 4 namespaces are listed 5. Performed --auto-resize-enabled=false using "ceph nvmeof ns set_auto_resize" on all 4 namespaces 6. Resize the RBD images 7. Verified namespace size is not increased because we set --auto-resize-enabled=false 8. using "ceph nvmeof ns refresh_size" command manually refreshed the namespaces 9. Verified the namespace size is increased 10. After manually refreshed the size, in initiator only 2 namespaces are connected. 11. Tried discovering and connect using connect-all and connect with each subsystem but still only 2 namespaces are connected and 2 namespaces are missing Actual results: Namespaces are missing in initiator side after manually refreshing the size Expected results: All namespaces should be visible in initiator when we connect through connect-all Additional info: [ceph: root@ceph-jp-90-98pymp-node1-installer /]# [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph -s cluster: id: 3c877b24-d4c6-11f0-8825-fa163e57215e health: HEALTH_WARN 1 stray daemon(s) not managed by cephadm services: mon: 3 daemons, quorum ceph-jp-90-98pymp-node1-installer,ceph-jp-90-98pymp-node2,ceph-jp-90-98pymp-node3 (age 28m) [leader: ceph-jp-90-98pymp-node1-installer] mgr: ceph-jp-90-98pymp-node1-installer.bqtagk(active, since 32m), standbys: ceph-jp-90-98pymp-node2.lsfgjh osd: 12 osds: 12 up (since 23m), 12 in (since 24m) data: pools: 1 pools, 1 pgs objects: 4 objects, 1.3 MiB usage: 324 MiB used, 240 GiB / 240 GiB avail pgs: 1 active+clean io: client: 767 B/s rd, 0 B/s wr, 0 op/s rd, 0 op/s wr [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph versions { "mon": { "ceph version 20.1.0-120.el9cp (810391a4e0bcc3734d005b357fbe501331ed1c8d) tentacle (rc - RelWithDebInfo)": 3 }, "mgr": { "ceph version 20.1.0-120.el9cp (810391a4e0bcc3734d005b357fbe501331ed1c8d) tentacle (rc - RelWithDebInfo)": 2 }, "osd": { "ceph version 20.1.0-120.el9cp (810391a4e0bcc3734d005b357fbe501331ed1c8d) tentacle (rc - RelWithDebInfo)": 12 }, "overall": { "ceph version 20.1.0-120.el9cp (810391a4e0bcc3734d005b357fbe501331ed1c8d) tentacle (rc - RelWithDebInfo)": 17 } } [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 66s ago 33m count:1 ceph-exporter ?:9926 7/7 67s ago 33m * crash 7/7 67s ago 33m * grafana ?:3000 1/1 66s ago 33m count:1 mgr 2/2 67s ago 30m label:mgr mon 3/3 67s ago 28m label:mon node-exporter ?:9100 7/7 67s ago 33m * osd.all-available-devices 12 67s ago 24m * prometheus ?:9095 1/1 66s ago 33m count:1 [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph osd pool ls .mgr [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph osd pool create pool1;ceph osd pool application enable pool1 rbd;rbd pool init -p pool1 pool 'pool1' created enabled application 'rbd' on pool 'pool1' [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph orch host ls HOST ADDR LABELS STATUS ceph-jp-90-98pymp-node1-installer 10.0.66.217 _admin,mgr,installer,mon ceph-jp-90-98pymp-node2 10.0.65.185 mgr,mon ceph-jp-90-98pymp-node3 10.0.65.238 osd,mon ceph-jp-90-98pymp-node4 10.0.67.228 mds,osd ceph-jp-90-98pymp-node5 10.0.67.164 mds,rgw,osd ceph-jp-90-98pymp-node6 10.0.66.191 nvmeof-gw ceph-jp-90-98pymp-node7 10.0.65.6 nvmeof-gw 7 hosts in cluster [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph orch apply nvmeof pool1 gw_group1 --placement='ceph-jp-90-98pymp-node6 ceph-jp-90-98pymp-node7' Scheduled nvmeof.pool1.gw_group1 update... [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 2m ago 35m count:1 ceph-exporter ?:9926 7/7 2m ago 35m * crash 7/7 2m ago 35m * grafana ?:3000 1/1 2m ago 35m count:1 mgr 2/2 2m ago 32m label:mgr mon 3/3 2m ago 30m label:mon node-exporter ?:9100 7/7 2m ago 35m * nvmeof.pool1.gw_group1 ?:4420,5500,8009,10008 0/2 - 11s ceph-jp-90-98pymp-node6;ceph-jp-90-98pymp-node7 osd.all-available-devices 12 2m ago 26m * prometheus ?:9095 1/1 2m ago 35m count:1 [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 9s ago 35m count:1 ceph-exporter ?:9926 7/7 3m ago 35m * crash 7/7 3m ago 35m * grafana ?:3000 1/1 9s ago 35m count:1 mgr 2/2 3m ago 32m label:mgr mon 3/3 3m ago 30m label:mon node-exporter ?:9100 7/7 3m ago 35m * nvmeof.pool1.gw_group1 ?:4420,5500,8009,10008 2/2 18s ago 40s ceph-jp-90-98pymp-node6;ceph-jp-90-98pymp-node7 osd.all-available-devices 12 3m ago 26m * prometheus ?:9095 1/1 9s ago 35m count:1 [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph orch ps | grep "nvme" nvmeof.pool1.gw_group1.ceph-jp-90-98pymp-node6.evhukn ceph-jp-90-98pymp-node6 *:5500,4420,8009,10008 running (35s) 26s ago 35s 52.6M - 1.5.14 cd940b865952 ba3d07e62bc9 nvmeof.pool1.gw_group1.ceph-jp-90-98pymp-node7.tsowpf ceph-jp-90-98pymp-node7 *:5500,4420,8009,10008 running (32s) 25s ago 32s 52.2M - 1.5.14 cd940b865952 141757d2d99f [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof gw info +-----------------------+-------+------------------------------------------------------------+---------+-----------+----+--------+-------------------------------------------------------------+ |Hostname |Version|Name |Group |Addr |Port|Lb Group|Spdk Version | +-----------------------+-------+------------------------------------------------------------+---------+-----------+----+--------+-------------------------------------------------------------+ |ceph-jp-90-98pymp-node6|1.5.14 |client.nvmeof.pool1.gw_group1.ceph-jp-90-98pymp-node6.evhukn|gw_group1|10.0.66.191|5500|1 |SPDK v25.05 git sha1 c19d07f46274d31201c0c4db3775a655d68f5f38| +-----------------------+-------+------------------------------------------------------------+---------+-----------+----+--------+-------------------------------------------------------------+ [ceph: root@ceph-jp-90-98pymp-node1-installer /]# [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof subsystem add --nqn nqn.2016-06.io.spdk:cnode1 --max-namespaces 32 --no-group-append --gw_group gw_group1 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof subsystem add --nqn nqn.2016-06.io.spdk:cnode2 --max-namespaces 32 --no-group-append --gw_group gw_group1 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof listener add --nqn nqn.2016-06.io.spdk:cnode1 --traddr 10.0.66.191 --trsvcid 4420 --host-name ceph-jp-90-98pymp-node6 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof listener add --nqn nqn.2016-06.io.spdk:cnode2 --traddr 10.0.66.191 --trsvcid 4420 --host-name ceph-jp-90-98pymp-node6 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof listener add --nqn nqn.2016-06.io.spdk:cnode1 --traddr 10.0.65.6 --trsvcid 4420 --host-name ceph-jp-90-98pymp-node7 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof listener add --nqn nqn.2016-06.io.spdk:cnode2 --traddr 10.0.65.6 --trsvcid 4420 --host-name ceph-jp-90-98pymp-node7 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof host add --nqn nqn.2016-06.io.spdk:cnode1 --host_nqn '*' --gw_group gw_group1 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof host add --nqn nqn.2016-06.io.spdk:cnode2 --host_nqn '*' --gw_group gw_group1 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# rbd create pool1/image1 --size 2G [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns add --nqn nqn.2016-06.io.spdk:cnode1 --rbd_pool pool1 --rbd_image_name image1 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# rbd create pool1/image2 --size 2G [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns add --nqn nqn.2016-06.io.spdk:cnode1 --rbd_pool pool1 --rbd_image_name image2 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# rbd create pool1/image3 --size 2G [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns add --nqn nqn.2016-06.io.spdk:cnode2 --rbd_pool pool1 --rbd_image_name image3 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# rbd create pool1/image4 --size 2G [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns add --nqn nqn.2016-06.io.spdk:cnode2 --rbd_pool pool1 --rbd_image_name image4 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof subsystem list +--------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ |Nqn |Serial Number |Model Number |Namespace Count|Subtype|Max Namespaces|Has Dhchap Key|Allow Any Host|Created Without Key| +--------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ |nqn.2016-06.io.spdk:cnode1|Ceph92221876096481|Ceph bdev Controller|2 |NVMe |32 |False |True |False | |nqn.2016-06.io.spdk:cnode2|Ceph66249215294062|Ceph bdev Controller|2 |NVMe |32 |False |True |False | +--------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns list +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn |Trash Image|Disable Auto Resize|Read Only| +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |bdev_a570b1be-44ec-45f7-b86c-f8e7e9f81e8f|image1 |pool1 |1 |2GB |512B |0 |0 |0 |0 |True |[] |1 |a570b1be-44ec-45f7-b86c-f8e7e9f81e8f|nqn.2016-06.io.spdk:cnode1|False |False |False | |bdev_5102cb69-d217-4e94-a94e-effcb1a31bfe|image2 |pool1 |2 |2GB |512B |0 |0 |0 |0 |True |[] |2 |5102cb69-d217-4e94-a94e-effcb1a31bfe|nqn.2016-06.io.spdk:cnode1|False |False |False | |bdev_10add56b-c24c-4d9e-b322-04949220ccde|image3 |pool1 |1 |2GB |512B |0 |0 |0 |0 |True |[] |1 |10add56b-c24c-4d9e-b322-04949220ccde|nqn.2016-06.io.spdk:cnode2|False |False |False | |bdev_5167f633-9ceb-4ec3-bf18-b466634f6841|image4 |pool1 |2 |2GB |512B |0 |0 |0 |0 |True |[] |2 |5167f633-9ceb-4ec3-bf18-b466634f6841|nqn.2016-06.io.spdk:cnode2|False |False |False | +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ [ceph: root@ceph-jp-90-98pymp-node1-installer /]# [ceph: root@ceph-jp-90-98pymp-node1-installer /]# [root@ceph-jp-90-98pymp-node8 ~]# modprobe nvme-fabrics [root@ceph-jp-90-98pymp-node8 ~]# nvme discover --transport tcp --traddr 10.0.66.191 --trsvcid 8009 Discovery Log Number of Records 4, Generation counter 1 =====Discovery Log Entry 0====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 0 trsvcid: 4420 subnqn: nqn.2016-06.io.spdk:cnode1 traddr: 10.0.66.191 eflags: none sectype: none =====Discovery Log Entry 1====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 1 trsvcid: 4420 subnqn: nqn.2016-06.io.spdk:cnode1 traddr: 10.0.65.6 eflags: none sectype: none =====Discovery Log Entry 2====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 2 trsvcid: 4420 subnqn: nqn.2016-06.io.spdk:cnode2 traddr: 10.0.66.191 eflags: none sectype: none =====Discovery Log Entry 3====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 3 trsvcid: 4420 subnqn: nqn.2016-06.io.spdk:cnode2 traddr: 10.0.65.6 eflags: none sectype: none [root@ceph-jp-90-98pymp-node8 ~]# nvme connect-all --trsvcid 8009 --transport tcp --traddr 10.0.66.191 --ctrl-loss-tmo 3600 [root@ceph-jp-90-98pymp-node8 ~]# nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- /dev/nvme1n1 /dev/ng1n1 Ceph92221876096481 Ceph bdev Controller 0x1 2.15 GB / 2.15 GB 512 B + 0 B 25.05 /dev/nvme1n2 /dev/ng1n2 Ceph92221876096481 Ceph bdev Controller 0x2 2.15 GB / 2.15 GB 512 B + 0 B 25.05 /dev/nvme3n1 /dev/ng3n1 Ceph66249215294062 Ceph bdev Controller 0x1 2.15 GB / 2.15 GB 512 B + 0 B 25.05 /dev/nvme3n2 /dev/ng3n2 Ceph66249215294062 Ceph bdev Controller 0x2 2.15 GB / 2.15 GB 512 B + 0 B 25.05 [root@ceph-jp-90-98pymp-node8 ~]# [ceph: root@ceph-jp-90-98pymp-node1-installer /]# [ceph: root@ceph-jp-90-98pymp-node1-installer /]# date;echo "ns set_auto_resize" Tue Dec 9 06:58:38 UTC 2025 ns set_auto_resize [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns set_auto_resize --nqn nqn.2016-06.io.spdk:cnode1 --nsid 1 --auto-resize-enabled=false --gw_group gw_group1 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns set_auto_resize --nqn nqn.2016-06.io.spdk:cnode1 --nsid 2 --auto-resize-enabled=false --gw_group gw_group1 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns set_auto_resize --nqn nqn.2016-06.io.spdk:cnode2 --nsid 1 --auto-resize-enabled=false --gw_group gw_group1 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns set_auto_resize --nqn nqn.2016-06.io.spdk:cnode2 --nsid 2 --auto-resize-enabled=false --gw_group gw_group1 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns list +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn |Trash Image|Disable Auto Resize|Read Only| +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |bdev_a570b1be-44ec-45f7-b86c-f8e7e9f81e8f|image1 |pool1 |1 |2GB |512B |0 |0 |0 |0 |True |[] |1 |a570b1be-44ec-45f7-b86c-f8e7e9f81e8f|nqn.2016-06.io.spdk:cnode1|False |True |False | |bdev_5102cb69-d217-4e94-a94e-effcb1a31bfe|image2 |pool1 |2 |2GB |512B |0 |0 |0 |0 |True |[] |2 |5102cb69-d217-4e94-a94e-effcb1a31bfe|nqn.2016-06.io.spdk:cnode1|False |True |False | |bdev_10add56b-c24c-4d9e-b322-04949220ccde|image3 |pool1 |1 |2GB |512B |0 |0 |0 |0 |True |[] |1 |10add56b-c24c-4d9e-b322-04949220ccde|nqn.2016-06.io.spdk:cnode2|False |True |False | |bdev_5167f633-9ceb-4ec3-bf18-b466634f6841|image4 |pool1 |2 |2GB |512B |0 |0 |0 |0 |True |[] |2 |5167f633-9ceb-4ec3-bf18-b466634f6841|nqn.2016-06.io.spdk:cnode2|False |True |False | +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ [ceph: root@ceph-jp-90-98pymp-node1-installer /]# echo "verify Disable Auto Resize is set to true in ns list output" verify Disable Auto Resize is set to true in ns list output [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns list +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn |Trash Image|Disable Auto Resize|Read Only| +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |bdev_a570b1be-44ec-45f7-b86c-f8e7e9f81e8f|image1 |pool1 |1 |2GB |512B |0 |0 |0 |0 |True |[] |1 |a570b1be-44ec-45f7-b86c-f8e7e9f81e8f|nqn.2016-06.io.spdk:cnode1|False |True |False | |bdev_5102cb69-d217-4e94-a94e-effcb1a31bfe|image2 |pool1 |2 |2GB |512B |0 |0 |0 |0 |True |[] |2 |5102cb69-d217-4e94-a94e-effcb1a31bfe|nqn.2016-06.io.spdk:cnode1|False |True |False | |bdev_10add56b-c24c-4d9e-b322-04949220ccde|image3 |pool1 |1 |2GB |512B |0 |0 |0 |0 |True |[] |1 |10add56b-c24c-4d9e-b322-04949220ccde|nqn.2016-06.io.spdk:cnode2|False |True |False | |bdev_5167f633-9ceb-4ec3-bf18-b466634f6841|image4 |pool1 |2 |2GB |512B |0 |0 |0 |0 |True |[] |2 |5167f633-9ceb-4ec3-bf18-b466634f6841|nqn.2016-06.io.spdk:cnode2|False |True |False | +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ [root@ceph-jp-90-98pymp-node8 ~]# date;echo "after Disable Auto Resize is set to true" Tue Dec 9 07:01:52 UTC 2025 after Disable Auto Resize is set to true [root@ceph-jp-90-98pymp-node8 ~]# nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- /dev/nvme1n1 /dev/ng1n1 Ceph92221876096481 Ceph bdev Controller 0x1 2.15 GB / 2.15 GB 512 B + 0 B 25.05 /dev/nvme1n2 /dev/ng1n2 Ceph92221876096481 Ceph bdev Controller 0x2 2.15 GB / 2.15 GB 512 B + 0 B 25.05 /dev/nvme3n1 /dev/ng3n1 Ceph66249215294062 Ceph bdev Controller 0x1 2.15 GB / 2.15 GB 512 B + 0 B 25.05 /dev/nvme3n2 /dev/ng3n2 Ceph66249215294062 Ceph bdev Controller 0x2 2.15 GB / 2.15 GB 512 B + 0 B 25.05 [root@ceph-jp-90-98pymp-node8 ~]# nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- /dev/nvme1n1 /dev/ng1n1 Ceph92221876096481 Ceph bdev Controller 0x1 2.15 GB / 2.15 GB 512 B + 0 B 25.05 /dev/nvme1n2 /dev/ng1n2 Ceph92221876096481 Ceph bdev Controller 0x2 2.15 GB / 2.15 GB 512 B + 0 B 25.05 /dev/nvme3n1 /dev/ng3n1 Ceph66249215294062 Ceph bdev Controller 0x1 2.15 GB / 2.15 GB 512 B + 0 B 25.05 /dev/nvme3n2 /dev/ng3n2 Ceph66249215294062 Ceph bdev Controller 0x2 2.15 GB / 2.15 GB 512 B + 0 B 25.05 [ceph: root@ceph-jp-90-98pymp-node1-installer /]# echo "Resize the RBD images to 3G" Resize the RBD images to 3G [ceph: root@ceph-jp-90-98pymp-node1-installer /]# rbd resize --size 3G pool1/image1 Resizing image: 100% complete...done. [ceph: root@ceph-jp-90-98pymp-node1-installer /]# rbd resize --size 3G pool1/image2 Resizing image: 100% complete...done. [ceph: root@ceph-jp-90-98pymp-node1-installer /]# rbd resize --size 3G pool1/image3 Resizing image: 100% complete...done. [ceph: root@ceph-jp-90-98pymp-node1-installer /]# rbd resize --size 3G pool1/image4 Resizing image: 100% complete...done. [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns list --nqn nqn.2016-06.io.spdk:cnode1 --gw_group gw_group1 +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn |Trash Image|Disable Auto Resize|Read Only| +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |bdev_a570b1be-44ec-45f7-b86c-f8e7e9f81e8f|image1 |pool1 |1 |2GB |512B |0 |0 |0 |0 |True |[] |1 |a570b1be-44ec-45f7-b86c-f8e7e9f81e8f|nqn.2016-06.io.spdk:cnode1|False |True |False | |bdev_5102cb69-d217-4e94-a94e-effcb1a31bfe|image2 |pool1 |2 |2GB |512B |0 |0 |0 |0 |True |[] |2 |5102cb69-d217-4e94-a94e-effcb1a31bfe|nqn.2016-06.io.spdk:cnode1|False |True |False | +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns list --nqn nqn.2016-06.io.spdk:cnode2 --gw_group gw_group1 +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn |Trash Image|Disable Auto Resize|Read Only| +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |bdev_10add56b-c24c-4d9e-b322-04949220ccde|image3 |pool1 |1 |2GB |512B |0 |0 |0 |0 |True |[] |1 |10add56b-c24c-4d9e-b322-04949220ccde|nqn.2016-06.io.spdk:cnode2|False |True |False | |bdev_5167f633-9ceb-4ec3-bf18-b466634f6841|image4 |pool1 |2 |2GB |512B |0 |0 |0 |0 |True |[] |2 |5167f633-9ceb-4ec3-bf18-b466634f6841|nqn.2016-06.io.spdk:cnode2|False |True |False | +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ [ceph: root@ceph-jp-90-98pymp-node1-installer /]# rbd df -p pool1 [ceph: root@ceph-jp-90-98pymp-node1-installer /]# rbd du -p pool1 NAME PROVISIONED USED image1 3 GiB 0 B image2 3 GiB 0 B image3 3 GiB 0 B image4 3 GiB 0 B <TOTAL> 12 GiB 0 B [ceph: root@ceph-jp-90-98pymp-node1-installer /]# echo "anually refresh namespace size" anually refresh namespace size [ceph: root@ceph-jp-90-98pymp-node1-installer /]# echo "manually refresh namespace size" manually refresh namespace size [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns refresh_size --nqn nqn.2016-06.io.spdk:cnode1 --nsid 1 --gw_group gw_group1 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns refresh_size --nqn nqn.2016-06.io.spdk:cnode1 --nsid 2 --gw_group gw_group1 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns refresh_size --nqn nqn.2016-06.io.spdk:cnode2 --nsid 1 --gw_group gw_group1 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns refresh_size --nqn nqn.2016-06.io.spdk:cnode2 --nsid 2 --gw_group gw_group1 Success [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns list --nqn nqn.2016-06.io.spdk:cnode2 --gw_group gw_group1 +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn |Trash Image|Disable Auto Resize|Read Only| +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |bdev_10add56b-c24c-4d9e-b322-04949220ccde|image3 |pool1 |1 |3GB |512B |0 |0 |0 |0 |True |[] |1 |10add56b-c24c-4d9e-b322-04949220ccde|nqn.2016-06.io.spdk:cnode2|False |True |False | |bdev_5167f633-9ceb-4ec3-bf18-b466634f6841|image4 |pool1 |2 |3GB |512B |0 |0 |0 |0 |True |[] |2 |5167f633-9ceb-4ec3-bf18-b466634f6841|nqn.2016-06.io.spdk:cnode2|False |True |False | +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns list --nqn nqn.2016-06.io.spdk:cnode1 --gw_group gw_group1 +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn |Trash Image|Disable Auto Resize|Read Only| +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |bdev_a570b1be-44ec-45f7-b86c-f8e7e9f81e8f|image1 |pool1 |1 |3GB |512B |0 |0 |0 |0 |True |[] |1 |a570b1be-44ec-45f7-b86c-f8e7e9f81e8f|nqn.2016-06.io.spdk:cnode1|False |True |False | |bdev_5102cb69-d217-4e94-a94e-effcb1a31bfe|image2 |pool1 |2 |3GB |512B |0 |0 |0 |0 |True |[] |2 |5102cb69-d217-4e94-a94e-effcb1a31bfe|nqn.2016-06.io.spdk:cnode1|False |True |False | +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ [ceph: root@ceph-jp-90-98pymp-node1-installer /]# ceph nvmeof ns list +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn |Trash Image|Disable Auto Resize|Read Only| +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ |bdev_a570b1be-44ec-45f7-b86c-f8e7e9f81e8f|image1 |pool1 |1 |3GB |512B |0 |0 |0 |0 |True |[] |1 |a570b1be-44ec-45f7-b86c-f8e7e9f81e8f|nqn.2016-06.io.spdk:cnode1|False |True |False | |bdev_5102cb69-d217-4e94-a94e-effcb1a31bfe|image2 |pool1 |2 |3GB |512B |0 |0 |0 |0 |True |[] |2 |5102cb69-d217-4e94-a94e-effcb1a31bfe|nqn.2016-06.io.spdk:cnode1|False |True |False | |bdev_10add56b-c24c-4d9e-b322-04949220ccde|image3 |pool1 |1 |3GB |512B |0 |0 |0 |0 |True |[] |1 |10add56b-c24c-4d9e-b322-04949220ccde|nqn.2016-06.io.spdk:cnode2|False |True |False | |bdev_5167f633-9ceb-4ec3-bf18-b466634f6841|image4 |pool1 |2 |3GB |512B |0 |0 |0 |0 |True |[] |2 |5167f633-9ceb-4ec3-bf18-b466634f6841|nqn.2016-06.io.spdk:cnode2|False |True |False | +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+--------------------------+-----------+-------------------+---------+ [ceph: root@ceph-jp-90-98pymp-node1-installer /]# [root@ceph-jp-90-98pymp-node8 ~]# date; echo "after maual refresh" Tue Dec 9 07:05:06 UTC 2025 after maual refresh [root@ceph-jp-90-98pymp-node8 ~]# nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- /dev/nvme1n1 /dev/ng1n1 Ceph92221876096481 Ceph bdev Controller 0x1 3.22 GB / 3.22 GB 512 B + 0 B 25.05 /dev/nvme3n1 /dev/ng3n1 Ceph66249215294062 Ceph bdev Controller 0x1 3.22 GB / 3.22 GB 512 B + 0 B 25.05 [root@ceph-jp-90-98pymp-node8 ~]# nvme list-subsys nvme-subsys1 - NQN=nqn.2016-06.io.spdk:cnode1 hostnqn=nqn.2014-08.org.nvmexpress:uuid:30b4860f-4580-44d8-8ce9-d56807b711d7 \ +- nvme1 tcp traddr=10.0.66.191,trsvcid=4420,src_addr=10.0.67.220 live +- nvme2 tcp traddr=10.0.65.6,trsvcid=4420,src_addr=10.0.67.220 live nvme-subsys3 - NQN=nqn.2016-06.io.spdk:cnode2 hostnqn=nqn.2014-08.org.nvmexpress:uuid:30b4860f-4580-44d8-8ce9-d56807b711d7 \ +- nvme3 tcp traddr=10.0.66.191,trsvcid=4420,src_addr=10.0.67.220 live +- nvme4 tcp traddr=10.0.65.6,trsvcid=4420,src_addr=10.0.67.220 live [root@ceph-jp-90-98pymp-node8 ~]# modprobe nvme-fabrics [root@ceph-jp-90-98pymp-node8 ~]# nvme discover --transport tcp --traddr 10.0.66.191 --trsvcid 8009 Discovery Log Number of Records 4, Generation counter 3 =====Discovery Log Entry 0====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 0 trsvcid: 4420 subnqn: nqn.2016-06.io.spdk:cnode1 traddr: 10.0.66.191 eflags: none sectype: none =====Discovery Log Entry 1====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 1 trsvcid: 4420 subnqn: nqn.2016-06.io.spdk:cnode1 traddr: 10.0.65.6 eflags: none sectype: none =====Discovery Log Entry 2====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 2 trsvcid: 4420 subnqn: nqn.2016-06.io.spdk:cnode2 traddr: 10.0.66.191 eflags: none sectype: none =====Discovery Log Entry 3====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 3 trsvcid: 4420 subnqn: nqn.2016-06.io.spdk:cnode2 traddr: 10.0.65.6 eflags: none sectype: none [root@ceph-jp-90-98pymp-node8 ~]# nvme connect-all --trsvcid 8009 --transport tcp --traddr 10.0.66.191 --ctrl-loss-tmo 3600 [root@ceph-jp-90-98pymp-node8 ~]# nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- /dev/nvme1n1 /dev/ng1n1 Ceph92221876096481 Ceph bdev Controller 0x1 3.22 GB / 3.22 GB 512 B + 0 B 25.05 /dev/nvme3n1 /dev/ng3n1 Ceph66249215294062 Ceph bdev Controller 0x1 3.22 GB / 3.22 GB 512 B + 0 B 25.05 [root@ceph-jp-90-98pymp-node8 ~]# nvme connect --transport tcp --traddr 10.0.66.191 --trsvcid 4420 --nqn nqn.2016-06.io.spdk:cnode1 already connected [root@ceph-jp-90-98pymp-node8 ~]# nvme connect --transport tcp --traddr 10.0.66.191 --trsvcid 4420 --nqn nqn.2016-06.io.spdk:cnode2 already connected [root@ceph-jp-90-98pymp-node8 ~]# nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- /dev/nvme1n1 /dev/ng1n1 Ceph92221876096481 Ceph bdev Controller 0x1 3.22 GB / 3.22 GB 512 B + 0 B 25.05 /dev/nvme3n1 /dev/ng3n1 Ceph66249215294062 Ceph bdev Controller 0x1 3.22 GB / 3.22 GB 512 B + 0 B 25.05 [root@ceph-jp-90-98pymp-node8 ~]# nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- /dev/nvme1n1 /dev/ng1n1 Ceph92221876096481 Ceph bdev Controller 0x1 3.22 GB / 3.22 GB 512 B + 0 B 25.05 /dev/nvme3n1 /dev/ng3n1 Ceph66249215294062 Ceph bdev Controller 0x1 3.22 GB / 3.22 GB 512 B + 0 B 25.05 [root@ceph-jp-90-98pymp-node8 ~]#