This site requires JavaScript to be enabled to function correctly, please enable it.
Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.
Summary:
ceph nvmeof cli fails with error 'failed to connect to all addresses' when mtls enabled
Product:
[Red Hat Storage] Red Hat Ceph Storage
Reporter:
Chaitanya <cdommeti>
Component:
Ceph-Dashboard Assignee:
thaskalo
Status:
CLOSED
UPSTREAM
QA Contact:
Chaitanya <cdommeti>
Severity:
high
Docs Contact:
Rivka Pollack <rpollack>
Priority:
high
Version:
9.0 CC:
acaro, afrahman, aindenba, bdavidov, bkunal, ceph-eng-bugs, cephqe-warriors, gbregman, kjosy, lchernin, linuxkidd, mmurthy, pdhange, rkachach, rpollack, thaskalo, vumrao
Target Milestone:
---
Target Release:
9.0z1
Hardware:
Unspecified
OS:
Unspecified
Whiteboard:
Fixed In Version:
Doc Type:
Known Issue
Doc Text:
.mTLS does not work with `ceph nvmeof` commands
Currently, mTLS does not work with the new `ceph nvmeof` commands.
As a workaround, to work with NVMe-oF mTLS, use the `nvmeof-cli` utility when running CLI commands.
Story Points:
---
Clone Of:
Environment:
Last Closed:
2026-03-04 09:57:43 UTC
Type:
Bug
Regression:
---
Mount Type:
---
Documentation:
---
CRM:
Verified Versions:
Category:
---
oVirt Team:
---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:
---
Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:
2388233
Description of problem: After redeploying NVMe service with mTLS enabled, ceph nvmeof cli commands fail with error "Error EINVAL: failed to connect to all addresses" . However, the gateway CLI commands are successful. Version-Release number of selected component (if applicable): 20.1.0-100.el9cp How reproducible: Always Steps to Reproduce: [ceph: root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer /]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 8m ago 3d count:1 ceph-exporter ?:9926 13/13 8m ago 3d * crash 13/13 8m ago 3d * grafana ?:3000 1/1 8m ago 3d count:1 mgr 2/2 8m ago 3d label:mgr mon 3/3 8m ago 3d label:mon node-exporter ?:9100 13/13 8m ago 3d * nvmeof.rbd.gw_group1 ?:4420,5500,8009,10008 2/2 8m ago 16s ceph-cd-nvmeinbbnd-x4fagt-node6;ceph-cd-nvmeinbbnd-x4fagt-node7 osd.all-available-devices 20 6m ago 3d * prometheus ?:9095 1/1 8m ago 3d count:1 Generate server cert/keys [root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer mtls]# openssl req -x509 -newkey rsa:4096 -nodes -keyout server.key -out server.crt -days 3650 -subj /CN=my.server -addext "subjectAltName=IP:10.0.64.12,IP:10.0.65.126" ..+......+.....+....+.........+..+......+...+....+...+...+..+...+....+.....+.......+.....+......+....+...+.....+.+...........+.+...+..............+...+...+.+...........+.......+........+...+......+++++++++++++++++++++++++++++++++++++++++++++*.+........+.+.....+...+......+.+.....+.+..+.+.....+.+...+...+..+......+....+......+..+...+.......+..+..........+..+..........+...+......+.........+.....+.+...........+.+..+.......+......+..+...+....+......+...+..+...+....+.....+..........+..+......+...+.+...+...........+.+...+...+........+...................+..+.+........+.+++++++++++++++++++++++++++++++++++++++++++++*..+...+..........+.........+...........+..........+..+.......+..+.+........+....+...+...+..+...................+..............+............+...+................+........+..................+..........+...+...............+...........+........................+....+.....+.+.............................+...+....+...+......+.........+.....+.......+........+...+.............+.....+.+.....+.+..+.......+......+............+.....+...+...+......+.+........+....+..+.+.........+++++ ....+......+.+++++++++++++++++++++++++++++++++++++++++++++*.....+.....+.+++++++++++++++++++++++++++++++++++++++++++++*.........+......+.....+............................+.........+...+...+.........+............+...+.....+......+.........+.+..............+...+.......+........+...+...................+..+...+..........+...............+...+.....+.........+.+........+.+.....+....+...........+.+..+............+.........+.......+.........+........+..........+......+......+.....+...+.+.....+......+.+......................................+....+........+............+...+.............+.........+...+...+..............+.+..+.......+........+...+......+....+.........+..+..................+...............+...........................+.....................+...+++++ ----- [root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer mtls]# ls server.crt server.key Generate client cert/keys: [root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer mtls]# openssl req -x509 -newkey rsa:4096 -nodes -keyout client.key -out client.crt -days 3650 -subj /CN=my.client .....+....................+...+.+..................+...+...+......+.....+...+.+..+...+.+...........+.........+..........+......+...+......+...+..+................+...+..+...+++++++++++++++++++++++++++++++++++++++++++++*.+.....+....+...........+.........+++++++++++++++++++++++++++++++++++++++++++++*.+...........+..........+............................................+..........+.........+..+....+..+....+......+........+.......+...+.....+.+...............+..............+.+..............+....+..+.+............+............+...........+....+........+.+..............+.......+........+...+++++ .......+.....+.+...........+....+.....+.........+.+...............+..............+.+........................+...........+.........+.+.........+..+....+.........+...+..+......+.+........+.+.....+.+.....+.........+.+...+...+.........+++++++++++++++++++++++++++++++++++++++++++++*..+.....+......+...+.+...+..+.............+...+...+.....+.........+...+....+.....+...+.+++++++++++++++++++++++++++++++++++++++++++++*......+...+.........+......+...+....+...+............+..+...+......+....+......+......+........+......+.......+.........+...+...+..+............+.+........+....+..+.........................+......+..+......+......+...................+......+...+...+.........+...........+.............+........................+...+...............+.........+........+.......+.....+..........+...+...+...+.........+......+.........+++++ ----- [root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer mtls]# ls client.crt client.key server.crt server.key [ceph: root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer /]# ceph orch ls nvmeof --export > gw-conf.yaml [ceph: root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer /]# cat gw-conf.yaml service_type: nvmeof service_id: rbd.gw_group1 service_name: nvmeof.rbd.gw_group1 placement: hosts: - ceph-cd-nvmeinbbnd-x4fagt-node6 - ceph-cd-nvmeinbbnd-x4fagt-node7 spec: abort_discovery_on_errors: true abort_on_errors: true allowed_consecutive_spdk_ping_failures: 1 break_update_interval_sec: 25 cluster_connections: 32 conn_retries: 10 discovery_port: 8009 enable_monitor_client: true enable_prometheus_exporter: true group: gw_group1 log_directory: /var/log/ceph/ log_files_enabled: true log_files_rotation_enabled: true log_level: INFO max_gws_in_grp: 16 max_hosts: 2048 max_hosts_per_namespace: 8 max_hosts_per_subsystem: 128 max_log_directory_backups: 10 max_log_file_size_in_mb: 10 max_log_files_count: 20 max_namespaces: 4096 max_namespaces_per_subsystem: 512 max_namespaces_with_netmask: 1000 max_ns_to_change_lb_grp: 8 max_subsystems: 128 monitor_timeout: 1.0 notifications_interval: 60 omap_file_lock_duration: 20 omap_file_lock_on_read: true omap_file_lock_retries: 30 omap_file_lock_retry_sleep_interval: 1.0 omap_file_update_reloads: 10 pool: rbd port: 5500 prometheus_connection_list_cache_expiration: 60 prometheus_port: 10008 prometheus_startup_delay: 240 prometheus_stats_interval: 10 rebalance_period_sec: 7 rpc_socket_dir: /var/tmp/ rpc_socket_name: spdk.sock spdk_path: /usr/local/bin/nvmf_tgt spdk_ping_interval_in_seconds: 2.0 spdk_protocol_log_level: WARNING spdk_timeout: 60.0 state_update_interval_sec: 5 state_update_notify: true subsystem_cache_expiration: 5 tgt_path: /usr/local/bin/nvmf_tgt transport_tcp_options: in_capsule_data_size: 8192 max_io_qpairs_per_ctrlr: 7 transports: tcp verbose_log_messages: true verify_keys: true verify_listener_ip: true verify_nqns: true [ceph: root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer /]# cp gw-conf.yaml mTLS_SPECIFICATION_FILE.yaml [ceph: root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer /]# cat >mTLS_SPECIFICATION_FILE.yaml service_type: nvmeof service_id: rbd.gw_group1 service_name: nvmeof.rbd.gw_group1 placement: hosts: - ceph-cd-nvmeinbbnd-x4fagt-node6 - ceph-cd-nvmeinbbnd-x4fagt-node7 spec: abort_discovery_on_errors: true abort_on_errors: true allowed_consecutive_spdk_ping_failures: 1 break_update_interval_sec: 25 cluster_connections: 32 conn_retries: 10 discovery_port: 8009 enable_monitor_client: true enable_prometheus_exporter: true group: gw_group1 log_directory: /var/log/ceph/ log_files_enabled: true log_files_rotation_enabled: true log_level: INFO max_gws_in_grp: 16 max_hosts: 2048 max_hosts_per_namespace: 8 max_hosts_per_subsystem: 128 max_log_directory_backups: 10 max_log_file_size_in_mb: 10 max_log_files_count: 20 max_namespaces: 4096 max_namespaces_per_subsystem: 512 max_namespaces_with_netmask: 1000 max_ns_to_change_lb_grp: 8 max_subsystems: 128 monitor_timeout: 1.0 notifications_interval: 60 omap_file_lock_duration: 20 omap_file_lock_on_read: true omap_file_lock_retries: 30 omap_file_lock_retry_sleep_interval: 1.0 omap_file_update_reloads: 10 pool: rbd port: 5500 prometheus_connection_list_cache_expiration: 60 prometheus_port: 10008 prometheus_startup_delay: 240 prometheus_stats_interval: 10 rebalance_period_sec: 7 rpc_socket_dir: /var/tmp/ rpc_socket_name: spdk.sock spdk_path: /usr/local/bin/nvmf_tgt spdk_ping_interval_in_seconds: 2.0 spdk_protocol_log_level: WARNING spdk_timeout: 60.0 state_update_interval_sec: 5 state_update_notify: true subsystem_cache_expiration: 5 tgt_path: /usr/local/bin/nvmf_tgt transport_tcp_options: in_capsule_data_size: 8192 max_io_qpairs_per_ctrlr: 7 transports: tcp verbose_log_messages: true verify_keys: true verify_listener_ip: true verify_nqns: true enable_auth: true root_ca_cert: | -----BEGIN CERTIFICATE----- MIIFIDCCAwigAwIBAgIUbwWf62pxZkp8x3XyLesdqHsGg8MwDQYJKoZIhvcNAQEL BQAwFDESMBAGA1UEAwwJbXkuc2VydmVyMB4XDTI1MTEyNDExMDMzNVoXDTM1MTEy MjExMDMzNVowFDESMBAGA1UEAwwJbXkuc2VydmVyMIICIjANBgkqhkiG9w0BAQEF AAOCAg8AMIICCgKCAgEA58IjSPf7/2kGpUk5QhSEij2QB196RORYG5O8oYads/FM D6QdudJnfZNTXU+9j3uZkt5TjyZupgyUBp/SpivoqHvTbhglIiVOWIxC2d7RhjJ8 lFKkdkYFpE2AsT86Ce0OIxKMHNqtvM6Zw0CXKOldrADhXLKj+nlVzDQeMchXQnSc e7cubrqhd2m3cTz9qmw/yudFQAc1F7yJQ893J/m3TXTTfLPq4sT3DRFzxcZSScfJ zeldNBNmsva0RHi3ct33C0FcotT2BGoZ3/oaBGtO2qncgcPUsR74fbB+aRThhSGr hrZNTmX2pyC+jWC4Jta1CSVf3bLHbSbQNTCVIefli/ORYAXHL6BUNpn5tB1jb3Zh S3T1gzbjTq6bFyz+D8l+GJkzyYsTWMfVD6ImKw8/RQjVu+Pi8SGotoGNrLo2uMYc gvkN8ZuEypYGYluYhrj059IdOKOgW5aBWV9nVj9sfnaAMJXU9cXxlT/QtgxOMl2h eU7wlHmojnlAUxwtevtY8E8FJibmcvTNRnH0yKrBt/bfrjCAWMCupRMNP/8McGFm 0bCi75wqzXH33CyGW0Pj0c9WImRt9u/BfnUuJ5rorH5xZ72PKWOROurh9eImvYMS Q+e+6edV7uYCpyQLdlIVUWMb3uWNgCgZMNOGjqQIvY8CRkwwZadRMmOqrNRpBnkC AwEAAaNqMGgwHQYDVR0OBBYEFOtPkAdnw/Z2Yu51G448h0aLudeNMB8GA1UdIwQY MBaAFOtPkAdnw/Z2Yu51G448h0aLudeNMA8GA1UdEwEB/wQFMAMBAf8wFQYDVR0R BA4wDIcECgBADIcECgBBfjANBgkqhkiG9w0BAQsFAAOCAgEA2GXxzFR3bFp7m9be EM+aaQnoo8bDp3Ng44kIKVc61wY4MTIXvB7qGbU2C3YomszTi+8uFPxBERgyP5PX OppWUKbNMcFSQkdSfqzGV6i8tLlm4BjnB3xlIDezxDU8XFPKaNvxp9aiGCpbKH2v QyN2rfHD5zgMjnieGdYmsmTTjJ7x2CozC/ju7ngPyzFeJN3ZjpyvD3ggjIubnmvQ GhR0bewvHhvYruYnHwaSey7Vxa937Bx2GDqKN/Q284ozCPWSlLK/iO/GfgeKVTYv +VdSlgYxZi7vUJ+llnBxLXt8oUy4jZDaNKPNDg62Zta8Et2OAY0wmmQ1R9soCWdM tbt9CQMRVTyKZQJ4XO9Fi8ZonacSI8tEkZiiT0Hny83v+YNa+xwz1/upQS7rLnsh PnTjKbuXXwvv+znq7Tumoq3nBJX06+isB3CC/1ggzpRFLLgzjTXPmjcJ8p65+hJM 0ZUyqWxoRsCptrZa+0eJ8COmJUwTNhtkeBjqjZRjBDpnsROcTs1CYb+05di4LnTq 1NFp7A1hZ29PGvA32p8pQ1jock5YLWQUZPw5+5OqzblB2HQisImVZIwj0L8lzBxr m4O7BKK31GKY9kkf/7YY26qkrAP6bmKdl+uZ7F5sGGNyoqyDNDPqbjjuVWCowdYb euzspgi9Wi6n2RFZt7lCkzDTGu0= -----END CERTIFICATE----- server_cert: | -----BEGIN CERTIFICATE----- MIIFIDCCAwigAwIBAgIUbwWf62pxZkp8x3XyLesdqHsGg8MwDQYJKoZIhvcNAQEL BQAwFDESMBAGA1UEAwwJbXkuc2VydmVyMB4XDTI1MTEyNDExMDMzNVoXDTM1MTEy MjExMDMzNVowFDESMBAGA1UEAwwJbXkuc2VydmVyMIICIjANBgkqhkiG9w0BAQEF AAOCAg8AMIICCgKCAgEA58IjSPf7/2kGpUk5QhSEij2QB196RORYG5O8oYads/FM D6QdudJnfZNTXU+9j3uZkt5TjyZupgyUBp/SpivoqHvTbhglIiVOWIxC2d7RhjJ8 lFKkdkYFpE2AsT86Ce0OIxKMHNqtvM6Zw0CXKOldrADhXLKj+nlVzDQeMchXQnSc e7cubrqhd2m3cTz9qmw/yudFQAc1F7yJQ893J/m3TXTTfLPq4sT3DRFzxcZSScfJ zeldNBNmsva0RHi3ct33C0FcotT2BGoZ3/oaBGtO2qncgcPUsR74fbB+aRThhSGr hrZNTmX2pyC+jWC4Jta1CSVf3bLHbSbQNTCVIefli/ORYAXHL6BUNpn5tB1jb3Zh S3T1gzbjTq6bFyz+D8l+GJkzyYsTWMfVD6ImKw8/RQjVu+Pi8SGotoGNrLo2uMYc gvkN8ZuEypYGYluYhrj059IdOKOgW5aBWV9nVj9sfnaAMJXU9cXxlT/QtgxOMl2h eU7wlHmojnlAUxwtevtY8E8FJibmcvTNRnH0yKrBt/bfrjCAWMCupRMNP/8McGFm 0bCi75wqzXH33CyGW0Pj0c9WImRt9u/BfnUuJ5rorH5xZ72PKWOROurh9eImvYMS Q+e+6edV7uYCpyQLdlIVUWMb3uWNgCgZMNOGjqQIvY8CRkwwZadRMmOqrNRpBnkC AwEAAaNqMGgwHQYDVR0OBBYEFOtPkAdnw/Z2Yu51G448h0aLudeNMB8GA1UdIwQY MBaAFOtPkAdnw/Z2Yu51G448h0aLudeNMA8GA1UdEwEB/wQFMAMBAf8wFQYDVR0R BA4wDIcECgBADIcECgBBfjANBgkqhkiG9w0BAQsFAAOCAgEA2GXxzFR3bFp7m9be EM+aaQnoo8bDp3Ng44kIKVc61wY4MTIXvB7qGbU2C3YomszTi+8uFPxBERgyP5PX OppWUKbNMcFSQkdSfqzGV6i8tLlm4BjnB3xlIDezxDU8XFPKaNvxp9aiGCpbKH2v QyN2rfHD5zgMjnieGdYmsmTTjJ7x2CozC/ju7ngPyzFeJN3ZjpyvD3ggjIubnmvQ GhR0bewvHhvYruYnHwaSey7Vxa937Bx2GDqKN/Q284ozCPWSlLK/iO/GfgeKVTYv +VdSlgYxZi7vUJ+llnBxLXt8oUy4jZDaNKPNDg62Zta8Et2OAY0wmmQ1R9soCWdM tbt9CQMRVTyKZQJ4XO9Fi8ZonacSI8tEkZiiT0Hny83v+YNa+xwz1/upQS7rLnsh PnTjKbuXXwvv+znq7Tumoq3nBJX06+isB3CC/1ggzpRFLLgzjTXPmjcJ8p65+hJM 0ZUyqWxoRsCptrZa+0eJ8COmJUwTNhtkeBjqjZRjBDpnsROcTs1CYb+05di4LnTq 1NFp7A1hZ29PGvA32p8pQ1jock5YLWQUZPw5+5OqzblB2HQisImVZIwj0L8lzBxr m4O7BKK31GKY9kkf/7YY26qkrAP6bmKdl+uZ7F5sGGNyoqyDNDPqbjjuVWCowdYb euzspgi9Wi6n2RFZt7lCkzDTGu0= -----END CERTIFICATE----- server_key: | -----BEGIN PRIVATE KEY----- MIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQDnwiNI9/v/aQal STlCFISKPZAHX3pE5Fgbk7yhhp2z8UwPpB250md9k1NdT72Pe5mS3lOPJm6mDJQG n9KmK+ioe9NuGCUiJU5YjELZ3tGGMnyUUqR2RgWkTYCxPzoJ7Q4jEowc2q28zpnD QJco6V2sAOFcsqP6eVXMNB4xyFdCdJx7ty5uuqF3abdxPP2qbD/K50VABzUXvIlD z3cn+bdNdNN8s+rixPcNEXPFxlJJx8nN6V00E2ay9rREeLdy3fcLQVyi1PYEahnf +hoEa07aqdyBw9SxHvh9sH5pFOGFIauGtk1OZfanIL6NYLgm1rUJJV/dssdtJtA1 MJUh5+WL85FgBccvoFQ2mfm0HWNvdmFLdPWDNuNOrpsXLP4PyX4YmTPJixNYx9UP oiYrDz9FCNW74+LxIai2gY2suja4xhyC+Q3xm4TKlgZiW5iGuPTn0h04o6BbloFZ X2dWP2x+doAwldT1xfGVP9C2DE4yXaF5TvCUeaiOeUBTHC16+1jwTwUmJuZy9M1G cfTIqsG39t+uMIBYwK6lEw0//wxwYWbRsKLvnCrNcffcLIZbQ+PRz1YiZG3278F+ dS4nmuisfnFnvY8pY5E66uH14ia9gxJD577p51Xu5gKnJAt2UhVRYxve5Y2AKBkw 04aOpAi9jwJGTDBlp1EyY6qs1GkGeQIDAQABAoICAGoOnqUazthQELl869YAm7LD f3DQUZzLPkLIde8ZMnwRNXSG7w+FaNS/ocu23RGXR209jtGYjdQxN2BcrUJcH6lS mSvtorigh9zfuwwG+q1F11rtAtQZMSXCTd1PCNlYATstlq7xp8YaaaU/nq0bNkGr WXZZTc8mzYuYYiT4AG6p/J5k8ru7bn6+AzOv7Imb5gmNklQ9uB83qT096NKDZU3t 44ZSs2+5ITF6xhdArZGT+xbe70cQhdPqMA6Ri6nmR9J/w8vEVx0oDNVBl7xj+v73 8q/zlRbFDIRAS2j9JEMPaHZKYNA6uwg6ZNk/5KXpZYlc8CT9VmPt+iQNKoIMpC9r gmm9qRKg1LJE+kTn/ipbY5Dyvu1ZfsM2fr4bUIgTnMQ8vVw+JTDZuaEEZqatFSh9 jN3kusfe5XG9PFiPI/Uc3X9yZ5s3VDOW9NcOp5GFAEVdia0qh4dSW4/OIzfCH5Vf czRGhvxUb3pqCLpuN4cN5mrj/j4zgO1cZznuXN1Mxz0loo1X+QC5EXSdhqe/15P8 esKZoJgOHn/D5KhL493x2TyLZS6Qu9uMY+pa9gcu5OcfC/d+/FllTHoNi93Jfa84 pvbPequuAUgBxDuo1m/5BRuUXvSAgKPlA/+ZZBQwzU5q2sMEJnp4e7U/5PLQMygA wpLREyTD+kCNdSVU87+tAoIBAQD1xeCJ+FvrCiCOxLPx8hLezF2fi32Xq3pU8GyP xrjaxzBMyOW2NGnXTURr9IXr7smQj0snqJhSU8aZT8wMT3SWOvoGEKp8vDyx8TGb 02ju46ZkLWDf7amrsYRaFbbykckqtccuEnRWNZUdt50uVp1+L2uMieQ+5BvP4bCi /BPfbvVOBAG9y6JbRyI8EE2rkQHg8ZGEmnsbyfLDLkBDpeobQ3UrESSl/ip7Womn k+lgTUAnWT391E+kBB9rJ+LSr1ujDY3kCBHox/uPPnkC5yJ7tCnxUsgRLSx1blxQ dO1W3ZwEJBPyBBR7lb1fqkwkbTZujAuhOXFlxlOcSd9+waTvAoIBAQDxZvf3lqiI 0IrrhpZ1fbf7MBDnrGnQCt1pxjcWvqoP9GAsGqTovRsLlct8DfSx21BQmObgH8cw txGin0qAz/hA0/xfU5Q1XabDYMrv8PSJ/YAaAvlrcrGyPtcL3ZygJQjWKT90JKTd iIE3lxdsykDtXRDWW14+lPJhoQYkW0HTyiRpoajvM2ANpz8QiYCm4jd2tlZ5Jn5n LsFDdv756E4/LglVhfZgOq8q5Jc3nfshOaFJLc6IQAGbzrjCfo/rqTCSRR4ooC+W d+Z7WPCvvKHnoWSuhR4phzBRFS3V1fF6j3SiJNZ5u+EnWh0qp6WiETOym8YNrsAX lqDPioBxQhsXAoIBAQDz5tG0WkJt+Hji9hZVzxQnAs0OnhGfSM94J7i03lNo2i/2 o5BDx7QZtFZohBu7The+XFYLH2c6nGUElIo81UUwKYsiJbvrWasqM+A1chI6sLTZ yabP0z2EZACindYW19K5jfgCotHNFBPNkABukgptxYWHzPgz6ZAoNyNiO6NFIwpR x+wqfEG9L9SrospRrvAcWd89p0jxBPLj+6xO9gJj2AgZG0QxbxCdpfXh1gnORhua cRfyJJZ8gJLldonE572bDMJ5QAD18DCXsTnKRbKU7PCKyso9MYVjbIg0WDWI9Rf1 jBA3JxFkq+9pVsmFhNR11IleEHGWvMEQ4sbnNikFAoIBAFH6ldsO3s0XhRXW7uGX gBNdj2AdvRFZnzd8iNjAhyBD7SUNJ1OiZLzace//DUoZIH0+b6lNIWf5w5NmZPa5 qaTaWKpGx94Xn7pZ9JDnUyRJYmdy+yDyIkZV8NGmwLnh2c5g05utlg9le/xU4y+e v83b/QvOm4LFMkrVVDCkjQZPb24v6B7TY5s/YLtM9tO1nwBBi7w4xlRkHF2823oi 2e+6wpC4naKwSyxV1MSecdomqA8h7zF2CxXObhAeyuFM46HItyU0wPRoY20gFxyX AqSWuopJyqRvtIjUmGw163ujnAg0WmGc3ClxANareCRBkzzkia272qAoFxRkBJpx yKECggEAc/XtEHM6g0nQIS9ITs9Y8k/z41wR25SKh489vJhoDXcT+CBWEoBGSMyU 5jPdz1G7GEKKCEUDw7j80qcs6ky5h43zhhET4cL14HfSp2m44cbditLJ0OjSEyWW Q4v2vCPAYmr61KrQ/qa+KyX8puZR9hY+IvFxMStY6QItognhD+vNWTtBeTdgQZoi EulTMXiaRbTSCQgHA54ibfGZlADVs0LyJqY3ZyfXJgOqcuqfwczqi0YTbt8LmkSc HZALPRUuCy//Fc+o3T+E2L19cicQy7OITF51XmRarA52AjQe2XQ4Bc7n8m0iY16f z0hH/UlNmMWygIFaBUDvPkPJBDrPmQ== -----END PRIVATE KEY----- client_cert: | -----BEGIN CERTIFICATE----- MIIFCTCCAvGgAwIBAgIUHnkO8XGw22ycMNsc1BBPZp9MbscwDQYJKoZIhvcNAQEL BQAwFDESMBAGA1UEAwwJbXkuY2xpZW50MB4XDTI1MTEyNDExMDYzMFoXDTM1MTEy MjExMDYzMFowFDESMBAGA1UEAwwJbXkuY2xpZW50MIICIjANBgkqhkiG9w0BAQEF AAOCAg8AMIICCgKCAgEA6WVbSnGzEbdZ5Tks9UuwhV7yUr/WTfTyLTQbNGK0aVmN wiryZNOYibtpXPIrb1qM1NtbXpquPPLV2tSGXHl8ZoP4ICdKs+36DDwykNiXBGIO 9LkDfMfan+caF2OHi+9ccY62kDVDBUl1No6DMsbBTbj2dUbS6yGxeiCVeOLFdpok onSSkIz1wBQ62VEGuz7pmlnpzOHX+2aWBT1T7jrOIMSEBVTFVvzTMIF/Oes3dvCY pwY5pXX9LBI+kIgqInDWAkiSBh5Dza/nrgHiIjjdBpvsDvjF1+4pdVmbd9Ktw86F nvcX1/8mShu+327Vs3zyrhJD45eOv1Y37PqGLOfPTMLtfBKTT9SB3jbz6ufEsOI6 FWlZDCnSWYaPujBaVhMo0x/Lx80x1XwISzzvQMbrprQuJf+si2Jrr8lL8humpYgW K3bkDp+pC8sJ6tvTU2tq1HkG45vIUJy+5WVu70XuH2Aknto4VnXJJ18BVDXaSaeR ln+lx+M3bJTCPN8eVh9QuRCsQx7z6JU1aD7wIdGshwM51epowWNo2juIP7aRdMgZ avcFhhQkKesCIUJZTo2fxDN0HzxMjLFxEF4R67yu4iIem8ObdGDSvx6L9yB7Wj6i /PwzM7d6qwTVcsGYp5qKSt+3vIjsIN+WgTutGd+qv74i++3xxw8Q/793yTA1b8EC AwEAAaNTMFEwHQYDVR0OBBYEFMBR1REF1Xgcgi9SFEplyvhN8aawMB8GA1UdIwQY MBaAFMBR1REF1Xgcgi9SFEplyvhN8aawMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI hvcNAQELBQADggIBACwnxi6pilqjS3y+hBmIrUTVLoSUT2t5YwCiwZOhrugaKx9l Sks3KQrNEmwNZfvcuL9CilMg3ITQ570ZJlHl4VBjfOpF1RcNC5c6RK7akcws5yNg Iimd4foD9IjAceavO7ryeRFjiYMd2KIaHEhF1LeCDRCEwEM4x6qlp4yaQR3FFLRv Gr9+48QHNGUsI7eK/3yao0UZLavb59DdqjYP2GDA9yy3yyy32uxFd9TcwKq2U/CO AyF1oDSfrK2HSAynNCti0oijhhC4w1uR8ZbEotfdMidmgGHFKa5XOoJ4ycknJk5X Y7XvSWPOXJZdCG5qyhGG1NSMKx2mLEh/Qfpp+xdZXRupCvaoAKuYEG+ZH/2Nz4es Lel67Ydbowa4NE6xKuY0U713r6We6iPE0ylFv96aJBoIDk524kWe5W6Y7QmMwJU3 sXDoI+S10MtNXTHUC9DPaAjJykrUZq+OvFNwtldn8LyB1T/a9ugjdqyKFZ5L/upQ qeo01UwSnR16Rx9KPIBHf7AYSSUvlU4B7Lg0IE35ckxO+ueo4Szrvy0/qy12kyW0 KWTAI8RetE3moH1ig+U6RGsvMWaLespQMBtkNARh92O4AN6cr1NHEN9as4IczQKP +R3V6x/aAJ9ykyVs5v8v2TAQvtd4rwhonsYNm2WS99XusRsvlPsZ/Fw8MR3L -----END CERTIFICATE----- client_key: | -----BEGIN PRIVATE KEY----- MIIJQQIBADANBgkqhkiG9w0BAQEFAASCCSswggknAgEAAoICAQDpZVtKcbMRt1nl OSz1S7CFXvJSv9ZN9PItNBs0YrRpWY3CKvJk05iJu2lc8itvWozU21temq488tXa 1IZceXxmg/ggJ0qz7foMPDKQ2JcEYg70uQN8x9qf5xoXY4eL71xxjraQNUMFSXU2 joMyxsFNuPZ1RtLrIbF6IJV44sV2miSidJKQjPXAFDrZUQa7PumaWenM4df7ZpYF PVPuOs4gxIQFVMVW/NMwgX856zd28JinBjmldf0sEj6QiCoicNYCSJIGHkPNr+eu AeIiON0Gm+wO+MXX7il1WZt30q3DzoWe9xfX/yZKG77fbtWzfPKuEkPjl46/Vjfs +oYs589Mwu18EpNP1IHeNvPq58Sw4joVaVkMKdJZho+6MFpWEyjTH8vHzTHVfAhL PO9AxuumtC4l/6yLYmuvyUvyG6aliBYrduQOn6kLywnq29NTa2rUeQbjm8hQnL7l ZW7vRe4fYCSe2jhWdcknXwFUNdpJp5GWf6XH4zdslMI83x5WH1C5EKxDHvPolTVo PvAh0ayHAznV6mjBY2jaO4g/tpF0yBlq9wWGFCQp6wIhQllOjZ/EM3QfPEyMsXEQ XhHrvK7iIh6bw5t0YNK/Hov3IHtaPqL8/DMzt3qrBNVywZinmopK37e8iOwg35aB O60Z36q/viL77fHHDxD/v3fJMDVvwQIDAQABAoICACZ3JumI/Zsw3abyml8+/zOr 50IODdLBSNx8VvN2RQeL09opW9OUL/KbKY1A+Dmc8dLu6AsacRboaqhrGE3LU95N Z6pc72niEbDMhwTqPVFim9ZnW5TIF1b9LoamR6OfalvoylvP+2C2r5OzPxoyOY+G qU/AwWgxIF/hWOEYdrTKSLlhfvpF0Qa8DsrcGMDpaHSg1vCGVCNBz/3DEr8Tf/9U g/Gg+/kPJ/s5jG9OPdDjbDUyyHNlk7kctKT7W/5bHFo5/Z72g/Q1RnieJNfBCWRa dDgvdePVg42sm/vOhAbeVZVSYeKeUYArPHT0flHbiN95cSjkRIlyOztaOpzSA47x hbf25SViPdSOtlXJubY6PErVw6FG4WqgYt+7BAqzbJEr/nE0PXSzhxcc9g5I5CTe FSOS4uWzLMDx5eyjC309giqEDKMXVIgkVnlltF2Rcc8bCEYQusqsvTMpBVKvQ+0x 98lgTSD+mrt6CpxwKGLavyY5aBgkBT+9+3uIQUqi66AHN8W8UUkuJ9NHFyHEQmEq wq2qjHfZQbZ3/JqO2YMmH40usk2wWtg0fPAlc/43cQl+qxCaJmZy0IsSe7Ze8Csu 8XkFRqN0hJWdd5fAJZoFG9oYUnXr5vqe4YdNoPLTjfJIWzDyX57lyRxY0ZTIdIKx Zv21NDa98J+MbORUjYiVAoIBAQD6EKqbVB41BT0WdbhUhE53ZNLd2bTVUgExRzlY WU+tE25UqbUPsKB+LYMEWqxq1PE4JMK5oSg1iCcIxNSZwHyhsAs9Xt0q9U9IvcMC 37bhMO3Y/lL2ghmmIOyA1cITAnOjHtZRKwusC/LLIFPHZYSszk//eabxYpsvH/ih 6QneARx2/EGA3W5mOHRzaEZIAvyVvTsdaEr+wbuj5mvYjIpL7TaaQkCQVSVpNWwX Pk9Xgs7FnJzmQoo/GnXAt+koItdy9b1xNQEb3UrzkXWA+TZV1jPkEHIEVUHznC+H VfBATW1Yqg1O0FWHDaMF4R1S9rARrObtjAlk8uSM0sMkxFNjAoIBAQDu72mRSs0M XdJ7+7uLdy72bDCBuKySgwLk1NZQY61piaKwQ8WlkwIa2LRz5rkfPJ0E6ajcUxgi eRyQG9Rdg6ZH8dh/smN9k00vor5ZCkmQwkQtwm4QpT+H3ObyGMTNs15tMkNOnaCY BREBze8up+SwYyqdUVO7+OW0JtEgiJZpgKa/kTga7FL+lug0/iGpRtBYBE82F1Mz 9OQCYcwC5adY6XQvRKAQD8HBdWBU1yvImyu2ZxAQ5tiVrWzev8muZIE2es4v5hzr yvQtswR2KiHy8n1jNX+ut2AoqTNR9MZeCQ7FqV9c+9aYLElp6wuBIUxHzvXIM/gP t6IO6n+fdgOLAoIBAAE9deshRqkSppPm74wTv+PdryIEg14PcwwQg1tJzhhYoVJw KQKz6IfXp++sl3BMMAoMPJ75Cp8BVtg3SaqdhsCyg8vBHa2p28PLlRz9iQWi3XsW /8hTKtdlEGcvKDA58eLcM0nhwcQPpxuUxHWXrknBGCp3KvnFeGmKagw7gJ4GOWLp q1V9Q5/kJcewu7B9u0zuzE1ACCctRMXyTZScVc88lOm87F8K3Q2hgzC2UWRotKfM OAb4S4E2m2DiSJi3W4QpbdAS6DeAKG/+x6tlEPzp/AXkgvw8hAA2RRIRvghDmgg3 h6tdYx8xVR6hWCjiKktcJSUGbcF/Zty/6tHS22sCggEAWk5+tf1xlpT3cwjoD4RS aazmgIEOKKn/tAGFpwiliFgO91QGJdh0WTVCrvOrUFUWe77cpbl6S70+wHDkSRvz Awoza8A+vj2oVLLMAtdSirYCEtGDzCLI6k7hsusNpzl+HQN8UQBerFZt4SK2ngmy 87/nYWJVpZwjJX3uWL+HKgUIASuf6v0bOUJMLYtHqbUsFbg7QBR8zAxpCUVxWD74 izBhS2RCow9xie8VI7vXvpB3aw91EQ5PGfNcTKxsS8+MzsyHla8Bi1VACO26Z/bw seRY4+Bai+yfGX8p+HXta6k8vhSQCJLevtEuRiwiQwIU1KcyWiLaRQaA5+ZVqi1u lQKCAQAMtTX7Z0panm9/nWstrPsAVvVODZ5zF68TuOK6Xv4/tzuyyb66/+JDhX/1 QSmbylpYPLSCIeXfNXDgs8uQoP5+i8048EQzED1wyHBPYUfYYqe9eD6P5oziqANB xc1uwSB8cqX7fyj7OruRTDAujDGNWwSFol6Piy7FeC7GoY/kll4B/JhJ6CEj6xAu qxPqBSGpKL+p1KJML7LKimKD5z0lMZUf/gF29uL+kkQEj48wdivDPJrfxxEB+tgp +syxQlO7wyd1K3R3ZBTu3ZbULcF8RLmbr/Zli0Gz2Tm0GJpIhHcXmshVilsmxvRk go4BxX3kQHHXc+m97U19Rl7315pP -----END PRIVATE KEY----- [ceph: root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer /]# ceph orch apply -i mTLS_SPECIFICATION_FILE.yaml Scheduled nvmeof.rbd.gw_group1 update... [ceph: root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer /]# ceph orch redeploy nvmeof.rbd.gw_group1 Scheduled to redeploy nvmeof.rbd.gw_group1.ceph-cd-nvmeinbbnd-x4fagt-node6.omdhuv on host 'ceph-cd-nvmeinbbnd-x4fagt-node6' Scheduled to redeploy nvmeof.rbd.gw_group1.ceph-cd-nvmeinbbnd-x4fagt-node7.cvxqpi on host 'ceph-cd-nvmeinbbnd-x4fagt-node7' [ceph: root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer /]# ceph orch ps | grep nvmeof nvmeof.rbd.gw_group1.ceph-cd-nvmeinbbnd-x4fagt-node6.omdhuv ceph-cd-nvmeinbbnd-x4fagt-node6 *:5500,4420,8009,10008 running (60m) 7m ago 80m 127M - 1.5.13 e2c28b957bcc bb737c858b05 nvmeof.rbd.gw_group1.ceph-cd-nvmeinbbnd-x4fagt-node7.cvxqpi ceph-cd-nvmeinbbnd-x4fagt-node7 *:5500,4420,8009,10008 running (60m) 7m ago 80m 131M - 1.5.13 e2c28b957bcc fe2df28ee4fd Now, ceph nvmeof cli commands fail as below [ceph: root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer /]# ceph nvmeof gateway info --gw_group gw_group1 Error EINVAL: failed to connect to all addresses [ceph: root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer /]# ceph nvmeof subsystem list Error EINVAL: failed to connect to all addresses [root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer mtls]# scp * root.64.12:/etc/mtls client.crt 100% 1805 4.2MB/s 00:00 client.key 100% 3268 6.7MB/s 00:00 server.crt 100% 1838 5.8MB/s 00:00 server.key 100% 3272 9.6MB/s 00:00 [root@ceph-cd-nvmeinbbnd-x4fagt-node1-installer mtls]# scp * root.65.126:/etc/mtls client.crt 100% 1805 3.7MB/s 00:00 client.key 100% 3268 7.6MB/s 00:00 server.crt 100% 1838 6.5MB/s 00:00 server.key 100% 3272 8.7MB/s 00:00 However nvmeof gateway cli works: [root@ceph-cd-nvmeinbbnd-x4fagt-node6 mtls]# podman login cp.stg.icr.io -u cp -p eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJJQk0gTWFya2V0cGxhY2UiLCJlbnYiOiJzdGFnZSIsImlhdCI6MTY4Mzc3NDg1MSwianRpIjoiZWM4ODUyZDFhOWE0NGE3ODkwYWE0YzA1MDdlOTk1NDgifQ.o9Z2h1lbbA_zaeV55QMoTsSlkk-gFt0yJHiTFyApkHs Login Succeeded! [root@ceph-cd-nvmeinbbnd-x4fagt-node6 mtls]# podman run -it -v /etc/mtls/server.crt:/root/server.crt:z -v /etc/mtls/client.crt:/root/client.crt:z -v /etc/mtls/client.key:/root/client.key:z cp.stg.icr.io/cp/ibm-ceph/nvmeof-cli-rhel9:v9.0-7673.20251120035518.6b9fdc7 --server-address 10.0.64.12 --client-key /root/client.key --client-cert /root/client.crt --server-cert /root/server.crt gw info Trying to pull cp.stg.icr.io/cp/ibm-ceph/nvmeof-cli-rhel9:v9.0-7673.20251120035518.6b9fdc7... Getting image source signatures Checking if image destination supports signatures Copying blob 6e3290d827a0 done | Copying blob 1153e061da4e skipped: already exists Copying blob c4319966581d done | Copying config b1fb9f90d0 done | Writing manifest to image destination Storing signatures Enable server auth since both --client-key and --client-cert are provided CLI's version: 1.5.13 Gateway's version: 1.5.13 Gateway's name: client.nvmeof.rbd.gw_group1.ceph-cd-nvmeinbbnd-x4fagt-node6.omdhuv Gateway's group: gw_group1 Gateway's host name: ceph-cd-nvmeinbbnd-x4fagt-node6 Gateway's load balancing group: 1 Gateway's address: 10.0.64.12 Gateway's port: 5500 Gateway's max subsystems: 128 Gateway's max hosts: 2048 Gateway's max namespaces: 4096 Gateway's max namespaces per subsystem: 512 Gateway's max hosts per subsystem: 128 SPDK version: SPDK v25.05 git sha1 c19d07f46274d31201c0c4db3775a655d68f5f38 [root@ceph-cd-nvmeinbbnd-x4fagt-node6 mtls]# podman run -it -v /etc/mtls/server.crt:/root/server.crt:z -v /etc/mtls/client.crt:/root/client.crt:z -v /etc/mtls/client.key:/root/client.key:z cp.stg.icr.io/cp/ibm-ceph/nvmeof-cli-rhel9:v9.0-7673.20251120035518.6b9fdc7 --server-address 10.0.64.12 --client-key /root/client.key --client-cert /root/client.crt --server-cert /root/server.crt subsystem list Enable server auth since both --client-key and --client-cert are provided No subsystems Actual results: ceph nvmeof cli fails with error "failed to connect to all addresses" Expected results: ceph nvmeof cli should work