Bug 2299705 - Invalid SSL key and Cert format found on NVMe Service configuration export using orchestration command [NEEDINFO]
Summary: Invalid SSL key and Cert format found on NVMe Service configuration export us...
Keywords:
Status: NEW
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: Cephadm
Version: 7.1
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 7.2
Assignee: Adam King
QA Contact: Mohit Bisht
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-07-24 15:44 UTC by Sunil Kumar Nagaraju
Modified: 2025-04-15 08:27 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
.Using `ceph orch ls` command with the `--export` flag corrupts the cert/key files format Previously, long multi-line strings like cert/key files format would be mangled when using `ceph orch ls` with the `--export` flag. Specifically, some newlines are stripped. As a result, if users re-apply a specification with a cert/key as they got it from `ceph orch ls` with `--export` provided, the cert/key will be unusable by the daemon. As a workaround, to modify a specification while using `ceph orch ls` with `--export` to get the current contents, you need to modify the formatting of the cert/key file before re-applying the specification. It's recommended to use the format with a '|' and an indented string. Example: ---- client_cert: | -----BEGIN CERTIFICATE----- MIIFCTCCAvGgAwIBAgIUO6yXXkNb1+1tJzxZDplvgKpwWkMwDQYJKoZIhvcNAQEL BQAwFDESMBAGA1UEAwwJbXkuY2xpZW50MB4XDTI0MDcyMzA3NDI1N1oXDTM0MDcy ... ----
Clone Of:
Environment:
Last Closed:
Embargoed:
sunnagar: needinfo? (nia)

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHCEPH-9402 0 None None None 2024-07-24 15:45:52 UTC

Description Sunil Kumar Nagaraju 2024-07-24 15:44:54 UTC 
Description of problem:

The SSL Cert and Key content within a service config exported using orchestration command are in invalid format for NVMe Gateway service and this could not be used further to apply changes onto the service.


Specification file created to deploy NVMe Service
---------------------------------------------------
>>>[root@ceph-sunilkumar-00-gmqual-node1-installer ~]# cat cluster-files/nvme-mtls.yaml
>>>service_type: nvmeof
>>>service_id: rbd
>>>service_name: nvmeof.rbd
>>>placement:
>>>  hosts:
>>>  - ceph-sunilkumar-00-gmqual-node6
>>>  - ceph-sunilkumar-00-gmqual-node7
>>>  - ceph-sunilkumar-00-gmqual-node8
>>>  - ceph-sunilkumar-00-gmqual-node9
>>>spec:
>>>  allowed_consecutive_spdk_ping_failures: 1
>>>  bdevs_per_cluster: 32
>>>  conn_retries: 10
>>>  discovery_port: 8009
>>>  enable_monitor_client: true
>>>  enable_prometheus_exporter: true
>>>  log_directory: /var/log/ceph/
>>>  log_files_enabled: true
>>>  log_files_rotation_enabled: true
>>>  log_level: INFO
>>>  max_log_directory_backups: 10
>>>  max_log_file_size_in_mb: 10
>>>  max_log_files_count: 20
>>>  monitor_timeout: 1.0
>>>  omap_file_lock_duration: 20
>>>  omap_file_lock_retries: 30
>>>  omap_file_lock_retry_sleep_interval: 1.0
>>>  omap_file_update_reloads: 10
>>>  pool: rbd
>>>  port: 5500
>>>  rpc_socket_dir: /var/tmp/
>>>  rpc_socket_name: spdk.sock
>>>  spdk_log_level: WARNING
>>>  spdk_path: /usr/local/bin/nvmf_tgt
>>>  spdk_ping_interval_in_seconds: 2.0
>>>  spdk_timeout: 60.0
>>>  state_update_interval_sec: 5
>>>  state_update_notify: true
>>>  tgt_cmd_extra_args: --cpumask=0xF
>>>  tgt_path: /usr/local/bin/nvmf_tgt
>>>  transport_tcp_options:
>>>    in_capsule_data_size: 8192
>>>    max_io_qpairs_per_ctrlr: 7
>>>  transports: tcp
>>>  verbose_log_messages: true
>>>  verify_nqns: true
>>>  enable_auth: true
>>>  root_ca_cert: |
>>>    -----BEGIN CERTIFICATE-----
>>>    MIIFLDCCAxSgAwIBAgIUQYBb/jgCHtcwpuy1bxKL3mGTEm8wDQYJKoZIhvcNAQEL
>>>    BQAwFDESMBAGA1UEAwwJbXkuc2VydmVyMB4XDTI0MDcyMzA3NDIwN1oXDTM0MDcy
>>>    MTA3NDIwN1owFDESMBAGA1UEAwwJbXkuc2VydmVyMIICIjANBgkqhkiG9w0BAQEF
>>>    AAOCAg8AMIICCgKCAgEAqukwSwpTUBgSKAaJiFs7t892HD0WGFU/KoOgV7o6Yvcv
>>>    H7zifYais19vS7YZVb1X6CqHlHk8UVRHZmOAvVHLfM2NEhz/qwErFF08ZYN4GWnj
>>>    9Ncs2mzSbdvwo6sgvNElrhVz1z1epk21GXYkpjZMLeThwJ8STM08gyheIJEG3QZ8
>>>    MNKCc4jot/Q5V5kimPF+P8nM/jXKPgCUw6Z/SMe8/cQ96Qo9tyMop+NIAIqomaJI
>>>    mW4cdN+d/zlmZyRZ+iY8u6MUL/7lJeVMx9Z89co+rSBqigKiXasC4qZ4pPSsqJ+p
>>>    f/qaLksz94TqDbMiGA+w8A7l9zqxfm2UU3HIZcPomQ8a/veJCZRcI7ZNqII8KT6K
>>>    4uSIV6iJ5c5WshDwr9G2xijYIUKeTV2C6CEst91P1XMH3W3LFzZItB/V5/RfOYH4
>>>    n0XzXzTTYsdA8AqzC40C0eQu9k3La2obXqjRceLZIaH/mJwNf04UeTy8Avd2qwNJ
>>>    ifWrG1rrqjuzp6Hw6zFT9qkEds86YpQC7EA0C6ECmC03joXkVc5Lv7AF4rQjDmc3
>>>    i3qGEprFkRDktySdAMedEHj9dgTEDk+OXwUdHWbGLzDxpXEA4wyD63/sCm0r4M/k
>>>    Fcwln6hL0Yubn2fAii9oxHHbVSFu4WYoMnV7LVyjS+QwNhISfXPFnbvWmelV7yUC
>>>    AwEAAaN2MHQwHQYDVR0OBBYEFIYZ3n3dJEH5Yz1Qyzj+aZauxfzhMB8GA1UdIwQY
>>>    MBaAFIYZ3n3dJEH5Yz1Qyzj+aZauxfzhMA8GA1UdEwEB/wQFMAMBAf8wIQYDVR0R
>>>    BBowGIcECgBB74cECgBBB4cECgBATYcECgBCozANBgkqhkiG9w0BAQsFAAOCAgEA
>>>    ZxrsB7ruClrci+Dy1mEP0ZqBgs/3Nir66tS16zQSspmHnGXHpdxfPyhl4kLQNHrp
>>>    4rk3Of6wkik1RZGsLRQ5ctnpbxylMX4BWU0RGzMnKTh76rDxhoGDCofUnL448i0H
>>>    7tzfU56VVJLs9lhcKVwSsoRQQR6FOwx9xhM7fq6Y/qRdWMQbPfuCOd2kakP6WZay
>>>    74DClvz53yXIuDcZXGRlqnDfZnEs/JTxAbWLuLAp8JaZC6nNDUUct3mxdIOWKQ05
>>>    mVRBwPrEeY/c8XRr/2172TGz/j/cU4YTpnu8tQ8Kcm86nmURRLknU5Ea+o4p6Gnb
>>>    4SQMuHoZFv2/u8dmA3sV/jEd308exTdq6rYO9yC7L0tsdY7dGtBqcQZdmiPP8SSb
>>>    axI8I7INZIKi4fX9xt824G4EBj27HwwvAMKqEsIC8uSettP/ptjafqZxkUWcrpjC
>>>    MVJjaYk0YveGlaJMx6evL8sLicPamPPwahe2TcyF8d/dd8h87LMIZV0pD17EchJi
>>>    /qqIX3VQNToHotjRscyj3cryHgpHO1+WpFUjLwRBkjRRs1aD5ynjaLyRGNR8oYHB
>>>    nK89OKDDcTO9ODCcW7kf+YcV1RXFShRFpLBjkS/6oVKNO/UYeXmzeKiLir1mfHrs
>>>    B91RCIFCX5CUDyiYqeIRsYrvrrdTTr/gzDFtRbye5Jo=
>>>    -----END CERTIFICATE-----
>>>  client_cert: |
>>>    -----BEGIN CERTIFICATE-----
>>>    MIIFCTCCAvGgAwIBAgIUO6yXXkNb1+1tJzxZDplvgKpwWkMwDQYJKoZIhvcNAQEL
>>>    BQAwFDESMBAGA1UEAwwJbXkuY2xpZW50MB4XDTI0MDcyMzA3NDI1N1oXDTM0MDcy
>>>    MTA3NDI1N1owFDESMBAGA1UEAwwJbXkuY2xpZW50MIICIjANBgkqhkiG9w0BAQEF
>>>    AAOCAg8AMIICCgKCAgEA2xjItrlk0TnKPdzhfBP5wOyv7a9OGZDJdErOhZNgu3nc
>>>    ufgvK6NnPkk8KESRwTJS/dn+vEGMrydDjTHN5xxZeNC6Zr/4tTB/UY8vJjD4E5nJ
>>>    DOx+XPnaKUeNYZC6D/7/zZfms6TpwzLbkOTIkYsMXfzPrNhSnqZpwaCF8Ek0U22W
>>>    DXFonY3r4onZl0JV6kGuqjJZ3PzQ2Qig0Wc6VDwh/N47N8T1Jm+d/lsY1RgCjy6k
>>>    kdwB0QXRjjGr+OEPgvobWNJ/bpaoRfpk02ERFIf9ox7a8qbelTjrbnc3+P1nUWLm
>>>    7OYDhMuaQEM8WPx1wuTJpqETubCIyZaiPvdHDBZDc1RQntKcgpkdFI+OUATg+OTQ
>>>    05YWYL1w+DvdnsPiYKG+/pAv12yLgr/XpZiClASTq5rjTChC00KGlRgWKFQiG6tl
>>>    IxpWoql6RET6dD5MhKRQuTVlhT5Oys6Pt3uevIn2mSUci7tXSWmq8rDb+IYzHv+A
>>>    iN1sA5ng/tUKXIU3vvVDm5veXFHRHd1jui/pGpAX6GqTy9AYJQuEisbZhDKXQpFr
>>>    Agb/+cegk9cIOhJBRhncl9NwHmegrR+WTDPbd/KoZV93261GSOezU/mq/5Aoz7uz
>>>    MkwMq18SM1HAb/smwdEKp6Tf8WZ9S1EPsg1f38yEri4bEhT2kPdtPjOpmiblhSkC
>>>    AwEAAaNTMFEwHQYDVR0OBBYEFNv3bpEtWpqgCj8nj3dgSVxUrXtVMB8GA1UdIwQY
>>>    MBaAFNv3bpEtWpqgCj8nj3dgSVxUrXtVMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI
>>>    hvcNAQELBQADggIBAECUDqhAA44FBIeqbUcChCCgqLVW6iNRNEtu9TzK9TQJKKek
>>>    YbJfEC5mxfSLI+EAui0g2ksmkuvv4yZ4MXtyTK0ylasUU6HtSicizFsWuJzVp4qy
>>>    Se5WYPQSqQIVN2rLQhCWY9mRAZYZ8dZTpxrNCanQkqmQLf1pTnFswbnyYb78VOFZ
>>>    a4A91TjjsgtCbA8GlRC9Adyl0xQUN3wy0YqZOkCfXeMUeQ1n83BqMX1oSUJiztTZ
>>>    xuuaRnukGWyu484wtPbDHJaDgQe+jsEw3sYGz6ZSMA9s8HgT4FenfB5HvZQILR2z
>>>    ymQpf0rIH6iCcus1MSmVeTUp8q82mbnq8nCzx7ZpgM0VHy8OJN8t6LA+ooTYVYUu
>>>    cJgNk6J8ZzME1v6IyBjRiQuA0bNaBGFykW8ABSvbAQX6iy02V7FEgzG5fYXacefv
>>>    sm0VsZ2IBgJXQp7xoO0N8ZoNFcgT79Zt3UNahheZ+Cu2FMDj4ydA1Q4dFyK6P0/h
>>>    6mnAS3ClCyZAvQ5gAWvOCdYreX75vrheQ4PDM5reh3QoTphIYlKdwYKOBoXrHxvl
>>>    /IaCZPU8tZQHJr86lnyvwE5Dt6fbkFe8kny37cDQrgx1sVWLg5aam+c3ukxyEIcp
>>>    kVv/xHS4KepLKr6xhubY5Nw5yn2Y0iF6B0YwNUYdAdEEdutJraMKVEOniOKf
>>>    -----END CERTIFICATE-----
>>>  client_key: |
>>>    -----BEGIN PRIVATE KEY-----
>>>    MIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQDbGMi2uWTROco9
>>>    3OF8E/nA7K/tr04ZkMl0Ss6Fk2C7edy5+C8ro2c+STwoRJHBMlL92f68QYyvJ0ON
>>>    Mc3nHFl40Lpmv/i1MH9Rjy8mMPgTmckM7H5c+dopR41hkLoP/v/Nl+azpOnDMtuQ
>>>    5MiRiwxd/M+s2FKepmnBoIXwSTRTbZYNcWidjeviidmXQlXqQa6qMlnc/NDZCKDR
>>>    ZzpUPCH83js3xPUmb53+WxjVGAKPLqSR3AHRBdGOMav44Q+C+htY0n9ulqhF+mTT
>>>    YREUh/2jHtrypt6VOOtudzf4/WdRYubs5gOEy5pAQzxY/HXC5MmmoRO5sIjJlqI+
>>>    90cMFkNzVFCe0pyCmR0Uj45QBOD45NDTlhZgvXD4O92ew+Jgob7+kC/XbIuCv9el
>>>    mIKUBJOrmuNMKELTQoaVGBYoVCIbq2UjGlaiqXpERPp0PkyEpFC5NWWFPk7Kzo+3
>>>    e568ifaZJRyLu1dJaarysNv4hjMe/4CI3WwDmeD+1QpchTe+9UObm95cUdEd3WO6
>>>    L+kakBfoapPL0BglC4SKxtmEMpdCkWsCBv/5x6CT1wg6EkFGGdyX03AeZ6CtH5ZM
>>>    M9t38qhlX3fbrUZI57NT+ar/kCjPu7MyTAyrXxIzUcBv+ybB0QqnpN/xZn1LUQ+y
>>>    DV/fzISuLhsSFPaQ920+M6maJuWFKQIDAQABAoICACSTduCAQzslg2R/9RmmaQ2i
>>>    xbzvhOGCOQtjuKX2hFHP5oAaVV+a3A+7IKKVTeLbkJzF0X+ijsanzqBp5daU9j/T
>>>    0qZ4GmA/LCYUufmNd8Cn7fctHOxPHrNA+XKF+3XTq2cqWm53k7nNSffepbgcZzBi
>>>    ldXkWCiZnfKf5nRGeEiMNR5h5iXVTUGIo01b2uVsWSkbFkLs5r2wsc47hjKNRlxF
>>>    SkwFYqqrvAu8MAFVDIpULk+WU6xOJR1wn+XAxrrmH34cx/7Rgl6KTl42aGhNypM6
>>>    gbE8F9EAC1ep6et0fFTLJsN6I7C9366mOtMdNFiYoqxhXB5mAPPsZIIP/GmhZtlx
>>>    AQ/exuQBMxipX0h2kATL33npQidePdOIjyFwRxZcB9ilYPICPDKWipvlZD4X4hQ2
>>>    nigarUIVRJGMJfiwS7NeZVBZGy+jxO+n47ZczFiKAPSVPBQYrnKVn1jrhji7EtEI
>>>    8Luj+1WqNtkITGjjPGCFDIyNhSISCF0fyLBHVkLLQZxBpJeZwBvg9IInPrQcv9ln
>>>    /Z/GrEaf9SENnu5XguzsjF+XEOsrqsE1HnGg0c/0AcjPh4Z5xBiGViNMr/PEEj6c
>>>    9YwSR9I4hRvwwSt0buY4h2oS8TquUJcO4sQQR6kifiOw1P8HvfvBfTRePu7BGyUy
>>>    30Rrc7Do8jHBrSZpegjtAoIBAQDdDZ0yQHm0cbSV2/oo5G12YaHJc7X8Xo0vB0cQ
>>>    eB5+YTTV6/shkezL5AuIsk0JtuKvex+fQg+7X1I1smFgejzCj2MNiBUdT1QaOU5v
>>>    60MJ3IZh02ktyPvo4WNqCvEsdt8JWuiiLl4HQJwa6lUSle/vb1XUs7ix5NobjzF2
>>>    QRzgZhK537xjaVPOipWyVnzd0rpK4AdYmjz2MUTpZ5u7GFOKEQW9JqtKaU/Exgh1
>>>    LWAr30+XZZlwvMFkP7FK7L+kCKJQmzHpHItp3XZ6y9qG2Y1XOcLDKTiB0KJzuPou
>>>    7pIsZ+iiANmwChQ4Ei9P7b+vfzZ7rSrYh+EWQuA1z+cXdlO9AoIBAQD9u/4aUceM
>>>    0oA5t4ik44Fj1wyUAMebeLAORz9cx9IpHh2nWFTK0im443PICN4WELjUC9rUoYlU
>>>    KfZZBuHNLFhG0VJ2u3QqsUusBKcu+5AuDazCQHsspx8uu8adTLn0m2Y6w+MouHkp
>>>    iF7gjV1VdWBmmZybj3EtRSxU3u7mTqCAboTbrjWNb6F1q72VKvl6pvu9BGkv5XBq
>>>    pJbWIJqmUnvBXV0GtjVx0Qul7jK7BhAoLTIFkpbDmTskloWBDbxxOwgk2+DqUrYe
>>>    adQFtFZsQrJK+m4+y4hC7Fm/CSbiY/GcAcIkN1ajWbppgCA8/cNuEjf21h1RMjjV
>>>    zA8/ACW8ElfdAoIBAEvOnmkcWWB7Wg3Giay6YZumccBPUUrB9jL5kPJh7zsyBNeT
>>>    /OSPUaC+jQIM+4RoVqcNV1PbaN++GchWazSV1v52A+chEvShPttbab3R9YSNPvyP
>>>    btOHyHQdz2UPaZPxFB32Mi9kLVTZYG+Oiyw7hTQcf1fZZm6DaBTpb3H5dHi0KJnb
>>>    05RYC6iaOCoXpOzBlqeU2pAVHPDlBpfwsyRffE6t9cTfXQDAyt1VeJMOw5toErh9
>>>    3DggU0YY3akvNa2AVIHGJ0ClTF4616XXARMbym/uXA9ntHw4W23KMuojYLefztS3
>>>    6++GFVcBTX5gr0r1Gm8hfL8TIirrNk7eBIc0ShUCggEBANWmxoBC/FSHq5DuChFN
>>>    wFXhbqXxQiKXrSdYtvHZEA2vXijbKArgjGEypPt6/AZ4bimfEdSZcJl9/AFa2id5
>>>    f5uiBl4T2e1Y/O8Lu/rDuNQOj7nBsfYXgxpG4ybwxGRDtgElkNFpBB0Ex18tqbmb
>>>    7GB4WHZkglgsNvgAAAnNiziZ1yVju5QsuzfqNOi2FvxAPxkZ7bh26gHiNWB4c8A5
>>>    FBvuEbblFnuQGs7Nk8oHgPNTmxs2I9zv2e0lEaVpe5OcsW7X9rtYiiFx2yYyg9MF
>>>    FjwsQ0h8+W/gtHE3s5ZNsdQ7Zs44CqvhkiXjzUGOLFEy5gzini0QrOrZuLzD0wJa
>>>    L8ECggEADI9EdCfI6CU5ZESI1TcoMIPtkd+pmDb5CzbKXZCp46jduIgobRNreAez
>>>    6Yrc+ASx+7CLtXO4gflVvwzZT6bb1aAWoXRLJTvKKUUGu87ZfUSNppOkgj9aecAl
>>>    TGB/ZfSI2NLTNoHmNf11s9HrBH+TFLDT4mCYV5fPEdbMnty1l0ZCL9INIXRjmlqy
>>>    fp1PgCHweo1aV7gHbN96trcbKiaIAFbSYtP6N8UAXmeAmzJKDLsHH4oZmQki9UDn
>>>    2tA+3vRyVNGzSqB9oI3p7q8nugvGx7pXa/5/BYNlR2CQipFhB8SOjwwxqRXpxBr1
>>>    Gkk+eWXzBVzBF3Grh+jw/l/t9lAfVg==
>>>    -----END PRIVATE KEY-----
>>>  server_cert: |
>>>    -----BEGIN CERTIFICATE-----
>>>    MIIFLDCCAxSgAwIBAgIUQYBb/jgCHtcwpuy1bxKL3mGTEm8wDQYJKoZIhvcNAQEL
>>>    BQAwFDESMBAGA1UEAwwJbXkuc2VydmVyMB4XDTI0MDcyMzA3NDIwN1oXDTM0MDcy
>>>    MTA3NDIwN1owFDESMBAGA1UEAwwJbXkuc2VydmVyMIICIjANBgkqhkiG9w0BAQEF
>>>    AAOCAg8AMIICCgKCAgEAqukwSwpTUBgSKAaJiFs7t892HD0WGFU/KoOgV7o6Yvcv
>>>    H7zifYais19vS7YZVb1X6CqHlHk8UVRHZmOAvVHLfM2NEhz/qwErFF08ZYN4GWnj
>>>    9Ncs2mzSbdvwo6sgvNElrhVz1z1epk21GXYkpjZMLeThwJ8STM08gyheIJEG3QZ8
>>>    MNKCc4jot/Q5V5kimPF+P8nM/jXKPgCUw6Z/SMe8/cQ96Qo9tyMop+NIAIqomaJI
>>>    mW4cdN+d/zlmZyRZ+iY8u6MUL/7lJeVMx9Z89co+rSBqigKiXasC4qZ4pPSsqJ+p
>>>    f/qaLksz94TqDbMiGA+w8A7l9zqxfm2UU3HIZcPomQ8a/veJCZRcI7ZNqII8KT6K
>>>    4uSIV6iJ5c5WshDwr9G2xijYIUKeTV2C6CEst91P1XMH3W3LFzZItB/V5/RfOYH4
>>>    n0XzXzTTYsdA8AqzC40C0eQu9k3La2obXqjRceLZIaH/mJwNf04UeTy8Avd2qwNJ
>>>    ifWrG1rrqjuzp6Hw6zFT9qkEds86YpQC7EA0C6ECmC03joXkVc5Lv7AF4rQjDmc3
>>>    i3qGEprFkRDktySdAMedEHj9dgTEDk+OXwUdHWbGLzDxpXEA4wyD63/sCm0r4M/k
>>>    Fcwln6hL0Yubn2fAii9oxHHbVSFu4WYoMnV7LVyjS+QwNhISfXPFnbvWmelV7yUC
>>>    AwEAAaN2MHQwHQYDVR0OBBYEFIYZ3n3dJEH5Yz1Qyzj+aZauxfzhMB8GA1UdIwQY
>>>    MBaAFIYZ3n3dJEH5Yz1Qyzj+aZauxfzhMA8GA1UdEwEB/wQFMAMBAf8wIQYDVR0R
>>>    BBowGIcECgBB74cECgBBB4cECgBATYcECgBCozANBgkqhkiG9w0BAQsFAAOCAgEA
>>>    ZxrsB7ruClrci+Dy1mEP0ZqBgs/3Nir66tS16zQSspmHnGXHpdxfPyhl4kLQNHrp
>>>    4rk3Of6wkik1RZGsLRQ5ctnpbxylMX4BWU0RGzMnKTh76rDxhoGDCofUnL448i0H
>>>    7tzfU56VVJLs9lhcKVwSsoRQQR6FOwx9xhM7fq6Y/qRdWMQbPfuCOd2kakP6WZay
>>>    74DClvz53yXIuDcZXGRlqnDfZnEs/JTxAbWLuLAp8JaZC6nNDUUct3mxdIOWKQ05
>>>    mVRBwPrEeY/c8XRr/2172TGz/j/cU4YTpnu8tQ8Kcm86nmURRLknU5Ea+o4p6Gnb
>>>    4SQMuHoZFv2/u8dmA3sV/jEd308exTdq6rYO9yC7L0tsdY7dGtBqcQZdmiPP8SSb
>>>    axI8I7INZIKi4fX9xt824G4EBj27HwwvAMKqEsIC8uSettP/ptjafqZxkUWcrpjC
>>>    MVJjaYk0YveGlaJMx6evL8sLicPamPPwahe2TcyF8d/dd8h87LMIZV0pD17EchJi
>>>    /qqIX3VQNToHotjRscyj3cryHgpHO1+WpFUjLwRBkjRRs1aD5ynjaLyRGNR8oYHB
>>>    nK89OKDDcTO9ODCcW7kf+YcV1RXFShRFpLBjkS/6oVKNO/UYeXmzeKiLir1mfHrs
>>>    B91RCIFCX5CUDyiYqeIRsYrvrrdTTr/gzDFtRbye5Jo=
>>>    -----END CERTIFICATE-----
>>>  server_key: |
>>>    -----BEGIN PRIVATE KEY-----
>>>    MIIJQQIBADANBgkqhkiG9w0BAQEFAASCCSswggknAgEAAoICAQCq6TBLClNQGBIo
>>>    BomIWzu3z3YcPRYYVT8qg6BXujpi9y8fvOJ9hqKzX29LthlVvVfoKoeUeTxRVEdm
>>>    Y4C9Uct8zY0SHP+rASsUXTxlg3gZaeP01yzabNJt2/CjqyC80SWuFXPXPV6mTbUZ
>>>    diSmNkwt5OHAnxJMzTyDKF4gkQbdBnww0oJziOi39DlXmSKY8X4/ycz+Nco+AJTD
>>>    pn9Ix7z9xD3pCj23Iyin40gAiqiZokiZbhx0353/OWZnJFn6Jjy7oxQv/uUl5UzH
>>>    1nz1yj6tIGqKAqJdqwLipnik9Kyon6l/+pouSzP3hOoNsyIYD7DwDuX3OrF+bZRT
>>>    cchlw+iZDxr+94kJlFwjtk2ogjwpPori5IhXqInlzlayEPCv0bbGKNghQp5NXYLo
>>>    ISy33U/VcwfdbcsXNki0H9Xn9F85gfifRfNfNNNix0DwCrMLjQLR5C72Tctrahte
>>>    qNFx4tkhof+YnA1/ThR5PLwC93arA0mJ9asbWuuqO7OnofDrMVP2qQR2zzpilALs
>>>    QDQLoQKYLTeOheRVzku/sAXitCMOZzeLeoYSmsWREOS3JJ0Ax50QeP12BMQOT45f
>>>    BR0dZsYvMPGlcQDjDIPrf+wKbSvgz+QVzCWfqEvRi5ufZ8CKL2jEcdtVIW7hZigy
>>>    dXstXKNL5DA2EhJ9c8Wdu9aZ6VXvJQIDAQABAoICAAYXCZK+8zGJLgpxkzCB6bqW
>>>    wKNodNNYpEfQSWQe3Wo92xbDLQHijIUeOzHEMxpoRQvfihW7DTBrs6CdHo8AbWMM
>>>    0wBIer6966pN2TNX0C19pbMJgM5mrsfd3LEETcYUoMZXetl6VRdRcQjPeiZXDHdd
>>>    LsMll+oXLRcSVupLqKYzDzVRJvdVc09suydZLE+tWT2OJAkR4xs6+5IOKn5uKZiU
>>>    R814QTM3pED5Ow1T44aX7wTSlN6AQlnAEptNwwwEFUltWfCM4V7skQQU6z553qpe
>>>    G5W/2sfCKnfnjSh5lHZsvAY0qI5KPTeOa6Wh0q7vTjN7oGujEoSKBI1Fommn5a+R
>>>    fQ9OnYODwdT5b//oy8fTf2YSBSpJPnyAafH4swHtsZFXQvl4L1tvkqVEzJrj7Vwr
>>>    q2KJ8f4iYjtdep0FQBaqPm1r60X0dmLgSngaE1gg92JyCJazrhVtX3FXu9O/sTln
>>>    RKWKVjOMGE/vILEZsnBqgRgn3e0EdFGbEqMuvGO44ZtqiwZQd2rKtYpl6HC/8zi/
>>>    uaBVB4YuaX6d9+qPiXkRFo0BqPbEgJEWrd1SDPZ7WFN+QKwAv2MeKG5wZl5GqRsu
>>>    eMjvvSPOVJZkDXGvicdKwkuVwNY1J1ryEV0w1zqK6ZKzKMmeOi34mG33a5IzfnK5
>>>    GEkW230m/WptINNbmfSRAoIBAQC292AsxNYBw4IhtV9PmG2YmZZu6rXlqVC1Pu/a
>>>    5tmzYW8NbIExQ7Gz7FW6O/oW/IObRiF9W6Ug67jYwpOHxmybQ/u5peElj2QJLxzW
>>>    w11UH9OSkrvBjwZitLOjeW2Dzi1wjOLw4BxqH5k1cMVcyxCSM0BgyqvFd/YXO/4O
>>>    PSuJwP2BsmRe5IAxqUexntcFWl8mnLN+Y0fTO4CGnpPMI7GgV0uyfFwAIoDDra7f
>>>    DzKZen+TxbstlL3Yk/wSJxSsCZQO+Cp3xFYV607krvDNmhEeBhfJKh9NSYFJZe3Y
>>>    +j9/6Oqs5cZRjIfZWQVhZpdTISBKtLi7cvGVLIilYsYqjRSRAoIBAQDvIeqF8SkZ
>>>    6q/bXDODvZ5LERwRuOUNlW96/jqQCs8bnt3vrFVCWPpUpdq7lYpigzb7S92AuYr7
>>>    UJEVXP9Q4J0Rjv4BQBrL5DqF53eNVznEaNOvFq46Co8FDvHtrHwGI/cSVjNv2BV9
>>>    nD2gc9vb5HVeF/EQcRIBVxXNWCUVIhAIxlO200Cedj/9S4RVRF4j5VEDDkib7HaY
>>>    vRfynXyf6ne12Atbn3DG8X1n1NYgvnKVM+6IgyaVJfTJWvRwXWr6HMqJYZslTA67
>>>    9Xd0skopUPy4BiG+a8nOBn7phGcguH3rXnUw6b4rCBDy2lnZWFkUNTe8V4pCJG/x
>>>    8G9Gtt0nTetVAoIBAEXZ00HEM9+ggEHWOHjuvIiRujjGKoNvTHyldnSfxWUSQY9i
>>>    aUAhk+oYw/HBjAzSgnv5Hjf5kOUsmPlH74LaYgQ0xBlwFrB8Obi3386t7yd6rw86
>>>    LawLqmhsjFhQt1qmj/p0Dv/ez3pEbf1U/EQc0kq+fkl1lnZhAfaE1M0i1UhWAXJO
>>>    ZLtFiRtsgQ5ykaxdgA4BrRUHWhhukvbR+6LnzZxlN2UAzS7P/VC3Mhe+eZbWiXYk
>>>    +vy/2GqitI1Jlj7HuVqtq6jlr+fWWdw8XXpg40APG22VQzYCZfVbkJ5Ri4SNq2ou
>>>    nrPrbsGvlrpErGj0MJvUuuH1U/2bbt+VAgqkqcECggEAXYErjPhBlAq6fNHOquLI
>>>    NJ4aFavwlKEsGDG0gIJZfXlfWspOBEV6jQLVRqgwllisdRKzKBRlDePw9yOICf3q
>>>    LOV22NJSKAuz/owFqVTEF9LNgxchRTy/s8CmRMM/M794askeD8u0i4s+ZbEn9RGd
>>>    +w7tifpMig2dTs/6oArTrjqunTx5dfzTBJ40voqfLPYL+V/qcF3D+fFmnc2K35T0
>>>    /jveJRNxK7ldye3lVKRjdYf665Is0gEneFyCXDWLEQWVBKcP3UOiN0jCTvUwHnu2
>>>    ddaf9y96l37evTyyfo7lXdVviZK8c2MSDYPnwh7Tph0xEApC254Xz8iuGj6Ct022
>>>    iQKCAQAs24asqU95uUlrX40+zAcnfYlcErfK3Q6UmhpNtDnvOcJlwn9yHwGP7xi+
>>>    VSjgpj2u5aMO4Q0T9QEu8WMs2RX+hpLRcjXsSWAoHKhv9v8hK0p6OEvodle02ze3
>>>    W1A0KUuWghpoTRY8hpzsE6ezbXkyCwd7TKPwj4y4RVfx8Uu/exCXzpg4AmkORIw0
>>>    N6VX9uKByKUCH7/dhlHKQSaZ5ZbaZBEHISiJf8PROzWzl8xapa332Q168f0lMZfN
>>>    BYeiQ50C8ZxKN3AAp++nuauh8NZ+lEu/vWs8Q8OwwHkP3cwM88n0QrhZBc0J/ZER
>>>    w6NRINExhgYasqGy0v1XsbYEOCB7
>>>    -----END PRIVATE KEY-----



But NVMe service specification content from export
---------------------------------------------------

>>>[root@ceph-sunilkumar-00-gmqual-node1-installer ~]# ceph orch ls --service-type nvmeof --service-name nvmeof.rbd --export > new-gw.yaml
>>>[root@ceph-sunilkumar-00-gmqual-node1-installer ~]# cat new-gw.yaml
>>>service_type: nvmeof
>>>service_id: rbd
>>>service_name: nvmeof.rbd
>>>placement:
>>>  hosts:
>>>  - ceph-sunilkumar-00-gmqual-node6
>>>  - ceph-sunilkumar-00-gmqual-node7
>>>  - ceph-sunilkumar-00-gmqual-node8
>>>  - ceph-sunilkumar-00-gmqual-node9
>>>spec:
>>>  allowed_consecutive_spdk_ping_failures: 1
>>>  bdevs_per_cluster: 32
>>>  client_cert: '-----BEGIN CERTIFICATE-----
>>>    MIIFCTCCAvGgAwIBAgIUO6yXXkNb1+1tJzxZDplvgKpwWkMwDQYJKoZIhvcNAQEL
>>>    BQAwFDESMBAGA1UEAwwJbXkuY2xpZW50MB4XDTI0MDcyMzA3NDI1N1oXDTM0MDcy
>>>    MTA3NDI1N1owFDESMBAGA1UEAwwJbXkuY2xpZW50MIICIjANBgkqhkiG9w0BAQEF
>>>    AAOCAg8AMIICCgKCAgEA2xjItrlk0TnKPdzhfBP5wOyv7a9OGZDJdErOhZNgu3nc
>>>    ufgvK6NnPkk8KESRwTJS/dn+vEGMrydDjTHN5xxZeNC6Zr/4tTB/UY8vJjD4E5nJ
>>>    DOx+XPnaKUeNYZC6D/7/zZfms6TpwzLbkOTIkYsMXfzPrNhSnqZpwaCF8Ek0U22W
>>>    DXFonY3r4onZl0JV6kGuqjJZ3PzQ2Qig0Wc6VDwh/N47N8T1Jm+d/lsY1RgCjy6k
>>>    kdwB0QXRjjGr+OEPgvobWNJ/bpaoRfpk02ERFIf9ox7a8qbelTjrbnc3+P1nUWLm
>>>    7OYDhMuaQEM8WPx1wuTJpqETubCIyZaiPvdHDBZDc1RQntKcgpkdFI+OUATg+OTQ
>>>    05YWYL1w+DvdnsPiYKG+/pAv12yLgr/XpZiClASTq5rjTChC00KGlRgWKFQiG6tl
>>>    IxpWoql6RET6dD5MhKRQuTVlhT5Oys6Pt3uevIn2mSUci7tXSWmq8rDb+IYzHv+A
>>>    iN1sA5ng/tUKXIU3vvVDm5veXFHRHd1jui/pGpAX6GqTy9AYJQuEisbZhDKXQpFr
>>>    Agb/+cegk9cIOhJBRhncl9NwHmegrR+WTDPbd/KoZV93261GSOezU/mq/5Aoz7uz
>>>    MkwMq18SM1HAb/smwdEKp6Tf8WZ9S1EPsg1f38yEri4bEhT2kPdtPjOpmiblhSkC
>>>    AwEAAaNTMFEwHQYDVR0OBBYEFNv3bpEtWpqgCj8nj3dgSVxUrXtVMB8GA1UdIwQY
>>>    MBaAFNv3bpEtWpqgCj8nj3dgSVxUrXtVMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI
>>>    hvcNAQELBQADggIBAECUDqhAA44FBIeqbUcChCCgqLVW6iNRNEtu9TzK9TQJKKek
>>>    YbJfEC5mxfSLI+EAui0g2ksmkuvv4yZ4MXtyTK0ylasUU6HtSicizFsWuJzVp4qy
>>>    Se5WYPQSqQIVN2rLQhCWY9mRAZYZ8dZTpxrNCanQkqmQLf1pTnFswbnyYb78VOFZ
>>>    a4A91TjjsgtCbA8GlRC9Adyl0xQUN3wy0YqZOkCfXeMUeQ1n83BqMX1oSUJiztTZ
>>>    xuuaRnukGWyu484wtPbDHJaDgQe+jsEw3sYGz6ZSMA9s8HgT4FenfB5HvZQILR2z
>>>    ymQpf0rIH6iCcus1MSmVeTUp8q82mbnq8nCzx7ZpgM0VHy8OJN8t6LA+ooTYVYUu
>>>    cJgNk6J8ZzME1v6IyBjRiQuA0bNaBGFykW8ABSvbAQX6iy02V7FEgzG5fYXacefv
>>>    sm0VsZ2IBgJXQp7xoO0N8ZoNFcgT79Zt3UNahheZ+Cu2FMDj4ydA1Q4dFyK6P0/h
>>>    6mnAS3ClCyZAvQ5gAWvOCdYreX75vrheQ4PDM5reh3QoTphIYlKdwYKOBoXrHxvl
>>>    /IaCZPU8tZQHJr86lnyvwE5Dt6fbkFe8kny37cDQrgx1sVWLg5aam+c3ukxyEIcp
>>>    kVv/xHS4KepLKr6xhubY5Nw5yn2Y0iF6B0YwNUYdAdEEdutJraMKVEOniOKf
>>>    -----END CERTIFICATE-----
>>>    '
>>>  client_key: '-----BEGIN PRIVATE KEY-----
>>>    MIIJQgIBADANBgkqhkiG9w0BAQEFAASCCSwwggkoAgEAAoICAQDbGMi2uWTROco9
>>>    3OF8E/nA7K/tr04ZkMl0Ss6Fk2C7edy5+C8ro2c+STwoRJHBMlL92f68QYyvJ0ON
>>>    Mc3nHFl40Lpmv/i1MH9Rjy8mMPgTmckM7H5c+dopR41hkLoP/v/Nl+azpOnDMtuQ
>>>    5MiRiwxd/M+s2FKepmnBoIXwSTRTbZYNcWidjeviidmXQlXqQa6qMlnc/NDZCKDR
>>>    ZzpUPCH83js3xPUmb53+WxjVGAKPLqSR3AHRBdGOMav44Q+C+htY0n9ulqhF+mTT
>>>    YREUh/2jHtrypt6VOOtudzf4/WdRYubs5gOEy5pAQzxY/HXC5MmmoRO5sIjJlqI+
>>>    90cMFkNzVFCe0pyCmR0Uj45QBOD45NDTlhZgvXD4O92ew+Jgob7+kC/XbIuCv9el
>>>    mIKUBJOrmuNMKELTQoaVGBYoVCIbq2UjGlaiqXpERPp0PkyEpFC5NWWFPk7Kzo+3
>>>    e568ifaZJRyLu1dJaarysNv4hjMe/4CI3WwDmeD+1QpchTe+9UObm95cUdEd3WO6
>>>    L+kakBfoapPL0BglC4SKxtmEMpdCkWsCBv/5x6CT1wg6EkFGGdyX03AeZ6CtH5ZM
>>>    M9t38qhlX3fbrUZI57NT+ar/kCjPu7MyTAyrXxIzUcBv+ybB0QqnpN/xZn1LUQ+y
>>>    DV/fzISuLhsSFPaQ920+M6maJuWFKQIDAQABAoICACSTduCAQzslg2R/9RmmaQ2i
>>>    xbzvhOGCOQtjuKX2hFHP5oAaVV+a3A+7IKKVTeLbkJzF0X+ijsanzqBp5daU9j/T
>>>    0qZ4GmA/LCYUufmNd8Cn7fctHOxPHrNA+XKF+3XTq2cqWm53k7nNSffepbgcZzBi
>>>    ldXkWCiZnfKf5nRGeEiMNR5h5iXVTUGIo01b2uVsWSkbFkLs5r2wsc47hjKNRlxF
>>>    SkwFYqqrvAu8MAFVDIpULk+WU6xOJR1wn+XAxrrmH34cx/7Rgl6KTl42aGhNypM6
>>>    gbE8F9EAC1ep6et0fFTLJsN6I7C9366mOtMdNFiYoqxhXB5mAPPsZIIP/GmhZtlx
>>>    AQ/exuQBMxipX0h2kATL33npQidePdOIjyFwRxZcB9ilYPICPDKWipvlZD4X4hQ2
>>>    nigarUIVRJGMJfiwS7NeZVBZGy+jxO+n47ZczFiKAPSVPBQYrnKVn1jrhji7EtEI
>>>    8Luj+1WqNtkITGjjPGCFDIyNhSISCF0fyLBHVkLLQZxBpJeZwBvg9IInPrQcv9ln
>>>    /Z/GrEaf9SENnu5XguzsjF+XEOsrqsE1HnGg0c/0AcjPh4Z5xBiGViNMr/PEEj6c
>>>    9YwSR9I4hRvwwSt0buY4h2oS8TquUJcO4sQQR6kifiOw1P8HvfvBfTRePu7BGyUy
>>>    30Rrc7Do8jHBrSZpegjtAoIBAQDdDZ0yQHm0cbSV2/oo5G12YaHJc7X8Xo0vB0cQ
>>>    eB5+YTTV6/shkezL5AuIsk0JtuKvex+fQg+7X1I1smFgejzCj2MNiBUdT1QaOU5v
>>>    60MJ3IZh02ktyPvo4WNqCvEsdt8JWuiiLl4HQJwa6lUSle/vb1XUs7ix5NobjzF2
>>>    QRzgZhK537xjaVPOipWyVnzd0rpK4AdYmjz2MUTpZ5u7GFOKEQW9JqtKaU/Exgh1
>>>    LWAr30+XZZlwvMFkP7FK7L+kCKJQmzHpHItp3XZ6y9qG2Y1XOcLDKTiB0KJzuPou
>>>    7pIsZ+iiANmwChQ4Ei9P7b+vfzZ7rSrYh+EWQuA1z+cXdlO9AoIBAQD9u/4aUceM
>>>    0oA5t4ik44Fj1wyUAMebeLAORz9cx9IpHh2nWFTK0im443PICN4WELjUC9rUoYlU
>>>    KfZZBuHNLFhG0VJ2u3QqsUusBKcu+5AuDazCQHsspx8uu8adTLn0m2Y6w+MouHkp
>>>    iF7gjV1VdWBmmZybj3EtRSxU3u7mTqCAboTbrjWNb6F1q72VKvl6pvu9BGkv5XBq
>>>    pJbWIJqmUnvBXV0GtjVx0Qul7jK7BhAoLTIFkpbDmTskloWBDbxxOwgk2+DqUrYe
>>>    adQFtFZsQrJK+m4+y4hC7Fm/CSbiY/GcAcIkN1ajWbppgCA8/cNuEjf21h1RMjjV
>>>    zA8/ACW8ElfdAoIBAEvOnmkcWWB7Wg3Giay6YZumccBPUUrB9jL5kPJh7zsyBNeT
>>>    /OSPUaC+jQIM+4RoVqcNV1PbaN++GchWazSV1v52A+chEvShPttbab3R9YSNPvyP
>>>    btOHyHQdz2UPaZPxFB32Mi9kLVTZYG+Oiyw7hTQcf1fZZm6DaBTpb3H5dHi0KJnb
>>>    05RYC6iaOCoXpOzBlqeU2pAVHPDlBpfwsyRffE6t9cTfXQDAyt1VeJMOw5toErh9
>>>    3DggU0YY3akvNa2AVIHGJ0ClTF4616XXARMbym/uXA9ntHw4W23KMuojYLefztS3
>>>    6++GFVcBTX5gr0r1Gm8hfL8TIirrNk7eBIc0ShUCggEBANWmxoBC/FSHq5DuChFN
>>>    wFXhbqXxQiKXrSdYtvHZEA2vXijbKArgjGEypPt6/AZ4bimfEdSZcJl9/AFa2id5
>>>    f5uiBl4T2e1Y/O8Lu/rDuNQOj7nBsfYXgxpG4ybwxGRDtgElkNFpBB0Ex18tqbmb
>>>    7GB4WHZkglgsNvgAAAnNiziZ1yVju5QsuzfqNOi2FvxAPxkZ7bh26gHiNWB4c8A5
>>>    FBvuEbblFnuQGs7Nk8oHgPNTmxs2I9zv2e0lEaVpe5OcsW7X9rtYiiFx2yYyg9MF
>>>    FjwsQ0h8+W/gtHE3s5ZNsdQ7Zs44CqvhkiXjzUGOLFEy5gzini0QrOrZuLzD0wJa
>>>    L8ECggEADI9EdCfI6CU5ZESI1TcoMIPtkd+pmDb5CzbKXZCp46jduIgobRNreAez
>>>    6Yrc+ASx+7CLtXO4gflVvwzZT6bb1aAWoXRLJTvKKUUGu87ZfUSNppOkgj9aecAl
>>>    TGB/ZfSI2NLTNoHmNf11s9HrBH+TFLDT4mCYV5fPEdbMnty1l0ZCL9INIXRjmlqy
>>>    fp1PgCHweo1aV7gHbN96trcbKiaIAFbSYtP6N8UAXmeAmzJKDLsHH4oZmQki9UDn
>>>    2tA+3vRyVNGzSqB9oI3p7q8nugvGx7pXa/5/BYNlR2CQipFhB8SOjwwxqRXpxBr1
>>>    Gkk+eWXzBVzBF3Grh+jw/l/t9lAfVg==
>>>    -----END PRIVATE KEY-----
>>>    '
>>>  conn_retries: 10
>>>  discovery_port: 8009
>>>  enable_auth: true
>>>  enable_monitor_client: true
>>>  enable_prometheus_exporter: true
>>>  log_directory: /var/log/ceph/
>>>  log_files_enabled: true
>>>  log_files_rotation_enabled: true
>>>  log_level: INFO
>>>  max_log_directory_backups: 10
>>>  max_log_file_size_in_mb: 10
>>>  max_log_files_count: 20
>>>  monitor_timeout: 1.0
>>>  omap_file_lock_duration: 20
>>>  omap_file_lock_retries: 30
>>>  omap_file_lock_retry_sleep_interval: 1.0
>>>  omap_file_update_reloads: 10
>>>  pool: rbd
>>>  port: 5500
>>>  root_ca_cert: "-----BEGIN CERTIFICATE-----\nMIIFLDCCAxSgAwIBAgIUQYBb/jgCHtcwpuy1bxKL3mGTEm8wDQYJKoZIhvcNAQEL\n\
>>>    BQAwFDESMBAGA1UEAwwJbXkuc2VydmVyMB4XDTI0MDcyMzA3NDIwN1oXDTM0MDcy\nMTA3NDIwN1owFDESMBAGA1UEAwwJbXkuc2VydmVyMIICIjANBgkqhkiG9w0BAQEF\n\
>>>    AAOCAg8AMIICCgKCAgEAqukwSwpTUBgSKAaJiFs7t892HD0WGFU/KoOgV7o6Yvcv\nH7zifYais19vS7YZVb1X6CqHlHk8UVRHZmOAvVHLfM2NEhz/qwErFF08ZYN4GWnj\n\
>>>    9Ncs2mzSbdvwo6sgvNElrhVz1z1epk21GXYkpjZMLeThwJ8STM08gyheIJEG3QZ8\nMNKCc4jot/Q5V5kimPF+P8nM/jXKPgCUw6Z/SMe8/cQ96Qo9tyMop+NIAIqomaJI\n\
>>>    mW4cdN+d/zlmZyRZ+iY8u6MUL/7lJeVMx9Z89co+rSBqigKiXasC4qZ4pPSsqJ+p\nf/qaLksz94TqDbMiGA+w8A7l9zqxfm2UU3HIZcPomQ8a/veJCZRcI7ZNqII8KT6K\n\
>>>    4uSIV6iJ5c5WshDwr9G2xijYIUKeTV2C6CEst91P1XMH3W3LFzZItB/V5/RfOYH4\nn0XzXzTTYsdA8AqzC40C0eQu9k3La2obXqjRceLZIaH/mJwNf04UeTy8Avd2qwNJ\n\
>>>    ifWrG1rrqjuzp6Hw6zFT9qkEds86YpQC7EA0C6ECmC03joXkVc5Lv7AF4rQjDmc3\ni3qGEprFkRDktySdAMedEHj9dgTEDk+OXwUdHWbGLzDxpXEA4wyD63/sCm0r4M/k\n\
>>>    Fcwln6hL0Yubn2fAii9oxHHbVSFu4WYoMnV7LVyjS+QwNhISfXPFnbvWmelV7yUC\nAwEAAaN2MHQwHQYDVR0OBBYEFIYZ3n3dJEH5Yz1Qyzj+aZauxfzhMB8GA1UdIwQY\n\
>>>    MBaAFIYZ3n3dJEH5Yz1Qyzj+aZauxfzhMA8GA1UdEwEB/wQFMAMBAf8wIQYDVR0R\nBBowGIcECgBB74cECgBBB4cECgBATYcECgBCozANBgkqhkiG9w0BAQsFAAOCAgEA\n\
>>>    ZxrsB7ruClrci+Dy1mEP0ZqBgs/3Nir66tS16zQSspmHnGXHpdxfPyhl4kLQNHrp\n4rk3Of6wkik1RZGsLRQ5ctnpbxylMX4BWU0RGzMnKTh76rDxhoGDCofUnL448i0H\n\
>>>    7tzfU56VVJLs9lhcKVwSsoRQQR6FOwx9xhM7fq6Y/qRdWMQbPfuCOd2kakP6WZay\n74DClvz53yXIuDcZXGRlqnDfZnEs/JTxAbWLuLAp8JaZC6nNDUUct3mxdIOWKQ05\n\
>>>    mVRBwPrEeY/c8XRr/2172TGz/j/cU4YTpnu8tQ8Kcm86nmURRLknU5Ea+o4p6Gnb\n4SQMuHoZFv2/u8dmA3sV/jEd308exTdq6rYO9yC7L0tsdY7dGtBqcQZdmiPP8SSb\n\
>>>    axI8I7INZIKi4fX9xt824G4EBj27HwwvAMKqEsIC8uSettP/ptjafqZxkUWcrpjC\nMVJjaYk0YveGlaJMx6evL8sLicPamPPwahe2TcyF8d/dd8h87LMIZV0pD17EchJi\n\
>>>    /qqIX3VQNToHotjRscyj3cryHgpHO1+WpFUjLwRBkjRRs1aD5ynjaLyRGNR8oYHB\nnK89OKDDcTO9ODCcW7kf+YcV1RXFShRFpLBjkS/6oVKNO/UYeXmzeKiLir1mfHrs\n\
>>>    B91RCIFCX5CUDyiYqeIRsYrvrrdTTr/gzDFtRbye5Jo=\n-----END CERTIFICATE-----   \n"
>>>  rpc_socket_dir: /var/tmp/
>>>  rpc_socket_name: spdk.sock
>>>  server_cert: '-----BEGIN CERTIFICATE-----
>>>    MIIFLDCCAxSgAwIBAgIUQYBb/jgCHtcwpuy1bxKL3mGTEm8wDQYJKoZIhvcNAQEL
>>>    BQAwFDESMBAGA1UEAwwJbXkuc2VydmVyMB4XDTI0MDcyMzA3NDIwN1oXDTM0MDcy
>>>    MTA3NDIwN1owFDESMBAGA1UEAwwJbXkuc2VydmVyMIICIjANBgkqhkiG9w0BAQEF
>>>    AAOCAg8AMIICCgKCAgEAqukwSwpTUBgSKAaJiFs7t892HD0WGFU/KoOgV7o6Yvcv
>>>    H7zifYais19vS7YZVb1X6CqHlHk8UVRHZmOAvVHLfM2NEhz/qwErFF08ZYN4GWnj
>>>    9Ncs2mzSbdvwo6sgvNElrhVz1z1epk21GXYkpjZMLeThwJ8STM08gyheIJEG3QZ8
>>>    MNKCc4jot/Q5V5kimPF+P8nM/jXKPgCUw6Z/SMe8/cQ96Qo9tyMop+NIAIqomaJI
>>>    mW4cdN+d/zlmZyRZ+iY8u6MUL/7lJeVMx9Z89co+rSBqigKiXasC4qZ4pPSsqJ+p
>>>    f/qaLksz94TqDbMiGA+w8A7l9zqxfm2UU3HIZcPomQ8a/veJCZRcI7ZNqII8KT6K
>>>    4uSIV6iJ5c5WshDwr9G2xijYIUKeTV2C6CEst91P1XMH3W3LFzZItB/V5/RfOYH4
>>>    n0XzXzTTYsdA8AqzC40C0eQu9k3La2obXqjRceLZIaH/mJwNf04UeTy8Avd2qwNJ
>>>    ifWrG1rrqjuzp6Hw6zFT9qkEds86YpQC7EA0C6ECmC03joXkVc5Lv7AF4rQjDmc3
>>>    i3qGEprFkRDktySdAMedEHj9dgTEDk+OXwUdHWbGLzDxpXEA4wyD63/sCm0r4M/k
>>>    Fcwln6hL0Yubn2fAii9oxHHbVSFu4WYoMnV7LVyjS+QwNhISfXPFnbvWmelV7yUC
>>>    AwEAAaN2MHQwHQYDVR0OBBYEFIYZ3n3dJEH5Yz1Qyzj+aZauxfzhMB8GA1UdIwQY
>>>    MBaAFIYZ3n3dJEH5Yz1Qyzj+aZauxfzhMA8GA1UdEwEB/wQFMAMBAf8wIQYDVR0R
>>>    BBowGIcECgBB74cECgBBB4cECgBATYcECgBCozANBgkqhkiG9w0BAQsFAAOCAgEA
>>>    ZxrsB7ruClrci+Dy1mEP0ZqBgs/3Nir66tS16zQSspmHnGXHpdxfPyhl4kLQNHrp
>>>    4rk3Of6wkik1RZGsLRQ5ctnpbxylMX4BWU0RGzMnKTh76rDxhoGDCofUnL448i0H
>>>    7tzfU56VVJLs9lhcKVwSsoRQQR6FOwx9xhM7fq6Y/qRdWMQbPfuCOd2kakP6WZay
>>>    74DClvz53yXIuDcZXGRlqnDfZnEs/JTxAbWLuLAp8JaZC6nNDUUct3mxdIOWKQ05
>>>    mVRBwPrEeY/c8XRr/2172TGz/j/cU4YTpnu8tQ8Kcm86nmURRLknU5Ea+o4p6Gnb
>>>    4SQMuHoZFv2/u8dmA3sV/jEd308exTdq6rYO9yC7L0tsdY7dGtBqcQZdmiPP8SSb
>>>    axI8I7INZIKi4fX9xt824G4EBj27HwwvAMKqEsIC8uSettP/ptjafqZxkUWcrpjC
>>>    MVJjaYk0YveGlaJMx6evL8sLicPamPPwahe2TcyF8d/dd8h87LMIZV0pD17EchJi
>>>    /qqIX3VQNToHotjRscyj3cryHgpHO1+WpFUjLwRBkjRRs1aD5ynjaLyRGNR8oYHB
>>>    nK89OKDDcTO9ODCcW7kf+YcV1RXFShRFpLBjkS/6oVKNO/UYeXmzeKiLir1mfHrs
>>>    B91RCIFCX5CUDyiYqeIRsYrvrrdTTr/gzDFtRbye5Jo=
>>>    -----END CERTIFICATE-----
>>>    '
>>>  server_key: '-----BEGIN PRIVATE KEY-----
>>>    MIIJQQIBADANBgkqhkiG9w0BAQEFAASCCSswggknAgEAAoICAQCq6TBLClNQGBIo
>>>    BomIWzu3z3YcPRYYVT8qg6BXujpi9y8fvOJ9hqKzX29LthlVvVfoKoeUeTxRVEdm
>>>    Y4C9Uct8zY0SHP+rASsUXTxlg3gZaeP01yzabNJt2/CjqyC80SWuFXPXPV6mTbUZ
>>>    diSmNkwt5OHAnxJMzTyDKF4gkQbdBnww0oJziOi39DlXmSKY8X4/ycz+Nco+AJTD
>>>    pn9Ix7z9xD3pCj23Iyin40gAiqiZokiZbhx0353/OWZnJFn6Jjy7oxQv/uUl5UzH
>>>    1nz1yj6tIGqKAqJdqwLipnik9Kyon6l/+pouSzP3hOoNsyIYD7DwDuX3OrF+bZRT
>>>    cchlw+iZDxr+94kJlFwjtk2ogjwpPori5IhXqInlzlayEPCv0bbGKNghQp5NXYLo
>>>    ISy33U/VcwfdbcsXNki0H9Xn9F85gfifRfNfNNNix0DwCrMLjQLR5C72Tctrahte
>>>    qNFx4tkhof+YnA1/ThR5PLwC93arA0mJ9asbWuuqO7OnofDrMVP2qQR2zzpilALs
>>>    QDQLoQKYLTeOheRVzku/sAXitCMOZzeLeoYSmsWREOS3JJ0Ax50QeP12BMQOT45f
>>>    BR0dZsYvMPGlcQDjDIPrf+wKbSvgz+QVzCWfqEvRi5ufZ8CKL2jEcdtVIW7hZigy
>>>    dXstXKNL5DA2EhJ9c8Wdu9aZ6VXvJQIDAQABAoICAAYXCZK+8zGJLgpxkzCB6bqW
>>>    wKNodNNYpEfQSWQe3Wo92xbDLQHijIUeOzHEMxpoRQvfihW7DTBrs6CdHo8AbWMM
>>>    0wBIer6966pN2TNX0C19pbMJgM5mrsfd3LEETcYUoMZXetl6VRdRcQjPeiZXDHdd
>>>    LsMll+oXLRcSVupLqKYzDzVRJvdVc09suydZLE+tWT2OJAkR4xs6+5IOKn5uKZiU
>>>    R814QTM3pED5Ow1T44aX7wTSlN6AQlnAEptNwwwEFUltWfCM4V7skQQU6z553qpe
>>>    G5W/2sfCKnfnjSh5lHZsvAY0qI5KPTeOa6Wh0q7vTjN7oGujEoSKBI1Fommn5a+R
>>>    fQ9OnYODwdT5b//oy8fTf2YSBSpJPnyAafH4swHtsZFXQvl4L1tvkqVEzJrj7Vwr
>>>    q2KJ8f4iYjtdep0FQBaqPm1r60X0dmLgSngaE1gg92JyCJazrhVtX3FXu9O/sTln
>>>    RKWKVjOMGE/vILEZsnBqgRgn3e0EdFGbEqMuvGO44ZtqiwZQd2rKtYpl6HC/8zi/
>>>    uaBVB4YuaX6d9+qPiXkRFo0BqPbEgJEWrd1SDPZ7WFN+QKwAv2MeKG5wZl5GqRsu
>>>    eMjvvSPOVJZkDXGvicdKwkuVwNY1J1ryEV0w1zqK6ZKzKMmeOi34mG33a5IzfnK5
>>>    GEkW230m/WptINNbmfSRAoIBAQC292AsxNYBw4IhtV9PmG2YmZZu6rXlqVC1Pu/a
>>>    5tmzYW8NbIExQ7Gz7FW6O/oW/IObRiF9W6Ug67jYwpOHxmybQ/u5peElj2QJLxzW
>>>    w11UH9OSkrvBjwZitLOjeW2Dzi1wjOLw4BxqH5k1cMVcyxCSM0BgyqvFd/YXO/4O
>>>    PSuJwP2BsmRe5IAxqUexntcFWl8mnLN+Y0fTO4CGnpPMI7GgV0uyfFwAIoDDra7f
>>>    DzKZen+TxbstlL3Yk/wSJxSsCZQO+Cp3xFYV607krvDNmhEeBhfJKh9NSYFJZe3Y
>>>    +j9/6Oqs5cZRjIfZWQVhZpdTISBKtLi7cvGVLIilYsYqjRSRAoIBAQDvIeqF8SkZ
>>>    6q/bXDODvZ5LERwRuOUNlW96/jqQCs8bnt3vrFVCWPpUpdq7lYpigzb7S92AuYr7
>>>    UJEVXP9Q4J0Rjv4BQBrL5DqF53eNVznEaNOvFq46Co8FDvHtrHwGI/cSVjNv2BV9
>>>    nD2gc9vb5HVeF/EQcRIBVxXNWCUVIhAIxlO200Cedj/9S4RVRF4j5VEDDkib7HaY
>>>    vRfynXyf6ne12Atbn3DG8X1n1NYgvnKVM+6IgyaVJfTJWvRwXWr6HMqJYZslTA67
>>>    9Xd0skopUPy4BiG+a8nOBn7phGcguH3rXnUw6b4rCBDy2lnZWFkUNTe8V4pCJG/x
>>>    8G9Gtt0nTetVAoIBAEXZ00HEM9+ggEHWOHjuvIiRujjGKoNvTHyldnSfxWUSQY9i
>>>    aUAhk+oYw/HBjAzSgnv5Hjf5kOUsmPlH74LaYgQ0xBlwFrB8Obi3386t7yd6rw86
>>>    LawLqmhsjFhQt1qmj/p0Dv/ez3pEbf1U/EQc0kq+fkl1lnZhAfaE1M0i1UhWAXJO
>>>    ZLtFiRtsgQ5ykaxdgA4BrRUHWhhukvbR+6LnzZxlN2UAzS7P/VC3Mhe+eZbWiXYk
>>>    +vy/2GqitI1Jlj7HuVqtq6jlr+fWWdw8XXpg40APG22VQzYCZfVbkJ5Ri4SNq2ou
>>>    nrPrbsGvlrpErGj0MJvUuuH1U/2bbt+VAgqkqcECggEAXYErjPhBlAq6fNHOquLI
>>>    NJ4aFavwlKEsGDG0gIJZfXlfWspOBEV6jQLVRqgwllisdRKzKBRlDePw9yOICf3q
>>>    LOV22NJSKAuz/owFqVTEF9LNgxchRTy/s8CmRMM/M794askeD8u0i4s+ZbEn9RGd
>>>    +w7tifpMig2dTs/6oArTrjqunTx5dfzTBJ40voqfLPYL+V/qcF3D+fFmnc2K35T0
>>>    /jveJRNxK7ldye3lVKRjdYf665Is0gEneFyCXDWLEQWVBKcP3UOiN0jCTvUwHnu2
>>>    ddaf9y96l37evTyyfo7lXdVviZK8c2MSDYPnwh7Tph0xEApC254Xz8iuGj6Ct022
>>>    iQKCAQAs24asqU95uUlrX40+zAcnfYlcErfK3Q6UmhpNtDnvOcJlwn9yHwGP7xi+
>>>    VSjgpj2u5aMO4Q0T9QEu8WMs2RX+hpLRcjXsSWAoHKhv9v8hK0p6OEvodle02ze3
>>>    W1A0KUuWghpoTRY8hpzsE6ezbXkyCwd7TKPwj4y4RVfx8Uu/exCXzpg4AmkORIw0
>>>    N6VX9uKByKUCH7/dhlHKQSaZ5ZbaZBEHISiJf8PROzWzl8xapa332Q168f0lMZfN
>>>    BYeiQ50C8ZxKN3AAp++nuauh8NZ+lEu/vWs8Q8OwwHkP3cwM88n0QrhZBc0J/ZER
>>>    w6NRINExhgYasqGy0v1XsbYEOCB7
>>>    -----END PRIVATE KEY-----
>>>    '
>>>  spdk_log_level: WARNING
>>>  spdk_path: /usr/local/bin/nvmf_tgt
>>>  spdk_ping_interval_in_seconds: 2.0
>>>  spdk_timeout: 60.0
>>>  state_update_interval_sec: 5
>>>  state_update_notify: true
>>>  tgt_cmd_extra_args: --cpumask=0xF
>>>  tgt_path: /usr/local/bin/nvmf_tgt
>>>  transport_tcp_options:
>>>    in_capsule_data_size: 8192
>>>    max_io_qpairs_per_ctrlr: 7
>>>  transports: tcp
>>>  verbose_log_messages: true
>>>  verify_nqns: true


If we use the above specification yaml content to reconfigure the service, then its results in error.

Every 0.1s: ceph orch ls --refresh;...  ceph-sunilkumar-00-gmqual-node1-installer: Wed Jul 24 11:38:54 2024

NAME                       PORTS             RUNNING  REFRESHED  AGE  PLACEMENT

mgr                                              2/2  0s ago     2d   label:mgr

mon                                              3/3  0s ago     2d   label:mon

node-proxy                                       0/0  -          2d   *

nvmeof.rbd                 ?:4420,5500,8009	 0/4  0s ago     19m  ceph-sunilkumar-00-gmqual-node6;ceph-
sunilkumar-00-gmqual-node7;ceph-sunilkumar-00-gmqual-node8;ceph-sunilkumar-00-gmqual-node9
osd.all-available-devices                         12  0s ago     2d   *

nvmeof.rbd.ceph-sunilkumar-00-gmqual-node6.zlfuuj     ceph-sunilkumar-00-gmqual-node6            *:5500,442
0,8009  error            0s ago   2d        -        -  <unknown>         <unknown>     <unknown>
nvmeof.rbd.ceph-sunilkumar-00-gmqual-node7.jvqoct     ceph-sunilkumar-00-gmqual-node7            *:5500,442
0,8009  error            1s ago   2d        -        -  <unknown>         <unknown>     <unknown>
nvmeof.rbd.ceph-sunilkumar-00-gmqual-node8.ydlgna     ceph-sunilkumar-00-gmqual-node8            *:5500,442
0,8009  error            0s ago   2d        -        -  <unknown>         <unknown>     <unknown>
nvmeof.rbd.ceph-sunilkumar-00-gmqual-node9.mvkqgg     ceph-sunilkumar-00-gmqual-node9            *:5500,442
0,8009  error            0s ago   2d        -        -  <unknown>         <unknown>     <unknown>



Gateway Logs Failure Messages
================================

>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 systemd[1]: Starting Ceph nvmeof.rbd.ceph-sunilkumar-00-gmqual-node6.zlfuuj for f0e272a6-4834-11ef-8f5e-fa163ee0af1c...
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 podman[1638413]: 2024-07-24 11:20:42.184571905 -0400 EDT m=+0.034504952 container create b735e9e92b88690f9036a0dd71872889cfbafe35bda88a7b93bbdf206c7a4771 (image=cp.stg.icr.io/cp/ibm-ceph/nvmeof-rhel9:1.2.16-6, name=ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj, description=Ceph NVMe over Fabrics Gateway, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/agreements, io.buildah.version=1.29.0, version=1.2.16, vcs-type=git, name=ceph-nvmeof, io.openshift.expose-services=, summary=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., architecture=x86_64, build-date=2024-07-19T20:12:26, release=6, vcs-ref=97d9065ee488afab264fd0c4330ee5c4f1b1b2ab, io.openshift.tags=minimal rhel9, io.k8s.description=Ceph NVMe over Fabrics Gateway, vendor=Red Hat, Inc., com.redhat.component=ceph-nvmeof-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ceph-nvmeof/images/1.2.16-6, io.k8s.display-name=Red Hat Universal Base Image 9 Minimal, maintainer=Aviv Caro <aviv.caro>)
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 podman[1638413]: 2024-07-24 11:20:42.220028061 -0400 EDT m=+0.069961134 container init b735e9e92b88690f9036a0dd71872889cfbafe35bda88a7b93bbdf206c7a4771 (image=cp.stg.icr.io/cp/ibm-ceph/nvmeof-rhel9:1.2.16-6, name=ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj, com.redhat.license_terms=https://www.redhat.com/agreements, io.k8s.display-name=Red Hat Universal Base Image 9 Minimal, build-date=2024-07-19T20:12:26, release=6, summary=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ceph-nvmeof/images/1.2.16-6, io.k8s.description=Ceph NVMe over Fabrics Gateway, name=ceph-nvmeof, com.redhat.component=ceph-nvmeof-container, maintainer=Aviv Caro <aviv.caro>, vcs-ref=97d9065ee488afab264fd0c4330ee5c4f1b1b2ab, architecture=x86_64, distribution-scope=public, vcs-type=git, io.openshift.expose-services=, io.buildah.version=1.29.0, io.openshift.tags=minimal rhel9, version=1.2.16, vendor=Red Hat, Inc., description=Ceph NVMe over Fabrics Gateway)
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 podman[1638413]: 2024-07-24 11:20:42.224026089 -0400 EDT m=+0.073959158 container start b735e9e92b88690f9036a0dd71872889cfbafe35bda88a7b93bbdf206c7a4771 (image=cp.stg.icr.io/cp/ibm-ceph/nvmeof-rhel9:1.2.16-6, name=ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj, version=1.2.16, description=Ceph NVMe over Fabrics Gateway, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ceph-nvmeof/images/1.2.16-6, io.buildah.version=1.29.0, maintainer=Aviv Caro <aviv.caro>, release=6, summary=Service to provide block storage on top of Ceph for platforms (e.g.: VMWare) without native Ceph support (RBD), replacing existing approaches (iSCSI) with a newer and more versatile standard (NVMe-oF)., architecture=x86_64, distribution-scope=public, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 9 Minimal, vcs-ref=97d9065ee488afab264fd0c4330ee5c4f1b1b2ab, io.openshift.tags=minimal rhel9, com.redhat.component=ceph-nvmeof-container, build-date=2024-07-19T20:12:26, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/agreements, name=ceph-nvmeof, io.openshift.expose-services=, io.k8s.description=Ceph NVMe over Fabrics Gateway)
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 bash[1638413]: b735e9e92b88690f9036a0dd71872889cfbafe35bda88a7b93bbdf206c7a4771
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 podman[1638413]: 2024-07-24 11:20:42.172842398 -0400 EDT m=+0.022775447 image pull 6e92d214ca9a3bc383bf22c2ef00e9186bb4637a66e4a67f444a34395614d363 cp.stg.icr.io/cp/ibm-ceph/nvmeof-rhel9:1.2.16-6
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 systemd[1]: Started Ceph nvmeof.rbd.ceph-sunilkumar-00-gmqual-node6.zlfuuj for f0e272a6-4834-11ef-8f5e-fa163ee0af1c.
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO utils.py:258: Initialize gateway log level to "INFO"
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO utils.py:271: Log files will be saved in /var/log/ceph/nvmeof-client.nvmeof.rbd.ceph-sunilkumar-00-gmqual-node6.zlfuuj, using rotation
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:78: Using NVMeoF gateway version 1.2.16
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:81: Configured SPDK version 24.01.1
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:59: Using configuration file ceph-nvmeof.conf
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:61: ====================================== Configuration file content ======================================
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: # This file is generated by cephadm.
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: [gateway]
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: name = client.nvmeof.rbd.ceph-sunilkumar-00-gmqual-node6.zlfuuj
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: group =
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: addr = 10.0.65.239
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: port = 5500
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: enable_auth = True
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: state_update_notify = True
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: state_update_interval_sec = 5
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: enable_spdk_discovery_controller = False
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: enable_prometheus_exporter = True
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: prometheus_exporter_ssl = False
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: prometheus_port = 10008
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: verify_nqns = True
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: omap_file_lock_duration = 20
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: omap_file_lock_retries = 30
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: omap_file_lock_retry_sleep_interval = 1.0
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: omap_file_update_reloads = 10
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: allowed_consecutive_spdk_ping_failures = 1
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: spdk_ping_interval_in_seconds = 2.0
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: ping_spdk_under_lock = False
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: enable_monitor_client = True
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65:
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: [gateway-logs]
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: log_level = INFO
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: log_files_enabled = True
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: log_files_rotation_enabled = True
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: verbose_log_messages = True
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: max_log_file_size_in_mb = 10
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: max_log_files_count = 20
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: max_log_directory_backups = 10
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: log_directory = /var/log/ceph/
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65:
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: [discovery]
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: addr = 10.0.65.239
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: port = 8009
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65:
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: [ceph]
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: pool = rbd
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: config_file = /etc/ceph/ceph.conf
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: id = nvmeof.rbd.ceph-sunilkumar-00-gmqual-node6.zlfuuj
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65:
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: [mtls]
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: server_key = /server.key
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: client_key = /client.key
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: server_cert = /server.cert
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: client_cert = /client.cert
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: root_ca_cert = /root.ca.cert
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65:
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: [spdk]
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: tgt_path = /usr/local/bin/nvmf_tgt
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: rpc_socket_dir = /var/tmp/
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: rpc_socket_name = spdk.sock
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: timeout = 60.0
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: bdevs_per_cluster = 32
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: log_level = WARNING
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: conn_retries = 10
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: transports = tcp
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: transport_tcp_options = {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: tgt_cmd_extra_args = --cpumask=0xF
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65:
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: [monitor]
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:65: timeout = 1.0
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO config.py:66: ========================================================================================================
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO server.py:90: Starting gateway client.nvmeof.rbd.ceph-sunilkumar-00-gmqual-node6.zlfuuj
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO server.py:156: Starting serve, monitor client version: ceph version 18.2.1-228.el9cp (26a6dfb9d5132badb606ef83e88da470a80452a6) reef (stable)
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO state.py:368: nvmeof.state OMAP object already exists.
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO server.py:237: Starting /usr/bin/ceph-nvmeof-monitor-client --gateway-name client.nvmeof.rbd.ceph-sunilkumar-00-gmqual-node6.zlfuuj --gateway-address 10.0.65.239:5500 --gateway-pool rbd --gateway-group  --monitor-group-address 10.0.65.239:5499 -c /etc/ceph/ceph.conf -n client.nvmeof.rbd.ceph-sunilkumar-00-gmqual-node6.zlfuuj -k /etc/ceph/keyring --server-cert /server.cert --client-key /client.key --client-cert /client.cert
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO server.py:241: monitor client process id: 19
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO server.py:298: mTLS authenciation has been enabled
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: E0724 15:20:42.487259634       2 ssl_transport_security.cc:772]        Invalid cert chain file.
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: E0724 15:20:42.487283682       2 ssl_security_connector.cc:271]        Handshaker factory creation failed with TSI_INVALID_ARGUMENT.
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: E0724 15:20:42.487447338       2 chttp2_server.cc:1045]                UNKNOWN:Unable to create secure server with credentials of type Ssl {created_time:"2024-07-24T15:20:42.487396108+00:00"}
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] ERROR server.py:245: Unable to start CEPH monitor client:
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: Traceback (most recent call last):
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/remote-source/ceph-nvmeof/app/control/server.py", line 243, in _start_monitor_client
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     self._wait_for_group_id()
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/remote-source/ceph-nvmeof/app/control/server.py", line 142, in _wait_for_group_id
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     self.monitor_server = self._grpc_server(self._monitor_address())
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/remote-source/ceph-nvmeof/app/control/server.py", line 321, in _grpc_server
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     server.add_secure_port(
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/usr/local/lib64/python3.9/site-packages/grpc/_server.py", line 1106, in add_secure_port
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     return _common.validate_port_binding_result(
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/usr/local/lib64/python3.9/site-packages/grpc/_common.py", line 175, in validate_port_binding_result
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address)
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: RuntimeError: Failed to bind to address 10.0.65.239:5499; set GRPC_VERBOSITY=debug environment variable to see detailed error message.
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] ERROR server.py:110: GatewayServer exception occurred:
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: Traceback (most recent call last):
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/remote-source/ceph-nvmeof/app/control/__main__.py", line 44, in <module>
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     gateway.serve()
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/remote-source/ceph-nvmeof/app/control/server.py", line 166, in serve
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     self._start_monitor_client()
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/remote-source/ceph-nvmeof/app/control/server.py", line 243, in _start_monitor_client
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     self._wait_for_group_id()
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/remote-source/ceph-nvmeof/app/control/server.py", line 142, in _wait_for_group_id
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     self.monitor_server = self._grpc_server(self._monitor_address())
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/remote-source/ceph-nvmeof/app/control/server.py", line 321, in _grpc_server
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     server.add_secure_port(
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/usr/local/lib64/python3.9/site-packages/grpc/_server.py", line 1106, in add_secure_port
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     return _common.validate_port_binding_result(
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/usr/local/lib64/python3.9/site-packages/grpc/_common.py", line 175, in validate_port_binding_result
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     raise RuntimeError(_ERROR_MESSAGE_PORT_BINDING_FAILED % address)
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: RuntimeError: Failed to bind to address 10.0.65.239:5499; set GRPC_VERBOSITY=debug environment variable to see detailed error message.
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO server.py:432: Aborting (client.nvmeof.rbd.ceph-sunilkumar-00-gmqual-node6.zlfuuj) pid 19...
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO server.py:131: Exiting the gateway process.
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: [24-Jul-2024 15:20:42] INFO utils.py:392: Will compress log file /var/log/ceph/nvmeof-client.nvmeof.rbd.ceph-sunilkumar-00-gmqual-node6.zlfuuj/nvmeof-log to /var/log/ceph/nvmeof-client.nvmeof.rbd.ceph-sunilkumar-00-gmqual-node6.zlfuuj/nvmeof-log.gz
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]: Traceback (most recent call last):
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     return _run_code(code, main_globals, None,
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     exec(code, run_globals)
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/remote-source/ceph-nvmeof/app/control/__main__.py", line 44, in <module>
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     gateway.serve()
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/remote-source/ceph-nvmeof/app/control/server.py", line 166, in serve
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     self._start_monitor_client()
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/remote-source/ceph-nvmeof/app/control/server.py", line 243, in _start_monitor_client
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:     self._wait_for_group_id()
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-fa163ee0af1c-nvmeof-rbd-ceph-sunilkumar-00-gmqual-node6-zlfuuj[1638454]:   File "/remote-source/ceph-nvmeof/app/control/server.py", line 142, in _wait_for_group_id
>>>Jul 24 11:20:42 ceph-sunilkumar-00-gmqual-node6 ceph-f0e272a6-4834-11ef-8f5e-


Version-Release number of selected component (if applicable): 
Ceph 18.2.1-228
NVMe 1.2.16-6


How reproducible: always
Note You need to log in before you can comment on or make changes to this bug.