Description of problem:

As mentioned in the title, when all Ganesha instance fails to get KMIP key from all configured hosts, instead of failing to mount, File share is able to mount in plain text. See the logs and output below:

kmip_key_id is set to export.

ceph nfs export info r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf /A1DA587DC124470884F24BCC7954E2A7
{
  "access_type": "none",
  "clients": [
    {
      "access_type": "rw",
      "addresses": [
        "32.0.201.0/24",
        "31.0.201.0/24",
        "33.0.201.0/24"
      ],
      "squash": "None"
    }
  ],
  "cluster_id": "r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf",
  "export_id": 1,
  "fsal": {
    "cmount_path": "/",
    "fs_name": "cephfs0",
    "name": "CEPH",
    "user_id": "nfs.r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf.cephfs0.236f1467"
  },
  "kmip_key_id": "df0564dd126042ebb03e0224728ce939_r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf_Ganesha",
  "path": "/volumes/_nogroup/r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf/b9c9a4ba-97a2-4e08-acf1-8acc6627a50c",
  "protocols": [
    3,
    4
  ],
  "pseudo": "/A1DA587DC124470884F24BCC7954E2A7",
  "qos_block": {
    "combined_rw_bw_control": true,
    "enable_bw_control": true,
    "enable_iops_control": true,
    "enable_qos": true,
    "max_export_combined_bw": "1.0MiB",
    "max_export_iops": 35000
  },
  "sectype": [
    "sys"
  ],
  "security_label": true,
  "squash": "none",
  "transports": [
    "TCP"
  ]
}

Directory is unencrypted

[root@dal1-qz2-sr2-rk044-s18 ~]# ls /mnt/cephfs/volumes/_nogroup/r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf
b9c9a4ba-97a2-4e08-acf1-8acc6627a50c

File is unencrypted

[root@dal1-qz2-sr2-rk044-s18 ~]# cat /mnt/cephfs/volumes/_nogroup/r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf/b9c9a4ba-97a2-4e08-acf1-8acc6627a50c/test.txt
This is a test

Ganesha is not able to connect to KMIP on all 3 zones/Ganesha instances.
zone1
23/07/2025 23:04:46 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip0.genctl.svc.cluster.local 5696
23/07/2025 23:04:46 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4073014BD87F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:46 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip0.genctl.svc.cluster.local:5696
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip1.genctl.svc.cluster.local 5696
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4073014BD87F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip1.genctl.svc.cluster.local:5696
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip3.genctl.svc.cluster.local 5696
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4073014BD87F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip3.genctl.svc.cluster.local:5696
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip2.genctl.svc.cluster.local 5696
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4073014BD87F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip2.genctl.svc.cluster.local:5696
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip4.genctl.svc.cluster.local 5696
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4073014BD87F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip4.genctl.svc.cluster.local:5696
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip5.genctl.svc.cluster.local 5696
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4073014BD87F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip5.genctl.svc.cluster.local:5696
23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :no available kmip hosts
zone2
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip0.genctl.svc.cluster.local 5696
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4063A101657F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip0.genctl.svc.cluster.local:5696
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip1.genctl.svc.cluster.local 5696
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4063A101657F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip1.genctl.svc.cluster.local:5696
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip3.genctl.svc.cluster.local 5696
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4063A101657F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip3.genctl.svc.cluster.local:5696
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip2.genctl.svc.cluster.local 5696
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4063A101657F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip2.genctl.svc.cluster.local:5696
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip4.genctl.svc.cluster.local 5696
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4063A101657F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip4.genctl.svc.cluster.local:5696
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip5.genctl.svc.cluster.local 5696
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4063A101657F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip5.genctl.svc.cluster.local:5696
23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :no available kmip hosts
zone 3
23/07/2025 23:04:22 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip0.genctl.svc.cluster.local 5696
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4003039CB17F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip0.genctl.svc.cluster.local:5696
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip1.genctl.svc.cluster.local 5696
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4003039CB17F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip1.genctl.svc.cluster.local:5696
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip3.genctl.svc.cluster.local 5696
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4003039CB17F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip3.genctl.svc.cluster.local:5696
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip2.genctl.svc.cluster.local 5696
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4003039CB17F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip2.genctl.svc.cluster.local:5696
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip4.genctl.svc.cluster.local 5696
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4003039CB17F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip4.genctl.svc.cluster.local:5696
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip5.genctl.svc.cluster.local 5696
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4003039CB17F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known>
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip5.genctl.svc.cluster.local:5696
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :no available kmip hosts
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] kmip_root_cb_func :FSAL :CRIT :keyset callback: failed to get key for kmip_key_id = df0564dd126042ebb03e0224728ce939_r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf_Ganesha, export = 1
23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] process_export_root_callbacks :EXPORT :CRIT :ExportId=1 callback callback failed rc=22

Version-Release number of selected component (if applicable):

nfs-ganesha-6.5-25
ceph-19.2.1-234

How reproducible:

Always

Steps to Reproduce:

1. Specify wrong KMIP hosts during NFS cluster creation.
2. Create exports with KMIP key ID.
3. Try the mount, it succeds even though key fetch failed
4. Check the data, they are in plain texts

Actual results:

Ganesha allows mount of encrypted shares in plain text.

Expected results:

Ganesha must not allow mount on encrypted shares (subvolumes) if the encryption key fetch failed. It should throw error to notify the key fetch failed and cannot initiate mount. 

Additional info:

If Ganesha allowed plain text mount of encrypted subvolumes, there is no recovering from it because the data will then never be encrypted without doing the re-encryption.