[8.1z backport] When all Ganesha instance fails to get KMIP key from all configured hosts, instead of failing to mount, File share is able to mount in plain text
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Red Hat Ceph Storage 8.1 security and bug fix updates), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2025:14015
We've verified that the latest version of Ceph fixes this issue. If the key fetch from KMIP server fails, mount fails with "no such file or directory" error message.
Description of problem: As mentioned in the title, when all Ganesha instance fails to get KMIP key from all configured hosts, instead of failing to mount, File share is able to mount in plain text. See the logs and output below: kmip_key_id is set to export. ceph nfs export info r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf /A1DA587DC124470884F24BCC7954E2A7 { "access_type": "none", "clients": [ { "access_type": "rw", "addresses": [ "32.0.201.0/24", "31.0.201.0/24", "33.0.201.0/24" ], "squash": "None" } ], "cluster_id": "r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf", "export_id": 1, "fsal": { "cmount_path": "/", "fs_name": "cephfs0", "name": "CEPH", "user_id": "nfs.r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf.cephfs0.236f1467" }, "kmip_key_id": "df0564dd126042ebb03e0224728ce939_r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf_Ganesha", "path": "/volumes/_nogroup/r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf/b9c9a4ba-97a2-4e08-acf1-8acc6627a50c", "protocols": [ 3, 4 ], "pseudo": "/A1DA587DC124470884F24BCC7954E2A7", "qos_block": { "combined_rw_bw_control": true, "enable_bw_control": true, "enable_iops_control": true, "enable_qos": true, "max_export_combined_bw": "1.0MiB", "max_export_iops": 35000 }, "sectype": [ "sys" ], "security_label": true, "squash": "none", "transports": [ "TCP" ] } Directory is unencrypted [root@dal1-qz2-sr2-rk044-s18 ~]# ls /mnt/cephfs/volumes/_nogroup/r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf b9c9a4ba-97a2-4e08-acf1-8acc6627a50c File is unencrypted [root@dal1-qz2-sr2-rk044-s18 ~]# cat /mnt/cephfs/volumes/_nogroup/r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf/b9c9a4ba-97a2-4e08-acf1-8acc6627a50c/test.txt This is a test Ganesha is not able to connect to KMIP on all 3 zones/Ganesha instances. zone1 23/07/2025 23:04:46 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip0.genctl.svc.cluster.local 5696 23/07/2025 23:04:46 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4073014BD87F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:46 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip0.genctl.svc.cluster.local:5696 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip1.genctl.svc.cluster.local 5696 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4073014BD87F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip1.genctl.svc.cluster.local:5696 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip3.genctl.svc.cluster.local 5696 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4073014BD87F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip3.genctl.svc.cluster.local:5696 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip2.genctl.svc.cluster.local 5696 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4073014BD87F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip2.genctl.svc.cluster.local:5696 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip4.genctl.svc.cluster.local 5696 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4073014BD87F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip4.genctl.svc.cluster.local:5696 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip5.genctl.svc.cluster.local 5696 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4073014BD87F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip5.genctl.svc.cluster.local:5696 23/07/2025 23:04:51 : epoch 68816a8b : dal3-qz2-sr3-rk247-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :no available kmip hosts zone2 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip0.genctl.svc.cluster.local 5696 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4063A101657F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip0.genctl.svc.cluster.local:5696 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip1.genctl.svc.cluster.local 5696 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4063A101657F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip1.genctl.svc.cluster.local:5696 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip3.genctl.svc.cluster.local 5696 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4063A101657F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip3.genctl.svc.cluster.local:5696 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip2.genctl.svc.cluster.local 5696 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4063A101657F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip2.genctl.svc.cluster.local:5696 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip4.genctl.svc.cluster.local 5696 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4063A101657F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip4.genctl.svc.cluster.local:5696 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip5.genctl.svc.cluster.local 5696 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4063A101657F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip5.genctl.svc.cluster.local:5696 23/07/2025 23:04:34 : epoch 68816a7f : dal2-qz2-sr2-rk523-s20 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :no available kmip hosts zone 3 23/07/2025 23:04:22 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip0.genctl.svc.cluster.local 5696 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4003039CB17F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip0.genctl.svc.cluster.local:5696 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip1.genctl.svc.cluster.local 5696 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4003039CB17F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip1.genctl.svc.cluster.local:5696 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip3.genctl.svc.cluster.local 5696 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4003039CB17F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip3.genctl.svc.cluster.local:5696 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip2.genctl.svc.cluster.local 5696 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4003039CB17F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip2.genctl.svc.cluster.local:5696 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip4.genctl.svc.cluster.local 5696 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4003039CB17F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip4.genctl.svc.cluster.local:5696 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] setup_kmip_connect :FSAL :CRIT :BIO_do_connect failed to kmip5.genctl.svc.cluster.local 5696 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] log_ssl_errors :FSAL :CRIT :ssl error: <4003039CB17F0000:error:10080002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known> 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :kmip can't connect to kmip5.genctl.svc.cluster.local:5696 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] make_kmip_connect :FSAL :CRIT :no available kmip hosts 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] kmip_root_cb_func :FSAL :CRIT :keyset callback: failed to get key for kmip_key_id = df0564dd126042ebb03e0224728ce939_r134-2aa68218-ff5c-4d50-9f0b-33e112647dcf_Ganesha, export = 1 23/07/2025 23:04:27 : epoch 68816a73 : dal1-qz2-sr2-rk044-s18 : ganesha.nfsd-2[main] process_export_root_callbacks :EXPORT :CRIT :ExportId=1 callback callback failed rc=22 Version-Release number of selected component (if applicable): nfs-ganesha-6.5-25 ceph-19.2.1-234 How reproducible: Always Steps to Reproduce: 1. Specify wrong KMIP hosts during NFS cluster creation. 2. Create exports with KMIP key ID. 3. Try the mount, it succeds even though key fetch failed 4. Check the data, they are in plain texts Actual results: Ganesha allows mount of encrypted shares in plain text. Expected results: Ganesha must not allow mount on encrypted shares (subvolumes) if the encryption key fetch failed. It should throw error to notify the key fetch failed and cannot initiate mount. Additional info: If Ganesha allowed plain text mount of encrypted subvolumes, there is no recovering from it because the data will then never be encrypted without doing the re-encryption.