Hide Forgot
Description of problem: on OCP 4.3 and 4.3 cluster with FIPS enabled, I met mount failure when mount azure-file share, and from the dmesg output, I see some error as below [81862.942474] CIFS: Attempting to mount //f799df282734e4a17963512.file.core.windows.net/anli3218-jdl6h-dynamic-pvc-65ae8537-30fc-4951-8905-18aae10974c5 [81863.003610] CIFS VFS: could not allocate crypto hmac(md5) [81863.007221] CIFS VFS: Error -2 during NTLMSSP authentication [81863.011203] CIFS VFS: Send error in SessSetup = -2 [81863.015864] CIFS VFS: cifs_mount failed w/return code = -2 Version-Release number of selected component (if applicable): 4.3.0-0.nightly-2020-02-17-205936 How reproducible: Always on azure FIPS enabled env Steps to Reproduce: 1.Set up OCP 4.3.z nightly build 2.create azure-file sc and pvc/pod 3.mount failed with following error: Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/7f135b03-12ec-4dee-adbe-a17ee1749535/volumes/kubernetes.io~azure-file/pvc-65ae8537-30fc-4951-8905-18aae10974c5 --scope -- mount -t cifs -o gid=1500,mfsymlinks,uid=1500,username=f799df282734e4a17963512,password=6/B4y+jaiSwsqoAgCXCo3JvH1xgaHslzZxyuyyAuDjwBDmio9LMhGzlUuNZ831WFoddgyHFEC3joU4Es+0u7Qw==,file_mode=0777,dir_mode=0777,vers=3.0 //f799df282734e4a17963512.file.core.windows.net/anli3218-jdl6h-dynamic-pvc-65ae8537-30fc-4951-8905-18aae10974c5 /var/lib/kubelet/pods/7f135b03-12ec-4dee-adbe-a17ee1749535/volumes/kubernetes.io~azure-file/pvc-65ae8537-30fc-4951-8905-18aae10974c5 Output: Running scope as unit: run-re883ec79097244889208aaeccfebf070.scope mount error(2): No such file or directory Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) 4.check node dmesg [46361.715906] CIFS: Attempting to mount //f799df282734e4a17963512.file.core.windows.net/anli3218-jdl6h-dynamic-pvc-65ae8537-30fc-4951-8905-18aae10974c5 [46361.782196] CIFS VFS: could not allocate crypto hmac(md5) [46361.786378] CIFS VFS: Error -2 during NTLMSSP authentication [46361.791587] CIFS VFS: Send error in SessSetup = -2 [46361.795626] CIFS VFS: cifs_mount failed w/return code = -2 Actual results: Mount failed Expected results: Mount should be successful. Additional info:
Created attachment 1664029 [details] dmesg_log
Hi Mrunal, it might be related to FIPS on , could you help check? Thanks!
This is not a bug. Kerberos Authentication is needed since the NTLM family of password hashing is not FIPS approved. FIPS mode only allows ciphers, hashes and algorithms that are approved. In the message, the "md5" hash fails to load. "[46361.782196] CIFS VFS: could not allocate crypto hmac(md5)" When FIPS mode is enabled, kernel crypto policies prevent the loading of unapproved crypto [2]. Per [2] in "1.2 Description of the Approved Modes" MD5 is not approved. The problem is described in [3], where the modules are loaded at mount. There's no fix. And finally in [4] an OpenShift issue was reported. [1] https://github.com/dracutdevs/dracut/blob/master/modules.d/01fips/module-setup.sh#L14-L40 [2] https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp2564.pdf [3] https://www.spinics.net/lists/linux-cifs/msg07740.html [4] https://www.spinics.net/lists/linux-cifs/msg07740.html
Hi Howard, Thanks for the update, just double confirm with you, CIFS mount does not support in OCP when FIPS enabled, and there is no w/a.
There's no action for the RHCOS team to take here; we can't change the list of allowed algorithms used by FIPS nor can we change the use of NTLM auth in CIFS. Please file a bug with the CIFS folks about the use of NTLM, if you would like to see that changed. I will edit the associated errata to drop this BZ.
I am facing a similar problem on Openshift 4.3.18 which I've reported here : https://github.com/openshift/origin/issues/24961 MountVolume.SetUp failed for volume "pvc-86c125f8-3c77-4bc2-b0ec-1c6035e5b2f6" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/3329b0d9-7491-4c32-b932-38c298d7bbd0/volumes/kubernetes.io~azure-file/pvc-86c125f8-3c77-4bc2-b0ec-1c6035e5b2f6 --scope -- mount -t cifs -o mfsymlinks,username=f21150e7e1f014d15a77c20,password=redactedxxxhtBA==,file_mode=0777,dir_mode=0777,vers=3.0 //f21150e7e1f014d15a77c20.file.core.windows.net/testcluster-d68nv-dyna-pvc-86c125f8-3c77-4bc2-b0ec-1c6035e5b2f6 /var/lib/kubelet/pods/3329b0d9-7491-4c32-b932-38c298d7bbd0/volumes/kubernetes.io~azure-file/pvc-86c125f8-3c77-4bc2-b0ec-1c6035e5b2f6 Output: Running scope as unit: run-r94f81185afed4319b44490d2a2f446ce.scope mount error(2): No such file or directory Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Abu, given the truncated nature of your message, I doubt that this is the same issue unless you have enabled FIPS mode. If you have enabled FIPS mode, then this is a known problem: you cannot use Azure Files and FIPS mode at the same time. If you have not enabled FIPS mode, then this is a different issue and a new bug should be filed.