Bug 1804698 - Azure-file storage doesn't work with ERROR "CIFS VFS: could not allocate crypto hmac(md5)" in fips on case
Summary: Azure-file storage doesn't work with ERROR "CIFS VFS: could not allocate cryp...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RHCOS
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 4.3.z
Assignee: Micah Abbott
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-19 13:17 UTC by Wei Duan
Modified: 2020-05-09 03:31 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-02-24 15:07:37 UTC
Target Upstream Version:


Attachments (Terms of Use)
dmesg_log (324.57 KB, text/plain)
2020-02-19 13:26 UTC, Wei Duan
no flags Details

Description Wei Duan 2020-02-19 13:17:39 UTC
Description of problem:
on OCP 4.3 and 4.3 cluster with FIPS enabled, I met mount failure when mount azure-file share, and from the dmesg output, I see some error as below
[81862.942474] CIFS: Attempting to mount //f799df282734e4a17963512.file.core.windows.net/anli3218-jdl6h-dynamic-pvc-65ae8537-30fc-4951-8905-18aae10974c5
[81863.003610] CIFS VFS: could not allocate crypto hmac(md5)
[81863.007221] CIFS VFS: Error -2 during NTLMSSP authentication
[81863.011203] CIFS VFS: Send error in SessSetup = -2
[81863.015864] CIFS VFS: cifs_mount failed w/return code = -2

Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2020-02-17-205936 

How reproducible:
Always on azure FIPS enabled env

Steps to Reproduce:
1.Set up OCP 4.3.z nightly build
2.create azure-file sc and pvc/pod
3.mount failed with following error:
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/7f135b03-12ec-4dee-adbe-a17ee1749535/volumes/kubernetes.io~azure-file/pvc-65ae8537-30fc-4951-8905-18aae10974c5 --scope -- mount -t cifs -o gid=1500,mfsymlinks,uid=1500,username=f799df282734e4a17963512,password=6/B4y+jaiSwsqoAgCXCo3JvH1xgaHslzZxyuyyAuDjwBDmio9LMhGzlUuNZ831WFoddgyHFEC3joU4Es+0u7Qw==,file_mode=0777,dir_mode=0777,vers=3.0 //f799df282734e4a17963512.file.core.windows.net/anli3218-jdl6h-dynamic-pvc-65ae8537-30fc-4951-8905-18aae10974c5 /var/lib/kubelet/pods/7f135b03-12ec-4dee-adbe-a17ee1749535/volumes/kubernetes.io~azure-file/pvc-65ae8537-30fc-4951-8905-18aae10974c5
Output: Running scope as unit: run-re883ec79097244889208aaeccfebf070.scope
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
4.check node dmesg
[46361.715906] CIFS: Attempting to mount
//f799df282734e4a17963512.file.core.windows.net/anli3218-jdl6h-dynamic-pvc-65ae8537-30fc-4951-8905-18aae10974c5
[46361.782196] CIFS VFS: could not allocate crypto hmac(md5)
[46361.786378] CIFS VFS: Error -2 during NTLMSSP authentication
[46361.791587] CIFS VFS: Send error in SessSetup = -2
[46361.795626] CIFS VFS: cifs_mount failed w/return code = -2

Actual results:
Mount failed 

Expected results:
Mount should be successful.

Additional info:

Comment 1 Wei Duan 2020-02-19 13:26:35 UTC
Created attachment 1664029 [details]
dmesg_log

Comment 2 Wei Sun 2020-02-19 14:06:45 UTC
Hi Mrunal, it might be related to FIPS on , could you help check? Thanks!

Comment 5 Ben Howard 2020-02-19 17:16:13 UTC
This is not a bug. Kerberos Authentication is needed since the NTLM family of password hashing is not FIPS approved.

FIPS mode only allows ciphers, hashes and algorithms that are approved. In the message, the "md5" hash fails to load.
"[46361.782196] CIFS VFS: could not allocate crypto hmac(md5)" 

When FIPS mode is enabled, kernel crypto policies prevent the loading of unapproved crypto [2].

Per [2] in "1.2 Description of the Approved Modes" MD5 is not approved. The problem is described in [3], where the modules are loaded at mount. There's no fix. 

And finally in [4] an OpenShift issue was reported. 


[1] https://github.com/dracutdevs/dracut/blob/master/modules.d/01fips/module-setup.sh#L14-L40
[2] https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp2564.pdf
[3] https://www.spinics.net/lists/linux-cifs/msg07740.html
[4] https://www.spinics.net/lists/linux-cifs/msg07740.html

Comment 8 Wei Duan 2020-02-20 06:42:09 UTC
Hi Howard,

Thanks for the update, just double confirm with you, CIFS mount does not support in OCP when FIPS enabled, and there is no w/a.

Comment 9 Micah Abbott 2020-02-24 15:07:37 UTC
There's no action for the RHCOS team to take here; we can't change the list of allowed algorithms used by FIPS nor can we change the use of NTLM auth in CIFS.

Please file a bug with the CIFS folks about the use of NTLM, if you would like to see that changed.

I will edit the associated errata to drop this BZ.

Comment 11 Abu Davis 2020-05-08 21:54:31 UTC
I am facing a similar problem on Openshift 4.3.18 which I've reported here : https://github.com/openshift/origin/issues/24961

MountVolume.SetUp failed for volume "pvc-86c125f8-3c77-4bc2-b0ec-1c6035e5b2f6" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/3329b0d9-7491-4c32-b932-38c298d7bbd0/volumes/kubernetes.io~azure-file/pvc-86c125f8-3c77-4bc2-b0ec-1c6035e5b2f6 --scope -- mount -t cifs -o mfsymlinks,username=f21150e7e1f014d15a77c20,password=redactedxxxhtBA==,file_mode=0777,dir_mode=0777,vers=3.0 //f21150e7e1f014d15a77c20.file.core.windows.net/testcluster-d68nv-dyna-pvc-86c125f8-3c77-4bc2-b0ec-1c6035e5b2f6 /var/lib/kubelet/pods/3329b0d9-7491-4c32-b932-38c298d7bbd0/volumes/kubernetes.io~azure-file/pvc-86c125f8-3c77-4bc2-b0ec-1c6035e5b2f6 Output: Running scope as unit: run-r94f81185afed4319b44490d2a2f446ce.scope mount error(2): No such file or directory Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Comment 12 Ben Howard 2020-05-09 03:31:48 UTC
Abu, given the truncated nature of your message, I doubt that this is the same issue unless you have enabled FIPS mode. If you have enabled FIPS mode, then this is a known problem: you cannot use Azure Files and FIPS mode at the same time. If you have not enabled FIPS mode, then this is a different issue and a new bug should be filed.


Note You need to log in before you can comment on or make changes to this bug.