Bug 1804698

Summary: Azure-file storage doesn't work with ERROR "CIFS VFS: could not allocate crypto hmac(md5)" in fips on case
Product: OpenShift Container Platform Reporter: Wei Duan <wduan>
Component: RHCOSAssignee: Micah Abbott <miabbott>
Status: CLOSED NOTABUG QA Contact: Michael Nguyen <mnguyen>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.3.zCC: abudavis, aos-bugs, bbreard, behoward, deparker, dustymabe, imcleod, jligon, mpatel, nstielau, tstellar, wsun
Target Milestone: ---Keywords: TestBlocker
Target Release: 4.3.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-02-24 15:07:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
dmesg_log none

Description Wei Duan 2020-02-19 13:17:39 UTC 
Description of problem:
on OCP 4.3 and 4.3 cluster with FIPS enabled, I met mount failure when mount azure-file share, and from the dmesg output, I see some error as below
[81862.942474] CIFS: Attempting to mount //f799df282734e4a17963512.file.core.windows.net/anli3218-jdl6h-dynamic-pvc-65ae8537-30fc-4951-8905-18aae10974c5
[81863.003610] CIFS VFS: could not allocate crypto hmac(md5)
[81863.007221] CIFS VFS: Error -2 during NTLMSSP authentication
[81863.011203] CIFS VFS: Send error in SessSetup = -2
[81863.015864] CIFS VFS: cifs_mount failed w/return code = -2

Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2020-02-17-205936 

How reproducible:
Always on azure FIPS enabled env

Steps to Reproduce:
1.Set up OCP 4.3.z nightly build
2.create azure-file sc and pvc/pod
3.mount failed with following error:
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/7f135b03-12ec-4dee-adbe-a17ee1749535/volumes/kubernetes.io~azure-file/pvc-65ae8537-30fc-4951-8905-18aae10974c5 --scope -- mount -t cifs -o gid=1500,mfsymlinks,uid=1500,username=f799df282734e4a17963512,password=6/B4y+jaiSwsqoAgCXCo3JvH1xgaHslzZxyuyyAuDjwBDmio9LMhGzlUuNZ831WFoddgyHFEC3joU4Es+0u7Qw==,file_mode=0777,dir_mode=0777,vers=3.0 //f799df282734e4a17963512.file.core.windows.net/anli3218-jdl6h-dynamic-pvc-65ae8537-30fc-4951-8905-18aae10974c5 /var/lib/kubelet/pods/7f135b03-12ec-4dee-adbe-a17ee1749535/volumes/kubernetes.io~azure-file/pvc-65ae8537-30fc-4951-8905-18aae10974c5
Output: Running scope as unit: run-re883ec79097244889208aaeccfebf070.scope
mount error(2): No such file or directory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
4.check node dmesg
[46361.715906] CIFS: Attempting to mount
//f799df282734e4a17963512.file.core.windows.net/anli3218-jdl6h-dynamic-pvc-65ae8537-30fc-4951-8905-18aae10974c5
[46361.782196] CIFS VFS: could not allocate crypto hmac(md5)
[46361.786378] CIFS VFS: Error -2 during NTLMSSP authentication
[46361.791587] CIFS VFS: Send error in SessSetup = -2
[46361.795626] CIFS VFS: cifs_mount failed w/return code = -2

Actual results:
Mount failed 

Expected results:
Mount should be successful.

Additional info:
Comment 1 Wei Duan 2020-02-19 13:26:35 UTC 
Created attachment 1664029 [details]
dmesg_log
Comment 2 Wei Sun 2020-02-19 14:06:45 UTC 
Hi Mrunal, it might be related to FIPS on , could you help check? Thanks!
Comment 5 Ben Howard 2020-02-19 17:16:13 UTC 
This is not a bug. Kerberos Authentication is needed since the NTLM family of password hashing is not FIPS approved.

FIPS mode only allows ciphers, hashes and algorithms that are approved. In the message, the "md5" hash fails to load.
"[46361.782196] CIFS VFS: could not allocate crypto hmac(md5)" 

When FIPS mode is enabled, kernel crypto policies prevent the loading of unapproved crypto [2].

Per [2] in "1.2 Description of the Approved Modes" MD5 is not approved. The problem is described in [3], where the modules are loaded at mount. There's no fix. 

And finally in [4] an OpenShift issue was reported. 


[1] https://github.com/dracutdevs/dracut/blob/master/modules.d/01fips/module-setup.sh#L14-L40
[2] https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp2564.pdf
[3] https://www.spinics.net/lists/linux-cifs/msg07740.html
[4] https://www.spinics.net/lists/linux-cifs/msg07740.html
Comment 8 Wei Duan 2020-02-20 06:42:09 UTC 
Hi Howard,

Thanks for the update, just double confirm with you, CIFS mount does not support in OCP when FIPS enabled, and there is no w/a.
Comment 9 Micah Abbott 2020-02-24 15:07:37 UTC 
There's no action for the RHCOS team to take here; we can't change the list of allowed algorithms used by FIPS nor can we change the use of NTLM auth in CIFS.

Please file a bug with the CIFS folks about the use of NTLM, if you would like to see that changed.

I will edit the associated errata to drop this BZ.
Comment 11 Abu Davis 2020-05-08 21:54:31 UTC 
I am facing a similar problem on Openshift 4.3.18 which I've reported here : https://github.com/openshift/origin/issues/24961

MountVolume.SetUp failed for volume "pvc-86c125f8-3c77-4bc2-b0ec-1c6035e5b2f6" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/3329b0d9-7491-4c32-b932-38c298d7bbd0/volumes/kubernetes.io~azure-file/pvc-86c125f8-3c77-4bc2-b0ec-1c6035e5b2f6 --scope -- mount -t cifs -o mfsymlinks,username=f21150e7e1f014d15a77c20,password=redactedxxxhtBA==,file_mode=0777,dir_mode=0777,vers=3.0 //f21150e7e1f014d15a77c20.file.core.windows.net/testcluster-d68nv-dyna-pvc-86c125f8-3c77-4bc2-b0ec-1c6035e5b2f6 /var/lib/kubelet/pods/3329b0d9-7491-4c32-b932-38c298d7bbd0/volumes/kubernetes.io~azure-file/pvc-86c125f8-3c77-4bc2-b0ec-1c6035e5b2f6 Output: Running scope as unit: run-r94f81185afed4319b44490d2a2f446ce.scope mount error(2): No such file or directory Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
Comment 12 Ben Howard 2020-05-09 03:31:48 UTC 
Abu, given the truncated nature of your message, I doubt that this is the same issue unless you have enabled FIPS mode. If you have enabled FIPS mode, then this is a known problem: you cannot use Azure Files and FIPS mode at the same time. If you have not enabled FIPS mode, then this is a different issue and a new bug should be filed.