== Comment: #0 - Sanjeev Patro <sanpatr1.com> - 2013-06-04 05:47:06 == Mounting the filesystem through cifs(samba) is failing with default security mode (ntmlssp) 1> We configured the samba by installing the following rpm # rpm -qa | grep samba samba-common-4.0.6-2.fc19.ppc64 samba-libs-4.0.6-2.fc19.ppc64 samba-4.0.6-2.fc19.ppc64 samba-client-4.0.6-2.fc19.ppc64 kernel version 3.9.2-301.fc19.ppc64p7 2> Attaching smb.conf, It has not been changed from the default once except folder entries 3> Folder entries are as such [exports] comment = Exported stuff browseable = yes path = /exports public = yes writable = yes [homes] comment = Home Directories browseable = yes path= /homes/ public = yes writable = yes valid users = root,smbguest 4> service smb restart 5> set the samba password of root smbpasswd -a root 6> smbclient output shows as below # smbclient -L localhost -U root Enter root's password: Domain=[MYGROUP] OS=[Unix] Server=[Samba 4.0.6] Sharename Type Comment --------- ---- ------- homes Disk Home Directories exports Disk Exported stuff IPC$ IPC IPC Service (Samba Server Version 4.0.6) root Disk Home Directories Domain=[MYGROUP] OS=[Unix] Server=[Samba 4.0.6] Server Comment --------- ------- Workgroup Master --------- ------- 7> When I try to mount these with default option with default security mode # mount -t cifs -o username=root,passwd=<password> //localhost/homes /smbhomes mount error(13): Permission denied Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) 8> Looking into the man page mount.cifs The default security mode in mainline kernel versions prior to v3.8 was sec=ntlm. In v3.8, the default was changed to sec=ntlmssp. 9> So we tried with sec=ntlm, then mount is successful # mount -t cifs -o username=root,passwd=xxx,sec=ntlm //localhost/homes /smbhomes # The concern over here is The default security mode of samba mounting in Fedora19 kernel is ntlmssp which uses NTLMv2 password hashing encapsulated in Raw NTLMSSP message. But with default security mode mount is failing # mount -t cifs -o username=root,passwd=<password> //localhost/homes /smbhomes mount error(13): Permission denied Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) # mount -t cifs -o username=root,passwd=xxx,sec=ntlmssp //localhost/homes /smbhomes mount error(13): Permission denied Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) User has to provide sec=ntlm to make mount successful Expected Output Mounting through cifs should support ntlmssp security mode or Default security mode should change to ntlm or In the error message we should provide some hint for using sec=ntlm. Of course, it is documented in man page but still for better usability we should provide this workaround in the error message itself. == Comment: #1 - Sanjeev Patro <sanpatr1.com> - 2013-06-04 05:48:05 == == Comment: #3 - Onkar N. Mahajan <onmahaja.com> - 2013-06-05 01:54:12 == Observed following kernel messages in /var/log/messages while doing the mount.cifs : (1) # mount -t cifs -o username=root,passwd=xxxxxxxx //localhost/homes /smbhomes mount error(13): Permission denied Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) (2) # mount -t cifs -o username=root,passwd=xxxxxxxx,sec=ntlmssp //localhost/homes /smbhomes mount error(13): Permission denied Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) (3) # mount -t cifs -o username=root,passwd=xxxxxxxx,sec=ntlm //localhost/homes /smbhomes # /var/log/messages : (1) Jun 5 01:17:21 ltcfbl6fb09 kernel: [ 1707.369537] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE Jun 5 01:17:21 ltcfbl6fb09 kernel: [ 1707.369554] CIFS VFS: Send error in SessSetup = -13 Jun 5 01:17:21 ltcfbl6fb09 kernel: [ 1707.369740] CIFS VFS: cifs_mount failed w/return code = -13 (2) Jun 5 01:17:40 ltcfbl6fb09 kernel: [ 1725.944926] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE Jun 5 01:17:40 ltcfbl6fb09 kernel: [ 1725.944942] CIFS VFS: Send error in SessSetup = -13 Jun 5 01:17:40 ltcfbl6fb09 kernel: [ 1725.945116] CIFS VFS: cifs_mount failed w/return code = -13 (3) No kernel debug messages Investigating more ... Regards, Onkar == Comment: #4 - Onkar N. Mahajan <onmahaja.com> - 2013-06-05 03:19:39 == Sanjeev, IMO - this is a documentation bug - the mount.cifs mentions more than what is acutally supported ( and what I can see from the source code ). For 'sec=' option only two values are supported : (1) "none" ? none - attempt to connection as a null user (no name) (2) "krb5" ? krb5 - Use Kerberos version 5 authentication also I see that the cifs.upcall (see man 8 cifs.upcall) supports only krb5 authentication protocol. Even on upstream cifs-utils I see that only these two options are supported. Please check. Regards, Onkar
Created attachment 757060 [details] /etc/samba/smb.conf
Moving this to a kernel bug for now. cifs-utils doesn't do much beyond send the password down to the kernel. That's where the real work happens. sec=ntlm is just a workaround. It's absolutely not secure enough for general usage these days since the hashes are crackable in seconds with rainbow tables. The real issue is why mounting samba with sec=ntlmssp is failing for you. It seems to work fine for me, but my client and server are x86_64 I'll note that you're testing this on ppc64. That makes me suspect that there's an endianness bug somewhere. Does it work if you attempt to mount this server using sec=ntlmssp from an x86_64 client? How about if you do the reverse and mount an x86_64 server from a ppc64 client?
------- Comment From sanpatr1.com 2013-06-07 11:51 EDT------- (In reply to comment #9) > Moving this to a kernel bug for now. cifs-utils doesn't do much beyond send > the password down to the kernel. That's where the real work happens. > > sec=ntlm is just a workaround. It's absolutely not secure enough for general > usage these days since the hashes are crackable in seconds with rainbow > tables. The real issue is why mounting samba with sec=ntlmssp is failing for > you. > > It seems to work fine for me, but my client and server are x86_64 I'll note > that you're testing this on ppc64. That makes me suspect that there's an > endianness bug somewhere. > > Does it work if you attempt to mount this server using sec=ntlmssp from an > x86_64 client? How about if you do the reverse and mount an x86_64 server > from a ppc64 client? As request I tried similar scenario and here are the results I have one pcc64 and one x86_64 Fedora19 systems. I configured samba server in both the system. Try to mount within and across the systems. Here are the results Samba Server mount client default sec=ntlm pcc64 ppc64 failure success x86_64 x86_64 success not required ppc64 x86_64 success not required x86_64 ppc64 failure success
Many thanks. Based on that, it does seem likely that there's an endianness problem in the cifs.ko NTLMSSP auth code. The trick now is to track down where it is. What might be good is to turn up samba debugging code, and see whether it gives us any hint as to which field isn't getting proper endianness conversion.
One other question... Do any of the failing configurations work if you use sec=ntlmv2? That sec= option and sec=ntlmssp share some code, so that may help us pinpoint where to look.
cc'ing Shirish too since he wrote most of this code. Shirish any thoughts on where an endianness bug in the NTLMSSP auth code might be?
------- Comment From sanpatr1.com 2013-06-07 13:02 EDT------- (In reply to comment #11) > Many thanks. Based on that, it does seem likely that there's an endianness > problem in the cifs.ko NTLMSSP auth code. The trick now is to track down > where it is. > > What might be good is to turn up samba debugging code, and see whether it > gives us any hint as to which field isn't getting proper endianness > conversion. > > One other question... > > Do any of the failing configurations work if you use sec=ntlmv2? No it is also not working. Only ntlm is working in case failing case >That sec= option and sec=ntlmssp share some code, so that may help us pinpoint where > to look.
Thanks. This bug then is probably somewhere down in the bowels of the ntlmv2 auth code.
Looking into it. I can spend time on this today and tomorrow but will be out of office starting June 9th through June 18th. And then I can resume afterwards if it is not resolved by then.
I have a setup where I can recreate this bug. It does not have to be a Samba server running on x86 box and a cifs client running on a ppc64 box, a Windows server on x86 and cifs client on ppc64 box is a good enough setup, which is what I have.
Yeah, I imagine that'll reproduce it. It might be easier to get info out of a samba server though if you (for instance) run it with -d10. With that, it may tell you something about why it's rejecting the auth request...
Focussing on nlmv2 code since it is failing and that is what ntlmssp uses as well. Comparing the wireshark traces of session setup from a Power box (be) and an x86 (le) box, I see that Unicode password lengths are different even though cifs client from both of the boxes is attempting to authenticate with the same server as the same user with the same password. Looking into it.
Unicode password lengths were different because I had older code in function build_avpair_blob() on my test machine running 2.6.32-279.el6.e1000.ppc64. Even after changing function build_avpair_blob() to make it same as mainline code, ntlmv2 authentication on this Power Big Endian machine still fails. Wondering whether hmac-md5 crypto code has any big endian bugs, it is possible but not sure, so will spend some time on that.
Just to be sure, is it possible to attach a wireshark trace of mount command with sec=ntlmv2 option on this ppc64 machine?
What I see is, in function calc_ntlmv2_hash(), for be and he boxes, nt_hash is same but ntlmv2_hash is different. Looking into that.
I think UniStrupr() function does not work on be architecture, outcome is still lowercase. Will investigate later today or tomorrow.
Nice work, Shirish. I think that makes sense... len = cifs_strtoUTF16((__le16 *)user, ses->user_name, len, nls_cp); UniStrupr(user); cifs_strtoUTF16 converts the string to LE, but UniStrupr doesn't handle endian converted strings properly. So, something like this patch might fix it? diff --git a/fs/cifs/cifs_unicode.h b/fs/cifs/cifs_unicode.h index 4fb0974..bec1137 100644 --- a/fs/cifs/cifs_unicode.h +++ b/fs/cifs/cifs_unicode.h @@ -334,7 +334,7 @@ UniStrupr(register wchar_t *upin) up = upin; while (*up) { /* For all characters */ - *up = UniToupper(*up); + *up = cpu_to_le16(UniToupper(le16_to_cpu(*up))); up++; } return upin; /* Return input pointer */
Jeff, Thanks. Sure enough, this patch works. I tried both sec=ntlmv2i and sec=ntlmsspi mount options and both worked.
Nice! Thanks again for chasing this down. I'll leave it to you to post the patch since you did most of the legwork. Also, while you're in there... The code that handles the domainName or serverName indicates that it should be converting those strings to uppercase, but it doesn't look like it does that. Do we need to fix those as well?
Jeff, I think that comment /* convert ses->domainName to unicode and uppercase */ should be changed to /* convert ses->domainName to unicode */ MS-NLMP section 3.3.2 does not say anything about Uppercase'ing UserDom. Define NTOWFv2(Passwd, User, UserDom) as HMAC_MD5( MD4(UNICODE(Passwd)), UNICODE(ConcatenationOf( Uppercase(User), UserDom ) ) ) EndDefine
------- Comment From onmahaja.com 2013-07-19 05:27 EDT------- Any progress here ?
Patch has been merged upstream and is going into 3.9 and 3.10 stable. It should make its way into Fedora in the near future.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 19 kernel bugs. Fedora 19 has now been rebased to 3.11.1-200.fc19. Please test this kernel update and let us know if you issue has been resolved or if it is still present with the newer kernel. If you experience different issues, please open a new bug report for those.
------- Comment From onmahaja.com 2013-12-02 04:33 EDT------- === EXTERNALIZING #c40 === Redhat , The bug reporter has tested this bug and confirmed that the bug stands resolved , hence closing the bug from our side , please do the needful from your side. Thanks [root@localhost /]# uname -a Linux localhost.localdomain 3.11.0-300.fc20.ppc64p7 #1 SMP Thu Sep 5 16:13:50 MST 2013 ppc64 ppc64 ppc64 GNU/Linux [root@localhost /]# service smb status Redirecting to /bin/systemctl status smb.service smb.service - Samba SMB Daemon Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled) Active: active (running) since Thu 2013-11-28 15:58:34 IST; 10min ago Process: 6665 ExecStart=/usr/sbin/smbd $SMBDOPTIONS (code=exited, status=0/SUCCESS) Main PID: 6666 (smbd) CGroup: /system.slice/smb.service ??6666 /usr/sbin/smbd ??6669 /usr/sbin/smbd ??6766 /usr/sbin/smbd [root@localhost /]# smbclient -L localhost -U root Domain=[MYGROUP] OS=[Unix] Server=[Samba 4.1.0rc2] IPC$ IPC IPC Service (Samba Server Version 4.1.0rc2) Domain=[MYGROUP] OS=[Unix] Server=[Samba 4.1.0rc2] [root@localhost /]# mount -t cifs -o username=root,passwd=.pasroot //localhost/homes /smbhomes [root@localhost /]# [root@localhost /]# mount -t cifs -o username=root,passwd=vpltest,sec=ntlm //localhost/homes /smbhomes [root@localhost /]#