Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 970891 - Fedora19:Beta: Mounting the filesystem through cifs(samba) is failing with default security mode (ntmlssp)
Fedora19:Beta: Mounting the filesystem through cifs(samba) is failing with de...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
19
ppc64 All
unspecified Severity high
: ---
: ---
Assigned To: Sachin Prabhu
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-05 03:30 EDT by IBM Bug Proxy
Modified: 2013-12-01 23:40 EST (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-08 13:13:42 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/etc/samba/smb.conf (11.61 KB, application/octet-stream)
2013-06-05 03:30 EDT, IBM Bug Proxy
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
IBM Linux Technology Center 94356 None None None Never

  None (edit)
Description IBM Bug Proxy 2013-06-05 03:30:40 EDT
== Comment: #0 - Sanjeev Patro <sanpatr1@in.ibm.com> - 2013-06-04 05:47:06 ==
Mounting the filesystem through cifs(samba) is failing with default security mode (ntmlssp)  

1> We configured the samba by installing the following rpm
# rpm -qa | grep samba
samba-common-4.0.6-2.fc19.ppc64
samba-libs-4.0.6-2.fc19.ppc64
samba-4.0.6-2.fc19.ppc64
samba-client-4.0.6-2.fc19.ppc64

kernel version
3.9.2-301.fc19.ppc64p7

2> Attaching smb.conf, It has not been changed from the default once except folder entries

3> Folder entries are as such

[exports]
        comment = Exported stuff
        browseable = yes
        path = /exports
        public = yes
        writable = yes

[homes]
        comment = Home Directories
        browseable = yes
        path= /homes/
        public = yes
        writable = yes
        valid users = root,smbguest


4> service smb restart

5> set the samba password of root 
smbpasswd  -a root

6> smbclient output shows as below
# smbclient -L localhost -U root 
Enter root's password: 
Domain=[MYGROUP] OS=[Unix] Server=[Samba 4.0.6]

	Sharename       Type      Comment
	---------       ----      -------
	homes           Disk      Home Directories
	exports         Disk      Exported stuff
	IPC$            IPC       IPC Service (Samba Server Version 4.0.6)
	root            Disk      Home Directories
Domain=[MYGROUP] OS=[Unix] Server=[Samba 4.0.6]

	Server               Comment
	---------            -------

	Workgroup            Master
	---------            -------

7> When I try to mount these with default option with default security mode
# mount -t cifs -o username=root,passwd=<password> //localhost/homes /smbhomes
mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

8> Looking into the man page mount.cifs
The default security mode in mainline kernel versions prior to v3.8 was sec=ntlm. In v3.8, the default was changed to sec=ntlmssp.

9> So we tried with sec=ntlm, then mount is successful
# mount -t cifs -o username=root,passwd=xxx,sec=ntlm //localhost/homes /smbhomes
#

The concern over here is
The default security mode of samba mounting in Fedora19 kernel is ntlmssp which uses NTLMv2 password hashing encapsulated in Raw NTLMSSP message. But with default security mode mount is failing
# mount -t cifs -o username=root,passwd=<password> //localhost/homes /smbhomes
mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

# mount -t cifs -o username=root,passwd=xxx,sec=ntlmssp //localhost/homes /smbhomes
mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

User has to provide sec=ntlm to make mount successful 

Expected Output
Mounting through cifs should support ntlmssp security mode
or
Default security mode should change to ntlm
or 
In the error message we should provide some hint for using sec=ntlm. Of course, it is documented in man page but still for better usability we should provide this workaround in the error message itself.

== Comment: #1 - Sanjeev Patro <sanpatr1@in.ibm.com> - 2013-06-04 05:48:05 ==


== Comment: #3 - Onkar N. Mahajan <onmahaja@in.ibm.com> - 2013-06-05 01:54:12 ==
Observed following kernel messages in /var/log/messages while doing the mount.cifs :

(1)
# mount -t cifs -o username=root,passwd=xxxxxxxx //localhost/homes /smbhomes
mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
(2)
# mount -t cifs -o username=root,passwd=xxxxxxxx,sec=ntlmssp //localhost/homes /smbhomes
mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
(3)
# mount -t cifs -o username=root,passwd=xxxxxxxx,sec=ntlm //localhost/homes /smbhomes
# 

/var/log/messages :
(1)
Jun  5 01:17:21 ltcfbl6fb09 kernel: [ 1707.369537] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
Jun  5 01:17:21 ltcfbl6fb09 kernel: [ 1707.369554] CIFS VFS: Send error in SessSetup = -13
Jun  5 01:17:21 ltcfbl6fb09 kernel: [ 1707.369740] CIFS VFS: cifs_mount failed w/return code = -13

(2)
Jun  5 01:17:40 ltcfbl6fb09 kernel: [ 1725.944926] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
Jun  5 01:17:40 ltcfbl6fb09 kernel: [ 1725.944942] CIFS VFS: Send error in SessSetup = -13
Jun  5 01:17:40 ltcfbl6fb09 kernel: [ 1725.945116] CIFS VFS: cifs_mount failed w/return code = -13

(3) 
No kernel debug messages 

Investigating more ... 


Regards,
Onkar

== Comment: #4 - Onkar N. Mahajan <onmahaja@in.ibm.com> - 2013-06-05 03:19:39 ==
Sanjeev, 
        IMO - this is a documentation bug - the mount.cifs mentions more than what is acutally supported ( and what I can see from the source code ). 

        For 'sec=' option only two values are supported :
        (1) "none"
         ?   none - attempt to connection as a null user (no name)

        (2) "krb5"
         ?   krb5 - Use Kerberos version 5 authentication

also I see that the cifs.upcall (see man 8 cifs.upcall) supports only krb5 authentication protocol. Even on upstream cifs-utils I see that only these two options are supported.
       
      Please check.

Regards,
Onkar
Comment 1 IBM Bug Proxy 2013-06-05 03:30:56 EDT
Created attachment 757060 [details]
/etc/samba/smb.conf
Comment 2 Jeff Layton 2013-06-05 08:07:49 EDT
Moving this to a kernel bug for now. cifs-utils doesn't do much beyond send the password down to the kernel. That's where the real work happens.

sec=ntlm is just a workaround. It's absolutely not secure enough for general usage these days since the hashes are crackable in seconds with rainbow tables. The real issue is why mounting samba with sec=ntlmssp is failing for you.

It seems to work fine for me, but my client and server are x86_64 I'll note that you're testing this on ppc64. That makes me suspect that there's an endianness bug somewhere.

Does it work if you attempt to mount this server using sec=ntlmssp from an x86_64 client? How about if you do the reverse and mount an x86_64 server from a ppc64 client?
Comment 3 IBM Bug Proxy 2013-06-07 08:01:42 EDT
------- Comment From sanpatr1@in.ibm.com 2013-06-07 11:51 EDT-------
(In reply to comment #9)
> Moving this to a kernel bug for now. cifs-utils doesn't do much beyond send
> the password down to the kernel. That's where the real work happens.
>
> sec=ntlm is just a workaround. It's absolutely not secure enough for general
> usage these days since the hashes are crackable in seconds with rainbow
> tables. The real issue is why mounting samba with sec=ntlmssp is failing for
> you.
>
> It seems to work fine for me, but my client and server are x86_64 I'll note
> that you're testing this on ppc64. That makes me suspect that there's an
> endianness bug somewhere.
>
> Does it work if you attempt to mount this server using sec=ntlmssp from an
> x86_64 client? How about if you do the reverse and mount an x86_64 server
> from a ppc64 client?

As request I tried similar scenario and here are the results
I have one pcc64 and one x86_64 Fedora19 systems. I configured samba server in both the system. Try to mount within and across the systems. Here are the results

Samba Server         mount client       default        sec=ntlm
pcc64                      ppc64                 failure          success
x86_64                   x86_64                success       not required
ppc64                     x86_64                success       not required
x86_64                   ppc64                  failure          success
Comment 4 Jeff Layton 2013-06-07 08:19:47 EDT
Many thanks. Based on that, it does seem likely that there's an endianness problem in the cifs.ko NTLMSSP auth code. The trick now is to track down where it is.

What might be good is to turn up samba debugging code, and see whether it gives us any hint as to which field isn't getting proper endianness conversion.
Comment 5 Jeff Layton 2013-06-07 08:21:33 EDT
One other question...

Do any of the failing configurations work if you use sec=ntlmv2? That sec= option and sec=ntlmssp share some code, so that may help us pinpoint where to look.
Comment 6 Jeff Layton 2013-06-07 08:31:09 EDT
cc'ing Shirish too since he wrote most of this code. Shirish any thoughts on where an endianness bug in the NTLMSSP auth code might be?
Comment 7 IBM Bug Proxy 2013-06-07 09:10:54 EDT
------- Comment From sanpatr1@in.ibm.com 2013-06-07 13:02 EDT-------
(In reply to comment #11)
> Many thanks. Based on that, it does seem likely that there's an endianness
> problem in the cifs.ko NTLMSSP auth code. The trick now is to track down
> where it is.
>
> What might be good is to turn up samba debugging code, and see whether it
> gives us any hint as to which field isn't getting proper endianness
> conversion.
>
> One other question...
>
> Do any of the failing configurations work if you use sec=ntlmv2?
No it is also not working. Only ntlm is working in case failing case
>That sec= option and sec=ntlmssp share some code, so that may help us pinpoint where
> to look.
Comment 8 Jeff Layton 2013-06-07 09:12:33 EDT
Thanks. This bug then is probably somewhere down in the bowels of the ntlmv2 auth code.
Comment 9 Shirish S. Pargaonkar 2013-06-07 09:48:10 EDT
Looking into it.  I can spend time on this today and tomorrow but
will be out of office starting June 9th through June 18th.
And then I can resume afterwards if it is not resolved by then.
Comment 10 Shirish S. Pargaonkar 2013-06-07 09:48:20 EDT
Looking into it.  I can spend time on this today and tomorrow but
will be out of office starting June 9th through June 18th.
And then I can resume afterwards if it is not resolved by then.
Comment 11 Shirish S. Pargaonkar 2013-06-07 12:13:19 EDT
I have a setup where I can recreate this bug.
It does not have to be a Samba server running on x86 box and
a cifs client running on a ppc64 box, a Windows server on x86 and
cifs client on ppc64 box is a good enough setup, which is what I have.
Comment 12 Jeff Layton 2013-06-07 13:06:52 EDT
Yeah, I imagine that'll reproduce it. It might be easier to get info out of a samba server though if you (for instance) run it with -d10. With that, it may tell you something about why it's rejecting the auth request...
Comment 13 Shirish S. Pargaonkar 2013-06-20 17:37:11 EDT
Focussing on nlmv2 code since it is failing and that is what
ntlmssp uses as well.
Comparing the wireshark traces of session setup from a Power box (be)
and an x86 (le) box, I see that Unicode password lengths are different
even though cifs client from both of the boxes is attempting to
authenticate with the same server as the same user with the same password.
Looking into it.
Comment 14 Shirish S. Pargaonkar 2013-06-21 01:25:09 EDT
Unicode password lengths were different because I had older code
in function build_avpair_blob() on my test machine running
2.6.32-279.el6.e1000.ppc64.

Even after changing function build_avpair_blob() to make it same as mainline
code, ntlmv2 authentication on this Power Big Endian machine still fails.

Wondering whether hmac-md5 crypto code has any big endian bugs, it is possible but not sure, so will spend some time on that.
Comment 15 Shirish S. Pargaonkar 2013-06-21 01:37:39 EDT
Just to be sure, is it possible to attach a wireshark trace
of mount command with sec=ntlmv2 option on this ppc64 machine?
Comment 16 Shirish S. Pargaonkar 2013-06-24 17:33:23 EDT
What I see is, in function calc_ntlmv2_hash(), for be and he boxes,
nt_hash is same but ntlmv2_hash is different. Looking into that.
Comment 17 Shirish S. Pargaonkar 2013-06-24 17:49:40 EDT
I think UniStrupr() function does not work on be architecture, outcome is
still lowercase.
Will investigate later today or tomorrow.
Comment 18 Jeff Layton 2013-06-24 20:44:01 EDT
Nice work, Shirish. I think that makes sense...

    len = cifs_strtoUTF16((__le16 *)user, ses->user_name, len, nls_cp);
    UniStrupr(user);

cifs_strtoUTF16 converts the string to LE, but UniStrupr doesn't handle endian converted strings properly. So, something like this patch might fix it?

diff --git a/fs/cifs/cifs_unicode.h b/fs/cifs/cifs_unicode.h
index 4fb0974..bec1137 100644
--- a/fs/cifs/cifs_unicode.h
+++ b/fs/cifs/cifs_unicode.h
@@ -334,7 +334,7 @@ UniStrupr(register wchar_t *upin)
 
        up = upin;
        while (*up) {           /* For all characters */
-               *up = UniToupper(*up);
+               *up = cpu_to_le16(UniToupper(le16_to_cpu(*up)));
                up++;
        }
        return upin;            /* Return input pointer */
Comment 19 Shirish S. Pargaonkar 2013-06-24 22:51:27 EDT
Jeff, Thanks. Sure enough, this patch works. I tried both sec=ntlmv2i
and sec=ntlmsspi mount options and both worked.
Comment 20 Jeff Layton 2013-06-25 06:44:47 EDT
Nice! Thanks again for chasing this down. I'll leave it to you to post the patch since you did most of the legwork.

Also, while you're in there...

The code that handles the domainName or serverName indicates that it should be converting those strings to uppercase, but it doesn't look like it does that. Do we need to fix those as well?
Comment 21 Shirish S. Pargaonkar 2013-06-25 09:41:20 EDT
Jeff, I think that comment 

 /* convert ses->domainName to unicode and uppercase */
should be changed to
 /* convert ses->domainName to unicode 
*/

MS-NLMP section 3.3.2 does not say anything about Uppercase'ing UserDom.

Define NTOWFv2(Passwd, User, UserDom) as HMAC_MD5(
MD4(UNICODE(Passwd)), UNICODE(ConcatenationOf( Uppercase(User),
UserDom ) ) )
EndDefine
Comment 22 IBM Bug Proxy 2013-07-19 01:34:41 EDT
------- Comment From onmahaja@in.ibm.com 2013-07-19 05:27 EDT-------
Any progress here ?
Comment 23 Jeff Layton 2013-07-19 05:59:59 EDT
Patch has been merged upstream and is going into 3.9 and 3.10 stable. It should make its way into Fedora in the near future.
Comment 24 Josh Boyer 2013-09-18 16:35:13 EDT
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 19 kernel bugs.

Fedora 19 has now been rebased to 3.11.1-200.fc19.  Please test this kernel update and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you experience different issues, please open a new bug report for those.
Comment 25 IBM Bug Proxy 2013-12-01 23:40:28 EST
------- Comment From onmahaja@in.ibm.com 2013-12-02 04:33 EDT-------
=== EXTERNALIZING #c40 ===

Redhat , The bug reporter has tested this bug and confirmed that the bug stands resolved , hence closing the bug from our side , please do the needful from your side.

Thanks

[root@localhost /]# uname -a
Linux localhost.localdomain 3.11.0-300.fc20.ppc64p7 #1 SMP Thu Sep 5 16:13:50 MST 2013 ppc64 ppc64 ppc64 GNU/Linux
[root@localhost /]# service smb status
Redirecting to /bin/systemctl status  smb.service
smb.service - Samba SMB Daemon
Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled)
Active: active (running) since Thu 2013-11-28 15:58:34 IST; 10min ago
Process: 6665 ExecStart=/usr/sbin/smbd $SMBDOPTIONS (code=exited, status=0/SUCCESS)
Main PID: 6666 (smbd)
CGroup: /system.slice/smb.service
??6666 /usr/sbin/smbd
??6669 /usr/sbin/smbd
??6766 /usr/sbin/smbd

[root@localhost /]#  smbclient -L localhost -U root
Domain=[MYGROUP] OS=[Unix] Server=[Samba 4.1.0rc2]

IPC$            IPC       IPC Service (Samba Server Version 4.1.0rc2)
Domain=[MYGROUP] OS=[Unix] Server=[Samba 4.1.0rc2]

[root@localhost /]# mount -t cifs -o username=root,passwd=.pasroot //localhost/homes /smbhomes
[root@localhost /]#

[root@localhost /]# mount -t cifs -o username=root,passwd=vpltest,sec=ntlm //localhost/homes /smbhomes
[root@localhost /]#

Note You need to log in before you can comment on or make changes to this bug.