Bug 728796

Summary: unable to mount XP machine C$, where XP is joined to samba domain
Product: [Fedora] Fedora Reporter: Carl Byington <carl>
Component: cifs-utilsAssignee: Jeff Layton <jlayton>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 15CC: jlayton, shirishp, sprabhu, ssorce, steved
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-29 11:11:37 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Attachments:
Description Flags
tcpdump of mount failure
none
patch to add lmv2 blob to ntlmssp type 3 message none

Description Carl Byington 2011-08-07 12:17:27 EDT
Description of problem:


Version-Release number of selected component (if applicable):
cifs-utils-5.0-1.fc15.x86_64

How reproducible:
always

Steps to Reproduce:
1. Samba (on F15 machine) running as PDC
2. Windows XP machine joined to samba domain
3. from the samba machine, use mount.cifs to mount xp machine C$ admin share using domain admin credentials. 
  
Actual results:
With domain admin credentials the cifs mount fails Using local xp machine admin credentials it works. On much older (centos4) samba, we could mount with the domain admin credentials.

Expected results:


Additional info:

We had a centos4 box running samba as the PDC, with XP workstations joined to the domain. The workstations could mount samba shares using domain credentials, and the samba box could mount the workstation admin shares C$ using mount.cifs with domain admin credentials.

After upgrading centos4 to F15, the XP workstations are still joined to the domain, they can still use their existing domain credentials to mount the samba shares. That was a really transparent upgrade. However, for backups, the samba machine tries to mount the workstation admin shares like C$ using the domain administrator credentials. That no longer works. 

[ -d /mnt/$MACH ] || mkdir /mnt/$MACH
/sbin/mount.cifs //$MACH/c$ /mnt/$MACH -o "username=$USR,dom=$DOM,password=$PASS,ip=$IP"

Is this actually a samba problem?
Comment 1 Jeff Layton 2011-08-31 08:53:55 EDT
It's difficult to know without more detail.

What user is running the mount.cifs command?

What error message do you get back when you try to mount the share by hand using the above mount.cifs command?

Does anything interesting show up in dmesg when you run that command?
Comment 2 Carl Byington 2011-08-31 12:08:57 EDT
This is all running as root, from a bash shell. Connecting with domain credentials failed:

DOM=ams2
MACH=stan3
USR=administrator

/sbin/mount.cifs //$MACH/c$ /mnt/$MACH -o "username=$USR,dom=$DOM,password=$PASS,ip=$IP"
mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

[3101329.432779] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[3101329.432786] CIFS VFS: Send error in SessSetup = -13
[3101329.432798] CIFS VFS: cifs_mount failed w/return code = -13


Connecting with XP local administrator credentials works:

/sbin/mount.cifs //$MACH/c$ /mnt/$MACH -o "username=$USR,dom=$MACH,password=$PASS,ip=$IP"
df | grep mnt
//stan3/c$            74413076  40611700  33801376  55% /mnt/stan3
Comment 3 Jeff Layton 2011-08-31 12:40:30 EDT
Does it work if you add 'sec=ntlmssp' on the command line? Also, what kernel are you testing this on?
Comment 4 Carl Byington 2011-08-31 23:19:18 EDT
/sbin/mount.cifs //$MACH/c$ /mnt/$MACH -o "username=$USR,dom=$DOM,password=$PASS,ip=$IP,sec=ntlmssp"
mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

[25413.166876] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[25413.166882] CIFS VFS: Send error in SessSetup = -13
[25413.171047] CIFS VFS: cifs_mount failed w/return code = -13

uname -a
Linux mail3.example.com 2.6.40.3-0.fc15.x86_64 #1 SMP Tue Aug 16 04:10:59 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
Comment 5 Jeff Layton 2011-09-01 08:28:23 EDT
Do you still have access to the rhel4 machine? If so, it would be helpful to know whether f15 is passing the password to the kernel incorrectly. One way to tell that would be to strace the mount on both machines with something like this:

# strace -f -e mount -s 4096 /sbin/mount.cifs //$MACH/c$

...and them compare whether the passwords being passed to the kernel are equivalent for both. This would help narrow down whether the problem is in the userspace mount helper, or the kernel.
Comment 6 Carl Byington 2011-09-01 11:19:09 EDT
No, the original centos4 box is gone, but I still have another client with the same setup (centos4 samba domain controller, xp joined to the domain, mount the xp c$ share from the domain controller using the domain credentials working).

So the two strace will be against two different xp machines in different domains. Does that still help?
Comment 7 Jeff Layton 2011-09-01 13:09:12 EDT
Probably not, unless they both use the same password. What we need to determine is whether the password is being passed to the kernel correctly in both cases.
Comment 8 Carl Byington 2011-09-01 20:23:18 EDT
Temporarily changing the admin domain password on both systems to abc123

From the F15 system:

strace -f -e mount -s 4096 /sbin/mount.cifs //$MACH/c$ /mnt/$MACH -o "username=$USR,dom=$DOM,password=$PASS,ip=$IP"
Process 17728 attached
Process 17727 suspended
Process 17727 resumed
Process 17728 detached
--- {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=17728, si_status=0, si_utime=0, si_stime=0} (Child exited) ---
mount("//stan3/c$", ".", "cifs", 0, "ip=192.168.2.99,unc=\\\\stan3\\c$,,ver=1,user=administrator,domain=ams2,pass=abc123") = -1 EACCES (Permission denied)
mount error(13): Permission denied
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)


From a different centos4 network:

strace -f -e mount -s 4096  /sbin/mount.cifs //$NTBIOS/c$ /mnt/$MOUNT -o "username=$USR,dom=$DOM,password=$PASS,ip=$NTDNS"
mount("//dell2400/c$", ".", "cifs", MS_MANDLOCK, "unc=//dell2400\\c$,ver=1,username=administrator,dom=510sg,ip=192.168.4.11,pass=abc123") = 0
Comment 9 Jeff Layton 2012-02-07 06:17:04 EST
Sorry for the delay in looking at this...

Yes, it looks like the correct pw is being passed down to the kernel in both cases. At this point, we'd probably need to look at network traces between the client and server in both cases. Ideally, you'd have them connect to the same
server to ensure that that isn't a factor. Instructions on how to do that are
here:

    https://wiki.samba.org/index.php/LinuxCIFS_troubleshooting


I will note this though:

On the f15 box, you're using "domain=ams2" and on the other one you're using "domain=510sg". If these machines use the same PDC, then why are you using a different domain name here?
Comment 10 Carl Byington 2012-02-08 12:22:35 EST
Yes, those were two different servers

f15 client connecting to XP machine joined to domain ams2 with pdc on the same f15 machine as the client.

centos4 client connecting to XP machine joined to domain 510sg with pdc on the same centos4 machine as the client.

However, the last centos4 machine has now been upgraded to centos6 and the mount with domain credentials now fails. So I no longer have a test system where the mount with domain credentials works.

I can get a network trace on either the F15 system or the centos6 system. Any preferences?
Comment 11 Jeff Layton 2012-02-17 15:08:05 EST
Either one is fine. Let's just start with any capture that shows the failure
occurring.
Comment 12 Carl Byington 2012-02-17 20:28:58 EST
Created attachment 564022 [details]
tcpdump of mount failure
Comment 13 Carl Byington 2012-02-17 20:30:06 EST
That tcpdump is from the centos6 system.
Comment 14 Jeff Layton 2012-02-18 07:50:43 EST
I suppose I'll need to know what key you used to encrypt this?
Comment 15 Carl Byington 2012-02-18 17:47:40 EST
Should be to your most recent key, RSA key 19568215, created: 2011-10-12
Comment 16 Jeff Layton 2012-02-19 10:16:29 EST
Ok, it's definitely falling down in the session setup code. The connection going
in the other direction seems to have worked fine though. Some questions:

Are you able to mount if you mount with sec=ntlmssp?

What kernel is the client running in this case?
Comment 17 Carl Byington 2012-02-19 12:51:50 EST
Adding sec=ntlmssp to the options does not change the behavior - mount still fails.

2.6.32-220.4.2.el6.x86_64 #1 SMP Tue Feb 14 04:00:16 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux
Comment 18 Jeff Layton 2012-02-20 06:42:03 EST
Ok, the capture shows two different machines. Each machine is apparently a
server as well as a client, and they are attempting to connect to each other's
servers. One is:

Native OS: Linux version 2.6.32-220.4.2.el6.x86_64
Native LAN Manager: CIFS VFS Client for Linux

...and the other is:

Native OS: Windows 2002 Service Pack 3 2600
Native LAN Manager: Windows 2002 5.1

The Linux client is using "normal" NTLMv1 auth, and is attempting to authenticate
with:

username = administrator
domain = 510sg

The windows machine is using NTLMSSP auth. Interestingly, the windows machine
sets the domain and username fields to NULL in the NTLMSSP auth request. It
doesn't appear to be authenticating as a user at all. It does however set the
"host name" field. I'm going to assume that this machine is doing anonymous
auth?

Unfortunately, I can't tell much from this info. I really need to see a
set of traces that allows me to compare the working and non-working cases
to the same server. Anything else leaves too many variables for me to know
where to start troubleshooting.

If you can get me that, I'll be happy to take a look.
Comment 19 Jeff Layton 2012-02-20 06:45:34 EST
Shirish, does this look at all familiar to you?
Comment 20 Shirish S. Pargaonkar 2012-02-20 08:48:17 EST
Jeff, looking into it.
Comment 21 Shirish S. Pargaonkar 2012-02-20 11:06:01 EST
Contents of smb.conf file could be different between centos4
and f15 (and centos6) and may offer some clue as to why the 
failure on f15 and centos6.

Without and with the patch cfbd6f84c2e26c13ded16b6bb0871edb7d75974f
applied to cifs client, does sec=ntlmv2 work? 
Perhaps Samba server does not allow ntlmv1 (current default or stated 
explicitly using mount option sec=ntlm) authentication mechanism!

This could be a case of a failing pass-through authentication.
Windows XP box just passes the (24 bytes) blob to the
Domain Controller (which is Samba server on f15 in this case)
for authentication.

In case of extended security authentication (e.g. sec=ntlmssp)
this blob needs to be LMv2 response (which is 24 bytes) but
cifs client does not generate and send along LMv2 response in
type 3 (auth) phase of the session setup (LM response is empty).
I had coded LMv2 response once, will see if I have that code
saved somewhere.  If not, will generate a patch that adds 
LMv2 response.

But meanwhile, we can see if smbclient from the f15 box succeeds
in authentication as a domain user.  smbclient can try employing
any/all of ntlm or ntlmv2 or spnego/ntlmssp authentication
mechanisms by choosing so in smb.conf file.

Still working on figuring out how to access tcpdump attached
as .pgp file.
So I am not sure whether Windows XP sends ntlm or ntlmv2 blob
within ntlmssp auth packet sent to the Samba server and whether it
also includes anything (24 bytes) in LM field of that auth packet!
Comment 22 Shirish S. Pargaonkar 2012-02-20 17:09:57 EST
Created attachment 564534 [details]
patch to add lmv2 blob to ntlmssp type 3 message

patch to add lmv2 auth blob to in ntlmssp type 3 response.
To be used with sec=ntlmssp/i mount option.
Comment 23 Jeff Layton 2012-05-14 07:24:44 EDT
Carl, have you had a chance to test Shirish's patch? Did it help?
Comment 24 Carl Byington 2012-05-19 13:34:12 EDT
I no longer have any XP machines here, or at the client, so I cannot test that patch. The migration to Linux is now complete.
Comment 25 Jeff Layton 2012-06-29 11:11:37 EDT
Ok then, at this point. I'm going to go ahead and close this with a resolution of INSUFFICIENT_DATA since we don't know enough about the problem to say whether it's fixed or not.

Please reopen if you or someone else finds a way to reproduce it and can test Shirish's patch.