Bug 479418
| Summary: | second cifs mount to samba server fails when samba using security=ADS | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 5 | Reporter: | Rob Henderson <robh> | ||||||||
| Component: | kernel | Assignee: | Jeff Layton <jlayton> | ||||||||
| Status: | CLOSED ERRATA | QA Contact: | yanfu,wang <yanwang> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | low | ||||||||||
| Version: | 5.2 | CC: | azelinka, bfields, dpal, gdeschner, jlayton, rbinkhor, shei, ssorce, steved, theo_nra, yanwang | ||||||||
| Target Milestone: | rc | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | i686 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | |||||||||||
| : | 604785 604786 (view as bug list) | Environment: | |||||||||
| Last Closed: | 2011-01-13 20:45:37 UTC | Type: | --- | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Bug Depends On: | 604785 | ||||||||||
| Bug Blocks: | 604786 | ||||||||||
| Attachments: |
|
||||||||||
Created attachment 328548 [details]
Transcript demonstrating problem and DebugData
The attachment shows a demonstration of the problem along with /proc/fs/cifs/DebugData output.
I figured out that the jtltest.57 kernels that were on my people page had a bad patch which prevented all mounts from working. I've just uploaded a jtltest.59 kernel to that page which should have a fix for that. If you're able, could you test that kernel and let me know whether the problem you had is still reproducible? The jtltest.59 kernel looks quite promising! I haven't done extensive testing but have tried a few things that would have caused problems before and it is working properly now. I'll beat on it some more and get back to you on Monday with an update. Thanks! Sounds good. Let me know how it goes... Well, it turns out that things are different but it still doesn't work. Previously, any second attempt to mount the same share as a different user (whether the authentication succeeded or not) stepped on the first mount and it started giving I/O errors. Now, the second mount attempt has no effect on the first mount BUT the second mount attempt is never able to authenticate. It just always fails whether you enter the correct password or not. The messages logged are exactly the same whether you enter the correct or incorrect password (NT_STATUS_LOGON_FAILURE on the client and NT_STATUS_WRONG_PASSWORD on the samba server). As soon as the first mount is umount'ed then it all works fine for the second user. I'm not able to reproduce this on my clients/servers. Could you attach the smb.conf from the RHEL5 server? It sounds like this might be a server-side problem. What samba package is the server running here? Well, it turns out that your new kernel (jtltest.59) does work with multiple users if I am using a local smbpasswd file (security=user) but not when using ADS (security=ADS). I will attach the smb.conf file on the server that is failing. I trimmed it down to the bare minimum that still demonstrates the problem. All I have to do is change just the one security line to security=user and it works for multiple users. Unfortunately, we need it to work with ADS authentication... Created attachment 328750 [details]
Samba server smb.conf
Here is the smb.conf from the server that uses ADS authentication. The server is RHEL5 with samba samba-3.0.28-1.el5_2.1 and running kernel 2.6.18-92.1.22.el5PAE.
I also just tried accessing this share as two different users using smbclient instead of a cifs mount and it worked. For example, I did the following for two different users simultaneously: smbclient -W ADS.IU.EDU -U user //curie.cs.indiana.edu/user I was able to login as two different users using ADS authentication and access their home directories simultaneously. So, it seems like the problem is limited to cifs mounts. Rob, can you raise the log level to 10 on your samba server and attach a log of what happens when you try the first and then the second mount? Maybe that will give us a clue at what is going wrong here. There's a big difference with 2 smbclient sessions. They use independent sockets. CIFS will share the socket to the server and just have separate smb sessions for each user. The problem is probably related to that, but it's not immediately clear to me. Created attachment 328918 [details]
samba server log file
Attached is the smbd.log file from the server with log level 10. Here are the times of the events in that log:
16:15 The smb service was started on the server named curie
16:16 The first (successful) mount was done from the client named test with:
mount //curie.cs.indiana.edu/shei /mnt/shei -o domain=ADS,user=shei,sec=ntlmv2
16:17 The second (unsuccessful) mount was done from the same workstation with:
mount //curie.cs.indiana.edu/robh /mnt/robh -o domain=ADS,user=robh,sec=ntlmv2
This failed with:
mount error 13 = Permission denied
16:18 The same mount as user robh was repeated and it failed again
16:20 /mnt/shei was umount'ed on test
16:21 The same mount as user robh was repeated and succeeded this time
16:22 /mnt/robh was umount'ed on test
16:23 The smb service was shut down on curie
Reassigning to GD for now since it looks like this is probably a samba bug. To reproduce, you'll need a samba server set up with security = ADS. The server should have 2 users configured. mount a share from the server as one user: # mount -t cifs -o user=user1 //sambaserver/share /mnt/cifs1 ...that should work. Now mount the share as another user: # mount -t cifs -o user=user2 //sambaserver/share /mnt/cifs2 ...that should fail. When the server is set up with security = user, apparently this works. The key here I think is that CIFS shares the TCP socket between multiple mounts. It looks like the second session setup call on the socket is failing when security = ADS. Rob, please correct me here if I'm missing anything. Jeff, your description is right on. But, I just want to make it clear that the behavior changes depending on the version of cifs being used. Case 1: When using the stock 2.6.18-92.1.22.el5 kernel with cifs version 1.50cRH on the client the second mount will actually succeed and work but the first mount goes into an I/O error state when you try to access it. Case 2: When using your development 2.6.18-128.el5.jtltest.59PAE kernel with cifs version 1.56RH the second mount always fails with "mount error 13 = Permission denied" but the first mount continues to work without any problem. Thanks! (In reply to comment #14) > Jeff, your description is right on. But, I just want to make it clear that the > behavior changes depending on the version of cifs being used. > > Case 1: When using the stock 2.6.18-92.1.22.el5 kernel with cifs version > 1.50cRH on the client the second mount will actually succeed and work but the > first mount goes into an I/O error state when you try to access it. I'm afraid I need to make a correction here. With the stock rhel5 kernel the first mount as user1 succeeds, the second mount as user2 fails with permission denied errors and also kills the first mount (it disappears from the df output and gives I/O errors on access), and a repeat of the second mount as user2 then succeeds. Yes the older kernel has a bug, the newer does not, but reveals a samba bug. Testing to see if it is an 3.0.28 bug or if it is also present in the newer 3.0.33 packages we built. I just wanted to let you know that there is no change in this problem when we upgraded the samba server to rhel 5.3 (2.6.18-128.el5 kernel with samba 3.0.33-3.7.el5). When the client is running the new 5.3 kernel (2.6.18-128.el5) we still get I/O errors after the second mount and when running Jeff's 2.6.18-128.el5.jtltest.59 kernel the second mount always gives permission denied errors. So, if the bug is in samba I'm afraid the upgrade to 3.0.33 didn't help. FYI, I tried installing samba 3.2.7 on a test rhel5.3 samba server and I got the same behavior. I'm not sure what else to try from this end so please let me know if there is anything I can do to help debug this. As an additional test, I tried the same thing with an ubuntu client and ubuntu samba server and can reproduce the problem there as well. So, it doesn't appear to be rhel-specific. Would it be more appropriate for me to post this in the samba.org bugzilla instead of here? I think it would make sense to open a samba bug upstream as well. If you do it would also be useful if you could cross reference the bugs (ie post a link to this bug in the upstream one, and vice versa) Ok, it looks like there was a bug in samba and in the kernel relating to this. I just pushed a kernel patch upstream to fix this there: http://lists.samba.org/archive/linux-cifs-client/2010-June/006293.html I'll plan to incorporate this into my test kernels. We also need to clone this bug for RHEL4 and RHEL6. in kernel-2.6.18-210.el5 You can download this test kernel from http://people.redhat.com/jwilson/el5 Detailed testing feedback is always welcomed. I configure krb4 domain realm with REDHAT.COM, then join Samba to the domain which need Administrator password (# net ads join -U Administrator). Hard to reproduce, so just do code review. the patch linux-2.6-fs-cifs-remove-bogus-check-in-ntlm-session-setup-code.patch is included with the new version kernel-2.6.18-236.el5.src.rpm and applies cleanly. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0017.html |
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.5) Gecko/2008120908 Red Hat/3.0.5-1.el5_2 Firefox/3.0.5 We are using cifs mounts on RHEL5 workstations to access user home directories on RHEL4 and RHEL5 samba servers. As long as there is only one user on the workstation, everything works fine. When a second user logs in and the home directory is mounted (via pam_mount) it steps on the first mount and all subsequent access of the original mount point result in I/O errors. This is easily repeatable by just running two mounts independent of pam_mount. Setting /proc/fs/cifs/MultiuserMount to 1 seems not to help. Reproducible: Always Steps to Reproduce: 1. Do the first mount for user1 with something like: mount -t cifs //server.name/user1 /mnt/user1 -o user=user1 2. Do the second mount similarly for user2 mount -t cifs //server.name/user2 /mnt/user2 -o user=user2 Actual Results: Any access of /mnt/user1 after the second mount will result in Input/Output errors. Attempting to umount things at this point will typically result in the system hanging or kernel panicing. Expected Results: Access to both mount points should work. I've tested this with both RHEL4, RHEL5, and Ubuntu clients as well as RHEL4, RHEL5, and Windows 2003 (R2) servers and they all fail in the same way. I'll attach a transcript showing a demonstration of the problem.