Bug 670511 - SSSD and sftp-only jailed users with pubkey login
SSSD and sftp-only jailed users with pubkey login
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: sssd (Show other bugs)
6.0
x86_64 Linux
low Severity medium
: rc
: ---
Assigned To: Stephen Gallagher
Chandrasekar Kannan
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-01-18 09:30 EST by kriberg
Modified: 2015-01-04 18:45 EST (History)
12 users (show)

See Also:
Fixed In Version: sssd-1.5.1-6.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-05-19 07:42:10 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
sftp without sssd log (11.73 KB, text/plain)
2011-01-18 09:31 EST, kriberg
no flags Details
sftp with sssd log (16.79 KB, text/plain)
2011-01-18 09:32 EST, kriberg
no flags Details
sshd without sssd log (17.53 KB, text/plain)
2011-01-18 09:32 EST, kriberg
no flags Details
sshd with sssd log (16.79 KB, text/plain)
2011-01-18 09:33 EST, kriberg
no flags Details
Logs from sssd with debug=9 (328 bytes, application/x-gzip)
2011-01-19 09:29 EST, kriberg
no flags Details
sssd configuration file (4.61 KB, text/plain)
2011-01-20 10:49 EST, kriberg
no flags Details

  None (edit)
Description kriberg 2011-01-18 09:30:57 EST
Description of problem:
I've created a user and made it part of the group "jailed". Home directory is set to /jails/user and sshd_config has the following entry:
Match Group jailed
	X11Forwarding no
	AllowTcpForwarding no
	ForceCommand internal-sftp
	ChrootDirectory /jails/%u

and

Subsystem	sftp	internal-sftp

The purpose is having the user able to log in only with SFTP and be jailed to the home directory. This works flawlessly. But when SSSD is running, the SFTP connection suddenly closes when the client requests a directory listing. If I shut down the sssd service, I get the correct behavior, I.E., my SFTP client lists the directory contents. 


Version-Release number of selected component (if applicable):
Name       : sssd
Arch       : x86_64
Version    : 1.2.1
Release    : 28.el6

Name       : openssh-server
Arch       : x86_64
Version    : 5.3p1
Release    : 20.el6.3


How reproducible:
create a user and a home directory according to sshd's requirements for jailing users. Upload a public key from a separate client. Stop the sssd service and connect with sftp to the server. do 'ls' in the sftp client and you should get directory listing from the server. disconnect sftp. start sssd on server. connect with sftp again and request directory listing. sftp client will close immediately 

  
Actual results:
SFTP client exits and reports "connection closed"

Expected results:
Directory listing shown in SFTP client.


Additional info:
I've attatched four logs.
Output from sshd with sssd running.
Output from sshd without sssd running
Output from sftp with sssd running.
Output from sftp without sssd running
Comment 1 kriberg 2011-01-18 09:31:49 EST
Created attachment 474059 [details]
sftp without sssd log
Comment 2 kriberg 2011-01-18 09:32:09 EST
Created attachment 474060 [details]
sftp with sssd log
Comment 3 kriberg 2011-01-18 09:32:32 EST
Created attachment 474061 [details]
sshd without sssd log
Comment 4 kriberg 2011-01-18 09:33:10 EST
Created attachment 474062 [details]
sshd with sssd log
Comment 6 Stephen Gallagher 2011-01-18 10:58:57 EST
Could you please include the output of /var/log/secure during these connections with SSSD as well?
Comment 7 Stephen Gallagher 2011-01-18 11:34:34 EST
Also, if you could set your debug_level = 9 in /etc/sssd/sssd.conf and tar.gz /var/log/sssd/* and attach it to the ticket (after sanitizing), that would be helpful too.
Comment 8 kriberg 2011-01-19 09:29:14 EST
Created attachment 474279 [details]
Logs from sssd with debug=9
Comment 9 kriberg 2011-01-19 09:30:43 EST
This is the output in /var/log/secure during the transactions:


Jan 19 09:17:54 alt-uxsupport01 sshd[14208]: Accepted publickey for bwacq-st from 194.213.160.231 port 33045 ssh2
Jan 19 09:17:54 alt-uxsupport01 sshd[14208]: pam_unix(sshd:session): session opened for user bwacq-st by (uid=0)
Jan 19 09:17:54 alt-uxsupport01 sshd[14211]: subsystem request for sftp
Jan 19 09:17:56 alt-uxsupport01 sshd[14211]: Received disconnect from 194.213.160.231: 11: disconnected by user
Jan 19 09:17:56 alt-uxsupport01 sshd[14208]: pam_unix(sshd:session): session closed for user bwacq-st
Comment 10 Stephen Gallagher 2011-01-19 09:38:19 EST
(In reply to comment #8)
> Created attachment 474279 [details]
> Logs from sssd with debug=9

According to these logs, SSSD failed to start correctly. Can you please include your sssd.conf as well?
Comment 11 kriberg 2011-01-20 10:49:35 EST
Created attachment 474482 [details]
sssd configuration file

This is my current sssd.conf, without the debug=9 attribute used while generating the logs.
Comment 12 Stephen Gallagher 2011-01-20 11:11:12 EST
Ok, nothing looks out of the ordinary with that config. Could you please regenerate the logs you submitted, but put debug_level=9 in the [sssd] section, rather than the [domain/edbapptest] section?

That will set the debug_level for all components and hopefully will show why the child processes are dying with:

(Wed Jan 19 09:16:32 2011) [sssd[be[edbapptest]]] [id_callback] (0): The Monitor returned an error [org.freedesktop.DBus.Error.NoReply]
Comment 13 kriberg 2011-01-20 11:27:19 EST
The logs were generated with debug=9 under [sssd], but I'll run the test again when I get to the office tomorrow.
Comment 14 Stephen Gallagher 2011-01-20 11:36:03 EST
(In reply to comment #13)
> The logs were generated with debug=9 under [sssd], but I'll run the test again
> when I get to the office tomorrow.

AH, that's probably the problem. It has to be debug_level = 9, not debug=9.

The logs you attached were consistent with debug_level = 0 (which is the default if it's not specified)
Comment 15 Vincent Danen 2011-01-20 18:14:40 EST
This probably doesn't help much, but  I'm unable to reproduce this here.  I've done a quick-n-dirty setup on Fedora 14 as I have it setup for LDAP authentication (I'm assuming that may be part of the puzzle because the bug is filed against 389-base):

# rpm -q openssh sssd
openssh-5.5p1-24.fc14.2.x86_64
sssd-1.5.0-1.fc14.x86_64

Versions are newer than what we have in RHEL6; given a bit of time to setup my RHEL6 VM I can test on there as well to see if I can reproduce it.

I cannot connect as my chosen user when SSSD is not running because the auth cannot proceed (not a local user), and when SSSD is running I get a directory listing, but it shows numeric uid/gid, not names (not sure if that is intentional, but it's not a big deal).

My sshd_config has:

Match Group vdanen
  X11Forwarding no
  AllowTcpForwarding no
  ForceCommand internal-sftp
  ChrootDirectory /jails/%u

And I created /jails/vdanen as root (so owned root:root).

Although I do wonder  about the source of the problem.  You haven't indicated what client you're using to connect, which might be important due to this bit in the logs:

Jan 19 09:17:56 alt-uxsupport01 sshd[14211]: Received disconnect from 194.213.160.231: 11: disconnected by user

That I think is more important than the sssd bits, because my /var/log/sssd/sssd.log shows a similar error message:

(Thu Jan 20 16:06:48 2011) [sssd] [monitor_quit] (0): Monitor received Terminated: terminating children

While that obviously isn't right, I think that is coincidental to the problem (or may be, at least -- will know more once I can test RHEL6 hopefully later tonight).

I do think the component this should be filed against should be either sssd or openssh though.

Also, I don't believe this is a security flaw, so it is incorrectly tagged with the Security keyword.  There is no breach of trust or authentication (i.e. someone can login who shouldn't be allowed), even if it's a pain that a proper user can't login, I think that would be considered a (fairly severe) bug, but not a security flaw.
Comment 16 kriberg 2011-01-21 04:24:46 EST
I ran the same test with "debug = 9" now and got the same output.

I've been using the openssh sftp-client shipping with ubuntu for the test, 5.3p1-3ubuntu3. I tested now from a rhel5.2 server with the same private key as on my ubuntu workstation towards the rhel6 server and got the exact same behavior. I.E., when sssd is running I get "connection closed" when requesting directory listing and when sssd isn't running I get the contents correctly. The 5.2 server is using openssh-4.3p2-26.el5. 

The bwacq-st user I try to connect with is a local user. Interestingly enough, authentication with LDAP actually works with that configuration. When sssd is running, I can login with an LDAP-stored user. When it isn't running, I can't. So configuration seems to work.
Comment 17 Stephen Gallagher 2011-01-21 08:03:57 EST
Ok, I'm going to try to summarize what I've read here, then go and see if I can reproduce it myself.

You are running SSSD to handle LDAP users, which works just fine with ssh. However, when you attempt to log in with a non-SSSD user (/etc/passwd-backed) to the system, using a private SSH key, you are granted access, but attempting to list a directory causes a client error that disconnects your SSH session.

Is this a correct summary?


p.s. Vincent: the message "[monitor_quit] (0): Monitor received Terminated: terminating children" is not a bug. It's telling you that at some point you shut down the SSSD the clean way (by sending SIGTERM to the 'sssd' process, which then in turn signals all of its children to exit before shutting itself down).
Comment 18 kriberg 2011-01-21 09:01:05 EST
Yes, the summary is correct. Please let me know if you need further debugging information, as I'm quite reliant on sftp-jails and pubkeys :)
Comment 19 Chandrasekar Kannan 2011-01-24 17:47:26 EST
changing component to sssd
Comment 20 Stephen Gallagher 2011-01-26 09:50:53 EST
Can you please check your /etc/nsswitch.conf file? I've been able to reproduce your issue, but only when the nsswitch.conf is configured incorrectly (meaning that the passwd and group lines list 'sss' before 'files'.

If this is the case, please move 'files' before 'sss'.


I suspect that you're operating under a mistaken impression that the 'local' provider deals with /etc/passwd. It does not, it's SSSD's local database, which is separate from /etc/passwd.
Comment 21 kriberg 2011-01-31 04:52:37 EST
/etc/nsswitch.conf has:
passwd:     files sss
shadow:     files sss
group:      files sss

It looks like it has the standard el6 config. 

Yeah, I thought local would deal with the normal authentication mechanism. I can remove that whole section, but I'm quite sure I've already tested that.
Comment 22 Stephen Gallagher 2011-01-31 13:07:23 EST
Can you describe more what your chroot environment setup looks like? I set mine up according to: http://www.fuschlberger.net/programs/ssh-scp-sftp-chroot-jail/

One thing I noticed above, you have a chroot directory set up as
ChrootDirectory /jails/%u

How is this chroot directory laid out?
Does it have /jails/username/home/username and /jails/username/bin etc?
Comment 23 kriberg 2011-02-01 02:40:18 EST
It doesn't use an environment like that at all. This jail is sftp-only, so it uses the internal sftp in sshd, which doesn't make it reliant on the character devices, shell, proc etc which a ssh jail would need set up.

The chroot is /jails/username/. Here's the permissions:
[root@alt-uxsupport01 /]# ls -sl
4 drwx--x--x.   4 root root  4096 Jan 18 14:23 jails
[root@alt-uxsupport01 jails]# ls -sl
4 drwxr-x---. 4 root bwacq-st 4096 Jan  7 10:46 bwacq-st
Comment 24 Dmitri Pal 2011-02-03 16:54:48 EST
We are still waiting for the debug logs with debug_level = 9 from SSSD.
Comment 25 Stephen Gallagher 2011-02-04 09:44:15 EST
I've been able to reproduce this finally. I think what's happening here (at least in part) is that the chroot environment has no access to the file descriptor that SSSD uses to perform identity lookups (it's in /var/lib/sss/pipes)

However, in all of my tests on the sss_client code, we're returning the appropriate error code (NSS_STATUS_UNAVAIL) when we can't open the socket, so there's no reason that the internal-sftp process should be choking on it.

I'm still investigating, but I'm starting to suspect this may be an openssh bug, not an SSSD bug.


(In reply to comment #24)
> We are still waiting for the debug logs with debug_level = 9 from SSSD.
Having reproduced the problem, I can verify that there ARE no debug logs applicable here, because the client cannot talk to the responders.
Comment 27 Simo Sorce 2011-02-07 09:49:36 EST
I believe this is really an SSH bug, as other system libraries may have open sockets for whatever reason.

However we are working on a workaround so that broken applications like this do not suffer.

See ticket: https://fedorahosted.org/sssd/ticket/790
Comment 30 Kaushik Banerjee 2011-04-11 08:55:37 EDT
In version 1.2.1-28:

# sftp user5@localhost
Connecting to localhost...
sftp> ls
Connection closed


In version 1.5.1-24:

# sftp user5@localhost
Connecting to localhost...
sftp> ls
1  2  3  a  b  c  
sftp>

The connection doesn't get closed now. I am able to list the directories after starting the sssd service.

Version:
# rpm -qi sssd | head
Name        : sssd                         Relocations: (not relocatable)
Version     : 1.5.1                             Vendor: Red Hat, Inc.
Release     : 24.el6                        Build Date: Sat 02 Apr 2011 01:24:54 AM IST
Install Date: Wed 06 Apr 2011 07:17:24 PM IST      Build Host: x86-012.build.bos.redhat.com
Group       : Applications/System           Source RPM: sssd-1.5.1-24.el6.src.rpm
Size        : 3462740                          License: GPLv3+
Signature   : (none)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
URL         : http://fedorahosted.org/sssd/
Summary     : System Security Services Daemon
Comment 31 errata-xmlrpc 2011-05-19 07:42:10 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0560.html
Comment 32 errata-xmlrpc 2011-05-19 09:09:09 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0560.html

Note You need to log in before you can comment on or make changes to this bug.