Bug 670511

Created attachment 474059 [details]
sftp without sssd log

Created attachment 474060 [details]
sftp with sssd log

Created attachment 474061 [details]
sshd without sssd log

Created attachment 474062 [details]
sshd with sssd log

Could you please include the output of /var/log/secure during these connections with SSSD as well?

Also, if you could set your debug_level = 9 in /etc/sssd/sssd.conf and tar.gz /var/log/sssd/* and attach it to the ticket (after sanitizing), that would be helpful too.

Created attachment 474279 [details]
Logs from sssd with debug=9

This is the output in /var/log/secure during the transactions:


Jan 19 09:17:54 alt-uxsupport01 sshd[14208]: Accepted publickey for bwacq-st from 194.213.160.231 port 33045 ssh2
Jan 19 09:17:54 alt-uxsupport01 sshd[14208]: pam_unix(sshd:session): session opened for user bwacq-st by (uid=0)
Jan 19 09:17:54 alt-uxsupport01 sshd[14211]: subsystem request for sftp
Jan 19 09:17:56 alt-uxsupport01 sshd[14211]: Received disconnect from 194.213.160.231: 11: disconnected by user
Jan 19 09:17:56 alt-uxsupport01 sshd[14208]: pam_unix(sshd:session): session closed for user bwacq-st

(In reply to comment #8)
> Created attachment 474279 [details]
> Logs from sssd with debug=9

According to these logs, SSSD failed to start correctly. Can you please include your sssd.conf as well?

Created attachment 474482 [details]
sssd configuration file

This is my current sssd.conf, without the debug=9 attribute used while generating the logs.

Ok, nothing looks out of the ordinary with that config. Could you please regenerate the logs you submitted, but put debug_level=9 in the [sssd] section, rather than the [domain/edbapptest] section?

That will set the debug_level for all components and hopefully will show why the child processes are dying with:

(Wed Jan 19 09:16:32 2011) [sssd[be[edbapptest]]] [id_callback] (0): The Monitor returned an error [org.freedesktop.DBus.Error.NoReply]

The logs were generated with debug=9 under [sssd], but I'll run the test again when I get to the office tomorrow.

(In reply to comment #13)
> The logs were generated with debug=9 under [sssd], but I'll run the test again
> when I get to the office tomorrow.

AH, that's probably the problem. It has to be debug_level = 9, not debug=9.

The logs you attached were consistent with debug_level = 0 (which is the default if it's not specified)

This probably doesn't help much, but  I'm unable to reproduce this here.  I've done a quick-n-dirty setup on Fedora 14 as I have it setup for LDAP authentication (I'm assuming that may be part of the puzzle because the bug is filed against 389-base):

# rpm -q openssh sssd
openssh-5.5p1-24.fc14.2.x86_64
sssd-1.5.0-1.fc14.x86_64

Versions are newer than what we have in RHEL6; given a bit of time to setup my RHEL6 VM I can test on there as well to see if I can reproduce it.

I cannot connect as my chosen user when SSSD is not running because the auth cannot proceed (not a local user), and when SSSD is running I get a directory listing, but it shows numeric uid/gid, not names (not sure if that is intentional, but it's not a big deal).

My sshd_config has:

Match Group vdanen
  X11Forwarding no
  AllowTcpForwarding no
  ForceCommand internal-sftp
  ChrootDirectory /jails/%u

And I created /jails/vdanen as root (so owned root:root).

Although I do wonder  about the source of the problem.  You haven't indicated what client you're using to connect, which might be important due to this bit in the logs:

Jan 19 09:17:56 alt-uxsupport01 sshd[14211]: Received disconnect from 194.213.160.231: 11: disconnected by user

That I think is more important than the sssd bits, because my /var/log/sssd/sssd.log shows a similar error message:

(Thu Jan 20 16:06:48 2011) [sssd] [monitor_quit] (0): Monitor received Terminated: terminating children

While that obviously isn't right, I think that is coincidental to the problem (or may be, at least -- will know more once I can test RHEL6 hopefully later tonight).

I do think the component this should be filed against should be either sssd or openssh though.

Also, I don't believe this is a security flaw, so it is incorrectly tagged with the Security keyword.  There is no breach of trust or authentication (i.e. someone can login who shouldn't be allowed), even if it's a pain that a proper user can't login, I think that would be considered a (fairly severe) bug, but not a security flaw.

I ran the same test with "debug = 9" now and got the same output.

I've been using the openssh sftp-client shipping with ubuntu for the test, 5.3p1-3ubuntu3. I tested now from a rhel5.2 server with the same private key as on my ubuntu workstation towards the rhel6 server and got the exact same behavior. I.E., when sssd is running I get "connection closed" when requesting directory listing and when sssd isn't running I get the contents correctly. The 5.2 server is using openssh-4.3p2-26.el5. 

The bwacq-st user I try to connect with is a local user. Interestingly enough, authentication with LDAP actually works with that configuration. When sssd is running, I can login with an LDAP-stored user. When it isn't running, I can't. So configuration seems to work.

Ok, I'm going to try to summarize what I've read here, then go and see if I can reproduce it myself.

You are running SSSD to handle LDAP users, which works just fine with ssh. However, when you attempt to log in with a non-SSSD user (/etc/passwd-backed) to the system, using a private SSH key, you are granted access, but attempting to list a directory causes a client error that disconnects your SSH session.

Is this a correct summary?


p.s. Vincent: the message "[monitor_quit] (0): Monitor received Terminated: terminating children" is not a bug. It's telling you that at some point you shut down the SSSD the clean way (by sending SIGTERM to the 'sssd' process, which then in turn signals all of its children to exit before shutting itself down).

Yes, the summary is correct. Please let me know if you need further debugging information, as I'm quite reliant on sftp-jails and pubkeys :)

changing component to sssd

Can you please check your /etc/nsswitch.conf file? I've been able to reproduce your issue, but only when the nsswitch.conf is configured incorrectly (meaning that the passwd and group lines list 'sss' before 'files'.

If this is the case, please move 'files' before 'sss'.


I suspect that you're operating under a mistaken impression that the 'local' provider deals with /etc/passwd. It does not, it's SSSD's local database, which is separate from /etc/passwd.

/etc/nsswitch.conf has:
passwd:     files sss
shadow:     files sss
group:      files sss

It looks like it has the standard el6 config. 

Yeah, I thought local would deal with the normal authentication mechanism. I can remove that whole section, but I'm quite sure I've already tested that.

Can you describe more what your chroot environment setup looks like? I set mine up according to: http://www.fuschlberger.net/programs/ssh-scp-sftp-chroot-jail/

One thing I noticed above, you have a chroot directory set up as
ChrootDirectory /jails/%u

How is this chroot directory laid out?
Does it have /jails/username/home/username and /jails/username/bin etc?

It doesn't use an environment like that at all. This jail is sftp-only, so it uses the internal sftp in sshd, which doesn't make it reliant on the character devices, shell, proc etc which a ssh jail would need set up.

The chroot is /jails/username/. Here's the permissions:
[root@alt-uxsupport01 /]# ls -sl
4 drwx--x--x.   4 root root  4096 Jan 18 14:23 jails
[root@alt-uxsupport01 jails]# ls -sl
4 drwxr-x---. 4 root bwacq-st 4096 Jan  7 10:46 bwacq-st

We are still waiting for the debug logs with debug_level = 9 from SSSD.

I've been able to reproduce this finally. I think what's happening here (at least in part) is that the chroot environment has no access to the file descriptor that SSSD uses to perform identity lookups (it's in /var/lib/sss/pipes)

However, in all of my tests on the sss_client code, we're returning the appropriate error code (NSS_STATUS_UNAVAIL) when we can't open the socket, so there's no reason that the internal-sftp process should be choking on it.

I'm still investigating, but I'm starting to suspect this may be an openssh bug, not an SSSD bug.


(In reply to comment #24)
> We are still waiting for the debug logs with debug_level = 9 from SSSD.
Having reproduced the problem, I can verify that there ARE no debug logs applicable here, because the client cannot talk to the responders.

I believe this is really an SSH bug, as other system libraries may have open sockets for whatever reason.

However we are working on a workaround so that broken applications like this do not suffer.

See ticket: https://fedorahosted.org/sssd/ticket/790

In version 1.2.1-28:

# sftp user5@localhost
Connecting to localhost...
sftp> ls
Connection closed


In version 1.5.1-24:

# sftp user5@localhost
Connecting to localhost...
sftp> ls
1  2  3  a  b  c  
sftp>

The connection doesn't get closed now. I am able to list the directories after starting the sssd service.

Version:
# rpm -qi sssd | head
Name        : sssd                         Relocations: (not relocatable)
Version     : 1.5.1                             Vendor: Red Hat, Inc.
Release     : 24.el6                        Build Date: Sat 02 Apr 2011 01:24:54 AM IST
Install Date: Wed 06 Apr 2011 07:17:24 PM IST      Build Host: x86-012.build.bos.redhat.com
Group       : Applications/System           Source RPM: sssd-1.5.1-24.el6.src.rpm
Size        : 3462740                          License: GPLv3+
Signature   : (none)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
URL         : http://fedorahosted.org/sssd/
Summary     : System Security Services Daemon

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0560.html

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0560.html