Bug 1313041 - ssh with sssd proxy fails with "Connection closed by remote host" if locale not available
Summary: ssh with sssd proxy fails with "Connection closed by remote host" if locale n...
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: sssd
Version: rawhide
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Jakub Hrozek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-02-29 19:31 UTC by Adam Williamson
Modified: 2016-03-30 21:19 UTC (History)
15 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2016-03-30 20:52:54 UTC


Attachments (Terms of Use)

Description Adam Williamson 2016-02-29 19:31:33 UTC
I hit this after the glibc locale split on Rawhide; the langpack for my locale was not installed, and ssh would fail to connect with "ssh_exchange_identification: Connection closed by remote host". Installing glibc-langpack-en solved the problem.

"LC_ALL=C ssh" works fine, but you can reproduce the bug with LC_ALL=(some locale that isn't installed), e.g.:

[adamw@adam comps (master %)]$ LC_ALL=de_DE.UTF-8 ssh www
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (de_DE.UTF-8)
ssh_exchange_identification: Connection closed by remote host
[adamw@adam comps (master %)]$ ssh www
Last login: Mon Feb 29 08:16:59 2016 from 192.168.1.5

I do not have the de langpack installed.

This is really on Rawhide, not 24; I haven't checked yet if F24 has the same bug.

Comment 1 Dennis Gilmore 2016-02-29 19:43:26 UTC
LC_ALL=de_DE.UTF-8 ssh hathor.ausil.us 
Last login: Thu Feb 11 20:43:39 2016 from 2607:ff50::ff6
-bash: warning: setlocale: LC_ALL: cannot change locale (de_DE.UTF-8)
-bash: warning: setlocale: LC_ALL: cannot change locale (de_DE.UTF-8)
/bin/sh: warning: setlocale: LC_ALL: cannot change locale (de_DE.UTF-8)
/bin/sh: warning: setlocale: LC_ALL: cannot change locale (de_DE.UTF-8)
/bin/sh: warning: setlocale: LC_ALL: cannot change locale (de_DE.UTF-8)
[dennis@hathor ~]$ 

ssh is noisy but works in f24

Comment 2 Dennis Gilmore 2016-02-29 19:44:45 UTC
too quick

[dennis@anubis ~]$ LC_ALL=de_DE.UTF-8 ssh hathor.ausil.us
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (de_DE.UTF-8)
ssh_exchange_identification: Connection closed by remote host

from f23 to f24 works but from f24 to f24 fails

Comment 3 Adam Williamson 2016-02-29 19:46:51 UTC
My case is 24 to 23, it seems the issue is on the client end.

Comment 4 Jakub Jelen 2016-03-01 09:24:49 UTC
4 comments from 2 important Fedora figures here, but no info about package version, no verbose log, noting. Sight ... 
but I guess it is somehow related to yesterday rebase to openssh-7.2p1.

I set up some f24 machine, but I can't reproduce your behaviour. SSH is noisy with bad locale, but no sign of failure. Can you please update the report at least with verbose ssh log (`-vvv`)?

Comment 5 Florian Weimer 2016-03-01 11:06:55 UTC
(In reply to Jakub Jelen from comment #4)
> 4 comments from 2 important Fedora figures here, but no info about package
> version, no verbose log, noting. Sight ... 
> but I guess it is somehow related to yesterday rebase to openssh-7.2p1.

It could also be a PAM or NSS module which is at fault.  The situation of SSH'ing to a host which does not support the locale passed through with AcceptEnv environment variables has seen ample testing with Debian servers at least (which only have a subset of locales, typically, and the subset will not match the client locales at least some of the time).

Comment 6 Adam Williamson 2016-03-01 15:57:15 UTC
sorry, I've been in a hurry trying to get things vaguely working the last couple of days and I figured since the bug seemed trivially reproducible you could get the details yourself, but apparently it isn't.

So a couple of missing bits:

[adamw@adam ~]$ LC_ALL=de_DE.UTF-8 ssh -vvvvv www
OpenSSH_7.1p2, OpenSSL 1.0.2f-fips  28 Jan 2016
debug1: Reading configuration data /home/adamw/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 www
debug1: permanently_drop_suid: 1001
debug1: identity file /home/adamw/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/adamw/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/adamw/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/adamw/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/adamw/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/adamw/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/adamw/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/adamw/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.1
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (de_DE.UTF-8)
ssh_exchange_identification: Connection closed by remote host
[adamw@adam ~]$ rpm -q openssh-clients
openssh-clients-7.1p2-4.fc25.x86_64

Most importantly, however: I'm using FreeIPA, which means I get the FreeIPA ssh proxy, which I'd forgotten about. /etc/ssh/config has these lines:

GlobalKnownHostsFile /var/lib/sss/pubconf/known_hosts
PubkeyAuthentication yes
ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h

Host *
        GSSAPIAuthentication yes

if I comment them out, it works fine. Thus this is sssd's fault, it appears.

sssd-common-1.13.3-5.fc25.x86_64

Comment 7 Jakub Hrozek 2016-03-01 16:09:19 UTC
This was fixed in upstream, but not backported to Fedora

Comment 8 Jakub Hrozek 2016-03-01 16:10:47 UTC
Upstream ticket:
https://fedorahosted.org/sssd/ticket/2785

Comment 9 Lukas Slebodnik 2016-03-01 16:36:34 UTC
(In reply to Jakub Hrozek from comment #7)
> This was fixed in upstream, but not backported to Fedora

I'm not sure it is issue in sssd.
I recently had a problem with locales on my fedora 24.
glibc-2.23.1-4.fc24 caused issues on my laptop.
glibc-2.23.1-5.fc24 works fine with extra installed glibc-langpack-en-2.23.1-5.fc24.x86_64.rpm.

Which version of glibc do you have?

Comment 10 Adam Williamson 2016-03-01 16:41:53 UTC
We know the glibc update is involved. The bug happens when the configured locale is not installed. That's more likely to happen now that locales have been split out in glibc. But the code *should not* crash in that case anyway. It's entirely avoidable and the situation should be handled better.

Comment 11 Jakub Hrozek 2016-03-01 16:52:05 UTC
Can you test this build please? http://koji.fedoraproject.org/koji/taskinfo?taskID=13190150

Comment 12 Lukas Slebodnik 2016-03-01 21:00:00 UTC
(In reply to Adam Williamson from comment #10)
> We know the glibc update is involved. The bug happens when the configured
> locale is not installed. That's more likely to happen now that locales have
> been split out in glibc. But the code *should not* crash in that case
> anyway.
It did not crash. It just failed.

> It's entirely avoidable and the situation should be handled better.
The upgrade of glibc *should not* create issues in other projects.
We might fix sssd with backport of https://fedorahosted.org/sssd/ticket/2785
But such change in glibc will affect many packages. And not just default packages in fedora but also other packages of from non-standard repositories.

Comment 13 Adam Williamson 2016-03-01 21:08:06 UTC
"It did not crash. It just failed." OK, true.

"The upgrade of glibc *should not* create issues in other projects."

It did no such thing. The issue always existed. The glibc upgrade just *exposed* it. The issue is "the proxy crashes when run with a locale configured that is not installed". That issue has been present all along. The glibc upgrade just made it more likely that someone would encounter it.

We know the glibc change affects many things, and the glibc package is being adjusted a bit as well, but it does not mean the underlying issues are invalid. Most things should cope gracefully with a non-existent locale.

Comment 14 Lukas Slebodnik 2016-03-01 21:18:29 UTC
(In reply to Adam Williamson from comment #13)
> We know the glibc change affects many things, and the glibc package is being
> adjusted a bit as well,
Good to know.

> but it does not mean the underlying issues are
> invalid. Most things should cope gracefully with a non-existent locale.
Fair enough.

Do you have any progress with testing scratch build?
@see Comment 11

Comment 15 Adam Williamson 2016-03-01 21:47:51 UTC
sorry, I got busy. let me grab it now and give it a shot...

..yep, that seems to work. thanks!

Comment 16 Fedora Update System 2016-03-22 09:49:02 UTC
sssd-1.13.3-6.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-319ed373cc

Comment 17 Fedora Update System 2016-03-22 09:49:32 UTC
sssd-1.13.3-6.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-526233c4aa

Comment 18 Fedora Update System 2016-03-22 09:50:01 UTC
sssd-1.13.3-6.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2016-dea583aee4

Comment 19 Fedora Update System 2016-03-22 16:54:54 UTC
sssd-1.13.3-6.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-319ed373cc

Comment 20 Fedora Update System 2016-03-22 21:25:42 UTC
sssd-1.13.3-6.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-dea583aee4

Comment 21 Fedora Update System 2016-03-22 21:31:10 UTC
sssd-1.13.3-6.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-526233c4aa

Comment 22 Fedora Update System 2016-03-26 17:54:52 UTC
sssd-1.13.3-6.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 23 Fedora Update System 2016-03-30 20:52:43 UTC
sssd-1.13.3-6.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 24 Fedora Update System 2016-03-30 21:19:11 UTC
sssd-1.13.3-6.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.