Bug 101157 - Errata packages cause slow login and bogus log entry
Errata packages cause slow login and bogus log entry
Status: CLOSED NOTABUG
Product: Red Hat Linux
Classification: Retired
Component: openssh (Show other bugs)
9
i386 Linux
high Severity medium
: ---
: ---
Assigned To: Nalin Dahyabhai
Brian Brock
:
: 101252 101662 102174 103935 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2003-07-29 14:07 EDT by Jordan Russell
Modified: 2007-03-27 00:08 EDT (History)
23 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2003-08-05 02:27:50 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Updated openssh-3.1p1-skip-initial.patch file (1.18 KB, patch)
2003-09-24 16:48 EDT, John Caruso
no flags Details | Diff
Updated openssh-3.5p1-skip-initial.patch file (1.20 KB, patch)
2003-09-24 16:49 EDT, John Caruso
no flags Details | Diff

  None (edit)
Description Jordan Russell 2003-07-29 14:07:21 EDT
Description of problem:
On my Red Hat Linux 9.0 boxes, I've found that after upgrading them to openssh-
3.5p1-6.9 (from openssh-3.5p1-6), logins take much longer than before, and a 
bogus log entry is created for each successful login.

Version-Release number of selected component (if applicable):
openssh-3.5p1-6.9

How reproducible:
Always

Steps to Reproduce:
To see the "slow login" problem, try the 'time' command before and after 
upgrading the packages.

--- BEFORE UPGRADING REMOTE MACHINE ---

# time ssh machine.name 'echo test'
test

real    0m0.295s
user    0m0.100s
sys     0m0.000s

--- AFTER UPGRADING REMOTE MACHINE ---

# time ssh machine.name 'echo test'
test

real    0m2.640s
user    0m0.090s
sys     0m0.010s

Notice that logins take 2+ additional seconds with the new packages.

To see the bogus log entry I mentioned, cat /var/log/messages after a 
successful login, and you should see:

Jul 29 12:55:49 machine.name sshd(pam_unix)[19658]: authentication failure; 
logname= uid=0 euid=0 tty=NODEV
ssh ruser= rhost=source.machine  user=root

Expected Results:  Logins should not take 2+ seconds, and there should be 
no "authentication failure" log entry for successful logins.
Comment 1 Jordan Russell 2003-07-29 14:21:45 EDT
Some other notes:

1. In my example, I was using RSA authentication with SSH protocol version 2, 
but I can reproduce the same issues with password authentication.

2. The issues are reproducable with the stock /etc/ssh/sshd_config file.

3. I'm not using any non-standard authentication scheme.

4. I can also reproduce the issues with a clean install of Red Hat Linux 9.0, 
with only the openssh errata packages installed. (My other boxes are upgrades 
from prior Red Hat Linux versions, and they have all errata packages applied.)
Comment 2 Andre 2003-07-30 12:31:53 EDT
If you don't need password authentication you can put in:
  PasswordAuthentication no
into /etc/ssh/sshd_config.  This seems to skip the bogus pam authentication
request (which avoids the bogus log entry and the delay).  But of course this
doesn't help if you need password authentication.
Comment 3 Tomas Kasparek 2003-07-31 03:34:33 EDT
The same behavior can be seen on RH7.3 after last openssh update
(openssh-3.1p1-8.i386.rpm)
Comment 4 Milan Kerslager 2003-07-31 03:56:50 EDT
The correct fix for this is probably patch PAM to produce same delay when accept
bad login-name or bad password with correct login-name. Patching ssh this way is
wrong as there could be another service that will have the same possibility to
see what account is wrong and what is not.
Comment 5 Mike Chambers 2003-07-31 04:35:22 EDT
Actually, I get the same thing going from a severn machine to another severn
machine, with stock packages...

real    0m5.039s
user    0m0.140s
sys     0m0.000s
Comment 6 Rich Graves 2003-07-31 17:55:35 EDT
This should be marked a duplicate of 101361 and marked critical. The patch is
simply incorrect and should not be used. It addresses only a minor brute-force
user enumeration non-problem.
Comment 7 Milan Kerslager 2003-08-01 06:57:37 EDT
Bug #101361 is not duplicate. This bug is another story about Kerberos but the
next bad behavior of this broken security update.
Comment 8 Nalin Dahyabhai 2003-08-05 02:27:50 EDT
Actually, this is what the fix is expected to do.

OpenSSH would skip PAM authentication for user accounts which didn't exist or
for passwords which didn't meet its internal requirements (no empty passwords,
and so on).

When PAM authentication fails, libpam, as configured, forces a delay.  OpenSSH,
by short-circuiting the PAM authentication check, leaks information about
whether or not an account exists for a given user through the length of the delay.

The fix implemented in 3.6.1 and backported to these errata packages is to
*always* attempt PAM authentication, even if the account or password fails
OpenSSH's internal checks (though if the authentication attempt has already
failed sshd's internal checks, regardless of whether or not libpam says "ok",
the authentication attempt will fail).
Comment 9 Jordan Russell 2003-08-05 12:54:12 EDT
I find a full 2 second delay for *successful logins* unacceptable. Likewise - 
to a lesser degree - "authentication failure" log messages for successful 
logins.

I find it very hard to believe that both of these side effects were 100% 
intentional, and further that they can't be worked around. Therefore, I can't 
help but disagree with your shrugging this off as "not a bug", and hope that 
you reconsider your stance. The majority seem to be believe that either 
this "fix" should be reverted, or that more work needs to be done on it.

As of now, I'm back using previous openssh packages, and apparently others here 
are as well. If this is not fixed, it's going to create a sticky situation when 
a "real" security issue is discovered in openssh and we're not able to upgrade.
Comment 10 Sanford Berger 2003-08-05 13:05:18 EDT
I agree with JR that this "fix" seems ridiculous. Why should I have login 
delays and generate log messages when there have been no login failures?!? We 
have a centralized location where all syslog entries on our network are sent. 
We have a periodic cron job that evaluates the common syslog and sends three 
system administrators e-mail with items of interest that should be 
investigated. Now we get all these authentication failures every time someone 
uses ssh or scp! There were so many entries, we had to modify our configuration 
to ignore these messages. Now if there are real authentication failures we 
won't see them! This seems to be a case where the "cure" is worse than the 
disease!
Comment 11 Andre 2003-08-06 03:52:03 EDT
If the only solution to the information leak is to have this delay, then so be
it.  But it seems like this should definitely be a configuration option for
those of us who aren't worried about this particular attack (the delay is very
annoying).  But the bogus authentication failure message is wrong in either
case.  As others have said, the cure is definitely worse than the disease.
Comment 12 Kevin M. Myer 2003-08-07 10:48:01 EDT
*** Bug 101662 has been marked as a duplicate of this bug. ***
Comment 13 Mircea Damian 2003-08-11 05:27:51 EDT
I'm hitting the same bug (this is definitely a bug - such a log entry should 
not exist) and I'm wondering why this message appears into the logs before the 
I even get the password prompt?

What does it try to authenticate?
Comment 14 Carwyn Edwards 2003-08-23 01:22:58 EDT
Bug: 102174 is a duplicate of this bug.
Comment 15 Jose Vicente Nunez Zuleta 2003-08-28 09:33:53 EDT
What will happen to this? We have a centralized syslog and we are getting all
these bogus messages and also the authentication is slow now (using password or
keys, it doesn't matter).

There are any plans to fix this, or this issue will be ignored?

How dangerous is not to apply the patch?

JV.
Comment 16 Kevin M. Myer 2003-08-28 11:09:03 EDT
I think the results of the patch are far worse than the risk/problem it was
meant to resolve.  The patch may technically be correct and do what its supposed
to but the behaviour it produces is flat out wrong - a successful login should
not generate a failed login error message.  No response from Red Hat since
8/5/2003 - that is LAME and I'm tired of having to back out patches that
introduce worse problems than the ones they're intended to resolve (ala the
OpenSSL errata blinding patch that breaks ubsec hardware crypto acceleration -
see https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=88018  - its still
marked as NEW nearly five months after being opened!!!!!!)
Comment 17 Seth Vidal 2003-09-11 10:50:31 EDT
if you add 'nodelay' to the pam auth for /etc/pam.d/sshd then the delay goes
away entirely.

not the best solution of course...

Comment 18 Need Real Name 2003-09-16 11:36:39 EDT
Why is this marked closed?  Has the bogus log entry been fixed?  Has the
delay been removed?  I don't understand the point about this being
intentional.  The problem was that there was no delay when checking a
nonexistant account.  Surely the fix would be to make that go through
PAM as well as successful logins.  Successful logins should not be affected
in any way.
Comment 19 Hunter Matthews 2003-09-16 18:23:15 EDT
The openssh errata of openssh-3.5p1-9 (RHSA-2003:279-01) appears to fix the
orginal two issues as well.

Well done guys.

Comment 20 John Caruso 2003-09-18 15:35:24 EDT
This bug also afflicts RedHat Advanced Server 2.1.  I couldn't possibly agree 
more with rocombs@cs.nmsu.edu: why is this marked closed?  Has the delay been 
removed?

Unfortunately, the answer is NO.  And now that there's another OpenSSH bugfix 
in the pipeline, the worst-case scenario has arrived: sites like ours are faced 
with a choice of installing the broken, dog-slow version of OpenSSH RedHat is 
now distributing, or going without an important OpenSSH bug fix.

I've filed a case against RHAS for this same bug, and I've been told that the 
bug--which they seemed to recognize as a bug, unlike in this call--must 
happen "upstream" (in the OpenSSH code, presumably).  And that's it; no 
apparent further action.  This is unacceptable, and RedHat needs to address 
this problem ASAP, especially with this important OpenSSH bugfix waiting in the 
pipeline.
Comment 21 John Caruso 2003-09-18 15:37:29 EDT
I should have added that the latest errata RPMs do not fix the slowdown bug (so 
I'm not sure what thm@duke.edu was talking about...).
Comment 22 Damian Menscher 2003-09-18 15:46:01 EDT
Both problems appear to be fixed for me.... (using RH9 with openssh-3.5p1-11).  
Maybe they fixed it in RHL but not in RHAS?  What was your testcase?
Comment 23 Philip Ross 2003-09-18 16:07:40 EDT
The new errata packages have fixed both problems on my Red Hat 7.1, 7.2 and 
8.0 machines.
Comment 24 John Caruso 2003-09-18 16:07:40 EDT
Password-based (interactive) or public-key (passwordless) authentication.  I'm 
testing between three verions: openssh-3.1p1-6 (which does not exhibit the 
bug), openssh-3.1p1-8 (where the bug first appeared), and openssl-3.1p1-14 (the 
current release level as of RHSA-2003:280, which still exhibits the bug).  
Simply by swapping RPMs, OpenSSH connections go from taking .4 seconds to 2.8 
seconds.

The difference seems to be that in RH9, the OpenSSH version is 3.5p1--so from 
what you're saying it sounds as though they've fixed this bug in that release.  
We're not that lucky in RHAS.  But if they can fix it for that release, 
presumably the same fix could be backported to RHAS (just as the bug was 
backported).  And to other 3.1p1-based releases (through RH7.3) as well.

In fact, it would be interesting to know if anyone other than RH9 users have 
seen this issue resolved by the latest RPMs.  It would seem that all 3.1p1-
based releases should still be having the problem.
Comment 25 Reuben Farrelly 2003-09-18 16:14:48 EDT
I've just downloaded the OpenSSH 3.6.1p2-4 SRPM, dropped the 3.7.1p1 tarball 
in it and rebuilt.  It was a clean swap with no other changes required.  The 
delay has gone...and problem fixed.

Lets hope that 3.7.1p1 appers in Rawhide real soon..
Comment 26 Philip Ross 2003-09-18 17:45:49 EDT
It looks like the new errata rpms have fixed both problems, but only if you 
are using SSH2 to connect. Connections using SSH1 are still subject to the 
delay and authentication failure message in the logs.
Comment 27 John Caruso 2003-09-18 17:56:19 EDT
Philip's comment applies to the RHAS RPMs as well: SSH1 still exhibits both 
bugs, but SSH2 works normally again.  And in fact the SRPM files for 7.2 and 
RHAS are actually the same file, so it wouldn't make sense for them to behave 
differently.  It would be interesting to know if RH9 (with openssh 3.5p1) 
exhibits the same behavior.

I see the following in the changelog from the openssh.spec file:

* Tue Sep 16 2003 Nalin Dahyabhai <nalin@redhat.com> 3.1p1-9
- apply patch to store the correct buffer size in allocated buffers
  (CAN-2003-0693)
- skip the initial PAM authentication attempt with an empty password if
  empty passwords are not permitted in our configuration (#103998)

So it looks like Nalin did throw in a fix for the bugs, but didn't update this 
particular bugID (not a surprise since it's "CLOSED").  But, his fix must not 
have addressed SSH1.  So Nalin, if you're watching this closed but very active 
bug: thanks for the SSH2 fix, and could you please see about making your bugfix 
work with SSH1 as well?
Comment 28 John Caruso 2003-09-24 16:39:23 EDT
In my RHAS suport case, I finally persuaded RedHat to fix this issue for SSH1 
as well as SSH2.  Unfortunately, they say that they will NOT be rereleasing the 
RPMs for RHAS or most other RedHat trains, so to get the SSH1 fix you'll need 
to rebuild from the SRPM using an updated patch file (there are two, one for 
3.5p1 and one for 3.1p1; I'll attach them to this case).  Nalin claims that 
you'll see this in RawHide even before a new OpenSSH RPM is released, but 
apparently all other releases will have to wait until there's yet another bug 
in OpenSSH before they get this fix as part of an official RedHat RPM.

I'm glad that this has now been addressed, but I find it a bit irritating that 
1) this case never had its NOTABUG status changed even though RedHat/Nalin 
clearly do recognize that it was a bug (as of course it was), 2) RedHat is 
trying to lay low with this bug and fix by not re-releasing the RPMs now that 
SSH1 has been repaired, which causes more work for all of us to get in put in 
place, and 3) I've had to go through this effort to get the patch files 
published in a place where other people can use them when it's something Nalin 
should be doing.  Maybe I should send RedHat an invoice at my normal hourly 
rate?

Below is Nalin's email to the person who was handling my RHAS case, with 
instructions on how to rebuild.

(Nalin, if you're watching this, please change the status to something more 
appropriate so that people searching Bugzilla for this issue in the future will 
be able to tell that it does in fact contain a fix for their problem.)

-----Forwarded Message-----
> From: Nalin Dahyabhai <nalin@redhat.com>
> To: Chris Kloiber <ckloiber@redhat.com>
> Subject: OpenSSH - updated skip-initial patch
> Date: Tue, 23 Sep 2003 16:34:56 -0400
> 
> Chris, this patch adds the necessary hunk for 
> making SSH1 skip the initial authentication 
> attempt, as we did for SSH2 in the last erratum.
> It's checked in on the appropriate branch, so if 
> another OpenSSH update comes down the 
> pipeline, it'll be incorporated, but until then 
> you'll only see this in Raw Hide.
> 
> To use it, install the source package with 'rpm -
> U', replace the patch file in 
> /src/redhat/SOURCES with the attached patch 
> [1], and run 'rpmbuild -bb 
> /usr/src/redhat/SPECS/openssh.spec'.  
> Assuming no build failures due to unmet build 
> dependencies or other unforeseen problems,
> you'll get a modified version of the package we 
> last shipped out which has the desired change 
> incorporated into it.
> 
> HTH,
> 
> Nalin
> 
> [1] There are two, actually, because the changes > for 3.1p1 are slightly 
> different from the 
> changes for 3.4p1, 3.5p1, and 3.6.1p2.
----End Forwarded Message-----
Comment 29 John Caruso 2003-09-24 16:48:23 EDT
Created attachment 94698 [details]
Updated openssh-3.1p1-skip-initial.patch file
Comment 30 John Caruso 2003-09-24 16:49:20 EDT
Created attachment 94699 [details]
Updated openssh-3.5p1-skip-initial.patch file
Comment 31 Jeremy Katz 2003-10-21 16:38:18 EDT
*** Bug 103935 has been marked as a duplicate of this bug. ***
Comment 32 Tomas Mraz 2005-02-04 10:35:15 EST
*** Bug 102174 has been marked as a duplicate of this bug. ***
Comment 33 Tomas Mraz 2005-02-04 10:40:51 EST
*** Bug 101252 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.