Red Hat Bugzilla – Bug 101157
Errata packages cause slow login and bogus log entry
Last modified: 2007-03-27 00:08:32 EDT
Description of problem:
On my Red Hat Linux 9.0 boxes, I've found that after upgrading them to openssh-
3.5p1-6.9 (from openssh-3.5p1-6), logins take much longer than before, and a
bogus log entry is created for each successful login.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
To see the "slow login" problem, try the 'time' command before and after
upgrading the packages.
--- BEFORE UPGRADING REMOTE MACHINE ---
# time ssh machine.name 'echo test'
--- AFTER UPGRADING REMOTE MACHINE ---
# time ssh machine.name 'echo test'
Notice that logins take 2+ additional seconds with the new packages.
To see the bogus log entry I mentioned, cat /var/log/messages after a
successful login, and you should see:
Jul 29 12:55:49 machine.name sshd(pam_unix): authentication failure;
logname= uid=0 euid=0 tty=NODEV
ssh ruser= rhost=source.machine user=root
Expected Results: Logins should not take 2+ seconds, and there should be
no "authentication failure" log entry for successful logins.
Some other notes:
1. In my example, I was using RSA authentication with SSH protocol version 2,
but I can reproduce the same issues with password authentication.
2. The issues are reproducable with the stock /etc/ssh/sshd_config file.
3. I'm not using any non-standard authentication scheme.
4. I can also reproduce the issues with a clean install of Red Hat Linux 9.0,
with only the openssh errata packages installed. (My other boxes are upgrades
from prior Red Hat Linux versions, and they have all errata packages applied.)
If you don't need password authentication you can put in:
into /etc/ssh/sshd_config. This seems to skip the bogus pam authentication
request (which avoids the bogus log entry and the delay). But of course this
doesn't help if you need password authentication.
The same behavior can be seen on RH7.3 after last openssh update
The correct fix for this is probably patch PAM to produce same delay when accept
bad login-name or bad password with correct login-name. Patching ssh this way is
wrong as there could be another service that will have the same possibility to
see what account is wrong and what is not.
Actually, I get the same thing going from a severn machine to another severn
machine, with stock packages...
This should be marked a duplicate of 101361 and marked critical. The patch is
simply incorrect and should not be used. It addresses only a minor brute-force
user enumeration non-problem.
Bug #101361 is not duplicate. This bug is another story about Kerberos but the
next bad behavior of this broken security update.
Actually, this is what the fix is expected to do.
OpenSSH would skip PAM authentication for user accounts which didn't exist or
for passwords which didn't meet its internal requirements (no empty passwords,
and so on).
When PAM authentication fails, libpam, as configured, forces a delay. OpenSSH,
by short-circuiting the PAM authentication check, leaks information about
whether or not an account exists for a given user through the length of the delay.
The fix implemented in 3.6.1 and backported to these errata packages is to
*always* attempt PAM authentication, even if the account or password fails
OpenSSH's internal checks (though if the authentication attempt has already
failed sshd's internal checks, regardless of whether or not libpam says "ok",
the authentication attempt will fail).
I find a full 2 second delay for *successful logins* unacceptable. Likewise -
to a lesser degree - "authentication failure" log messages for successful
I find it very hard to believe that both of these side effects were 100%
intentional, and further that they can't be worked around. Therefore, I can't
help but disagree with your shrugging this off as "not a bug", and hope that
you reconsider your stance. The majority seem to be believe that either
this "fix" should be reverted, or that more work needs to be done on it.
As of now, I'm back using previous openssh packages, and apparently others here
are as well. If this is not fixed, it's going to create a sticky situation when
a "real" security issue is discovered in openssh and we're not able to upgrade.
I agree with JR that this "fix" seems ridiculous. Why should I have login
delays and generate log messages when there have been no login failures?!? We
have a centralized location where all syslog entries on our network are sent.
We have a periodic cron job that evaluates the common syslog and sends three
system administrators e-mail with items of interest that should be
investigated. Now we get all these authentication failures every time someone
uses ssh or scp! There were so many entries, we had to modify our configuration
to ignore these messages. Now if there are real authentication failures we
won't see them! This seems to be a case where the "cure" is worse than the
If the only solution to the information leak is to have this delay, then so be
it. But it seems like this should definitely be a configuration option for
those of us who aren't worried about this particular attack (the delay is very
annoying). But the bogus authentication failure message is wrong in either
case. As others have said, the cure is definitely worse than the disease.
*** Bug 101662 has been marked as a duplicate of this bug. ***
I'm hitting the same bug (this is definitely a bug - such a log entry should
not exist) and I'm wondering why this message appears into the logs before the
I even get the password prompt?
What does it try to authenticate?
Bug: 102174 is a duplicate of this bug.
What will happen to this? We have a centralized syslog and we are getting all
these bogus messages and also the authentication is slow now (using password or
keys, it doesn't matter).
There are any plans to fix this, or this issue will be ignored?
How dangerous is not to apply the patch?
I think the results of the patch are far worse than the risk/problem it was
meant to resolve. The patch may technically be correct and do what its supposed
to but the behaviour it produces is flat out wrong - a successful login should
not generate a failed login error message. No response from Red Hat since
8/5/2003 - that is LAME and I'm tired of having to back out patches that
introduce worse problems than the ones they're intended to resolve (ala the
OpenSSL errata blinding patch that breaks ubsec hardware crypto acceleration -
see https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=88018 - its still
marked as NEW nearly five months after being opened!!!!!!)
if you add 'nodelay' to the pam auth for /etc/pam.d/sshd then the delay goes
not the best solution of course...
Why is this marked closed? Has the bogus log entry been fixed? Has the
delay been removed? I don't understand the point about this being
intentional. The problem was that there was no delay when checking a
nonexistant account. Surely the fix would be to make that go through
PAM as well as successful logins. Successful logins should not be affected
in any way.
The openssh errata of openssh-3.5p1-9 (RHSA-2003:279-01) appears to fix the
orginal two issues as well.
Well done guys.
This bug also afflicts RedHat Advanced Server 2.1. I couldn't possibly agree
more with email@example.com: why is this marked closed? Has the delay been
Unfortunately, the answer is NO. And now that there's another OpenSSH bugfix
in the pipeline, the worst-case scenario has arrived: sites like ours are faced
with a choice of installing the broken, dog-slow version of OpenSSH RedHat is
now distributing, or going without an important OpenSSH bug fix.
I've filed a case against RHAS for this same bug, and I've been told that the
bug--which they seemed to recognize as a bug, unlike in this call--must
happen "upstream" (in the OpenSSH code, presumably). And that's it; no
apparent further action. This is unacceptable, and RedHat needs to address
this problem ASAP, especially with this important OpenSSH bugfix waiting in the
I should have added that the latest errata RPMs do not fix the slowdown bug (so
I'm not sure what firstname.lastname@example.org was talking about...).
Both problems appear to be fixed for me.... (using RH9 with openssh-3.5p1-11).
Maybe they fixed it in RHL but not in RHAS? What was your testcase?
The new errata packages have fixed both problems on my Red Hat 7.1, 7.2 and
Password-based (interactive) or public-key (passwordless) authentication. I'm
testing between three verions: openssh-3.1p1-6 (which does not exhibit the
bug), openssh-3.1p1-8 (where the bug first appeared), and openssl-3.1p1-14 (the
current release level as of RHSA-2003:280, which still exhibits the bug).
Simply by swapping RPMs, OpenSSH connections go from taking .4 seconds to 2.8
The difference seems to be that in RH9, the OpenSSH version is 3.5p1--so from
what you're saying it sounds as though they've fixed this bug in that release.
We're not that lucky in RHAS. But if they can fix it for that release,
presumably the same fix could be backported to RHAS (just as the bug was
backported). And to other 3.1p1-based releases (through RH7.3) as well.
In fact, it would be interesting to know if anyone other than RH9 users have
seen this issue resolved by the latest RPMs. It would seem that all 3.1p1-
based releases should still be having the problem.
I've just downloaded the OpenSSH 3.6.1p2-4 SRPM, dropped the 3.7.1p1 tarball
in it and rebuilt. It was a clean swap with no other changes required. The
delay has gone...and problem fixed.
Lets hope that 3.7.1p1 appers in Rawhide real soon..
It looks like the new errata rpms have fixed both problems, but only if you
are using SSH2 to connect. Connections using SSH1 are still subject to the
delay and authentication failure message in the logs.
Philip's comment applies to the RHAS RPMs as well: SSH1 still exhibits both
bugs, but SSH2 works normally again. And in fact the SRPM files for 7.2 and
RHAS are actually the same file, so it wouldn't make sense for them to behave
differently. It would be interesting to know if RH9 (with openssh 3.5p1)
exhibits the same behavior.
I see the following in the changelog from the openssh.spec file:
* Tue Sep 16 2003 Nalin Dahyabhai <email@example.com> 3.1p1-9
- apply patch to store the correct buffer size in allocated buffers
- skip the initial PAM authentication attempt with an empty password if
empty passwords are not permitted in our configuration (#103998)
So it looks like Nalin did throw in a fix for the bugs, but didn't update this
particular bugID (not a surprise since it's "CLOSED"). But, his fix must not
have addressed SSH1. So Nalin, if you're watching this closed but very active
bug: thanks for the SSH2 fix, and could you please see about making your bugfix
work with SSH1 as well?
In my RHAS suport case, I finally persuaded RedHat to fix this issue for SSH1
as well as SSH2. Unfortunately, they say that they will NOT be rereleasing the
RPMs for RHAS or most other RedHat trains, so to get the SSH1 fix you'll need
to rebuild from the SRPM using an updated patch file (there are two, one for
3.5p1 and one for 3.1p1; I'll attach them to this case). Nalin claims that
you'll see this in RawHide even before a new OpenSSH RPM is released, but
apparently all other releases will have to wait until there's yet another bug
in OpenSSH before they get this fix as part of an official RedHat RPM.
I'm glad that this has now been addressed, but I find it a bit irritating that
1) this case never had its NOTABUG status changed even though RedHat/Nalin
clearly do recognize that it was a bug (as of course it was), 2) RedHat is
trying to lay low with this bug and fix by not re-releasing the RPMs now that
SSH1 has been repaired, which causes more work for all of us to get in put in
place, and 3) I've had to go through this effort to get the patch files
published in a place where other people can use them when it's something Nalin
should be doing. Maybe I should send RedHat an invoice at my normal hourly
Below is Nalin's email to the person who was handling my RHAS case, with
instructions on how to rebuild.
(Nalin, if you're watching this, please change the status to something more
appropriate so that people searching Bugzilla for this issue in the future will
be able to tell that it does in fact contain a fix for their problem.)
> From: Nalin Dahyabhai <firstname.lastname@example.org>
> To: Chris Kloiber <email@example.com>
> Subject: OpenSSH - updated skip-initial patch
> Date: Tue, 23 Sep 2003 16:34:56 -0400
> Chris, this patch adds the necessary hunk for
> making SSH1 skip the initial authentication
> attempt, as we did for SSH2 in the last erratum.
> It's checked in on the appropriate branch, so if
> another OpenSSH update comes down the
> pipeline, it'll be incorporated, but until then
> you'll only see this in Raw Hide.
> To use it, install the source package with 'rpm -
> U', replace the patch file in
> /src/redhat/SOURCES with the attached patch
> , and run 'rpmbuild -bb
> Assuming no build failures due to unmet build
> dependencies or other unforeseen problems,
> you'll get a modified version of the package we
> last shipped out which has the desired change
> incorporated into it.
>  There are two, actually, because the changes > for 3.1p1 are slightly
> different from the
> changes for 3.4p1, 3.5p1, and 3.6.1p2.
----End Forwarded Message-----
Created attachment 94698 [details]
Updated openssh-3.1p1-skip-initial.patch file
Created attachment 94699 [details]
Updated openssh-3.5p1-skip-initial.patch file
*** Bug 103935 has been marked as a duplicate of this bug. ***
*** Bug 102174 has been marked as a duplicate of this bug. ***
*** Bug 101252 has been marked as a duplicate of this bug. ***