Description of problem: On my Red Hat Linux 9.0 boxes, I've found that after upgrading them to openssh- 3.5p1-6.9 (from openssh-3.5p1-6), logins take much longer than before, and a bogus log entry is created for each successful login. Version-Release number of selected component (if applicable): openssh-3.5p1-6.9 How reproducible: Always Steps to Reproduce: To see the "slow login" problem, try the 'time' command before and after upgrading the packages. --- BEFORE UPGRADING REMOTE MACHINE --- # time ssh machine.name 'echo test' test real 0m0.295s user 0m0.100s sys 0m0.000s --- AFTER UPGRADING REMOTE MACHINE --- # time ssh machine.name 'echo test' test real 0m2.640s user 0m0.090s sys 0m0.010s Notice that logins take 2+ additional seconds with the new packages. To see the bogus log entry I mentioned, cat /var/log/messages after a successful login, and you should see: Jul 29 12:55:49 machine.name sshd(pam_unix)[19658]: authentication failure; logname= uid=0 euid=0 tty=NODEV ssh ruser= rhost=source.machine user=root Expected Results: Logins should not take 2+ seconds, and there should be no "authentication failure" log entry for successful logins.
Some other notes: 1. In my example, I was using RSA authentication with SSH protocol version 2, but I can reproduce the same issues with password authentication. 2. The issues are reproducable with the stock /etc/ssh/sshd_config file. 3. I'm not using any non-standard authentication scheme. 4. I can also reproduce the issues with a clean install of Red Hat Linux 9.0, with only the openssh errata packages installed. (My other boxes are upgrades from prior Red Hat Linux versions, and they have all errata packages applied.)
If you don't need password authentication you can put in: PasswordAuthentication no into /etc/ssh/sshd_config. This seems to skip the bogus pam authentication request (which avoids the bogus log entry and the delay). But of course this doesn't help if you need password authentication.
The same behavior can be seen on RH7.3 after last openssh update (openssh-3.1p1-8.i386.rpm)
The correct fix for this is probably patch PAM to produce same delay when accept bad login-name or bad password with correct login-name. Patching ssh this way is wrong as there could be another service that will have the same possibility to see what account is wrong and what is not.
Actually, I get the same thing going from a severn machine to another severn machine, with stock packages... real 0m5.039s user 0m0.140s sys 0m0.000s
This should be marked a duplicate of 101361 and marked critical. The patch is simply incorrect and should not be used. It addresses only a minor brute-force user enumeration non-problem.
Bug #101361 is not duplicate. This bug is another story about Kerberos but the next bad behavior of this broken security update.
Actually, this is what the fix is expected to do. OpenSSH would skip PAM authentication for user accounts which didn't exist or for passwords which didn't meet its internal requirements (no empty passwords, and so on). When PAM authentication fails, libpam, as configured, forces a delay. OpenSSH, by short-circuiting the PAM authentication check, leaks information about whether or not an account exists for a given user through the length of the delay. The fix implemented in 3.6.1 and backported to these errata packages is to *always* attempt PAM authentication, even if the account or password fails OpenSSH's internal checks (though if the authentication attempt has already failed sshd's internal checks, regardless of whether or not libpam says "ok", the authentication attempt will fail).
I find a full 2 second delay for *successful logins* unacceptable. Likewise - to a lesser degree - "authentication failure" log messages for successful logins. I find it very hard to believe that both of these side effects were 100% intentional, and further that they can't be worked around. Therefore, I can't help but disagree with your shrugging this off as "not a bug", and hope that you reconsider your stance. The majority seem to be believe that either this "fix" should be reverted, or that more work needs to be done on it. As of now, I'm back using previous openssh packages, and apparently others here are as well. If this is not fixed, it's going to create a sticky situation when a "real" security issue is discovered in openssh and we're not able to upgrade.
I agree with JR that this "fix" seems ridiculous. Why should I have login delays and generate log messages when there have been no login failures?!? We have a centralized location where all syslog entries on our network are sent. We have a periodic cron job that evaluates the common syslog and sends three system administrators e-mail with items of interest that should be investigated. Now we get all these authentication failures every time someone uses ssh or scp! There were so many entries, we had to modify our configuration to ignore these messages. Now if there are real authentication failures we won't see them! This seems to be a case where the "cure" is worse than the disease!
If the only solution to the information leak is to have this delay, then so be it. But it seems like this should definitely be a configuration option for those of us who aren't worried about this particular attack (the delay is very annoying). But the bogus authentication failure message is wrong in either case. As others have said, the cure is definitely worse than the disease.
*** Bug 101662 has been marked as a duplicate of this bug. ***
I'm hitting the same bug (this is definitely a bug - such a log entry should not exist) and I'm wondering why this message appears into the logs before the I even get the password prompt? What does it try to authenticate?
Bug: 102174 is a duplicate of this bug.
What will happen to this? We have a centralized syslog and we are getting all these bogus messages and also the authentication is slow now (using password or keys, it doesn't matter). There are any plans to fix this, or this issue will be ignored? How dangerous is not to apply the patch? JV.
I think the results of the patch are far worse than the risk/problem it was meant to resolve. The patch may technically be correct and do what its supposed to but the behaviour it produces is flat out wrong - a successful login should not generate a failed login error message. No response from Red Hat since 8/5/2003 - that is LAME and I'm tired of having to back out patches that introduce worse problems than the ones they're intended to resolve (ala the OpenSSL errata blinding patch that breaks ubsec hardware crypto acceleration - see https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=88018 - its still marked as NEW nearly five months after being opened!!!!!!)
if you add 'nodelay' to the pam auth for /etc/pam.d/sshd then the delay goes away entirely. not the best solution of course...
Why is this marked closed? Has the bogus log entry been fixed? Has the delay been removed? I don't understand the point about this being intentional. The problem was that there was no delay when checking a nonexistant account. Surely the fix would be to make that go through PAM as well as successful logins. Successful logins should not be affected in any way.
The openssh errata of openssh-3.5p1-9 (RHSA-2003:279-01) appears to fix the orginal two issues as well. Well done guys.
This bug also afflicts RedHat Advanced Server 2.1. I couldn't possibly agree more with rocombs.edu: why is this marked closed? Has the delay been removed? Unfortunately, the answer is NO. And now that there's another OpenSSH bugfix in the pipeline, the worst-case scenario has arrived: sites like ours are faced with a choice of installing the broken, dog-slow version of OpenSSH RedHat is now distributing, or going without an important OpenSSH bug fix. I've filed a case against RHAS for this same bug, and I've been told that the bug--which they seemed to recognize as a bug, unlike in this call--must happen "upstream" (in the OpenSSH code, presumably). And that's it; no apparent further action. This is unacceptable, and RedHat needs to address this problem ASAP, especially with this important OpenSSH bugfix waiting in the pipeline.
I should have added that the latest errata RPMs do not fix the slowdown bug (so I'm not sure what thm was talking about...).
Both problems appear to be fixed for me.... (using RH9 with openssh-3.5p1-11). Maybe they fixed it in RHL but not in RHAS? What was your testcase?
The new errata packages have fixed both problems on my Red Hat 7.1, 7.2 and 8.0 machines.
Password-based (interactive) or public-key (passwordless) authentication. I'm testing between three verions: openssh-3.1p1-6 (which does not exhibit the bug), openssh-3.1p1-8 (where the bug first appeared), and openssl-3.1p1-14 (the current release level as of RHSA-2003:280, which still exhibits the bug). Simply by swapping RPMs, OpenSSH connections go from taking .4 seconds to 2.8 seconds. The difference seems to be that in RH9, the OpenSSH version is 3.5p1--so from what you're saying it sounds as though they've fixed this bug in that release. We're not that lucky in RHAS. But if they can fix it for that release, presumably the same fix could be backported to RHAS (just as the bug was backported). And to other 3.1p1-based releases (through RH7.3) as well. In fact, it would be interesting to know if anyone other than RH9 users have seen this issue resolved by the latest RPMs. It would seem that all 3.1p1- based releases should still be having the problem.
I've just downloaded the OpenSSH 3.6.1p2-4 SRPM, dropped the 3.7.1p1 tarball in it and rebuilt. It was a clean swap with no other changes required. The delay has gone...and problem fixed. Lets hope that 3.7.1p1 appers in Rawhide real soon..
It looks like the new errata rpms have fixed both problems, but only if you are using SSH2 to connect. Connections using SSH1 are still subject to the delay and authentication failure message in the logs.
Philip's comment applies to the RHAS RPMs as well: SSH1 still exhibits both bugs, but SSH2 works normally again. And in fact the SRPM files for 7.2 and RHAS are actually the same file, so it wouldn't make sense for them to behave differently. It would be interesting to know if RH9 (with openssh 3.5p1) exhibits the same behavior. I see the following in the changelog from the openssh.spec file: * Tue Sep 16 2003 Nalin Dahyabhai <nalin> 3.1p1-9 - apply patch to store the correct buffer size in allocated buffers (CAN-2003-0693) - skip the initial PAM authentication attempt with an empty password if empty passwords are not permitted in our configuration (#103998) So it looks like Nalin did throw in a fix for the bugs, but didn't update this particular bugID (not a surprise since it's "CLOSED"). But, his fix must not have addressed SSH1. So Nalin, if you're watching this closed but very active bug: thanks for the SSH2 fix, and could you please see about making your bugfix work with SSH1 as well?
In my RHAS suport case, I finally persuaded RedHat to fix this issue for SSH1 as well as SSH2. Unfortunately, they say that they will NOT be rereleasing the RPMs for RHAS or most other RedHat trains, so to get the SSH1 fix you'll need to rebuild from the SRPM using an updated patch file (there are two, one for 3.5p1 and one for 3.1p1; I'll attach them to this case). Nalin claims that you'll see this in RawHide even before a new OpenSSH RPM is released, but apparently all other releases will have to wait until there's yet another bug in OpenSSH before they get this fix as part of an official RedHat RPM. I'm glad that this has now been addressed, but I find it a bit irritating that 1) this case never had its NOTABUG status changed even though RedHat/Nalin clearly do recognize that it was a bug (as of course it was), 2) RedHat is trying to lay low with this bug and fix by not re-releasing the RPMs now that SSH1 has been repaired, which causes more work for all of us to get in put in place, and 3) I've had to go through this effort to get the patch files published in a place where other people can use them when it's something Nalin should be doing. Maybe I should send RedHat an invoice at my normal hourly rate? Below is Nalin's email to the person who was handling my RHAS case, with instructions on how to rebuild. (Nalin, if you're watching this, please change the status to something more appropriate so that people searching Bugzilla for this issue in the future will be able to tell that it does in fact contain a fix for their problem.) -----Forwarded Message----- > From: Nalin Dahyabhai <nalin> > To: Chris Kloiber <ckloiber> > Subject: OpenSSH - updated skip-initial patch > Date: Tue, 23 Sep 2003 16:34:56 -0400 > > Chris, this patch adds the necessary hunk for > making SSH1 skip the initial authentication > attempt, as we did for SSH2 in the last erratum. > It's checked in on the appropriate branch, so if > another OpenSSH update comes down the > pipeline, it'll be incorporated, but until then > you'll only see this in Raw Hide. > > To use it, install the source package with 'rpm - > U', replace the patch file in > /src/redhat/SOURCES with the attached patch > [1], and run 'rpmbuild -bb > /usr/src/redhat/SPECS/openssh.spec'. > Assuming no build failures due to unmet build > dependencies or other unforeseen problems, > you'll get a modified version of the package we > last shipped out which has the desired change > incorporated into it. > > HTH, > > Nalin > > [1] There are two, actually, because the changes > for 3.1p1 are slightly > different from the > changes for 3.4p1, 3.5p1, and 3.6.1p2. ----End Forwarded Message-----
Created attachment 94698 [details] Updated openssh-3.1p1-skip-initial.patch file
Created attachment 94699 [details] Updated openssh-3.5p1-skip-initial.patch file
*** Bug 103935 has been marked as a duplicate of this bug. ***
*** Bug 102174 has been marked as a duplicate of this bug. ***
*** Bug 101252 has been marked as a duplicate of this bug. ***