Bug 674747 - cannot login with rsa key on FIPS environment.
cannot login with rsa key on FIPS environment.
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: openssh (Show other bugs)
5.6
All Linux
urgent Severity urgent
: rc
: ---
Assigned To: Radek Vokal
Miroslav Vadkerti
: Regression, ZStream
: 674614 (view as bug list)
Depends On:
Blocks: BaseOS-FIPS-Tracker 700845 730652 739822 758797
  Show dependency treegraph
 
Reported: 2011-02-03 02:02 EST by Masahiro Matsuya
Modified: 2012-02-21 01:12 EST (History)
10 users (show)

See Also:
Fixed In Version: openssh-4.3p2-76.el5
Doc Type: Bug Fix
Doc Text:
When Federal Information Processing Standards (FIPS) mode was enabled on a system, key-based authentication was always unsuccessful. This was caused by the newly introduced pubkey_key_verify() verification function, which did not take into consideration the fact that it was running in a FIPS environment. With this update, the pubkey_key_verify() function has been modified to respect FIPS, and authentication using an RSA key is now successful without any issues when FIPS mode is enabled.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-02-21 01:12:06 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
proposed patch (553 bytes, patch)
2011-02-03 02:05 EST, Masahiro Matsuya
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Legacy) 46628 None None None Never

  None (edit)
Description Masahiro Matsuya 2011-02-03 02:02:19 EST
Description of problem:
On FIPS enabled environment, login with rsa key doesn't work.
prelink is disabled already on the system, and sshd daemon can start without any error. So, this is not related with Bugzilla 515854.

The following is strace log.

14367 16:07:59 connect(4, {sa_family=AF_FILE, path="/dev/log"...}, 110) = 0
14367 16:07:59 sendto(4, "<87>Jan 27 16:07:59 sshd[14367]: debug1: ssh_rsa_verify: signature correct", 74, MSG_NOSIGNAL, NULL, 0 <unfinished ...>
14369 16:07:59 <... sendto resumed> )   = 83
14369 16:07:59 stat("/etc/localtime", {st_dev=makedev(252, 0), st_ino=1910888, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=16, st_size=265, st_atime=2011/01/27-16:07:58, st_mtime=2010/06/24-23:09:52, st_ctime=2011/01/24-12:24:59}) = 0
14369 16:07:59 stat("/etc/localtime", {st_dev=makedev(252, 0), st_ino=1910888, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=16, st_size=265, st_atime=2011/01/27-16:07:58, st_mtime=2010/06/24-23:09:52, st_ctime=2011/01/24-12:24:59}) = 0
14369 16:07:59 stat("/etc/localtime", {st_dev=makedev(252, 0), st_ino=1910888, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=16, st_size=265, st_atime=2011/01/27-16:07:58, st_mtime=2010/06/24-23:09:52, st_ctime=2011/01/24-12:24:59}) = 0
14369 16:07:59 sendto(6, "<87>Jan 27 16:07:59 sshd[14369]: debug3: mm_request_receive entering", 68, MSG_NOSIGNAL, NULL, 0 <unfinished ...>
14367 16:07:59 <... sendto resumed> )   = 74
14367 16:07:59 close(4)                 = 0
14367 16:07:59 write(2, "digest.c(151): OpenSSL internal error, assertion failed: Digest update previous FIPS forbidden algorithm error ignored\n", 119) = 119
14367 16:07:59 rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0

And, this is backtrace.

(gdb) bt
#0  0x00002ba73f12b265 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00002ba73f12cd10 in abort () at abort.c:88
#2  0x00002ba73d7e9ccf in OpenSSLDie (file=<value optimized out>, line=<value optimized out>, assertion=<value optimized out>) at cryptlib.c:542
#3  0x00002ba73d7ec98c in bad_update (ctx=<value optimized out>, data=<value optimized out>, count=<value optimized out>) at digest.c:151
#4  0x00002ba73c67c833 in key_fingerprint_raw (k=0x2ba74441a960, dgst_type=<value optimized out>, dgst_raw_length=0x7fff23902ddc) at key.c:282
#5  0x00002ba73c67c95a in key_fingerprint (k=0xf06, dgst_type=3846, dgst_rep=SSH_FP_HEX) at key.c:365
#6  0x00002ba73c65bf01 in pubkey_key_verify (key=0x2ba74441a960, sig=<value optimized out>, slen=<value optimized out>, data=<value optimized out>, 
    datalen=<value optimized out>) at auth2-pubkey.c:117
#7  0x00002ba73c65f230 in mm_answer_keyverify (sock=5, m=0x7fff23902f50) at monitor.c:1228
#8  0x00002ba73c65fd68 in monitor_read (pmonitor=0x2ba744408810, ent=0x2ba73c8a0e00, pent=0x7fff23902fb8) at monitor.c:458
#9  0x00002ba73c65ffc7 in monitor_child_preauth (_authctxt=0x2ba744408780, pmonitor=0x2ba744408810) at monitor.c:344
#10 0x00002ba73c64c863 in privsep_preauth (ac=<value optimized out>, av=<value optimized out>) at sshd.c:609
#11 main (ac=<value optimized out>, av=<value optimized out>) at sshd.c:1730

auth2-pubkey.c
-------------------------------
    100 int
    101 pubkey_key_verify(const Key *key, const u_char *sig, u_int slen, const u_char *data, u_int datalen)
    102 {
...
    115          rv = key_verify(key, sig, slen, data, datalen);
    116 #ifdef HAVE_LINUX_AUDIT
    117         fp = key_fingerprint(key, SSH_FP_MD5, SSH_FP_HEX);

SSH_FP_MD5 is always used here.


key.c
----------------------
    235 u_char*
    236 key_fingerprint_raw(const Key *k, enum fp_type dgst_type,
    237     u_int *dgst_raw_length)
    238 {
...
    248         switch (dgst_type) {
    249         case SSH_FP_MD5:
    250                 md = EVP_md5();
    251                 break;

EVP_md5() is defined in openssl.

crypto/evp/m_md5.c of openssl
---------------------------
static const EVP_MD md5_md=
        {
        NID_md5,
        NID_md5WithRSAEncryption,
        MD5_DIGEST_LENGTH,
        0,      <<============= no any flags (no EVP_MD_FLAG_FIPS)
        init,
        update,
        final,
        NULL,
        NULL,
        EVP_PKEY_RSA_method,
        MD5_CBLOCK,
        sizeof(EVP_MD *)+sizeof(MD5_CTX),
        };

const EVP_MD *EVP_md5(void)
        {
        return(&md5_md);
        }

I created a simple patch to use SHA1 on FIPS enabled environment, and it worked on my test machines. I will attach it on this bugzilla.

Version-Release number of selected component (if applicable):
openssh-4.3p2-72.el5

How reproducible:
Always

Steps to Reproduce:
1. create two users (t0 and t1) on RHEL5.6
2. run ssh-genkey with t0 user, copy the public key to ~t1/.ssh/authorized_keys
3. run chmod 600 ~/.ssh/authorized_keys with t1 user.
4. run this with t0
     # ssh -l t1 hostname

Actual results:
The connection is closed soon.

Expected results:
The connection is not closed, and can log into successfully.

Additional info:
Comment 1 Masahiro Matsuya 2011-02-03 02:05:48 EST
Created attachment 476712 [details]
proposed patch
Comment 8 Jan F. Chadima 2011-03-08 03:20:53 EST
*** Bug 674614 has been marked as a duplicate of this bug. ***
Comment 11 Cameron Harr 2011-04-20 17:44:05 EDT
Is there an ETA on release of an updated package?
Comment 18 Miroslav Svoboda 2011-08-17 04:58:43 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
When Federal Information Processing Standards (FIPS) mode was enabled on a system, key-based authentication was always unsuccessful. This was caused by the newly introduced pubkey_key_verify() verification function, which did not take into consideration the fact that it was running in a FIPS environment. With this update, the pubkey_key_verify() function has been modified to respect FIPS, and authentication using an RSA key is now successful without any issues when FIPS mode is enabled.
Comment 23 errata-xmlrpc 2012-02-21 01:12:06 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0237.html

Note You need to log in before you can comment on or make changes to this bug.