Description of problem:
On FIPS enabled environment, login with rsa key doesn't work.
prelink is disabled already on the system, and sshd daemon can start without any error. So, this is not related with Bugzilla 515854.

The following is strace log.

14367 16:07:59 connect(4, {sa_family=AF_FILE, path="/dev/log"...}, 110) = 0
14367 16:07:59 sendto(4, "<87>Jan 27 16:07:59 sshd[14367]: debug1: ssh_rsa_verify: signature correct", 74, MSG_NOSIGNAL, NULL, 0 <unfinished ...>
14369 16:07:59 <... sendto resumed> )   = 83
14369 16:07:59 stat("/etc/localtime", {st_dev=makedev(252, 0), st_ino=1910888, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=16, st_size=265, st_atime=2011/01/27-16:07:58, st_mtime=2010/06/24-23:09:52, st_ctime=2011/01/24-12:24:59}) = 0
14369 16:07:59 stat("/etc/localtime", {st_dev=makedev(252, 0), st_ino=1910888, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=16, st_size=265, st_atime=2011/01/27-16:07:58, st_mtime=2010/06/24-23:09:52, st_ctime=2011/01/24-12:24:59}) = 0
14369 16:07:59 stat("/etc/localtime", {st_dev=makedev(252, 0), st_ino=1910888, st_mode=S_IFREG|0644, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=16, st_size=265, st_atime=2011/01/27-16:07:58, st_mtime=2010/06/24-23:09:52, st_ctime=2011/01/24-12:24:59}) = 0
14369 16:07:59 sendto(6, "<87>Jan 27 16:07:59 sshd[14369]: debug3: mm_request_receive entering", 68, MSG_NOSIGNAL, NULL, 0 <unfinished ...>
14367 16:07:59 <... sendto resumed> )   = 74
14367 16:07:59 close(4)                 = 0
14367 16:07:59 write(2, "digest.c(151): OpenSSL internal error, assertion failed: Digest update previous FIPS forbidden algorithm error ignored\n", 119) = 119
14367 16:07:59 rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0

And, this is backtrace.

(gdb) bt
#0  0x00002ba73f12b265 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00002ba73f12cd10 in abort () at abort.c:88
#2  0x00002ba73d7e9ccf in OpenSSLDie (file=<value optimized out>, line=<value optimized out>, assertion=<value optimized out>) at cryptlib.c:542
#3  0x00002ba73d7ec98c in bad_update (ctx=<value optimized out>, data=<value optimized out>, count=<value optimized out>) at digest.c:151
#4  0x00002ba73c67c833 in key_fingerprint_raw (k=0x2ba74441a960, dgst_type=<value optimized out>, dgst_raw_length=0x7fff23902ddc) at key.c:282
#5  0x00002ba73c67c95a in key_fingerprint (k=0xf06, dgst_type=3846, dgst_rep=SSH_FP_HEX) at key.c:365
#6  0x00002ba73c65bf01 in pubkey_key_verify (key=0x2ba74441a960, sig=<value optimized out>, slen=<value optimized out>, data=<value optimized out>, 
    datalen=<value optimized out>) at auth2-pubkey.c:117
#7  0x00002ba73c65f230 in mm_answer_keyverify (sock=5, m=0x7fff23902f50) at monitor.c:1228
#8  0x00002ba73c65fd68 in monitor_read (pmonitor=0x2ba744408810, ent=0x2ba73c8a0e00, pent=0x7fff23902fb8) at monitor.c:458
#9  0x00002ba73c65ffc7 in monitor_child_preauth (_authctxt=0x2ba744408780, pmonitor=0x2ba744408810) at monitor.c:344
#10 0x00002ba73c64c863 in privsep_preauth (ac=<value optimized out>, av=<value optimized out>) at sshd.c:609
#11 main (ac=<value optimized out>, av=<value optimized out>) at sshd.c:1730

auth2-pubkey.c
-------------------------------
    100 int
    101 pubkey_key_verify(const Key *key, const u_char *sig, u_int slen, const u_char *data, u_int datalen)
    102 {
...
    115          rv = key_verify(key, sig, slen, data, datalen);
    116 #ifdef HAVE_LINUX_AUDIT
    117         fp = key_fingerprint(key, SSH_FP_MD5, SSH_FP_HEX);

SSH_FP_MD5 is always used here.


key.c
----------------------
    235 u_char*
    236 key_fingerprint_raw(const Key *k, enum fp_type dgst_type,
    237     u_int *dgst_raw_length)
    238 {
...
    248         switch (dgst_type) {
    249         case SSH_FP_MD5:
    250                 md = EVP_md5();
    251                 break;

EVP_md5() is defined in openssl.

crypto/evp/m_md5.c of openssl
---------------------------
static const EVP_MD md5_md=
        {
        NID_md5,
        NID_md5WithRSAEncryption,
        MD5_DIGEST_LENGTH,
        0,      <<============= no any flags (no EVP_MD_FLAG_FIPS)
        init,
        update,
        final,
        NULL,
        NULL,
        EVP_PKEY_RSA_method,
        MD5_CBLOCK,
        sizeof(EVP_MD *)+sizeof(MD5_CTX),
        };

const EVP_MD *EVP_md5(void)
        {
        return(&md5_md);
        }

I created a simple patch to use SHA1 on FIPS enabled environment, and it worked on my test machines. I will attach it on this bugzilla.

Version-Release number of selected component (if applicable):
openssh-4.3p2-72.el5

How reproducible:
Always

Steps to Reproduce:
1. create two users (t0 and t1) on RHEL5.6
2. run ssh-genkey with t0 user, copy the public key to ~t1/.ssh/authorized_keys
3. run chmod 600 ~/.ssh/authorized_keys with t1 user.
4. run this with t0
     # ssh -l t1 hostname

Actual results:
The connection is closed soon.

Expected results:
The connection is not closed, and can log into successfully.

Additional info: