Bug 1197666

+++ This bug was initially created as a clone of Bug #1197072 +++

Description of problem:
ssh client aborts with HostbasedAuthentication fails if there is /etc/ssh/ssh_host_key present. This is somehow similar to bz1009959.

Version-Release number of selected component (if applicable):
openssh-5.3p1-105.el6

How reproducible:
always

Steps to Reproduce:
1. setup hostbased authentication for server
2. ssh -p 6296 -o PubkeyAuthentication=no -o BatchMode=yes -o HostbasedAuthentication=yes -v <USER>@<SERVER>

Actual results:
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
FIPS mode initialized
debug1: Connecting to sheep-29.lab.eng.brq.redhat.com [10.34.88.29] port 6296.
debug1: Connection established.
md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
Aborted (core dumped)

Expected results:
no core dump, authentication success

Additional info:
If I just remove /etc/ssh/ssh_host_key, everything works fine.

Configuration:

I am using same machine for both server and client for simplicity.
* add these lines to default /etc/ssh/sshd_config
HostbasedAuthentication yes
HostbasedUsesNameFromPacketOnly yes
IgnoreRhosts no
* add machine to global known_hosts
ssh-keyscan <HOSTNAME> >/etc/ssh/ssh_known_hosts
* add root to user's ~/.shosts file
# su - <USER> -c -c 'echo "<HOSTNAME> root" >.shosts; chmod 600 .shosts'

Backtrace:
(gdb) bt
#0  0x00007fb4b613e625 in raise () from /lib64/libc.so.6
#1  0x00007fb4b613fe05 in abort () from /lib64/libc.so.6
#2  0x00007fb4b7c25fbf in OpenSSLDie () from /usr/lib64/libcrypto.so.10
#3  0x00007fb4b7c2c729 in MD5_Init () from /usr/lib64/libcrypto.so.10
#4  0x00007fb4b860f85c in cipher_set_key_string (cc=0x7fff7c31a4d0, cipher=0x7fb4b884f4e0, 
    passphrase=0x7fb4b8640602 "", do_encrypt=0) at cipher.c:314
#5  0x00007fb4b8603a47 in key_load_private_rsa1 (fd=4, filename=0x7fb4b8631e2d "/etc/ssh/ssh_host_key", 
    passphrase=0x7fb4b8640602 "", commentp=0x0) at authfile.c:430
#6  0x00007fb4b85ee36a in main (ac=<value optimized out>, av=<value optimized out>) at ssh.c:829

I am attaching patch that solves problem for me.

--- Additional comment from Jakub Jelen on 2015-03-01 20:30:30 CET ---

This makes sense to me. We can't load RSA1 key in FIPS mode. Also sounds reasonable for RHEL-6.7.
But there is that comment "closes fd" so we should close fd as well even if we are in fips mode (as it is solved in your referenced bugzilla). Also we need to return and not to let it fallback into key_load_private_pem and generate errors

Unfortunately, I don't have FIPS machine around. Stanislav, can you try it with this patch?

--- Additional comment from Jakub Jelen on 2015-03-02 11:53:14 CET ---

OK. Tested with rhel6 and it solves this issue for me.

This issue applies also for rhel7 in this border condition so I'm cloning bug to rhel7.