Connecting redmine issue http://projects.theforeman.org/issues/23731 from this bug

Actual results:
always checking host - rex job stuck at host key check

Expected results:
reasonable way to ignore checking only for the first time - rex job passes

Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/23731 has been resolved.

VERIFIED.

@satellite-6.4.0-9.beta.el7sat.noarch
tfm-rubygem-foreman_ansible-2.2.0-1.el7sat.noarch
tfm-rubygem-foreman_ansible_core-2.1.0-1.el7sat.noarch
rubygem-smart_proxy_ansible-2.0.2-3.el7sat.noarch
ansible-2.6.0-0.5.rc5.el7ae.noarch

0) Have a host registered to SAT and REX ssh key set up on the host
# curl -k https://$SAT:9090/ssh/pubkey >> ~/.ssh/authorized_keys

1) Turn hostkey verification on:
(since we have BZ about ignoring per user config, set it in global ansible conf)
# grep ^host_key /etc/ansible/ansible.cfg 
host_key_checking = True

2) Clear foreman-proxy known hosts (to be sure)
# echo '' > ~foreman-proxy/.ssh/known_hosts

3) Run "Ansible Command" REX job with command 'uname'
>>> 100% success

4) Check content of knowhn hosts
# cat ~foreman-proxy/.ssh/known_hosts 
<FQDN>,<IP> ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwl/drCLTzg0prgoPlCV7WTBg4xJCdXoiAw7uCDLZhtmkyzf7l0X0+1hiBpAY+fhDBBabH/hWuTljK80zVq4l6fCNqNf5o4C83w4K7LGGERz+XX8xIXF5O65jv2QUPIiOIK5FwkkXxyK6yS7RcecnCxbGl8JHorfOLGNkzinDsLnhWKNXg4C6nKJLaHK7G59WxMoj01vuCI59RJB9elk8e+eTUz7dnmVNmDF/traDHQ5o1R9q9pJGL+L/RN6Lim8K3vUDyG2E59Hw27D0GSBzZR8lbXvcMyrZKY3K0dhOdcff8hUZx0uf3Zany41rdFqPR3LMqgKre5Ie4Ze3wTn3Vw==
<FQDN> ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwl/drCLTzg0prgoPlCV7WTBg4xJCdXoiAw7uCDLZhtmkyzf7l0X0+1hiBpAY+fhDBBabH/hWuTljK80zVq4l6fCNqNf5o4C83w4K7LGGERz+XX8xIXF5O65jv2QUPIiOIK5FwkkXxyK6yS7RcecnCxbGl8JHorfOLGNkzinDsLnhWKNXg4C6nKJLaHK7G59WxMoj01vuCI59RJB9elk8e+eTUz7dnmVNmDF/traDHQ5o1R9q9pJGL+L/RN6Lim8K3vUDyG2E59Hw27D0GSBzZR8lbXvcMyrZKY3K0dhOdcff8hUZx0uf3Zany41rdFqPR3LMqgKre5Ie4Ze3wTn3Vw==

5) @HOST: Regenerate SSH host keys:
# rm -rfv /etc/ssh/*key* && /sbin/service sshd restart
removed `/etc/ssh/ssh_host_dsa_key'
removed `/etc/ssh/ssh_host_dsa_key.pub'
removed `/etc/ssh/ssh_host_key'
removed `/etc/ssh/ssh_host_key.pub'
removed `/etc/ssh/ssh_host_rsa_key'
removed `/etc/ssh/ssh_host_rsa_key.pub'
Stopping sshd:                                             [  OK  ]
Generating SSH2 RSA host key:                              [  OK  ]
Generating SSH1 RSA host key:                              [  OK  ]
Generating SSH2 DSA host key:                              [  OK  ]
Starting sshd:                                             [  OK  ]

6) Rerun the job that succeeded in step 3)
>>> 100% fail

----
   1:
PLAY [all] *********************************************************************
   2:
TASK [Gathering Facts] *********************************************************
   3:
fatal: [<HOST_FQDN>]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nIT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!\r\nSomeone could be eavesdropping on you right now (man-in-the-middle attack)!\r\nIt is also possible that a host key has just been changed.\r\nThe fingerprint for the RSA key sent by the remote host is\nSHA256:Y459EjGU2E7ognLLHOrxOgBQACiMYLUlfYW3UHSK9CQ.\r\nPlease contact your system administrator.\r\nAdd correct host key in /usr/share/foreman-proxy/.ssh/known_hosts to get rid of this message.\r\nOffending RSA key in /usr/share/foreman-proxy/.ssh/known_hosts:3\r\nRSA host key for <HOST_FQDN> has changed and you have requested strict checking.\r\nHost key verification failed.\r\n", "unreachable": true}
   4:
to retry, use: --limit @/tmp/foreman-playbook-9d7b02f7-bdf9-4d0c-9fea-16ba16fb22da.retry
   5:
   6:
PLAY RECAP *********************************************************************
   7:
<HOST_FQDN> : ok=0 changed=0 unreachable=1 failed=0
   8:
Exit status: 4
---

>>> Though Ansible REX provide reasonable way to store SSH hostkey on 1st run, it still prevents mitm attacks (i.e to connect when ssh host key has changed)

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927