Created attachment 1512950 [details]
Prereq task after changing the ssh keys

Description of problem:

Unable to install openshift 3.9.51 as the prereq task fails at the task in subject.

(0, '/home/ec2-user\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 28629\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 4\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit 

Tried regenerating the keys and it still fails. 

Passwordless ssh works fine outside of ansible. Even via ansible, all the tasks work except for anything to do with yum. All the yum tasks like yum info also take time to display an output. 

1. Any local command works immediately.
2. Any command run via ansible and retrieving information from the host returns immediately.
3. Any yum commands run via ansible hang indefinitely and the same command run over ssh work fine.
4. Running

ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ec2-user -o ConnectTimeout=10 -o ControlPath=/home/ec2-user/.ansible/cp/c839dbe1af ip-10-156-26-246.ap-southeast-2.compute.internal


outside the playbook works fine

5.ansible-playbook --timeout=40 -c paramiko <playbook>

also fails at the same point


Version-Release number of the following components:

openshift-ansible-3.9.51-1.git.0.c4968ca.el7.noarch
ansible-playbook 2.4.3.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/ec2-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]


Actual results:

The task hang indefinitely, the last few lines are

<hostname.example.com> (0, 'sftp> put /tmp/tmpsOBWcU /home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/yum.py\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 7926\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename" revision 1\r\ndebug2: Server supports extension "statvfs" revision 2\r\ndebug2: Server supports extension "fstatvfs" revision 2\r\ndebug2: Server supports extension "hardlink" revision 1\r\ndebug2: Server supports extension "fsync" revision 1\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/ec2-user size 0\r\ndebug3: Looking up /tmp/tmpsOBWcU\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/yum.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:851\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 851 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ec2-user
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ec2-user -o ConnectTimeout=10 -o ControlPath=/home/ec2-user/.ansible/cp/c839dbe1af hostname.example.com '/bin/sh -c '"'"'chmod u+x /home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/ /home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/yum.py && sleep 0'"'"''
<hostname.example.com> (0, '', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 7926\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ec2-user
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ec2-user -o ConnectTimeout=10 -o ControlPath=/home/ec2-user/.ansible/cp/c839dbe1af -tt hostname.example.com '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-bgkbbjyyiqbhhmbwpqexikpmmskhtpsu; /usr/bin/python /home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/yum.py; rm -rf "/home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"''