Bug 1657610 - Openshift installation fails at task "Ensure openshift-ansible installer package deps are installed"
Summary: Openshift installation fails at task "Ensure openshift-ansible installer pack...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: ---
Assignee: Scott Dodson
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-10 05:58 UTC by Venkata Tadimarri
Modified: 2018-12-13 21:59 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-12-13 21:59:18 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Venkata Tadimarri 2018-12-10 05:58:24 UTC
Created attachment 1512950 [details]
Prereq task after changing the ssh keys

Description of problem:

Unable to install openshift 3.9.51 as the prereq task fails at the task in subject.

(0, '/home/ec2-user\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 28629\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 4\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit 

Tried regenerating the keys and it still fails. 

Passwordless ssh works fine outside of ansible. Even via ansible, all the tasks work except for anything to do with yum. All the yum tasks like yum info also take time to display an output. 

1. Any local command works immediately.
2. Any command run via ansible and retrieving information from the host returns immediately.
3. Any yum commands run via ansible hang indefinitely and the same command run over ssh work fine.
4. Running

ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ec2-user -o ConnectTimeout=10 -o ControlPath=/home/ec2-user/.ansible/cp/c839dbe1af ip-10-156-26-246.ap-southeast-2.compute.internal


outside the playbook works fine

5.ansible-playbook --timeout=40 -c paramiko <playbook>

also fails at the same point


Version-Release number of the following components:

openshift-ansible-3.9.51-1.git.0.c4968ca.el7.noarch
ansible-playbook 2.4.3.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/ec2-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]


Actual results:

The task hang indefinitely, the last few lines are

<hostname.example.com> (0, 'sftp> put /tmp/tmpsOBWcU /home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/yum.py\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 7926\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename@openssh.com" revision 1\r\ndebug2: Server supports extension "statvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "fstatvfs@openssh.com" revision 2\r\ndebug2: Server supports extension "hardlink@openssh.com" revision 1\r\ndebug2: Server supports extension "fsync@openssh.com" revision 1\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/ec2-user size 0\r\ndebug3: Looking up /tmp/tmpsOBWcU\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/yum.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:851\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 851 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ec2-user
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ec2-user -o ConnectTimeout=10 -o ControlPath=/home/ec2-user/.ansible/cp/c839dbe1af hostname.example.com '/bin/sh -c '"'"'chmod u+x /home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/ /home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/yum.py && sleep 0'"'"''
<hostname.example.com> (0, '', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 7926\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ec2-user
<hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ec2-user -o ConnectTimeout=10 -o ControlPath=/home/ec2-user/.ansible/cp/c839dbe1af -tt hostname.example.com '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-bgkbbjyyiqbhhmbwpqexikpmmskhtpsu; /usr/bin/python /home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/yum.py; rm -rf "/home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"''

Comment 4 Scott Dodson 2018-12-13 21:59:18 UTC
This support case is currently being debugged by the Ansible team investigating the yum module stalling on network transport. I'm closing this as there's no clear indication that there's a problem with openshift-ansible.


Note You need to log in before you can comment on or make changes to this bug.