Bug 1657610
| Summary: | Openshift installation fails at task "Ensure openshift-ansible installer package deps are installed" | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Venkata Tadimarri <ktadimar> |
| Component: | Installer | Assignee: | Scott Dodson <sdodson> |
| Installer sub component: | openshift-ansible | QA Contact: | Johnny Liu <jialiu> |
| Status: | CLOSED NOTABUG | Docs Contact: | |
| Severity: | urgent | ||
| Priority: | urgent | CC: | knakai, mperz, nikjain, openshift-bugs-escalate, sdodson |
| Version: | 3.9.0 | ||
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-12-13 21:59:18 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This support case is currently being debugged by the Ansible team investigating the yum module stalling on network transport. I'm closing this as there's no clear indication that there's a problem with openshift-ansible. |
Created attachment 1512950 [details] Prereq task after changing the ssh keys Description of problem: Unable to install openshift 3.9.51 as the prereq task fails at the task in subject. (0, '/home/ec2-user\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 28629\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 4\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit Tried regenerating the keys and it still fails. Passwordless ssh works fine outside of ansible. Even via ansible, all the tasks work except for anything to do with yum. All the yum tasks like yum info also take time to display an output. 1. Any local command works immediately. 2. Any command run via ansible and retrieving information from the host returns immediately. 3. Any yum commands run via ansible hang indefinitely and the same command run over ssh work fine. 4. Running ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ec2-user -o ConnectTimeout=10 -o ControlPath=/home/ec2-user/.ansible/cp/c839dbe1af ip-10-156-26-246.ap-southeast-2.compute.internal outside the playbook works fine 5.ansible-playbook --timeout=40 -c paramiko <playbook> also fails at the same point Version-Release number of the following components: openshift-ansible-3.9.51-1.git.0.c4968ca.el7.noarch ansible-playbook 2.4.3.0 config file = /etc/ansible/ansible.cfg configured module search path = [u'/home/ec2-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible-playbook python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] Actual results: The task hang indefinitely, the last few lines are <hostname.example.com> (0, 'sftp> put /tmp/tmpsOBWcU /home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/yum.py\n', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 7926\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "posix-rename" revision 1\r\ndebug2: Server supports extension "statvfs" revision 2\r\ndebug2: Server supports extension "fstatvfs" revision 2\r\ndebug2: Server supports extension "hardlink" revision 1\r\ndebug2: Server supports extension "fsync" revision 1\r\ndebug3: Sent message fd 5 T:16 I:1\r\ndebug3: SSH_FXP_REALPATH . -> /home/ec2-user size 0\r\ndebug3: Looking up /tmp/tmpsOBWcU\r\ndebug3: Sent message fd 5 T:17 I:2\r\ndebug3: Received stat reply T:101 I:2\r\ndebug1: Couldn\'t stat remote file: No such file or directory\r\ndebug3: Sent message SSH2_FXP_OPEN I:3 P:/home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/yum.py\r\ndebug3: Sent message SSH2_FXP_WRITE I:4 O:0 S:32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 4 32768 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:5 O:32768 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:65536 S:32768\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:98304 S:851\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 5 32768 bytes at 32768\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 32768 bytes at 65536\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 851 bytes at 98304\r\ndebug3: Sent message SSH2_FXP_CLOSE I:4\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n') <hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ec2-user <hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ec2-user -o ConnectTimeout=10 -o ControlPath=/home/ec2-user/.ansible/cp/c839dbe1af hostname.example.com '/bin/sh -c '"'"'chmod u+x /home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/ /home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/yum.py && sleep 0'"'"'' <hostname.example.com> (0, '', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 7926\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n') <hostname.example.com> ESTABLISH SSH CONNECTION FOR USER: ec2-user <hostname.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ec2-user -o ConnectTimeout=10 -o ControlPath=/home/ec2-user/.ansible/cp/c839dbe1af -tt hostname.example.com '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-bgkbbjyyiqbhhmbwpqexikpmmskhtpsu; /usr/bin/python /home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/yum.py; rm -rf "/home/ec2-user/.ansible/tmp/ansible-tmp-1544409345.8-49105038154697/" > /dev/null 2>&1'"'"'"'"'"'"'"'"' && sleep 0'"'"''