Bug 1377170

Summary: Upgrade from oc v3.1.1.6 to 3.2.24 hangs
Product: OpenShift Container Platform Reporter: Jaspreet Kaur <jkaur>
Component: Cluster Version OperatorAssignee: Scott Dodson <sdodson>
Status: CLOSED CURRENTRELEASE QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: high    
Version: 3.1.0CC: aos-bugs, jiajliu, jkaur, jokerman, mmccomas, pep
Target Milestone: ---Keywords: Reopened
Target Release: 3.2.1Flags: jobarker: needinfo? (jkaur)
jobarker: needinfo? (jkaur)
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously we had shipped release candidate versions of Ansible 2.2 in order to address significant performance regressions in Ansible 2.0. However those release candidate versions suffered from reliability problems that have since been fixed in the GA release of Ansible 2.2.0.0. We have also updated the openshift-ansible code base to be compatible with Ansible 2.2.0.0.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-27 15:49:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Jaspreet Kaur 2016-09-19 04:58:47 UTC
Description of problem: When running upgrade playbook it always at the gathering facts phase.

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_2/upgrade.yml


Upgrade to :
ansible-2.2.0-0.5.prerelease.el7.noarch
openshift-ansible-lookup-plugins-3.2.24-1.git.0.337259b.el7.noarch


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 4 liujia 2016-09-20 09:43:56 UTC
I could not re-produce the bug in my env.

steps:
1, prepare 12 hosts(2 master+node,6 node,3 etcd,1 lb+nfs)
2, install ose v3.1.1.6(docker version:1.8.2,atomic-openshift-utils-3.0.94-1.git.0.67a822a.el7.noarch.rpm) successfully.
3, new app successfully
4, yum update with repo ose3.2 on my install machine(ansible-2.2.0-0.5.prerelease.el7.noarch,atomic-openshift-utils-3.2.24-1.git.0.337259b.el7.noarch) 
5, replace repo with ose3.2 on above 12 hosts
6, run upgrade playbook

result:
It can upgrade successfully.

Comment 5 Anping Li 2016-09-20 09:59:35 UTC
How many pods are there on your unconnected hosts? Have you reserved cpu/memory for ssh?  Pods may crush into other nodes due to evacuate node during upgrade.  Be sure there are enough free resource before upgrade.

Comment 23 John Barker 2016-10-19 06:54:35 UTC
All Ansible bug reports should include the following:
 * Output of ansible -v
 * Output of "ansible-playbook -vvv ..." of a failing run
 * Full ansible-playbook commandline from a failing run
 * Link to the full source of playbooks would be useful
 * If this is a regression - what version of Ansible did this used to work, what version did it first fail on
 * Details of the machine (e.g. OS) running ansible (local)
 * Details of the machine (e.g. OS) ansible is configuring (remote)


For this specific bug report:
 * Do any tasks *before* gathering facts work
 * What is the purpose of openshift-ansible-lookup-plugins
 * Can you please point me at the source for openshift-ansible-lookup-plugins-3.2.24-1.git.0.337259b.el7.noarch

Comment 24 John Barker 2016-10-19 06:54:47 UTC
All Ansible bug reports should include the following:
 * Output of ansible -v
 * Output of "ansible-playbook -vvv ..." of a failing run
 * Full ansible-playbook commandline from a failing run
 * Link to the full source of playbooks would be useful
 * If this is a regression - what version of Ansible did this used to work, what version did it first fail on
 * Details of the machine (e.g. OS) running ansible (local)
 * Details of the machine (e.g. OS) ansible is configuring (remote)


For this specific bug report:
 * Do any tasks *before* gathering facts work
 * What is the purpose of openshift-ansible-lookup-plugins
 * Can you please point me at the source for openshift-ansible-lookup-plugins-3.2.24-1.git.0.337259b.el7.noarch

Comment 25 Scott Dodson 2016-10-19 13:14:33 UTC
John,

The ansible code in question would be https://github.com/openshift/openshift-ansible/tree/openshift-ansible-3.2.24-1

The ansible version was from devel branch at e244895.

Comment 8 and the logs attached from the debugging attempts to run the setup module are probably the only thing we can work with right now.

Comment 26 John Barker 2016-10-19 13:23:28 UTC
Scott,
Thanks for the details

Comment 29 Anping Li 2016-10-31 07:56:00 UTC
It work during QE testing, so Move to verified.

Comment 32 Anping Li 2016-11-04 06:25:11 UTC
it should be debug_level=2 rather than openshift_debug_level="2".