Bug 1377170 - Upgrade from oc v3.1.1.6 to 3.2.24 hangs [NEEDINFO]
Summary: Upgrade from oc v3.1.1.6 to 3.2.24 hangs
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 3.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.2.1
Assignee: Scott Dodson
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-19 04:58 UTC by Jaspreet Kaur
Modified: 2019-12-16 06:48 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously we had shipped release candidate versions of Ansible 2.2 in order to address significant performance regressions in Ansible 2.0. However those release candidate versions suffered from reliability problems that have since been fixed in the GA release of Ansible 2.2.0.0. We have also updated the openshift-ansible code base to be compatible with Ansible 2.2.0.0.
Clone Of:
Environment:
Last Closed: 2017-02-27 15:49:04 UTC
Target Upstream Version:
jobarker: needinfo? (jkaur)
jobarker: needinfo? (jkaur)


Attachments (Terms of Use)

Description Jaspreet Kaur 2016-09-19 04:58:47 UTC
Description of problem: When running upgrade playbook it always at the gathering facts phase.

ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_2/upgrade.yml


Upgrade to :
ansible-2.2.0-0.5.prerelease.el7.noarch
openshift-ansible-lookup-plugins-3.2.24-1.git.0.337259b.el7.noarch


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 4 liujia 2016-09-20 09:43:56 UTC
I could not re-produce the bug in my env.

steps:
1, prepare 12 hosts(2 master+node,6 node,3 etcd,1 lb+nfs)
2, install ose v3.1.1.6(docker version:1.8.2,atomic-openshift-utils-3.0.94-1.git.0.67a822a.el7.noarch.rpm) successfully.
3, new app successfully
4, yum update with repo ose3.2 on my install machine(ansible-2.2.0-0.5.prerelease.el7.noarch,atomic-openshift-utils-3.2.24-1.git.0.337259b.el7.noarch) 
5, replace repo with ose3.2 on above 12 hosts
6, run upgrade playbook

result:
It can upgrade successfully.

Comment 5 Anping Li 2016-09-20 09:59:35 UTC
How many pods are there on your unconnected hosts? Have you reserved cpu/memory for ssh?  Pods may crush into other nodes due to evacuate node during upgrade.  Be sure there are enough free resource before upgrade.

Comment 23 John Barker 2016-10-19 06:54:35 UTC
All Ansible bug reports should include the following:
 * Output of ansible -v
 * Output of "ansible-playbook -vvv ..." of a failing run
 * Full ansible-playbook commandline from a failing run
 * Link to the full source of playbooks would be useful
 * If this is a regression - what version of Ansible did this used to work, what version did it first fail on
 * Details of the machine (e.g. OS) running ansible (local)
 * Details of the machine (e.g. OS) ansible is configuring (remote)


For this specific bug report:
 * Do any tasks *before* gathering facts work
 * What is the purpose of openshift-ansible-lookup-plugins
 * Can you please point me at the source for openshift-ansible-lookup-plugins-3.2.24-1.git.0.337259b.el7.noarch

Comment 24 John Barker 2016-10-19 06:54:47 UTC
All Ansible bug reports should include the following:
 * Output of ansible -v
 * Output of "ansible-playbook -vvv ..." of a failing run
 * Full ansible-playbook commandline from a failing run
 * Link to the full source of playbooks would be useful
 * If this is a regression - what version of Ansible did this used to work, what version did it first fail on
 * Details of the machine (e.g. OS) running ansible (local)
 * Details of the machine (e.g. OS) ansible is configuring (remote)


For this specific bug report:
 * Do any tasks *before* gathering facts work
 * What is the purpose of openshift-ansible-lookup-plugins
 * Can you please point me at the source for openshift-ansible-lookup-plugins-3.2.24-1.git.0.337259b.el7.noarch

Comment 25 Scott Dodson 2016-10-19 13:14:33 UTC
John,

The ansible code in question would be https://github.com/openshift/openshift-ansible/tree/openshift-ansible-3.2.24-1

The ansible version was from devel branch at e244895.

Comment 8 and the logs attached from the debugging attempts to run the setup module are probably the only thing we can work with right now.

Comment 26 John Barker 2016-10-19 13:23:28 UTC
Scott,
Thanks for the details

Comment 29 Anping Li 2016-10-31 07:56:00 UTC
It work during QE testing, so Move to verified.

Comment 32 Anping Li 2016-11-04 06:25:11 UTC
it should be debug_level=2 rather than openshift_debug_level="2".


Note You need to log in before you can comment on or make changes to this bug.