Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1536839 - Upgrade against containerized ocp failed at Verify masters are already upgraded
Upgrade against containerized ocp failed at Verify masters are already upgraded
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Upgrade (Show other bugs)
3.9.0
Unspecified Unspecified
high Severity high
: ---
: 3.9.0
Assigned To: Michael Gugino
Weihua Meng
: TestBlocker
Depends On: 1551388 1540537
Blocks:
  Show dependency treegraph
 
Reported: 2018-01-21 10:45 EST by Weihua Meng
Modified: 2018-03-28 10:22 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-03-28 10:21:18 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0489 None None None 2018-03-28 10:22 EDT

  None (edit)
Comment 1 liujia 2018-01-22 21:11:52 EST
Block container upgrade
Comment 2 Michael Gugino 2018-01-23 16:29:59 EST
PR Created: https://github.com/openshift/openshift-ansible/pull/6842
Comment 3 Weihua Meng 2018-01-25 06:55:12 EST
Fixed.
openshift-ansible-3.9.0-0.24.0.git.0.735690f.el7.noarch

PLAY [Verify masters are already upgraded] ************************************************************************************************************************************************************************

TASK [fail] *******************************************************************************************************************************************************************************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/pre/config.yml:60
skipping: [host-8-244-181.host.centralci.eng.rdu2.redhat.com] => {
    "changed": false,
    "skip_reason": "Conditional result was False"
}
META: ran handlers
META: ran handlers
PLAY [Validate configuration for rolling restart] *****************************************************************************************************************************************************************
Comment 4 Weihua Meng 2018-01-25 07:09:44 EST
Sorry. Wrong steps used above.

Still got error.
openshift-ansible-3.9.0-0.24.0.git.0.735690f.el7.noarch

TASK [fail] *******************************************************************************************************************************************************************************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/pre/config.yml:60
fatal: [host-8-244-181.host.centralci.eng.rdu2.redhat.com]: FAILED! => {
    "changed": false, 
    "msg": "Master running 3.9 must be upgraded to 3.9.0 before node upgrade can be run."
}
	to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_nodes.retry
Comment 5 Michael Gugino 2018-01-25 09:20:07 EST
Please include full verbose output of the play run.

Also, how are the masters being upgraded?  Is that happening during the same test?  I need to see inventory, procedure steps, and output for that as well.
Comment 9 Weihua Meng 2018-01-26 03:11:14 EST
1. set up OCP 3.7 on Atomic Host
2. ansible-playbook -vvv -i ah3726.inv /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_control_plane.yml | tee up39024ah_ctrl_01.log
3. ansible-playbook -vvv -i ah3726.inv /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_nodes.yml | tee up39024ah_nodes.log
Comment 10 Michael Gugino 2018-01-26 12:32:41 EST
Weihua,

  Thank you for the attachments, they are very helpful.

It seems this is an edge cause when upgrading from 3.8 to 3.9 instead of 3.7 to 3.9.

I have created additional logic to account for both scenarios regarding the usage of openshift_image_tag: https://github.com/openshift/openshift-ansible/pull/6896
Comment 11 liujia 2018-01-29 04:12:47 EST
Still hit the issue on latest build openshift-ansible-3.9.0-0.31.0.git.0.e0a0ad8.el7.noarch.
Comment 12 liujia 2018-01-29 04:17:35 EST
Blocker containerized ocp's upgrade test
Comment 13 Scott Dodson 2018-01-29 10:28:30 EST
PR from comment 10 has been merged but not in a tagged build yet, MODIFIED
Comment 15 Weihua Meng 2018-02-01 01:42:17 EST
not in latest build.
waiting for new build.
Comment 16 Xiaoli Tian 2018-02-01 22:29:20 EST
Has been merged since openshift-ansible-3.9.0-0.32
Comment 17 Weihua Meng 2018-02-08 08:24:46 EST
There is still something wrong. Please check it. Thanks.

openshift-ansible-3.9.0-0.41.0.git.0.8290c01.el7.noarch

containerized installation on Atomic Host
with openshift_image_tag=v3.9.0-0.41.0 in inventory file.

other parameters are same with before.

TASK [fail] *******************************************************************************************************************************************************************************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/pre/config.yml:60
fatal: [host-xxx.redhat.com]: FAILED! => {
    "changed": false, 
    "msg": "Master running 3.9.0 must be upgraded to 3.6.173.0.96 before node upgrade can be run."


Failure summary:


  1. Hosts:    host-xxx.redhat.com
     Play:     Verify masters are already upgraded
     Task:     fail
     Message:  Master running 3.9.0 must be upgraded to 3.6.173.0.96 before node upgrade can be run.
Comment 18 Michael Gugino 2018-02-15 10:50:20 EST
This should be fixed by: https://github.com/openshift/openshift-ansible/pull/7124

PR Merged.
Comment 19 Weihua Meng 2018-02-15 22:22:39 EST
Fixed.
openshift-ansible-3.9.0-0.45.0.git.0.05f6826.el7.noarch

Upgrade succeeded without error.
Thanks.
Comment 22 errata-xmlrpc 2018-03-28 10:21:18 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489

Note You need to log in before you can comment on or make changes to this bug.