Bug 1577960

Summary: [3.9] Fail to upgrade ocp without docker registry deployed
Product: OpenShift Container Platform Reporter: Scott Dodson <sdodson>
Component: Cluster Version OperatorAssignee: Vadim Rutkovsky <vrutkovs>
Status: CLOSED ERRATA QA Contact: liujia <jiajliu>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.9.0CC: aos-bugs, byount, dmoessne, dzhukous, fshaikh, jiajliu, jkaur, jokerman, mmccomas, rbost, sdodson, vrutkovs, vwalek, wmeng
Target Milestone: ---   
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1559248 Environment:
Last Closed: 2018-06-15 13:54:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1559248, 1581243    
Bug Blocks:    

Comment 2 liujia 2018-05-18 07:07:30 UTC
Still failed at openshift-ansible-3.9.29-1.git.0.051bc5c.el7.noarch
Checked that the pr has not in the latest build.

Comment 4 Vadim Rutkovsky 2018-05-22 12:18:39 UTC
*** Bug 1581243 has been marked as a duplicate of this bug. ***

Comment 5 Vadim Rutkovsky 2018-05-22 13:10:53 UTC
Additional fix to resolve remaining issues: https://github.com/openshift/openshift-ansible/pull/8474

Comment 6 Vadim Rutkovsky 2018-05-28 08:17:02 UTC
Fix is available in openshift-ansible-3.9.30-1

Comment 7 liujia 2018-05-29 06:05:00 UTC
Version:
openshift-ansible-3.9.30-1.git.0.a91a657.el7.noarch


Upgrade still failed at a new task [openshift_web_console : Verify that the web console is running], original failure after this task couldn't be checked. 

TASK [openshift_web_console : Verify that the web console is running] **********
task path: /usr/share/ansible/openshift-ansible/roles/openshift_web_console/tasks/install.yml:137
FAILED - RETRYING: Verify that the web console is running (60 retries left).
...
FAILED - RETRYING: Verify that the web console is running (1 retries left).
fatal: [x]: FAILED! => {"attempts": 60, "changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "https://webconsole.openshift-web-console.svc/healthz"}
<--snip-->
TASK [openshift_web_console : Report console errors] ***************************
task path: /usr/share/ansible/openshift-ansible/roles/openshift_web_console/tasks/install.yml:190
fatal: [x]: FAILED! => {"changed": false, "failed": true, "msg": "Console install failed."}

This was because web-console needed to be installed during upgrade from v3.7 to v3.9, and web-console can not be deployed successfully without docker-registry.  But our upgrade process would be abort if web-console was not running.

Comment 8 Vadim Rutkovsky 2018-05-29 15:22:16 UTC
(In reply to liujia from comment #7)
> This was because web-console needed to be installed during upgrade from v3.7
> to v3.9, and web-console can not be deployed successfully without
> docker-registry.  But our upgrade process would be abort if web-console was
> not running.

I can't reproduce this - both upgrades with 'openshift_web_console_install' set to 'true' and 'false' have succeeded here.

Please attach inventory, 3.7 install log and 3.9 upgrade log

Comment 11 liujia 2018-05-31 07:37:26 UTC
@ Vadim Rutkovsky

Thx for the remind, I just forgot the image setting issue for 3.7-3.9 upgrade. Sorry for the delay verify. Now original issue can be verified.

Verified on openshift-ansible-3.9.30-1.git.0.a91a657.el7.noarch

TASK [Redeploy docker registry] ************************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/openshift-hosted/private/redeploy-registry-certificates.yml:90
skipping: [x] => {"changed": false, "skip_reason": "Conditional result was False", "skipped": true}

Comment 13 Scott Dodson 2018-06-15 13:54:40 UTC
Fixed in openshift-ansible-3.9.30-1 which has already shipped.