Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1470338

Summary: [3.5] Error upgrading control_plane
Product: OpenShift Container Platform Reporter: Scott Dodson <sdodson>
Component: Cluster Version OperatorAssignee: Scott Dodson <sdodson>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: high    
Version: 3.5.1CC: anli, aos-bugs, erich, jiajliu, jkaur, jokerman, mmccomas, sdodson
Target Milestone: ---   
Target Release: 3.5.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, the upgrade playbooks would use the default kubeconfig which may have been modified since creation to use a non admin user. Now the upgrade playbooks use the admin kubeconfig which avoids this problem.
Story Points: ---
Clone Of: 1468572 Environment:
Last Closed: 2017-07-27 18:02:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1468572, 1470339    
Bug Blocks:    

Comment 2 Anping Li 2017-07-13 07:04:39 UTC
@scott, I hit similar issue when Drain Node for Kubelet upgrade. I think there should be similar issue with import template, redeploy-certificates, deploy logging and metrics and etc. If we fix all of them, QE need a regression testing.

Reproduce steps:
1. oc login as a normal user
2. run upgrade playbook.

TASK [Mark node unschedulable] *************************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/upgrade_control_plane.yml:264
changed: [openshift-214.lab.eng.nay.redhat.com -> openshift-214.lab.eng.nay.redhat.com] => {
    "attempts": 1, 
    "changed": true, 
    "results": {
        "cmd": "/usr/bin/oc adm manage-node openshift-214.lab.eng.nay.redhat.com --schedulable=False", 
        "nodes": [
            {
                "name": "openshift-214.lab.eng.nay.redhat.com", 
                "schedulable": false
            }
        ], 
        "results": "NAME                                   STATUS                     AGE\nopenshift-214.lab.eng.nay.redhat.com   Ready,SchedulingDisabled   85d\n", 
        "returncode": 0
    }, 
    "state": "present"
}

TASK [Drain Node for Kubelet upgrade] ******************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/upgrade_control_plane.yml:274
fatal: [openshift-214.lab.eng.nay.redhat.com -> openshift-214.lab.eng.nay.redhat.com]: FAILED! => {
    "changed": true, 
    "cmd": [
        "oadm", 
        "drain", 
        "openshift-214.lab.eng.nay.redhat.com", 
        "--force", 
        "--delete-local-data", 
        "--ignore-daemonsets"
    ], 
    "delta": "0:00:00.307291", 
    "end": "2017-07-13 02:50:34.099727", 
    "failed": true, 
    "rc": 1, 
    "start": "2017-07-13 02:50:33.792436", 
    "warnings": []
}

STDERR:

Error from server (Forbidden): User "anli" cannot get nodes at the cluster scope

Comment 4 Anping Li 2017-07-17 02:02:47 UTC
Verified and pass with openshift-ansible-3.5.99

Comment 5 Anping Li 2017-07-18 08:41:05 UTC
Drain node are in upgrade_control_plane.yml, upgrade_nodes.yml and docker_upgrade.yml.

The admin kubeconfig also need to be added to [1] [2], so re-open this bug.

[1]playbooks/common/openshift-cluster/upgrades/upgrade_nodes.yml
[2]playbooks/byo/openshift-cluster/upgrades/docker/docker_upgrade.yml

Comment 7 liujia 2017-07-21 08:21:03 UTC
Version:
atomic-openshift-utils-3.5.101-1.git.0.0107544.el7.noarch

Steps:
1, Install ocp3.4
2, oc login with non system:admin user on master host
...
current-context: /x.x.x.x:8443/jliu
...
3, upgrade 3.4 to 3.5
#upgrade_control_plane.yml
#upgrade_nodes.yml

Upgrade succeed.

Comment 9 errata-xmlrpc 2017-07-27 18:02:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1810