Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1468572

Summary: [3.6] Error upgrading control_plane
Product: OpenShift Container Platform Reporter: Jaspreet Kaur <jkaur>
Component: Cluster Version OperatorAssignee: Scott Dodson <sdodson>
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: high Docs Contact:
Priority: high    
Version: 3.4.1CC: aos-bugs, erich, jokerman, mmccomas, sdodson
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, the upgrade playbooks would use the default kubeconfig which may have been modified since creation to use a non admin user. Now the upgrade playbooks use the admin kubeconfig which avoids this problem.
Story Points: ---
Clone Of:
: 1470338 1470339 (view as bug list) Environment:
Last Closed: 2017-08-10 05:29:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1470338, 1470339    

Description Jaspreet Kaur 2017-07-07 12:24:45 UTC
Description of problem: Upgrading from 3.4 to 3.5 (3.5.5.15), we've hit the following error.

TASK [Reconcile Security Context Constraints] **********************************
fatal: [SQY05TM001]: FAILED! => {
    "changed": false,
    "cmd": [
        "oc",
        "adm",
        "policy",
        "reconcile-sccs",
        "--confirm",
        "--additive-only=true",
        "-o",
        "name"
    ],
    "delta": "0:00:00.158302",
    "end": "2017-07-07 11:50:36.765357",
    "failed": true,
    "rc": 1,
    "start": "2017-07-07 11:50:36.607055",
    "warnings": []
}

STDERR:

error: openshift-infra is not a valid namespace
See 'oc adm policy reconcile-sccs -h' for help and examples.

When user is not system:admin the above issue is faced to resolve this we have to manually add perform :

cp -f /etc/origin/master/admin.kubeconfig /home/wnradm/.kube/config
oc whoami
system:admin



Version-Release number of the following components:


How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
It fails to upgrade if the user is not system:admin

Expected results: 

It should run independent of which user is logged in

Additional info:
PR : https://github.com/openshift/openshift-ansible/pull/4710

Comment 8 Anping Li 2017-07-18 06:45:40 UTC
Failed to Drian node when the default user is a normal user 
TASK [Drain Node for Kubelet upgrade] ******************************************
fatal: [openshift-217.lab.eng.nay.redhat.com -> openshift-181.lab.eng.nay.redhat.com]: FAILED! => {
    "changed": true,
    "cmd": [
        "oadm",
        "drain",
        "openshift-217.lab.eng.nay.redhat.com",
        "--force",
        "--delete-local-data",
        "--ignore-daemonsets"
    ],
    "delta": "0:00:00.338681",
    "end": "2017-07-18 02:40:32.096593",
    "failed": true,
    "rc": 1,
    "start": "2017-07-18 02:40:31.757912",
    "warnings": []
}

STDERR:

Error from server (Forbidden): User "anli" cannot get nodes at the cluster scope

Comment 9 Scott Dodson 2017-07-21 13:56:42 UTC
More changes merged.

Comment 10 Anping Li 2017-07-24 05:06:49 UTC
Scott, which puddle/branch can I use? 

atomic-openshift-utils-3.6.162-1.git.0.50e29bd.el7.noarch still fail to drain nodes.

fatal: [openshift-217.lab.eng.nay.redhat.com -> openshift-181.lab.eng.nay.redhat.com]: FAILED! => {
    "attempts": 60, 
    "changed": true, 
    "cmd": [
        "oadm", 
        "drain", 
        "openshift-217.lab.eng.nay.redhat.com", 
        "--force", 
        "--delete-local-data", 
        "--ignore-daemonsets"
    ], 
    "delta": "0:00:00.338451", 
    "end": "2017-07-24 00:26:06.314641", 
    "failed": true, 
    "rc": 1, 
    "start": "2017-07-24 00:26:05.976190", 
    "warnings": []
}

STDERR:

Error from server (Forbidden): User "anli" cannot get nodes at the cluster scope

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************
	to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade.retry

Comment 11 Scott Dodson 2017-07-24 13:36:10 UTC
The next puddle will have it, 3.6.168 or later when it becomes available. I had to add a commit for this fix today. Moving back to MODIFIED until we have a new build.

Comment 12 Scott Dodson 2017-07-25 12:50:09 UTC
changes in v3.6.169

Comment 13 Anping Li 2017-07-28 04:42:12 UTC
Failed on atomic-openshift-utils-3.6.171-1.git.0.9d04eee.el7.noarch

failed: [openshift-181.lab.eng.nay.redhat.com] (item=/usr/share/openshift/examples/image-streams/dotnet_imagestreams.json) => {
    "changed": false, 
    "cmd": [
        "oc", 
        "create", 
        "-n", 
        "openshift", 
        "-f", 
        "/usr/share/openshift/examples/image-streams/dotnet_imagestreams.json"
    ], 
    "delta": "0:00:00.306632", 
    "end": "2017-07-27 23:45:44.090470", 
    "failed": true, 
    "failed_when_result": true, 
    "item": "/usr/share/openshift/examples/image-streams/dotnet_imagestreams.json", 
    "rc": 1, 
    "start": "2017-07-27 23:45:43.783838", 
    "warnings": []
}

STDERR:

Error from server (Forbidden): User "anli" cannot create imagestreams in project "openshift"

Comment 14 Scott Dodson 2017-07-28 12:41:17 UTC
The change was missed on release-3.6 branch, I've verified that there's no diffs between master and releae-3.6 after this pull request https://github.com/openshift/openshift-ansible/pull/4924

Comment 15 Anping Li 2017-08-01 04:18:28 UTC
Pass in atomic-openshift-utils-3.6.173.0.0-1.git.0.18ecb68.el7.noarch

Comment 17 errata-xmlrpc 2017-08-10 05:29:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716