Bug 1468572 - [3.6] Error upgrading control_plane
Summary: [3.6] Error upgrading control_plane
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 3.4.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Scott Dodson
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks: 1470338 1470339
TreeView+ depends on / blocked
 
Reported: 2017-07-07 12:24 UTC by Jaspreet Kaur
Modified: 2020-12-14 09:03 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, the upgrade playbooks would use the default kubeconfig which may have been modified since creation to use a non admin user. Now the upgrade playbooks use the admin kubeconfig which avoids this problem.
Clone Of:
: 1470338 1470339 (view as bug list)
Environment:
Last Closed: 2017-08-10 05:29:50 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1716 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.6 RPM Release Advisory 2017-08-10 09:02:50 UTC

Description Jaspreet Kaur 2017-07-07 12:24:45 UTC
Description of problem: Upgrading from 3.4 to 3.5 (3.5.5.15), we've hit the following error.

TASK [Reconcile Security Context Constraints] **********************************
fatal: [SQY05TM001]: FAILED! => {
    "changed": false,
    "cmd": [
        "oc",
        "adm",
        "policy",
        "reconcile-sccs",
        "--confirm",
        "--additive-only=true",
        "-o",
        "name"
    ],
    "delta": "0:00:00.158302",
    "end": "2017-07-07 11:50:36.765357",
    "failed": true,
    "rc": 1,
    "start": "2017-07-07 11:50:36.607055",
    "warnings": []
}

STDERR:

error: openshift-infra is not a valid namespace
See 'oc adm policy reconcile-sccs -h' for help and examples.

When user is not system:admin the above issue is faced to resolve this we have to manually add perform :

cp -f /etc/origin/master/admin.kubeconfig /home/wnradm/.kube/config
oc whoami
system:admin



Version-Release number of the following components:


How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
It fails to upgrade if the user is not system:admin

Expected results: 

It should run independent of which user is logged in

Additional info:
PR : https://github.com/openshift/openshift-ansible/pull/4710

Comment 8 Anping Li 2017-07-18 06:45:40 UTC
Failed to Drian node when the default user is a normal user 
TASK [Drain Node for Kubelet upgrade] ******************************************
fatal: [openshift-217.lab.eng.nay.redhat.com -> openshift-181.lab.eng.nay.redhat.com]: FAILED! => {
    "changed": true,
    "cmd": [
        "oadm",
        "drain",
        "openshift-217.lab.eng.nay.redhat.com",
        "--force",
        "--delete-local-data",
        "--ignore-daemonsets"
    ],
    "delta": "0:00:00.338681",
    "end": "2017-07-18 02:40:32.096593",
    "failed": true,
    "rc": 1,
    "start": "2017-07-18 02:40:31.757912",
    "warnings": []
}

STDERR:

Error from server (Forbidden): User "anli" cannot get nodes at the cluster scope

Comment 9 Scott Dodson 2017-07-21 13:56:42 UTC
More changes merged.

Comment 10 Anping Li 2017-07-24 05:06:49 UTC
Scott, which puddle/branch can I use? 

atomic-openshift-utils-3.6.162-1.git.0.50e29bd.el7.noarch still fail to drain nodes.

fatal: [openshift-217.lab.eng.nay.redhat.com -> openshift-181.lab.eng.nay.redhat.com]: FAILED! => {
    "attempts": 60, 
    "changed": true, 
    "cmd": [
        "oadm", 
        "drain", 
        "openshift-217.lab.eng.nay.redhat.com", 
        "--force", 
        "--delete-local-data", 
        "--ignore-daemonsets"
    ], 
    "delta": "0:00:00.338451", 
    "end": "2017-07-24 00:26:06.314641", 
    "failed": true, 
    "rc": 1, 
    "start": "2017-07-24 00:26:05.976190", 
    "warnings": []
}

STDERR:

Error from server (Forbidden): User "anli" cannot get nodes at the cluster scope

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************
	to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade.retry

Comment 11 Scott Dodson 2017-07-24 13:36:10 UTC
The next puddle will have it, 3.6.168 or later when it becomes available. I had to add a commit for this fix today. Moving back to MODIFIED until we have a new build.

Comment 12 Scott Dodson 2017-07-25 12:50:09 UTC
changes in v3.6.169

Comment 13 Anping Li 2017-07-28 04:42:12 UTC
Failed on atomic-openshift-utils-3.6.171-1.git.0.9d04eee.el7.noarch

failed: [openshift-181.lab.eng.nay.redhat.com] (item=/usr/share/openshift/examples/image-streams/dotnet_imagestreams.json) => {
    "changed": false, 
    "cmd": [
        "oc", 
        "create", 
        "-n", 
        "openshift", 
        "-f", 
        "/usr/share/openshift/examples/image-streams/dotnet_imagestreams.json"
    ], 
    "delta": "0:00:00.306632", 
    "end": "2017-07-27 23:45:44.090470", 
    "failed": true, 
    "failed_when_result": true, 
    "item": "/usr/share/openshift/examples/image-streams/dotnet_imagestreams.json", 
    "rc": 1, 
    "start": "2017-07-27 23:45:43.783838", 
    "warnings": []
}

STDERR:

Error from server (Forbidden): User "anli" cannot create imagestreams in project "openshift"

Comment 14 Scott Dodson 2017-07-28 12:41:17 UTC
The change was missed on release-3.6 branch, I've verified that there's no diffs between master and releae-3.6 after this pull request https://github.com/openshift/openshift-ansible/pull/4924

Comment 15 Anping Li 2017-08-01 04:18:28 UTC
Pass in atomic-openshift-utils-3.6.173.0.0-1.git.0.18ecb68.el7.noarch

Comment 17 errata-xmlrpc 2017-08-10 05:29:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716


Note You need to log in before you can comment on or make changes to this bug.