Bug 1468572 - [3.6] Error upgrading control_plane
[3.6] Error upgrading control_plane
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Upgrade (Show other bugs)
3.4.1
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Scott Dodson
Anping Li
:
Depends On:
Blocks: 1470339 1470338
  Show dependency treegraph
 
Reported: 2017-07-07 08:24 EDT by Jaspreet Kaur
Modified: 2017-08-16 15 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, the upgrade playbooks would use the default kubeconfig which may have been modified since creation to use a non admin user. Now the upgrade playbooks use the admin kubeconfig which avoids this problem.
Story Points: ---
Clone Of:
: 1470338 1470339 (view as bug list)
Environment:
Last Closed: 2017-08-10 01:29:50 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jaspreet Kaur 2017-07-07 08:24:45 EDT
Description of problem: Upgrading from 3.4 to 3.5 (3.5.5.15), we've hit the following error.

TASK [Reconcile Security Context Constraints] **********************************
fatal: [SQY05TM001]: FAILED! => {
    "changed": false,
    "cmd": [
        "oc",
        "adm",
        "policy",
        "reconcile-sccs",
        "--confirm",
        "--additive-only=true",
        "-o",
        "name"
    ],
    "delta": "0:00:00.158302",
    "end": "2017-07-07 11:50:36.765357",
    "failed": true,
    "rc": 1,
    "start": "2017-07-07 11:50:36.607055",
    "warnings": []
}

STDERR:

error: openshift-infra is not a valid namespace
See 'oc adm policy reconcile-sccs -h' for help and examples.

When user is not system:admin the above issue is faced to resolve this we have to manually add perform :

cp -f /etc/origin/master/admin.kubeconfig /home/wnradm/.kube/config
oc whoami
system:admin



Version-Release number of the following components:


How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual results:
It fails to upgrade if the user is not system:admin

Expected results: 

It should run independent of which user is logged in

Additional info:
PR : https://github.com/openshift/openshift-ansible/pull/4710
Comment 8 Anping Li 2017-07-18 02:45:40 EDT
Failed to Drian node when the default user is a normal user 
TASK [Drain Node for Kubelet upgrade] ******************************************
fatal: [openshift-217.lab.eng.nay.redhat.com -> openshift-181.lab.eng.nay.redhat.com]: FAILED! => {
    "changed": true,
    "cmd": [
        "oadm",
        "drain",
        "openshift-217.lab.eng.nay.redhat.com",
        "--force",
        "--delete-local-data",
        "--ignore-daemonsets"
    ],
    "delta": "0:00:00.338681",
    "end": "2017-07-18 02:40:32.096593",
    "failed": true,
    "rc": 1,
    "start": "2017-07-18 02:40:31.757912",
    "warnings": []
}

STDERR:

Error from server (Forbidden): User "anli" cannot get nodes at the cluster scope
Comment 9 Scott Dodson 2017-07-21 09:56:42 EDT
More changes merged.
Comment 10 Anping Li 2017-07-24 01:06:49 EDT
Scott, which puddle/branch can I use? 

atomic-openshift-utils-3.6.162-1.git.0.50e29bd.el7.noarch still fail to drain nodes.

fatal: [openshift-217.lab.eng.nay.redhat.com -> openshift-181.lab.eng.nay.redhat.com]: FAILED! => {
    "attempts": 60, 
    "changed": true, 
    "cmd": [
        "oadm", 
        "drain", 
        "openshift-217.lab.eng.nay.redhat.com", 
        "--force", 
        "--delete-local-data", 
        "--ignore-daemonsets"
    ], 
    "delta": "0:00:00.338451", 
    "end": "2017-07-24 00:26:06.314641", 
    "failed": true, 
    "rc": 1, 
    "start": "2017-07-24 00:26:05.976190", 
    "warnings": []
}

STDERR:

Error from server (Forbidden): User "anli" cannot get nodes at the cluster scope

NO MORE HOSTS LEFT *************************************************************

NO MORE HOSTS LEFT *************************************************************
	to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade.retry
Comment 11 Scott Dodson 2017-07-24 09:36:10 EDT
The next puddle will have it, 3.6.168 or later when it becomes available. I had to add a commit for this fix today. Moving back to MODIFIED until we have a new build.
Comment 12 Scott Dodson 2017-07-25 08:50:09 EDT
changes in v3.6.169
Comment 13 Anping Li 2017-07-28 00:42:12 EDT
Failed on atomic-openshift-utils-3.6.171-1.git.0.9d04eee.el7.noarch

failed: [openshift-181.lab.eng.nay.redhat.com] (item=/usr/share/openshift/examples/image-streams/dotnet_imagestreams.json) => {
    "changed": false, 
    "cmd": [
        "oc", 
        "create", 
        "-n", 
        "openshift", 
        "-f", 
        "/usr/share/openshift/examples/image-streams/dotnet_imagestreams.json"
    ], 
    "delta": "0:00:00.306632", 
    "end": "2017-07-27 23:45:44.090470", 
    "failed": true, 
    "failed_when_result": true, 
    "item": "/usr/share/openshift/examples/image-streams/dotnet_imagestreams.json", 
    "rc": 1, 
    "start": "2017-07-27 23:45:43.783838", 
    "warnings": []
}

STDERR:

Error from server (Forbidden): User "anli" cannot create imagestreams in project "openshift"
Comment 14 Scott Dodson 2017-07-28 08:41:17 EDT
The change was missed on release-3.6 branch, I've verified that there's no diffs between master and releae-3.6 after this pull request https://github.com/openshift/openshift-ansible/pull/4924
Comment 15 Anping Li 2017-08-01 00:18:28 EDT
Pass in atomic-openshift-utils-3.6.173.0.0-1.git.0.18ecb68.el7.noarch
Comment 17 errata-xmlrpc 2017-08-10 01:29:50 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716

Note You need to log in before you can comment on or make changes to this bug.