Bug 1583151 - [3.7]upgrade failed at TASK [Drain Node for Kubelet upgrade]
Summary: [3.7]upgrade failed at TASK [Drain Node for Kubelet upgrade]
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 3.7.1
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 3.7.z
Assignee: Vadim Rutkovsky
QA Contact: Weihua Meng
URL:
Whiteboard:
: 1585657 (view as bug list)
Depends On:
Blocks: 1579321 1583152
TreeView+ depends on / blocked
 
Reported: 2018-05-28 10:32 UTC by Weihua Meng
Modified: 2018-06-27 07:59 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1583152 (view as bug list)
Environment:
Last Closed: 2018-06-27 07:59:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:2009 0 None None None 2018-06-27 07:59:54 UTC

Description Weihua Meng 2018-05-28 10:32:48 UTC
Description of problem:
[3.7]upgrade failed at TASK [Drain Node for Kubelet upgrade]
error is caused by https://github.com/openshift/openshift-ansible/blob/release-3.6/playbooks/common/openshift-cluster/upgrades/upgrade_control_plane.yml#L386

introduced by 
https://github.com/openshift/openshift-ansible/commit/ba74ec43b9e4d743f62b89ad4f316a45e7fd09c9#diff-6e1f944c172a66b8294fa8cc2b081a97

Version-Release number of the following components:
openshift-ansible-3.7.51-1.git.0.f9b681c.el7.noarch

How reproducible:
Always

Steps to Reproduce:
1. upgrade OCP 3.6 to 3.7

Actual results:
TASK [Drain Node for Kubelet upgrade] ******************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/upgrade_control_plane.yml:408
FAILED - RETRYING: Drain Node for Kubelet upgrade (1 retries left).
fatal: [host-8-246-90.host.centralci.eng.rdu2.redhat.com -> host-8-246-90.host.centralci.eng.rdu2.redhat.com]: FAILED! => {"attempts": 1, "changed": true, "cmd": ["oadm", "adm", "drain", "host-8-246-90.host.centralci.eng.rdu2.redhat.com", "--config=/etc/origin/master/admin.kubeconfig", "--force", "--delete-local-data", "--ignore-daemonsets", "--timeout=0s"], "delta": "0:00:00.241473", "end": "2018-05-28 05:12:14.862622", "failed": true, "failed_when_result": true, "rc": 1, "start": "2018-05-28 05:12:14.621149", "stderr": "Error: unknown command \"adm\" for \"oadm\"\nRun 'oadm --help' for usage.", "stderr_lines": ["Error: unknown command \"adm\" for \"oadm\"", "Run 'oadm --help' for usage."], "stdout": "", "stdout_lines": []}

Expected results:
Upgrade succeeds.

Comment 1 Vadim Rutkovsky 2018-05-28 11:40:53 UTC
Created https://github.com/openshift/openshift-ansible/pull/8547 to fix this

Comment 2 Weihua Meng 2018-06-02 06:04:52 UTC
Fixed.
openshift-ansible-3.7.52-1.git.0.3fddee4.el7.noarch


TASK [Drain Node for Kubelet upgrade] ******************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/upgrade_control_plane.yml:408

changed: [qe-wmengah36etcd-master-etcd-1.0601-51o.qe.rhcloud.com -> qe-wmengah36etcd-master-etcd-1.0601-51o.qe.rhcloud.com] => {
    "attempts": 1,
    "changed": true,
    "cmd": [
        "/usr/local/bin/oadm",
        "drain",
        "qe-wmengah36etcd-master-etcd-1",
        "--config=/etc/origin/master/admin.kubeconfig",
        "--force",
        "--delete-local-data",
        "--ignore-daemonsets",
        "--timeout=0s"
    ],
    "delta": "0:00:00.319085",
    "end": "2018-06-01 22:41:54.408099",
    "failed": false,
    "failed_when_result": false,
    "invocation": {
        "module_args": {
            "_raw_params": "/usr/local/bin/oadm drain qe-wmengah36etcd-master-etcd-1 --config=/etc/origin/master/admin.kubeconfig --force --delete-local-data --ignore-daemonsets --timeout=0s",

Comment 3 liujia 2018-06-25 06:58:19 UTC
*** Bug 1585657 has been marked as a duplicate of this bug. ***

Comment 5 errata-xmlrpc 2018-06-27 07:59:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2009


Note You need to log in before you can comment on or make changes to this bug.