Bug 1583151

Summary: [3.7]upgrade failed at TASK [Drain Node for Kubelet upgrade]
Product: OpenShift Container Platform Reporter: Weihua Meng <wmeng>
Component: Cluster Version OperatorAssignee: Vadim Rutkovsky <vrutkovs>
Status: CLOSED ERRATA QA Contact: Weihua Meng <wmeng>
Severity: high Docs Contact:
Priority: urgent    
Version: 3.7.1CC: aos-bugs, dyan, jiajliu, jokerman, mmccomas, vrutkovs
Target Milestone: ---   
Target Release: 3.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1583152 (view as bug list) Environment:
Last Closed: 2018-06-27 07:59:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1579321, 1583152    

Description Weihua Meng 2018-05-28 10:32:48 UTC 
Description of problem:
[3.7]upgrade failed at TASK [Drain Node for Kubelet upgrade]
error is caused by https://github.com/openshift/openshift-ansible/blob/release-3.6/playbooks/common/openshift-cluster/upgrades/upgrade_control_plane.yml#L386

introduced by 
https://github.com/openshift/openshift-ansible/commit/ba74ec43b9e4d743f62b89ad4f316a45e7fd09c9#diff-6e1f944c172a66b8294fa8cc2b081a97

Version-Release number of the following components:
openshift-ansible-3.7.51-1.git.0.f9b681c.el7.noarch

How reproducible:
Always

Steps to Reproduce:
1. upgrade OCP 3.6 to 3.7

Actual results:
TASK [Drain Node for Kubelet upgrade] ******************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/upgrade_control_plane.yml:408
FAILED - RETRYING: Drain Node for Kubelet upgrade (1 retries left).
fatal: [host-8-246-90.host.centralci.eng.rdu2.redhat.com -> host-8-246-90.host.centralci.eng.rdu2.redhat.com]: FAILED! => {"attempts": 1, "changed": true, "cmd": ["oadm", "adm", "drain", "host-8-246-90.host.centralci.eng.rdu2.redhat.com", "--config=/etc/origin/master/admin.kubeconfig", "--force", "--delete-local-data", "--ignore-daemonsets", "--timeout=0s"], "delta": "0:00:00.241473", "end": "2018-05-28 05:12:14.862622", "failed": true, "failed_when_result": true, "rc": 1, "start": "2018-05-28 05:12:14.621149", "stderr": "Error: unknown command \"adm\" for \"oadm\"\nRun 'oadm --help' for usage.", "stderr_lines": ["Error: unknown command \"adm\" for \"oadm\"", "Run 'oadm --help' for usage."], "stdout": "", "stdout_lines": []}

Expected results:
Upgrade succeeds.
Comment 1 Vadim Rutkovsky 2018-05-28 11:40:53 UTC 
Created https://github.com/openshift/openshift-ansible/pull/8547 to fix this
Comment 2 Weihua Meng 2018-06-02 06:04:52 UTC 
Fixed.
openshift-ansible-3.7.52-1.git.0.3fddee4.el7.noarch


TASK [Drain Node for Kubelet upgrade] ******************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/upgrade_control_plane.yml:408

changed: [qe-wmengah36etcd-master-etcd-1.0601-51o.qe.rhcloud.com -> qe-wmengah36etcd-master-etcd-1.0601-51o.qe.rhcloud.com] => {
    "attempts": 1,
    "changed": true,
    "cmd": [
        "/usr/local/bin/oadm",
        "drain",
        "qe-wmengah36etcd-master-etcd-1",
        "--config=/etc/origin/master/admin.kubeconfig",
        "--force",
        "--delete-local-data",
        "--ignore-daemonsets",
        "--timeout=0s"
    ],
    "delta": "0:00:00.319085",
    "end": "2018-06-01 22:41:54.408099",
    "failed": false,
    "failed_when_result": false,
    "invocation": {
        "module_args": {
            "_raw_params": "/usr/local/bin/oadm drain qe-wmengah36etcd-master-etcd-1 --config=/etc/origin/master/admin.kubeconfig --force --delete-local-data --ignore-daemonsets --timeout=0s",
Comment 3 liujia 2018-06-25 06:58:19 UTC 
*** Bug 1585657 has been marked as a duplicate of this bug. ***
Comment 5 errata-xmlrpc 2018-06-27 07:59:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2009