Bug 1585657

Summary: Fail to upgrade ocp to v3.7 due to wrong client binary
Product: OpenShift Container Platform Reporter: liujia <jiajliu>
Component: Cluster Version OperatorAssignee: Scott Dodson <sdodson>
Status: CLOSED DUPLICATE QA Contact: liujia <jiajliu>
Severity: high Docs Contact:
Priority: high    
Version: 3.7.1CC: aos-bugs, gpei, jokerman, mmccomas
Target Milestone: ---   
Target Release: 3.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-25 06:58:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1579321    

Description liujia 2018-06-04 10:36:00 UTC 
Description of problem:
Upgrade ocp v3.6 to v3.7 failed at task [Drain Node for Kubelet upgrade] due to wrong client binary.

TASK [Drain Node for Kubelet upgrade] ***************************************************************************************************************************************
FAILED - RETRYING: Drain Node for Kubelet upgrade (1 retries left).
fatal: [x -> x]: FAILED! => {"attempts": 1, "changed": true, "cmd": ["oadm", "adm", "drain", "qe-jliu-rp36-master-etcd-1", "--config=/etc/origin/master/admin.kubeconfig", "--force", "--delete-local-data", "--ignore-daemonsets", "--timeout=0s"], "delta": "0:00:00.232031", "end": "2018-06-04 05:54:21.810048", "failed": true, "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2018-06-04 05:54:21.578017", "stderr": "Error: unknown command \"adm\" for \"oadm\"\nRun 'oadm --help' for usage.", "stderr_lines": ["Error: unknown command \"adm\" for \"oadm\"", "Run 'oadm --help' for usage."], "stdout": "", "stdout_lines": []}


# vim roles/openshift_facts/library/openshift_facts.py
    if safe_get_bool(facts['common']['is_containerized']):
        facts['common']['admin_binary'] = '/usr/local/bin/oadm'
        facts['common']['client_binary'] = '/usr/local/bin/oc'

# vim playbooks/common/openshift-cluster/upgrades/upgrade_control_plane.yml
  - name: Drain Node for Kubelet upgrade
    command: >
      {{ hostvars[groups.oo_first_master.0].openshift.common.admin_binary }} adm drain {{ openshift.node.nodename | lower }}
      --config={{ openshift.common.config_base }}/master/admin.kubeconfig
      --force --delete-local-data --ignore-daemonsets
      --timeout={{ openshift_upgrade_nodes_drain_timeout | default(0) }}s

Version-Release number of the following components:
openshift-ansible-3.7.51-1.git.0.f9b681c.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. Run upgrade from ocp v3.6 to v3.7
2.
3.

Actual results:
Upgrade failed.

Expected results:
Upgrade succeed.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag
Comment 1 liujia 2018-06-25 06:58:19 UTC 

*** This bug has been marked as a duplicate of bug 1583151 ***