Bug 1566238

Summary: upgrade from v3.7 to v3.9 fails with openshift-ansible-3.9.20-1.git.0.f99fb43.el7
Product: OpenShift Container Platform Reporter: Jason Montleon <jmontleo>
Component: InstallerAssignee: Vadim Rutkovsky <vrutkovs>
Status: CLOSED CURRENTRELEASE QA Contact: Weihua Meng <wmeng>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.9.0CC: aos-bugs, jiajliu, jmontleo, jokerman, mmccomas, smunilla, wmeng
Target Milestone: ---   
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift-ansible-3.9.24-1.git.0.d0289ea.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-18 18:18:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1561247    

Description Jason Montleon 2018-04-11 20:08:54 UTC
Description of problem:
upgrade from v3.7 to v3.9 fails with openshift-ansible-3.9.20-1.git.0.f99fb43.el7

Version-Release number of the following components:
openshift-ansible-3.9.20-1.git.0.f99fb43.el7

How reproducible:
Always

Steps to Reproduce:
1. Install v3.7
2. Try to upgrade to v3.9 with openshift-ansible-3.9.20-1.git.0.f99fb43.el7

Actual results:
TASK [etcd : Generate etcd backup] *********************************************
    fatal: [192.168.120.4.nip.io]: FAILED! => {"changed": false, "cmd": "runc exec etcd etcdctl backup --data-dir=/var/lib/etcd/ --backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411080955", "msg": "[Errno 2] No such file or directory", "rc": 2}

Expected results:
Upgrade worls

Additional info:
Works with the previous version:
openshift-ansible-3.9.19-1.git.0.34f4090

Comment 2 Weihua Meng 2018-04-12 01:55:56 UTC
I did an upgrade with openshift-ansible-3.9.20-1.git.0.f99fb43.el7.noarch
Not meet this error

<debug-wmengah37-master-etcd-3.0411-l0i.qe.rhcloud.com> (0, '\r\n{"changed": true, "end": "2018-04-11 08:36:21.802860", "stdout": "", "cmd": ["docker", "exec", "etcd_container", "etcdctl", "backup", "--data-dir=/var/lib/etcd/", "--backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411043609"], "rc": 0, "start": "2018-04-11 08:36:20.701460", "stderr": "", "delta": "0:00:01.101400", "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "docker exec etcd_container etcdctl backup --data-dir=/var/lib/etcd/ --backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411043609", "removes": null, "creates": null, "chdir": null, "stdin": null}}}\r\n', 'Shared connection to debug-wmengah37-master-etcd-3.0411-l0i.qe.rhcloud.com closed.\r\n')
changed: [debug-wmengah37-master-etcd-3.0411-l0i.qe.rhcloud.com] => {
    "changed": true,
    "cmd": [
        "docker",
        "exec",
        "etcd_container",
        "etcdctl",
        "backup",
        "--data-dir=/var/lib/etcd/",
        "--backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411043609"
    ],
    "delta": "0:00:01.101400",
    "end": "2018-04-11 08:36:21.802860",
    "failed": false,
    "invocation": {
        "module_args": {
            "_raw_params": "docker exec etcd_container etcdctl backup --data-dir=/var/lib/etcd/ --backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411043609",
            "_uses_shell": false,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "rc": 0,
    "start": "2018-04-11 08:36:20.701460",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}

PLAY RECAP ********************************************************************************************************************************************************************************************************
debug-wmengah37-lb-1.0411-l0i.qe.rhcloud.com : ok=59   changed=2    unreachable=0    failed=0   
debug-wmengah37-master-etcd-1.0411-l0i.qe.rhcloud.com : ok=774  changed=149  unreachable=0    failed=0   
debug-wmengah37-master-etcd-2.0411-l0i.qe.rhcloud.com : ok=523  changed=72   unreachable=0    failed=0   
debug-wmengah37-master-etcd-3.0411-l0i.qe.rhcloud.com : ok=523  changed=72   unreachable=0    failed=0   
debug-wmengah37-node-primary-1.0411-l0i.qe.rhcloud.com : ok=94   changed=17   unreachable=0    failed=0   
debug-wmengah37-node-primary-2.0411-l0i.qe.rhcloud.com : ok=94   changed=17   unreachable=0    failed=0   
debug-wmengah37-nrri-1.0411-l0i.qe.rhcloud.com : ok=94   changed=17   unreachable=0    failed=0   
debug-wmengah37-nrri-2.0411-l0i.qe.rhcloud.com : ok=94   changed=17   unreachable=0    failed=0   
localhost                  : ok=38   changed=0    unreachable=0    failed=0

Comment 3 Scott Dodson 2018-04-12 21:37:59 UTC
Can you confirm this is a system container based install?

Comment 4 Jason Montleon 2018-04-13 12:38:53 UTC
It is an RPM install.

Comment 5 Weihua Meng 2018-04-16 09:53:39 UTC
reproduced by RPM install upgrade.
containerized install (RHEL and Atomic Host) upgrade does not have this issue.

openshift-ansible-3.9.22-1.git.0.2e15102.el7.noarch

Comment 6 Vadim Rutkovsky 2018-04-16 10:16:53 UTC
(In reply to Jason Montleon from comment #4)
> It is an RPM install.

It seems the code incorrectly decides its a system container installation.

Could you attach the full update log?

Comment 7 Vadim Rutkovsky 2018-04-16 11:28:49 UTC
Nvm, can reproduce on any update.

Created https://github.com/openshift/openshift-ansible/pull/7974 to fix it

Comment 8 Russell Teague 2018-04-17 13:17:10 UTC
*** Bug 1566435 has been marked as a duplicate of this bug. ***

Comment 9 Vadim Rutkovsky 2018-04-19 07:53:03 UTC
Fix is available in openshift-ansible-3.9.24-1

Comment 10 Weihua Meng 2018-04-19 14:44:01 UTC
Fixed.
openshift-ansible-3.9.24-1.git.0.d0289ea.el7.noarch

###  RPM  ###
  Operating System: Red Hat Enterprise Linux Server 7.5 (Maipo)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:7.5:GA:server
            Kernel: Linux 3.10.0-862.el7.x86_64
      Architecture: x86-64

changed: [qe-wmengrpm37-master-etcd-1.0419-u40.qe.rhcloud.com] => {
    "changed": true,
    "cmd": [
        "etcdctl",
        "backup",
        "--data-dir=/var/lib/etcd/",
        "--backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180419095653"
    ],


###  containerized  ###
  Operating System: Red Hat Enterprise Linux Atomic Host 7.5.0
       CPE OS Name: cpe:/o:redhat:enterprise_linux:7.5:GA:atomic-host
            Kernel: Linux 3.10.0-862.el7.x86_64
      Architecture: x86-64

changed: [qe-wmengah75-372-master-etcd-1.0419-q09.qe.rhcloud.com] => {
    "changed": true,
    "cmd": [
        "docker",
        "exec",
        "etcd_container",
        "etcdctl",
        "backup",
        "--data-dir=/var/lib/etcd/",
        "--backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180419100212"
    ],