Bug 1566238 - upgrade from v3.7 to v3.9 fails with openshift-ansible-3.9.20-1.git.0.f99fb43.el7
Summary: upgrade from v3.7 to v3.9 fails with openshift-ansible-3.9.20-1.git.0.f99fb43...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.9.z
Assignee: Vadim Rutkovsky
QA Contact: Weihua Meng
URL:
Whiteboard:
: 1566435 (view as bug list)
Depends On:
Blocks: 1561247
TreeView+ depends on / blocked
 
Reported: 2018-04-11 20:08 UTC by Jason Montleon
Modified: 2018-06-18 18:18 UTC (History)
7 users (show)

Fixed In Version: openshift-ansible-3.9.24-1.git.0.d0289ea.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-18 18:18:27 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1566435 0 medium CLOSED fail to upgrade when backup etcd due to wrong etcdctl command was set 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHSA-2018:2013 0 normal SHIPPED_LIVE Important: OpenShift Container Platform 3.9 security, bug fix, and enhancement update 2018-06-27 22:01:43 UTC

Internal Links: 1566435

Description Jason Montleon 2018-04-11 20:08:54 UTC
Description of problem:
upgrade from v3.7 to v3.9 fails with openshift-ansible-3.9.20-1.git.0.f99fb43.el7

Version-Release number of the following components:
openshift-ansible-3.9.20-1.git.0.f99fb43.el7

How reproducible:
Always

Steps to Reproduce:
1. Install v3.7
2. Try to upgrade to v3.9 with openshift-ansible-3.9.20-1.git.0.f99fb43.el7

Actual results:
TASK [etcd : Generate etcd backup] *********************************************
    fatal: [192.168.120.4.nip.io]: FAILED! => {"changed": false, "cmd": "runc exec etcd etcdctl backup --data-dir=/var/lib/etcd/ --backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411080955", "msg": "[Errno 2] No such file or directory", "rc": 2}

Expected results:
Upgrade worls

Additional info:
Works with the previous version:
openshift-ansible-3.9.19-1.git.0.34f4090

Comment 2 Weihua Meng 2018-04-12 01:55:56 UTC
I did an upgrade with openshift-ansible-3.9.20-1.git.0.f99fb43.el7.noarch
Not meet this error

<debug-wmengah37-master-etcd-3.0411-l0i.qe.rhcloud.com> (0, '\r\n{"changed": true, "end": "2018-04-11 08:36:21.802860", "stdout": "", "cmd": ["docker", "exec", "etcd_container", "etcdctl", "backup", "--data-dir=/var/lib/etcd/", "--backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411043609"], "rc": 0, "start": "2018-04-11 08:36:20.701460", "stderr": "", "delta": "0:00:01.101400", "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "docker exec etcd_container etcdctl backup --data-dir=/var/lib/etcd/ --backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411043609", "removes": null, "creates": null, "chdir": null, "stdin": null}}}\r\n', 'Shared connection to debug-wmengah37-master-etcd-3.0411-l0i.qe.rhcloud.com closed.\r\n')
changed: [debug-wmengah37-master-etcd-3.0411-l0i.qe.rhcloud.com] => {
    "changed": true,
    "cmd": [
        "docker",
        "exec",
        "etcd_container",
        "etcdctl",
        "backup",
        "--data-dir=/var/lib/etcd/",
        "--backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411043609"
    ],
    "delta": "0:00:01.101400",
    "end": "2018-04-11 08:36:21.802860",
    "failed": false,
    "invocation": {
        "module_args": {
            "_raw_params": "docker exec etcd_container etcdctl backup --data-dir=/var/lib/etcd/ --backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411043609",
            "_uses_shell": false,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "rc": 0,
    "start": "2018-04-11 08:36:20.701460",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}

PLAY RECAP ********************************************************************************************************************************************************************************************************
debug-wmengah37-lb-1.0411-l0i.qe.rhcloud.com : ok=59   changed=2    unreachable=0    failed=0   
debug-wmengah37-master-etcd-1.0411-l0i.qe.rhcloud.com : ok=774  changed=149  unreachable=0    failed=0   
debug-wmengah37-master-etcd-2.0411-l0i.qe.rhcloud.com : ok=523  changed=72   unreachable=0    failed=0   
debug-wmengah37-master-etcd-3.0411-l0i.qe.rhcloud.com : ok=523  changed=72   unreachable=0    failed=0   
debug-wmengah37-node-primary-1.0411-l0i.qe.rhcloud.com : ok=94   changed=17   unreachable=0    failed=0   
debug-wmengah37-node-primary-2.0411-l0i.qe.rhcloud.com : ok=94   changed=17   unreachable=0    failed=0   
debug-wmengah37-nrri-1.0411-l0i.qe.rhcloud.com : ok=94   changed=17   unreachable=0    failed=0   
debug-wmengah37-nrri-2.0411-l0i.qe.rhcloud.com : ok=94   changed=17   unreachable=0    failed=0   
localhost                  : ok=38   changed=0    unreachable=0    failed=0

Comment 3 Scott Dodson 2018-04-12 21:37:59 UTC
Can you confirm this is a system container based install?

Comment 4 Jason Montleon 2018-04-13 12:38:53 UTC
It is an RPM install.

Comment 5 Weihua Meng 2018-04-16 09:53:39 UTC
reproduced by RPM install upgrade.
containerized install (RHEL and Atomic Host) upgrade does not have this issue.

openshift-ansible-3.9.22-1.git.0.2e15102.el7.noarch

Comment 6 Vadim Rutkovsky 2018-04-16 10:16:53 UTC
(In reply to Jason Montleon from comment #4)
> It is an RPM install.

It seems the code incorrectly decides its a system container installation.

Could you attach the full update log?

Comment 7 Vadim Rutkovsky 2018-04-16 11:28:49 UTC
Nvm, can reproduce on any update.

Created https://github.com/openshift/openshift-ansible/pull/7974 to fix it

Comment 8 Russell Teague 2018-04-17 13:17:10 UTC
*** Bug 1566435 has been marked as a duplicate of this bug. ***

Comment 9 Vadim Rutkovsky 2018-04-19 07:53:03 UTC
Fix is available in openshift-ansible-3.9.24-1

Comment 10 Weihua Meng 2018-04-19 14:44:01 UTC
Fixed.
openshift-ansible-3.9.24-1.git.0.d0289ea.el7.noarch

###  RPM  ###
  Operating System: Red Hat Enterprise Linux Server 7.5 (Maipo)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:7.5:GA:server
            Kernel: Linux 3.10.0-862.el7.x86_64
      Architecture: x86-64

changed: [qe-wmengrpm37-master-etcd-1.0419-u40.qe.rhcloud.com] => {
    "changed": true,
    "cmd": [
        "etcdctl",
        "backup",
        "--data-dir=/var/lib/etcd/",
        "--backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180419095653"
    ],


###  containerized  ###
  Operating System: Red Hat Enterprise Linux Atomic Host 7.5.0
       CPE OS Name: cpe:/o:redhat:enterprise_linux:7.5:GA:atomic-host
            Kernel: Linux 3.10.0-862.el7.x86_64
      Architecture: x86-64

changed: [qe-wmengah75-372-master-etcd-1.0419-q09.qe.rhcloud.com] => {
    "changed": true,
    "cmd": [
        "docker",
        "exec",
        "etcd_container",
        "etcdctl",
        "backup",
        "--data-dir=/var/lib/etcd/",
        "--backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180419100212"
    ],


Note You need to log in before you can comment on or make changes to this bug.