Bug 1566238 - upgrade from v3.7 to v3.9 fails with openshift-ansible-3.9.20-1.git.0.f99fb43.el7
Summary: upgrade from v3.7 to v3.9 fails with openshift-ansible-3.9.20-1.git.0.f99fb43...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.9.z
Assignee: Vadim Rutkovsky
QA Contact: Weihua Meng
: 1566435 (view as bug list)
Depends On:
Blocks: 1561247
TreeView+ depends on / blocked
Reported: 2018-04-11 20:08 UTC by Jason Montleon
Modified: 2018-06-18 18:18 UTC (History)
7 users (show)

Fixed In Version: openshift-ansible-3.9.24-1.git.0.d0289ea.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2018-06-18 18:18:27 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1566435 None CLOSED fail to upgrade when backup etcd due to wrong etcdctl command was set 2019-03-13 06:55:14 UTC
Red Hat Product Errata RHSA-2018:2013 normal SHIPPED_LIVE Important: OpenShift Container Platform 3.9 security, bug fix, and enhancement update 2018-06-27 22:01:43 UTC

Internal Links: 1566435

Description Jason Montleon 2018-04-11 20:08:54 UTC
Description of problem:
upgrade from v3.7 to v3.9 fails with openshift-ansible-3.9.20-1.git.0.f99fb43.el7

Version-Release number of the following components:

How reproducible:

Steps to Reproduce:
1. Install v3.7
2. Try to upgrade to v3.9 with openshift-ansible-3.9.20-1.git.0.f99fb43.el7

Actual results:
TASK [etcd : Generate etcd backup] *********************************************
    fatal: []: FAILED! => {"changed": false, "cmd": "runc exec etcd etcdctl backup --data-dir=/var/lib/etcd/ --backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411080955", "msg": "[Errno 2] No such file or directory", "rc": 2}

Expected results:
Upgrade worls

Additional info:
Works with the previous version:

Comment 2 Weihua Meng 2018-04-12 01:55:56 UTC
I did an upgrade with openshift-ansible-3.9.20-1.git.0.f99fb43.el7.noarch
Not meet this error

<debug-wmengah37-master-etcd-3.0411-l0i.qe.rhcloud.com> (0, '\r\n{"changed": true, "end": "2018-04-11 08:36:21.802860", "stdout": "", "cmd": ["docker", "exec", "etcd_container", "etcdctl", "backup", "--data-dir=/var/lib/etcd/", "--backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411043609"], "rc": 0, "start": "2018-04-11 08:36:20.701460", "stderr": "", "delta": "0:00:01.101400", "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "docker exec etcd_container etcdctl backup --data-dir=/var/lib/etcd/ --backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411043609", "removes": null, "creates": null, "chdir": null, "stdin": null}}}\r\n', 'Shared connection to debug-wmengah37-master-etcd-3.0411-l0i.qe.rhcloud.com closed.\r\n')
changed: [debug-wmengah37-master-etcd-3.0411-l0i.qe.rhcloud.com] => {
    "changed": true,
    "cmd": [
    "delta": "0:00:01.101400",
    "end": "2018-04-11 08:36:21.802860",
    "failed": false,
    "invocation": {
        "module_args": {
            "_raw_params": "docker exec etcd_container etcdctl backup --data-dir=/var/lib/etcd/ --backup-dir=/var/lib/etcd//openshift-backup-pre-upgrade-20180411043609",
            "_uses_shell": false,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
    "rc": 0,
    "start": "2018-04-11 08:36:20.701460",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []

PLAY RECAP ********************************************************************************************************************************************************************************************************
debug-wmengah37-lb-1.0411-l0i.qe.rhcloud.com : ok=59   changed=2    unreachable=0    failed=0   
debug-wmengah37-master-etcd-1.0411-l0i.qe.rhcloud.com : ok=774  changed=149  unreachable=0    failed=0   
debug-wmengah37-master-etcd-2.0411-l0i.qe.rhcloud.com : ok=523  changed=72   unreachable=0    failed=0   
debug-wmengah37-master-etcd-3.0411-l0i.qe.rhcloud.com : ok=523  changed=72   unreachable=0    failed=0   
debug-wmengah37-node-primary-1.0411-l0i.qe.rhcloud.com : ok=94   changed=17   unreachable=0    failed=0   
debug-wmengah37-node-primary-2.0411-l0i.qe.rhcloud.com : ok=94   changed=17   unreachable=0    failed=0   
debug-wmengah37-nrri-1.0411-l0i.qe.rhcloud.com : ok=94   changed=17   unreachable=0    failed=0   
debug-wmengah37-nrri-2.0411-l0i.qe.rhcloud.com : ok=94   changed=17   unreachable=0    failed=0   
localhost                  : ok=38   changed=0    unreachable=0    failed=0

Comment 3 Scott Dodson 2018-04-12 21:37:59 UTC
Can you confirm this is a system container based install?

Comment 4 Jason Montleon 2018-04-13 12:38:53 UTC
It is an RPM install.

Comment 5 Weihua Meng 2018-04-16 09:53:39 UTC
reproduced by RPM install upgrade.
containerized install (RHEL and Atomic Host) upgrade does not have this issue.


Comment 6 Vadim Rutkovsky 2018-04-16 10:16:53 UTC
(In reply to Jason Montleon from comment #4)
> It is an RPM install.

It seems the code incorrectly decides its a system container installation.

Could you attach the full update log?

Comment 7 Vadim Rutkovsky 2018-04-16 11:28:49 UTC
Nvm, can reproduce on any update.

Created https://github.com/openshift/openshift-ansible/pull/7974 to fix it

Comment 8 Russell Teague 2018-04-17 13:17:10 UTC
*** Bug 1566435 has been marked as a duplicate of this bug. ***

Comment 9 Vadim Rutkovsky 2018-04-19 07:53:03 UTC
Fix is available in openshift-ansible-3.9.24-1

Comment 10 Weihua Meng 2018-04-19 14:44:01 UTC

###  RPM  ###
  Operating System: Red Hat Enterprise Linux Server 7.5 (Maipo)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:7.5:GA:server
            Kernel: Linux 3.10.0-862.el7.x86_64
      Architecture: x86-64

changed: [qe-wmengrpm37-master-etcd-1.0419-u40.qe.rhcloud.com] => {
    "changed": true,
    "cmd": [

###  containerized  ###
  Operating System: Red Hat Enterprise Linux Atomic Host 7.5.0
       CPE OS Name: cpe:/o:redhat:enterprise_linux:7.5:GA:atomic-host
            Kernel: Linux 3.10.0-862.el7.x86_64
      Architecture: x86-64

changed: [qe-wmengah75-372-master-etcd-1.0419-q09.qe.rhcloud.com] => {
    "changed": true,
    "cmd": [

Note You need to log in before you can comment on or make changes to this bug.