Bug 1391608 - Upgrade Playbook from 3.3.0.35 to 3.3.1.3 failed on checking embedded etcd on multi-master/etcd environment
Summary: Upgrade Playbook from 3.3.0.35 to 3.3.1.3 failed on checking embedded etcd on...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 3.3.1
Assignee: Devan Goodwin
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-03 15:46 UTC by Eric Jones
Modified: 2019-12-16 07:18 UTC (History)
7 users (show)

Fixed In Version: openshift-ansible-3.3.50-1.git.0.5bdbeaa.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-15 19:11:02 UTC
Target Upstream Version:


Attachments (Terms of Use)
Ansible hosts and ansible logs (80.57 KB, text/plain)
2016-11-14 05:48 UTC, Anping Li
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:2778 0 normal SHIPPED_LIVE Moderate: atomic-openshift-utils security and bug fix update 2016-11-16 00:08:29 UTC

Description Eric Jones 2016-11-03 15:46:27 UTC
While running the upgrade playbook on a multi-master and multi-etcd environment, the playbook failed on checking the current embedded etcd disk usage [0] when the etcd should not be considered embedded.

Attaching hosts file shortly


[0]
TASK [Check current embedded etcd disk usage] **********************************
fatal: [<IP>]: FAILED! => {
    "failed": true
}

MSG:

the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: 'dict object' has no attribute 'etcd_data_dir'

The error appears to have been in '/usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/upgrade_control_plane.yml': line 47, column 5, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  # TODO: replace shell module with command and update later checks
  - name: Check current embedded etcd disk usage
changed: [<IP>]
changed: [<IP>

Comment 3 Devan Goodwin 2016-11-04 16:08:07 UTC
I think I have found a reproducer, if I install a cluster and then remove /etc/ansible/facts.d/openshift.fact on each master, then try to re-run a 3_3 upgrade, it will fail with exactly this error.

It appears openshift_master_etcd_hosts is not being set during upgrade, but the error is hidden if you have the cached fact present from running original config.yml from cluster setup. 

Looks as though their fact cache was removed or somehow they hit a way for the cached value to disappear. Working on a fix now.

Comment 4 Devan Goodwin 2016-11-04 17:28:13 UTC
If customer uses config.yml playbook (used for installation) for continued maintenance, it looks like re-running this will re-generate the facts cache, after which upgrade should complete. However my understanding is customers seldom use this playbook for ongoing maintenance.

Comment 5 Devan Goodwin 2016-11-04 19:14:09 UTC
Proposed fix:

https://github.com/openshift/openshift-ansible/pull/2730

Steps to reproduce for QE:

ansible masters -i ./hosts -a "rm /etc/ansible/facts.d/openshift.fact"

Comment 6 Devan Goodwin 2016-11-09 17:26:03 UTC
I'm not 100% sure how customer hit this but I believe the above step is the best way to reproduce this bug.

The problem likely cannot affect embedded etcd deployments, or deployments with etcd on entirely separate hosts. I believe it will only trigger when etcd is colocated on the masters.

We found the issue was master facts not being fully loaded and defaulting to embedded etcd true, which causes the etcd fact loading to fail due to a missing file. (as it's not actually embedded etcd)

Fix: https://github.com/openshift/openshift-ansible/pull/2730

I have tested on containerized co-located etcd, rpm embedded etcd, rpm separate etcd hosts, and rpm co-located etcd.

Comment 8 Anping Li 2016-11-14 05:48:36 UTC
Created attachment 1220260 [details]
Ansible hosts and ansible logs

Upgrade failed.

AnsibleUndefinedVariable: 'dict object' has no attribute 'debug_level'
fatal: [openshift-190.lab.eng.nay.redhat.com]: FAILED! => {
    "changed": false,
    "failed": true
}

MSG:

AnsibleUndefinedVariable: 'dict object' has no attribute 'debug_level'
        to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_3/upgrade.retry

Comment 11 Anping Li 2016-11-15 07:12:06 UTC
It works well on atomic-openshift-utils-3.4.25-1

Comment 12 errata-xmlrpc 2016-11-15 19:11:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:2778


Note You need to log in before you can comment on or make changes to this bug.