Red Hat Bugzilla – Bug 1391608
Upgrade Playbook from 3.3.0.35 to 3.3.1.3 failed on checking embedded etcd on multi-master/etcd environment
Last modified: 2016-11-15 14:11:02 EST
While running the upgrade playbook on a multi-master and multi-etcd environment, the playbook failed on checking the current embedded etcd disk usage [0] when the etcd should not be considered embedded. Attaching hosts file shortly [0] TASK [Check current embedded etcd disk usage] ********************************** fatal: [<IP>]: FAILED! => { "failed": true } MSG: the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: 'dict object' has no attribute 'etcd_data_dir' The error appears to have been in '/usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/upgrade_control_plane.yml': line 47, column 5, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: # TODO: replace shell module with command and update later checks - name: Check current embedded etcd disk usage changed: [<IP>] changed: [<IP>
I think I have found a reproducer, if I install a cluster and then remove /etc/ansible/facts.d/openshift.fact on each master, then try to re-run a 3_3 upgrade, it will fail with exactly this error. It appears openshift_master_etcd_hosts is not being set during upgrade, but the error is hidden if you have the cached fact present from running original config.yml from cluster setup. Looks as though their fact cache was removed or somehow they hit a way for the cached value to disappear. Working on a fix now.
If customer uses config.yml playbook (used for installation) for continued maintenance, it looks like re-running this will re-generate the facts cache, after which upgrade should complete. However my understanding is customers seldom use this playbook for ongoing maintenance.
Proposed fix: https://github.com/openshift/openshift-ansible/pull/2730 Steps to reproduce for QE: ansible masters -i ./hosts -a "rm /etc/ansible/facts.d/openshift.fact"
I'm not 100% sure how customer hit this but I believe the above step is the best way to reproduce this bug. The problem likely cannot affect embedded etcd deployments, or deployments with etcd on entirely separate hosts. I believe it will only trigger when etcd is colocated on the masters. We found the issue was master facts not being fully loaded and defaulting to embedded etcd true, which causes the etcd fact loading to fail due to a missing file. (as it's not actually embedded etcd) Fix: https://github.com/openshift/openshift-ansible/pull/2730 I have tested on containerized co-located etcd, rpm embedded etcd, rpm separate etcd hosts, and rpm co-located etcd.
Created attachment 1220260 [details] Ansible hosts and ansible logs Upgrade failed. AnsibleUndefinedVariable: 'dict object' has no attribute 'debug_level' fatal: [openshift-190.lab.eng.nay.redhat.com]: FAILED! => { "changed": false, "failed": true } MSG: AnsibleUndefinedVariable: 'dict object' has no attribute 'debug_level' to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_3/upgrade.retry
https://github.com/openshift/openshift-ansible/pull/2794
It works well on atomic-openshift-utils-3.4.25-1
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2016:2778