Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1433378

Summary: [3.2] ansible-playbook fails with dict object has no attribute oo_etcd_to_config
Product: OpenShift Container Platform Reporter: Joel Rosental R. <jrosenta>
Component: Cluster Version OperatorAssignee: Andrew Butcher <abutcher>
Status: CLOSED ERRATA QA Contact: liujia <jiajliu>
Severity: high Docs Contact:
Priority: medium    
Version: 3.2.1CC: abutcher, anli, aos-bugs, jiajliu, jokerman, mmccomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-25 17:42:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Upgrade logs and inventory file none

Description Joel Rosental R. 2017-03-17 13:47:46 UTC
Description of problem:

While trying to update to the latest 3.2 version using asynchronous upgrade procedure running the ansible-playbook like this:  /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_2/upgrade.yml, the playbook fails with the following error:

ASK [Verify Ansible version is greater than or equal to 2.1.0.0] **************
skipping: [localhost]
... ... ... 
TASK [openshift_facts] *********************************************************
fatal: [master.example.com]: FAILED! => {"failed": true, "msg": "the field 'args' has an invalid value, which appears to include a variable that is undefined. The error was: 'dict object' has no attribute 'oo_etcd_to_config'\n\nThe error appears to have been in '/usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/pre.yml': line 243, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n  tasks:\n  - openshift_facts:\n    ^ here\n"}
        to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_2/upgrade.retry

Version-Release number of selected component (if applicable):

openshift-ansible-3.2.47-1.git.0.34a924d.el7.noarch
atomic-openshift-utils-3.2.47-1.git.0.34a924d.el7.noarch

How reproducible:

Always.

Steps to Reproduce:
1. Follow official documentation steps for doing an asynchronous upgrade through the ansible-playbook

2. Running the ansible playbook like this:  ansible-playbook -i /etc/ansible/hosts /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_2/upgrade.yml


Actual results:
The error described above.

Expected results:
Async upgrade performed without issues.

Additional info:

The ansible hosts used is like this:

# Create an OSEv3 group that contains the masters and nodes groups
[OSEv3:children]
masters
nodes
#new_nodes

# Set variables common for all OSEv3 hosts
[OSEv3:vars]

# SSH user, this user should allow ssh based auth without requiring a password
ansible_ssh_user=root

# If ansible_ssh_user is not root, ansible_sudo must be set to true
#ansible_sudo=true
deployment_type=openshift-enterprise
osm_use_cockpit=true
osm_default_node_selector="region=primary"
osm_default_subdomain=cloudapps.example.com
osm_cluster_network_cidr=11.1.0.0/16
openshift_node_kubelet_args={'minimum-container-ttl-duration': ['10s'],'maximum-dead-containers-per-container': ['2'],'maximum-dead-containers': ['10'],'image-gc-high-threshold': ['90'],'image-gc-low-threshold': ['80']}

# '': [''],

# uncomment the following to enable htpasswd authentication; defaultsto DenyAllPasswordIdentityProvider
openshift_master_identity_providers=[{'name': 'htpasswd_auth','login': 'true', 'challenge': 'true', 'kind':'HTPasswdPasswordIdentityProvider', 'filename':'/etc/origin/openshift-passwd'}]

# host group for masters
[masters]
master.example.com

# host group for nodes, includes region info
[nodes]
master.example.com openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
node1.example.com openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
node2.example.com openshift_node_labels="{'region': 'primary', 'zone': 'west'}"
#[new_nodes]
#node3.example.com openshift_node_labels="{'region': 'primary', 'zone': 'west'}"


The same result is happening with the latest atomic-openshift-utils package:

atomic-openshift-utils-3.2.53-1.git.0.2fefc17.el7.noarch

Comment 1 Scott Dodson 2017-03-17 15:47:47 UTC
I've tried adding etcd to [OSEv3:vars] and defining an etcd group, the same thing happens.

Comment 2 Scott Dodson 2017-03-17 15:48:06 UTC
I meant OSEv3:children.

Comment 5 Anping Li 2017-04-10 07:40:15 UTC
Created attachment 1270405 [details]
Upgrade logs and inventory file

No task was executed in playbook Backup etcd. I guess etcd_hosts_to_backup[1] was blank.

PLAY [Backup etcd] *************************************************************

PLAY [Gate on etcd backup] *****************************************************




[1]- name: Backup etcd
  hosts: etcd_hosts_to_backup
  vars:
    embedded_etcd: "{{ groups.oo_etcd_to_config | default([]) | length == 0 }}"
    timestamp: "{{ lookup('pipe', 'date +%Y%m%d%H%M%S') }}"
  roles:
  - openshift_facts
  tasks:
  # Ensure we persist the etcd role for this host in openshift_facts

Comment 6 Scott Dodson 2017-04-20 20:31:40 UTC
Anping,

I can't reproduce what you're seeing, I've tried this both with and without etcd group members and it's working as expected. What's the exact rpm version of ansible?

rpm -q ansible

Comment 7 Anping Li 2017-04-24 07:35:54 UTC
Scott, I was using ansible-2.2.1.0-2.el7.noarch.

Comment 8 Scott Dodson 2017-04-24 17:11:02 UTC
Anping,

Andrew figured out that this only happens when docker < 1.10 is installed on the etcd host but the repo with etcd in it has since been disabled. This happens even if etcd is not containerized and docker isn't necessary on the etcd host but is installed. While this seems un-necessary, we feel that if docker is installed the repo should be left enabled.

Can you please test with the extras repo enabled? We believe that this bug is fixed otherwise.

--
Scott

Comment 9 liujia 2017-04-25 08:31:41 UTC
Version:
atomic-openshift-utils-3.2.55-1.git.0.5feab7c.el7.noarch

Step:
1. Container install OCP v3.2.1.23 with one master/node/etcd + one nfs
# openshift version
openshift v3.2.1.23
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5

2. Update installer to latest version of 3.2.55
# yum -y update atomic-openshift-utils

3. Upgrade ocp to latest 3.2
# openshift version
openshift v3.2.1.31
kubernetes v1.2.0-36-g4a3f9c5
etcd 2.2.5

Upgrade successfully.

Comment 12 errata-xmlrpc 2017-04-25 17:42:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1140