Bug 1601792 - Failed to fetch recently created AMI due to the return object of module ec2_ami_find has no attribute 'creationDate'
Summary: Failed to fetch recently created AMI due to the return object of module ec2_a...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 3.11.0
Assignee: Chris Callegari
QA Contact: sheng.lao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-17 09:08 UTC by sheng.lao
Modified: 2018-10-11 07:22 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2018-10-11 07:21:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:2652 0 None None None 2018-10-11 07:22:03 UTC

Description sheng.lao 2018-07-17 09:08:12 UTC
Description of problem:
When using the ansible-2.6, to fetch recently created AMI, with provision.yml. Installer failed to get due to the return object of module ec2_ami_find has no attribute 'creationDate'

The issue happens on both ansible-2.4 and ansible-2.6

In the build_node_group.yml:8
  8   - name: fetch recently created AMI
  9     ec2_ami_find:
 10       region: "{{ openshift_aws_region }}"
 11       sort: creationDate
<--snip-->

We want to sort the recently created AMI by 'creationDate', but the return object has no attribute 'creationDate'

Version-Release number of the following components:

openshift-ansible: git master branch: a7cab9f1218b7

# rpm -qa |grep ansible
ansible-2.6.1-1.el7ae.noarch

# ansible --version
ansible 2.6.1
  config file = /root/openshift-ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Apr 11 2018, 07:36:10) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

How reproducible:
always

Steps to Reproduce:
1. ansible-playbook -i inventory.yml playbooks/aws/openshift-cluster/prerequisites.yml -e @provisioning_vars.yml
2. ansible-playbook -i inventory.yml playbooks/aws/openshift-cluster/build_ami.yml -e @provisioning_vars.yml
3. ansible-playbook -i inventory.yml playbooks/aws/openshift-cluster/provision.yml -e @provisioning_vars.yml

Actual results:
TASK [openshift_aws : fetch recently created AMI] ************************************************************************************************
task path: /root/openshift-ansible/roles/openshift_aws/tasks/build_node_group.yml:8
Tuesday 17 July 2018  04:30:00 -0400 (0:00:00.067)       0:00:43.718 **********
Using module file /usr/lib/python2.7/site-packages/ansible/modules/cloud/amazon/_ec2_ami_find.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python2 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_FFix1g/ansible_module_ec2_ami_find.py", line 423, in <module>
    main()
  File "/tmp/ansible_FFix1g/ansible_module_ec2_ami_find.py", line 380, in main
    'creationDate': image.creationDate,
AttributeError: 'Image' object has no attribute 'creationDate'


Expected results:
asnsible job is success


Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 2 Scott Dodson 2018-08-01 19:38:32 UTC
Apparently ec2_ami_find is notoriously flakey, in our CI jobs that build AMIs we've had to add some retries with up to 10 minute wait time in order to ensure high rates of success.

See https://github.com/openshift/origin-ci-tool/pull/172

Comment 3 Chris Callegari 2018-08-08 14:06:22 UTC
I have never once hit this problem in 9 months of testing plays in installer / playbooks/aws.

A couple things ...

1) What operating system is being used to execute the installer?  Linux is the only supported OS.  Mac and Windows is NOT supported.

2) Let's find out what version of boto and boto3 is installed.

Please run the following...
$ python
a=['boto', 'boto3']
for i in a:
    pkg_resources.get_distribution(i).version

3) Re openshift-ansible: git master branch: a7cab9f1218b7 ... I dev/test this code using a feature branch from master

$ git describe
openshift-ansible-3.11.0-0.11.0-61-ga7f25ca

Comment 4 sheng.lao 2018-08-09 11:41:38 UTC
(In reply to Chris C from comment #3)
> 1) What operating system is being used to execute the installer?  Linux is
> the only supported OS.  Mac and Windows is NOT supported.

# cat /etc/system-release
CentOS Linux release 7.5.1804 (Core)


> 2) Let's find out what version of boto and boto3 is installed.

>>> import pkg_resources 
>>> a=['boto', 'boto3']
>>> for i in a:
...   pkg_resources.get_distribution(i).version
... 
'2.34.0'
'1.4.0'

> 3) Re openshift-ansible: git master branch: a7cab9f1218b7 
# git pull
Already up-to-date.
# git checkout -b NewTest a7cab9f1218b7
Switched to a new branch 'NewTest'
# git describe
openshift-ansible-3.11.0-0.1.0-387-ga7cab9f

the same result

Comment 5 Chris Callegari 2018-08-09 12:57:10 UTC
re 2) I run boto 2.48.0 and boto3 1.5.24.  Please upgrade the two modules then try running build_ami.yml again.

Comment 6 sheng.lao 2018-08-10 12:18:04 UTC
What I can find the package of python-boto, in RHEL7, is python-boto-2.34.0-5.el7, and RHLE8, is python-boto-2.45.0-5.el8.

The suffix ,'el8', means this package belongs to next generation of OS(RHEL8).

Beside, this package is used by openshift-ansible project, so I don't have the right to choose the release version of python-boto with OCP.

I try my best to solve this problem, with the mostly approximate version, the minimal dependency of python2-botocore:
  # yum install python2-dateutil-2.6.1-3.el8+7.noarch.rpm python2-six-1.11.0-7.el8+7.noarch.rpm
  # yum upgrade python2-botocore-1.9.1-1.el8+7.noarch.rpm

** As a reference (Not official), works well.
pip install --upgrade  botocore==1.7.0

Comment 8 errata-xmlrpc 2018-10-11 07:21:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2652


Note You need to log in before you can comment on or make changes to this bug.