Created attachment 1477615 [details]
inventory

Description of problem:

The upgrade to 3.11 on a cluster with AWS as the cloud provider fails.   The node service on the master fails to restart with the following message in the log:

F0821 15:15:44.944704    8325 server.go:262] failed to run Kubelet: could not init cloud provider "aws": error finding instance i-061c93b1f9408ea6e: "error listing AWS instances: \"NoCredentialProviders: no valid providers in chain. Deprecated.\\n\\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors\""

At this point of the install, the contents of /etc/sysconfig/atomic-openshift-node has been reset:

OPTIONS=
DEBUG_LOGLEVEL=2
IMAGE_VERSION=v3.11


My inventory (attached) contains env vars for AWS credentials and the env vars are set correctly



Version-Release number of the following components:

root@ip-172-31-38-45: ~ # rpm -q openshift-ansible
openshift-ansible-3.11.0-0.19.0.git.0.ebd1bf9None.noarch
root@ip-172-31-38-45: ~ # rpm -q ansible
ansible-2.6.2-1.el7ae.noarch
root@ip-172-31-38-45: ~ # ansible --version
ansible 2.6.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Jul 16 2018, 19:52:45) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]


How reproducible: Always with AWS as the cloud provider

Steps to Reproduce:
1.  Install cluster at 3.10.28 with properly configured AWS cloud provider
2.  Upgrade to 3.11.0-0.19.0


Actual results:
Upgrade fails - see attached ansible log and inventory