Bug 2034245

Summary: installing a cluster on aws, gcp always fails with "Error: Incompatible provider version"
Product: OpenShift Container Platform Reporter: Lukasz Szaszkiewicz <lszaszki>
Component: InstallerAssignee: Matthew Staebler <mstaeble>
Installer sub component: openshift-installer QA Contact: Yunfei Jiang <yunjiang>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: mstaeble
Version: 4.10   
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-10 16:35:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Description Flags
installer log file none

Description Lukasz Szaszkiewicz 2021-12-20 13:37:48 UTC
Created attachment 1847033 [details]
installer log file

Thanks for opening a bug report!
Before hitting the button, please fill in as much of the template below as you can.
If you leave out information, it's harder to help you.
Be ready for follow-up questions, and please respond in a timely manner.
If we can't reproduce a bug we might close your issue.
If we're wrong, PLEASE feel free to reopen it and explain why.


$ openshift-install version
./openshift-install 4.10.0-0.ci-2021-12-20-064625
built from commit 9f37ece3620d14e48507f2afc5cf6a667ca2cef0
release image registry.ci.openshift.org/ocp/release@sha256:767e9a37ac51902d0d74957378b120250345a6d1347410fa43f0120bed5d7efa
release architecture amd64

Platform: aws, gcp

#Please specify the platform type: aws, libvirt, openstack or baremetal etc.

Please specify: IPI
* IPI (automated install with `openshift-install`. If you don't know, then it's IPI)
* UPI (semi-manual installation on customized infrastructure)

What happened?

Installing a cluster on AWS or GCP always fails with the following error:

level=debug msg=Initializing the backend...
level=debug msg=Initializing provider plugins...
level=debug msg=- Finding latest version of openshift/local/aws...
level=error msg=Error: Incompatible provider version
level=error msg=Provider openshift/local/aws v1.0.0 does not have a package available for your
level=error msg=current platform, darwin_amd64.
level=error msg=Provider releases are separate from Terraform CLI releases, so not all
level=error msg=providers are available for all platforms. Other versions of this provider may
level=error msg=have different platforms supported.
level=fatal msg=failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to initialize Terraform

It used to work in the past (~last week). I had more luck with openshift-install-mac-4.10.0-0.nightly-2021-12-14-083101

#See the troubleshooting documentation (https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md) for ideas about what information to collect.

#For example, 

# If the installer fails to create resources (https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md#installer-fails-to-create-resources), attach the relevant portions of your `.openshift_install.log.`
# If the installer fails to bootstrap the cluster (https://github.com/openshift/installer/blob/master/docs/user/troubleshootingbootstrap.md), attach the bootstrap log bundle.
# If the installer fails to complete installation after bootstrapping completes (https://github.com/openshift/installer/blob/master/docs/user/troubleshooting.md#installer-fails-to-initialize-the-cluster), attach the must-gather log bundle using `oc adm must-gather`

# Always at least include the `.openshift_install.log`

What did you expect to happen?

The installer should be able to load plugins for MacOS

How to reproduce it (as minimally and precisely as possible)?

1. Take a MacOS machine
2. Download an installer from https://amd64.ocp.releases.ci.openshift.org/#4.10.0-0.ci
3. I tried with https://openshift-release-artifacts.apps.ci.l2s4.p1.openshiftapps.com/4.10.0-0.ci-2021-12-20-064625/
4. Try to create a cluster on AWS or GCP

Anything else we need to know?

I think it only applies to MacOS. It used to work in the past. Something must have changed recenlty.

Comment 2 Matthew Staebler 2021-12-20 14:23:13 UTC
Terraform is determining that the architecture of the providers are linux_amd64 instead of darwin_amd64.

time="2021-12-20T15:16:31+01:00" level=debug msg="2021/12/20 15:16:31 [TRACE] getproviders.SearchLocalDirectory: found openshift/local/alicloud v1.0.0 for linux_amd64 at /private/var/folders/4x/d8vbcg5s6lx229_q7l5wbw0h0000gn/T/openshift-install-cluster-1937995839/plugins/openshift/local/alicloud/1.0.0/linux_amd64"

Comment 3 Matthew Staebler 2021-12-20 14:37:27 UTC
The symlinks that the installer creates need to use the appropriate platform. Currently, it is always putting the symlinks in the linux_amd64 directory.

time="2021-12-20T14:58:13+01:00" level=debug msg="Symlinking plugin terraform-provider-aws src: \"/Users/lszaszki/workspace/Downloads/openshift-install-mac-4.10.0-0.ci-2021-12-20-064625/openshift-install\" dst: \"/var/folders/4x/d8vbcg5s6lx229_q7l5wbw0h0000gn/T/openshift-install-cluster-203806641/plugins/openshift/local/aws/1.0.0/linux_amd64/terraform-provider-aws\""

Comment 7 Yunfei Jiang 2021-12-27 07:31:38 UTC
verified. PASS.
OCP version: 4.10.0-0.nightly-2021-12-23-153012

Comment 10 errata-xmlrpc 2022-03-10 16:35:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.