When creating a new cluster, the Installer will use any terraform-provider-openstack found in path, instead of the one embedded in the release binary. How to reproduce: ``` # Here GOBIN is a directory listed in $PATH cat > $GOBIN/terraform-provider-openstack <<EOF #!/usr/bin/env bash echo 'trololol' EOF chmod +x $GOBIN/terraform-provider-openstack openshift-install create cluster ``` Result: ``` ERROR ERROR Error: Failed to instantiate provider "openstack" to obtain schema: Unrecognized remote plugin message: trololol ERROR ERROR This usually means that the plugin is either invalid or simply ERROR needs to be recompiled to support the latest protocol. ``` Reproduced with: ``` $ openshift-install version /home/pierre/code/bin/openshift-install-linux-4.7.0 4.7.0 built from commit 98e11541c24e95c864328b9b35c64b77836212ed release image quay.io/openshift-release-dev/ocp-release@sha256:d74b1cfa81f8c9cc23336aee72d8ae9c9905e62c4874b071317a078c316f8a70 ```
This might be happening for all platforms, and not just OpenStack; to be investigated.
Deferring to an upcoming sprint
Removing the Triaged keyword because: * the target release value is missing * the QE automation assessment (flag qe_test_coverage) is missing
Reproduced with openshift-install-linux-4.10.0-0.nightly-2021-12-14-083101 Assigning to the Installer team.
I am unable to reproduce this for the aws provider. Could you provide the .openshift_install.log file when running with the "TF_LOG=trace" environment variable?
Created attachment 1846730 [details] openshift_install.log when installing with TF_LOG=trace openshift_install.log Installer run after: export TF_LOG=trace Installing on OpenStack with: openshift-install-linux-4.10.0-0.nightly-2021-12-14-083101 built from commit af575af283ccb9bb39ec17cfc437e39ae1069de9 release image quay.io/openshift-release-dev/ocp-release-nightly@sha256:8207b4e6371144d8a715617ddf1f5958b87e26a015da23cfec7ccbefab9cd49f release architecture amd64
OK. It is not that the provider is in the PATH. It is that the provider is in the same directory that the openshift-install binary is in. ~~~~ time="2021-12-17T15:47:57+01:00" level=debug msg="2021/12/17 15:47:57 [DEBUG] checking for provider in \".\"" time="2021-12-17T15:47:57+01:00" level=debug msg="2021/12/17 15:47:57 [DEBUG] checking for provider in \"/var/home/pierre/code/bin\"" time="2021-12-17T15:47:57+01:00" level=debug msg="2021/12/17 15:47:57 [WARN] found legacy provider \"terraform-provider-openstack\"" time="2021-12-17T15:47:57+01:00" level=debug msg="2021/12/17 15:47:57 [DEBUG] checking for provider in \"/tmp/openshift-install--116931073/plugins\"" ~~~~
Perhaps we can set dev_overrides in the terraform.rc file to force terraform to only look in the plugins directory. https://www.terraform.io/cli/config/config-file#development-overrides-for-provider-developers
I have updated the bug summary to reflect Matthew's findings.
This will likely be OBE by https://github.com/openshift/installer/pull/5507.
We believe this is fixed by recent terraform changes in the master branch of the installer. The most recent merged PR is https://github.com/openshift/installer/pull/5666
Relevant JIRA that should fix this: https://issues.redhat.com/browse/CORS-1907
The nightly openshift-install-linux-4.10.0-0.nightly-2021-12-14-083101(mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1932812#c8) is not available now, so using stable build 4.7.0(mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1932812#c0) to reproduce this issue. Download an old aws terraform provider from https://releases.hashicorp.com/terraform-provider-aws/0.1.0/ to local testing directory as "terraform-provider-aws", starting the installation with 4.7.0 installer. [root@preserve-gpei-worker test2]# ./openshift-install version ./openshift-install 4.7.0 built from commit 98e11541c24e95c864328b9b35c64b77836212ed release image quay.io/openshift-release-dev/ocp-release@sha256:d74b1cfa81f8c9cc23336aee72d8ae9c9905e62c4874b071317a078c316f8a70 [root@preserve-gpei-worker test2]# ./openshift-install create cluster ... INFO Creating infrastructure resources... ERROR ERROR Error: Failed to instantiate provider "aws" to obtain schema: Incompatible API version with plugin. Plugin version: 4, Client versions: [5] ERROR ERROR ERROR Failed to read tfstate: open /tmp/openshift-install-769641212/terraform.tfstate: no such file or directory FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change In .openshift_install.log: time="2022-03-13T09:07:06-04:00" level=debug msg="2022/03/13 09:07:06 [DEBUG] checking for provider in \".\"" time="2022-03-13T09:07:06-04:00" level=debug msg="2022/03/13 09:07:06 [WARN] found legacy provider \"terraform-provider-aws\"" time="2022-03-13T09:07:06-04:00" level=debug msg="2022/03/13 09:07:06 [DEBUG] checking for provider in \"/root/artifacts/test2\"" time="2022-03-13T09:07:06-04:00" level=debug msg="2022/03/13 09:07:06 [WARN] found legacy provider \"terraform-provider-aws\"" Retried with 4.11.0-0.nightly-2022-03-13-123528 in the same case, no issue happened, mark this issue as VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069