Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1932812

Summary: Installer uses the terraform-provider in the Installer's directory if it exists
Product: OpenShift Container Platform Reporter: Pierre Prinetti <pprinett>
Component: InstallerAssignee: Nobody <nobody>
Installer sub component: openshift-installer QA Contact: Gaoyun Pei <gpei>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: low CC: padillon
Version: 4.7   
Target Milestone: ---   
Target Release: 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: the installer would use any terraform-provider in the $PATH on the installer host Consequence: if there were terraform providers in the $PATH, the installer could use the wrong version or a broken provider--instead of the provider embedded in the installer--causing the install to fail Fix: the installer now unpacks the providers to a known directory and explicitly sets terraform to use that directory for providers Result: the installer will always use the provider embedded in the binary and install will succeed
 Story Points: ---
Clone Of: Environment:
Last Closed: 2022-08-10 10:36:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
openshift_install.log when installing with TF_LOG=trace none

Description Pierre Prinetti 2021-02-25 10:43:07 UTC 
When creating a new cluster, the Installer will use any terraform-provider-openstack found in path, instead of the one embedded in the release binary.

How to reproduce:

```
# Here GOBIN is a directory listed in $PATH

cat > $GOBIN/terraform-provider-openstack <<EOF
#!/usr/bin/env bash
echo 'trololol'
EOF
chmod +x $GOBIN/terraform-provider-openstack
openshift-install create cluster
```

Result:
```
ERROR
ERROR Error: Failed to instantiate provider "openstack" to obtain schema: Unrecognized remote plugin message: trololol
ERROR
ERROR This usually means that the plugin is either invalid or simply
ERROR needs to be recompiled to support the latest protocol.
```

Reproduced with:

```
$ openshift-install version
/home/pierre/code/bin/openshift-install-linux-4.7.0 4.7.0
built from commit 98e11541c24e95c864328b9b35c64b77836212ed
release image quay.io/openshift-release-dev/ocp-release@sha256:d74b1cfa81f8c9cc23336aee72d8ae9c9905e62c4874b071317a078c316f8a70
```
Comment 1 Pierre Prinetti 2021-02-25 10:57:08 UTC 
This might be happening for all platforms, and not just OpenStack; to be investigated.
Comment 3 Pierre Prinetti 2021-03-22 13:13:18 UTC 
Deferring to an upcoming sprint
Comment 6 ShiftStack Bugwatcher 2021-11-25 16:11:25 UTC 
Removing the Triaged keyword because:
* the target release value is missing

* the QE automation assessment (flag qe_test_coverage) is missing
Comment 8 Pierre Prinetti 2021-12-16 13:35:14 UTC 
Reproduced with openshift-install-linux-4.10.0-0.nightly-2021-12-14-083101

Assigning to the Installer team.
Comment 9 Matthew Staebler 2021-12-16 19:09:24 UTC 
I am unable to reproduce this for the aws provider. Could you provide the .openshift_install.log file when running with the "TF_LOG=trace" environment variable?
Comment 11 Pierre Prinetti 2021-12-17 14:52:21 UTC 
Created attachment 1846730 [details]
openshift_install.log when installing with TF_LOG=trace

openshift_install.log

Installer run after:
export TF_LOG=trace

Installing on OpenStack with:

openshift-install-linux-4.10.0-0.nightly-2021-12-14-083101
built from commit af575af283ccb9bb39ec17cfc437e39ae1069de9
release image quay.io/openshift-release-dev/ocp-release-nightly@sha256:8207b4e6371144d8a715617ddf1f5958b87e26a015da23cfec7ccbefab9cd49f
release architecture amd64
Comment 13 Matthew Staebler 2021-12-17 17:36:44 UTC 
OK. It is not that the provider is in the PATH. It is that the provider is in the same directory that the openshift-install binary is in.

~~~~
time="2021-12-17T15:47:57+01:00" level=debug msg="2021/12/17 15:47:57 [DEBUG] checking for provider in \".\""
time="2021-12-17T15:47:57+01:00" level=debug msg="2021/12/17 15:47:57 [DEBUG] checking for provider in \"/var/home/pierre/code/bin\""
time="2021-12-17T15:47:57+01:00" level=debug msg="2021/12/17 15:47:57 [WARN] found legacy provider \"terraform-provider-openstack\""
time="2021-12-17T15:47:57+01:00" level=debug msg="2021/12/17 15:47:57 [DEBUG] checking for provider in \"/tmp/openshift-install--116931073/plugins\""
~~~~
Comment 14 Matthew Staebler 2021-12-17 18:12:19 UTC 
Perhaps we can set dev_overrides in the terraform.rc file to force terraform to only look in the plugins directory.

https://www.terraform.io/cli/config/config-file#development-overrides-for-provider-developers
Comment 15 Pierre Prinetti 2022-02-23 16:29:53 UTC 
I have updated the bug summary to reflect Matthew's findings.
Comment 16 Matthew Staebler 2022-02-23 17:09:44 UTC 
This will likely be OBE by https://github.com/openshift/installer/pull/5507.
Comment 17 Patrick Dillon 2022-03-11 19:37:28 UTC 
We believe this is fixed by recent terraform changes in the master branch of the installer. The most recent merged PR is https://github.com/openshift/installer/pull/5666
Comment 18 Patrick Dillon 2022-03-11 19:56:02 UTC 
Relevant JIRA that should fix this: https://issues.redhat.com/browse/CORS-1907
Comment 19 Gaoyun Pei 2022-03-13 14:47:54 UTC 
The nightly openshift-install-linux-4.10.0-0.nightly-2021-12-14-083101(mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1932812#c8) is not available now, so using stable build 4.7.0(mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1932812#c0) to reproduce this issue. 

Download an old aws terraform provider from https://releases.hashicorp.com/terraform-provider-aws/0.1.0/ to local testing directory as "terraform-provider-aws", starting the installation with 4.7.0 installer.

[root@preserve-gpei-worker test2]# ./openshift-install version
./openshift-install 4.7.0
built from commit 98e11541c24e95c864328b9b35c64b77836212ed
release image quay.io/openshift-release-dev/ocp-release@sha256:d74b1cfa81f8c9cc23336aee72d8ae9c9905e62c4874b071317a078c316f8a70

[root@preserve-gpei-worker test2]# ./openshift-install create cluster
...

INFO Creating infrastructure resources...         
ERROR                                              
ERROR Error: Failed to instantiate provider "aws" to obtain schema: Incompatible API version with plugin. Plugin version: 4, Client versions: [5] 
ERROR                                              
ERROR                                              
ERROR Failed to read tfstate: open /tmp/openshift-install-769641212/terraform.tfstate: no such file or directory 
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change 


In .openshift_install.log:
time="2022-03-13T09:07:06-04:00" level=debug msg="2022/03/13 09:07:06 [DEBUG] checking for provider in \".\""
time="2022-03-13T09:07:06-04:00" level=debug msg="2022/03/13 09:07:06 [WARN] found legacy provider \"terraform-provider-aws\""
time="2022-03-13T09:07:06-04:00" level=debug msg="2022/03/13 09:07:06 [DEBUG] checking for provider in \"/root/artifacts/test2\""
time="2022-03-13T09:07:06-04:00" level=debug msg="2022/03/13 09:07:06 [WARN] found legacy provider \"terraform-provider-aws\""


Retried with 4.11.0-0.nightly-2022-03-13-123528 in the same case, no issue happened, mark this issue as VERIFIED.
Comment 25 errata-xmlrpc 2022-08-10 10:36:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069