Bug 2046277

Summary:	openshift-installer intermittent failure on AWS with "Error: Provider produced inconsistent result after apply" when creating the module.vpc.aws_subnet.private_subnet[0] resource
Product:	OpenShift Container Platform	Reporter:	Greg Sheremeta <gshereme>
Component:	Installer	Assignee:	Nobody <nobody>
Installer sub component:	openshift-installer	QA Contact:	Yunfei Jiang <yunjiang>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	cblecker, padillon, yunjiang
Version:	4.10	Keywords:	ServiceDeliveryBlocker
Target Milestone:	---
Target Release:	4.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: there was an eventual consistency issue in the aws-terraform-provider when trying to update newly created subnets Consequence: installs would fail trying to access subnet Fix: installer updated to upstream terraform-provider which has fix to respect eventual consistency Result: install does not fail	Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-10 10:43:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2047390

Description Greg Sheremeta 2022-01-26 13:59:58 UTC

$ openshift-install version
4.9.x

Platform: AWS (seen in CI here: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-csi-4.10/1486071548644167680)

Please specify:
IPI

What happened?
Error: Provider produced inconsistent result after apply

What did you expect to happen?
Successful install

How to reproduce it (as minimally and precisely as possible)?
It is random and rare. AWS eventual consistency / raciness bug. AWS needs to be having a bad day to reproduce it.

Flow seems to be:
1 Installer creates a thing
2 AWS creates it
3 AWS says it doesn't exist
4 Terrform dies

log excerpt:

level=error msg=Error: Provider produced inconsistent result after apply
level=error
level=error msg=When applying changes to module.vpc.aws_subnet.private_subnet[0], provider
level=error msg="openshift/local/aws" produced an unexpected new value: Root resource was
level=error msg=present, but now absent.
level=error
level=error msg=This is a bug in the provider, which should be reported in the provider's own
level=error msg=issue tracker.

Comment 1 Matthew Staebler 2022-01-31 18:47:44 UTC

The upstream fix for this is https://github.com/hashicorp/terraform-provider-aws/pull/18392.

Comment 5 Patrick Dillon 2022-05-03 17:55:38 UTC

The upstream fix for this has been brought in with https://github.com/openshift/installer/pull/5666

CI search is timing out for searches for the error. Moving to MODIFIED for QE verification.

Comment 7 Yunfei Jiang 2022-05-09 03:25:13 UTC

searched ci log [1], no errors found in 4.11

[1] https://search.ci.openshift.org/?search=When+applying+changes+to+module.vpc.aws_subnet.private_subnet&maxAge=168h&context=10&type=build-log&name=.%2B4%5C.11.%2Baws.%2B&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Comment 10 errata-xmlrpc 2022-08-10 10:43:43 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069