Bug 1759617 - NoSuchBucket: The specified bucket does not exist" error during install
Summary: NoSuchBucket: The specified bucket does not exist" error during install
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.0
Assignee: John Hixson
QA Contact: Yunfei Jiang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-10-08 17:05 UTC by Vadim Rutkovsky
Modified: 2020-07-13 17:12 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-13 17:11:31 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Failures related to this issue (271.14 KB, image/png)
2019-10-08 20:03 UTC, W. Trevor King
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 2745 0 None closed Go Modules Migration 2020-08-05 12:00:25 UTC
Github openshift installer pull 3323 0 None closed bug 1759617: vendor: bump tf provider aws to v2.54.0 2020-08-05 12:00:21 UTC
Github openshift installer pull 3621 0 None closed Bug 1759617: vendor: update to use openshift fork of terraform-provider-aws 2020-08-05 12:00:22 UTC
Github terraform-providers terraform-provider-aws issues 10068 0 'None' closed Error creating S3 bucket with tags - error getting S3 bucket tags: NoSuchBucket: The specified bucket does not exist 2020-08-05 12:00:20 UTC
Github terraform-providers terraform-provider-aws pull 12418 0 None closed resource_aws_s3_bucket.go: retry NoSuchBucket when updating tags for s3 bucket 2020-08-05 12:00:19 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:12:02 UTC

Description Vadim Rutkovsky 2019-10-08 17:05:42 UTC
Happens in both master, 4.3 and 4.2 builds:

```
level=error msg="Error: \"terraform-20191008145747927600000001\": NoSuchBucket: The specified bucket does not exist"
level=error msg="\tstatus code: 404, request id: BCC3822AF0933B5C, host id: QVhV5kSr/ztRk3204PNQoVKN6jT7LY0Pk2vJHEKBdG5W4Eok9T+JmZXj8M/1XtBDP5fYEC4HcOI="
level=error
level=error msg="  on ../tmp/openshift-install-302978166/bootstrap/main.tf line 1, in resource \"aws_s3_bucket\" \"ignition\":"
level=error msg="   1: resource \"aws_s3_bucket\" \"ignition\" {"
```

https://ci-search-ci-search-next.svc.ci.openshift.org/chart?search=Error:%20.*terraform-.*NoSuchBucket
First occurrence: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/codeready-toolchain_toolchain-e2e/14/pull-ci-codeready-toolchain-toolchain-e2e-master-e2e/72

Comment 1 Vadim Rutkovsky 2019-10-08 17:06:55 UTC
Potential duplicates:

level=error msg="Error: \"terraform-20191008145747927600000001\": NoSuchBucket: The specified bucket does not exist"
level=error msg="\tstatus code: 404, request id: BCC3822AF0933B5C, host id: QVhV5kSr/ztRk3204PNQoVKN6jT7LY0Pk2vJHEKBdG5W4Eok9T+JmZXj8M/1XtBDP5fYEC4HcOI="
level=error
level=error msg="  on ../tmp/openshift-install-302978166/bootstrap/main.tf line 1, in resource \"aws_s3_bucket\" \"ignition\":"
level=error msg="   1: resource \"aws_s3_bucket\" \"ignition\" {"

and

level=error msg="Error: \"terraform-20191008153631350500000001\": OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again."
level=error msg="\tstatus code: 409, request id: 381EAC2E9D3AE892, host id: 7n5vDvOMPkT6Yy4N5zzjrwWYuDwYrdTdhevzNEVjrdtx+BklXk+diu9pGBJosviWveLJXaGQS9Q="
level=error
level=error msg="  on ../tmp/openshift-install-026849280/bootstrap/main.tf line 1, in resource \"aws_s3_bucket\" \"ignition\":"
level=error msg="   1: resource \"aws_s3_bucket\" \"ignition\" {"

Comment 2 W. Trevor King 2019-10-08 17:12:37 UTC
Looks like it came from:

  $ AWS_PROFILE=ci aws health describe-event-details --event-arns arn:aws:health:us-east-1::event/S3/AWS_S3_OPERATIONAL_ISSUE/AWS_S3_OPERATIONAL_ISSUE_DEEWO_1570551545 --query 'successfulSet[*].eventDescription' --output text
  [RESOLVED] Increased Error Rates for Bucket Operations

  [09:19 AM PDT] We have identified the cause of the increased error rates for customers creating and managing Amazon S3 buckets and are working towards resolution. Only a small subset of requests are affected and retries are working.

  [09:30 AM PDT] Between 7:54 AM and 9:18 AM PDT, S3 customers saw elevated errors and latency for a subset of API requests to create and manage Amazon S3 buckets. The issue has been resolved and the service is operating normally.

Hopefully more recent runs start succeeding.

Comment 5 W. Trevor King 2019-10-18 10:16:22 UTC
Five more in the past 24 hours [1], but no new AWS events claiming responsibility:

$ AWS_PROFILE=ci aws health describe-events | jq -r '.events[] | select(.statusCode == "open" or (.endTime | gmtime | mktime) > now - 12*60*60).arn'
arn:aws:health:global::event/BILLING/AWS_BILLING_NOTIFICATION/AWS_BILLING_NOTIFICATION_ded8edff-0517-40ca-b6e5-366a95ab679e

Same issue though.  Picking terraform-20191018044945211500000001 from [2] and searching with Athena:

SELECT eventtime,
       eventname,
       useridentity.username,
       useragent,
       requestid,
       requestparameters,
       responseelements,
       errorcode,
       errormessage
FROM "default"."cloudtrail_logs_cloud_trail_test_clayton"
WHERE eventname LIKE '%Bucket%'
  AND requestparameters LIKE '%terraform-20191018044945211500000001%'
ORDER BY eventtime

gives:

"eventtime","eventname","username","useragent","requestid","requestparameters","responseelements","errorcode","errormessage"
"2019-10-18T04:49:45Z","PutBucketTagging","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","198F4F0D9303D3D8","{""tagging"":[""""],""bucketName"":""terraform-20191018044945211500000001"",""Tagging"":{""xmlns"":""http://s3.amazonaws.com/doc/2006-03-01/"",""TagSet"":{""Tag"":[{""Value"":""2019-10-18T08:49+0000"",""Key"":""expirationDate""},{""Value"":""owned"",""Key"":""kubernetes.io/cluster/ci-op-9g7ssdjx-281c8-bb2tv""},{""Value"":""ci-op-9g7ssdjx-281c8-bb2tv-bootstrap"",""Key"":""Name""}]}},""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""]}","null",,
"2019-10-18T04:49:45Z","CreateBucket","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","371265472816F568","{""x-amz-acl"":[""private""],""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001""}","null",,
"2019-10-18T04:49:45Z","GetBucketWebsite","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","3F0A01A019B4E5F8","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001"",""website"":[""""]}","null","NoSuchWebsiteConfiguration","The specified bucket does not have a website configuration"
"2019-10-18T04:49:45Z","GetBucketCors","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","760059BF39DE77BB","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001"",""cors"":[""""]}","null","NoSuchCORSConfiguration","The CORS configuration does not exist"
"2019-10-18T04:49:46Z","GetBucketLocation","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","75FBE0D398055F0C","{""host"":[""s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001"",""location"":[""""]}","null",,
"2019-10-18T04:49:46Z","GetBucketRequestPayment","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","1151353FE423EFCD","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001"",""requestPayment"":[""""]}","null",,
"2019-10-18T04:49:46Z","GetBucketLifecycle","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","1ADE2241FFCA2A14","{""lifecycle"":[""""],""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001""}","null",,
"2019-10-18T04:49:46Z","GetBucketLogging","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","A532B2D915BD3088","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""logging"":[""""],""bucketName"":""terraform-20191018044945211500000001""}","null",,
"2019-10-18T04:49:46Z","GetBucketTagging","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","A0A0025ED6B03B65","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""tagging"":[""""],""bucketName"":""terraform-20191018044945211500000001""}","null",,
"2019-10-18T04:49:46Z","GetBucketEncryption","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","6BDF478607D29424","{""encryption"":[""""],""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001""}","null","ServerSideEncryptionConfigurationNotFoundError","The server side encryption configuration was not found"
"2019-10-18T04:49:46Z","GetBucketObjectLockConfiguration","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","6BA5320ABBB53121","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001"",""object-lock"":[""""]}","null","ObjectLockConfigurationNotFoundError","Object Lock configuration does not exist for this bucket"
"2019-10-18T04:49:46Z","GetBucketReplication","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","DB90D38B63C68219","{""replication"":[""""],""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001""}","null","ReplicationConfigurationNotFoundError","The replication configuration was not found"
"2019-10-18T04:49:46Z","GetBucketVersioning","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","2A2D7C49593532EB","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001"",""versioning"":[""""]}","null",,
"2019-10-18T04:55:17Z","GetBucketTagging","prune_bot","[Boto3/1.9.199 Python/3.7.3 Linux/3.10.0-862.2.3.el7.x86_64 Botocore/1.12.199]","855FAF5188073296","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""tagging"":[""""],""bucketName"":""terraform-20191018044945211500000001""}","null",,
"2019-10-18T04:58:24Z","DeleteBucket","origin-ci-robot-provision","[aws-sdk-go/1.16.14 (go1.10.8; linux; amd64) OpenShift/4.x Installer/v4.1.20-201910102034-dirty OpenShift/4.x Destroyer/v4.1.20-201910102034-dirty]","935E0585988358D7","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001""}","null",,
"2019-10-18T05:01:50Z","PutBucketTagging","prune_bot","AWS Internal","7C027850D3F45F40","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""tagging"":[""""],""bucketName"":""terraform-20191018044945211500000001""}","null","NoSuchBucket","The specified bucket does not exist"
...

From [3]:

time="2019-10-18T04:49:47Z" level=debug msg="2019-10-18T04:49:47.201Z [DEBUG] plugin.terraform-provider-aws: <Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>terraform-20191018044945211500000001</BucketName><RequestId>938E77A11B4E4831</RequestId><HostId>tPx2OtBG0QvfXICF19Ri9QI07Pi9NYG4sJP5ugPqtZgxsdwHy2//kHu18Tcmf9HWoXBgIrOEZx4=</HostId></Error>"

I've linked the upstream issue, but until that gets fixed I think this is either a low-probability AWS eventual-consistency flake or an out-and-out AWS backend bug.  The lack of a request ID 938E77A11B4E4831 in our CloudTrail logs leads me to suspect an AWS bug.

[1]: https://ci-search-ci-search-next.svc.ci.openshift.org/?search=error.*NoSuchBucket&maxAge=24h&context=-1&type=all
[2]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.1/631
[3]: https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.1/631/artifacts/e2e-aws-upgrade/installer/.openshift_install.log

Comment 7 Scott Dodson 2019-10-21 17:24:26 UTC
Moving to 4.4 unless we receive feedback from either AWS or Upstream Terraform indicating that this should be fixed.

Comment 9 Tomáš Nožička 2019-11-13 17:37:12 UTC
apologies, ^ belong to another BZ

Comment 11 W. Trevor King 2019-11-15 14:57:47 UTC
Gonna wait for the 4.4 fork: https://github.com/openshift/installer/pull/2676#issuecomment-554361016

Comment 12 Scott Dodson 2020-02-04 19:51:33 UTC
Going to be part of https://github.com/openshift/installer/pull/2745

Comment 18 Johnny Liu 2020-03-02 09:50:55 UTC
In past 24 hours, still found 1 similar failure.
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/19675

Comment 19 W. Trevor King 2020-03-16 16:31:50 UTC
Dropping my assignment, because I'm not going to have time to take another look at this.

Comment 23 John Hixson 2020-05-19 17:56:25 UTC
This seems to be fixed in https://github.com/terraform-providers/terraform-provider-aws/pull/13009. I've grabbed the commit and placed it into a branch of the 2.62.0 release here: https://github.com/openshift/terraform-provider-aws/pull/1 and a pull request against the installer to use our fork here: https://github.com/openshift/installer/pull/3621.

Comment 24 John Hixson 2020-05-26 23:31:51 UTC
Updated the AWS terraform provider PR here: https://github.com/openshift/terraform-provider-aws/pull/3

Comment 26 John Hixson 2020-05-28 20:39:42 UTC
Waiting for https://github.com/openshift/installer/pull/3621 to be merged.

Comment 28 Yunfei Jiang 2020-06-03 03:32:48 UTC
Not found “NoSuchBucket” error in the last 3 days, mark this bug as VERIFIED.

Comment 30 errata-xmlrpc 2020-07-13 17:11:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.