Happens in both master, 4.3 and 4.2 builds: ``` level=error msg="Error: \"terraform-20191008145747927600000001\": NoSuchBucket: The specified bucket does not exist" level=error msg="\tstatus code: 404, request id: BCC3822AF0933B5C, host id: QVhV5kSr/ztRk3204PNQoVKN6jT7LY0Pk2vJHEKBdG5W4Eok9T+JmZXj8M/1XtBDP5fYEC4HcOI=" level=error level=error msg=" on ../tmp/openshift-install-302978166/bootstrap/main.tf line 1, in resource \"aws_s3_bucket\" \"ignition\":" level=error msg=" 1: resource \"aws_s3_bucket\" \"ignition\" {" ``` https://ci-search-ci-search-next.svc.ci.openshift.org/chart?search=Error:%20.*terraform-.*NoSuchBucket First occurrence: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/codeready-toolchain_toolchain-e2e/14/pull-ci-codeready-toolchain-toolchain-e2e-master-e2e/72
Potential duplicates: level=error msg="Error: \"terraform-20191008145747927600000001\": NoSuchBucket: The specified bucket does not exist" level=error msg="\tstatus code: 404, request id: BCC3822AF0933B5C, host id: QVhV5kSr/ztRk3204PNQoVKN6jT7LY0Pk2vJHEKBdG5W4Eok9T+JmZXj8M/1XtBDP5fYEC4HcOI=" level=error level=error msg=" on ../tmp/openshift-install-302978166/bootstrap/main.tf line 1, in resource \"aws_s3_bucket\" \"ignition\":" level=error msg=" 1: resource \"aws_s3_bucket\" \"ignition\" {" and level=error msg="Error: \"terraform-20191008153631350500000001\": OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again." level=error msg="\tstatus code: 409, request id: 381EAC2E9D3AE892, host id: 7n5vDvOMPkT6Yy4N5zzjrwWYuDwYrdTdhevzNEVjrdtx+BklXk+diu9pGBJosviWveLJXaGQS9Q=" level=error level=error msg=" on ../tmp/openshift-install-026849280/bootstrap/main.tf line 1, in resource \"aws_s3_bucket\" \"ignition\":" level=error msg=" 1: resource \"aws_s3_bucket\" \"ignition\" {"
Looks like it came from: $ AWS_PROFILE=ci aws health describe-event-details --event-arns arn:aws:health:us-east-1::event/S3/AWS_S3_OPERATIONAL_ISSUE/AWS_S3_OPERATIONAL_ISSUE_DEEWO_1570551545 --query 'successfulSet[*].eventDescription' --output text [RESOLVED] Increased Error Rates for Bucket Operations [09:19 AM PDT] We have identified the cause of the increased error rates for customers creating and managing Amazon S3 buckets and are working towards resolution. Only a small subset of requests are affected and retries are working. [09:30 AM PDT] Between 7:54 AM and 9:18 AM PDT, S3 customers saw elevated errors and latency for a subset of API requests to create and manage Amazon S3 buckets. The issue has been resolved and the service is operating normally. Hopefully more recent runs start succeeding.
Created attachment 1623608 [details] Failures related to this issue AWS seems to have gotten things back together and we're smooth again. Here's a chart from [1] showing things pretty smooth since 2019-10-08T18:37:59Z. [1]: https://ci-search-ci-search-next.svc.ci.openshift.org/chart?maxAge=8h&name=aws&search=error.*NoSuchBucket&search=Error%20creating%20S3%20bucket:%20BucketAlreadyExists&search=Error:.*terraform-.*OperationAborted:%20A%20conflicting%20conditional%20operation%20is%20currently%20in%20progress%20against%20this%20resource&search=timeout%20while%20waiting%20for%20state&search=Error%20finding%20matching%20ingress%20Security%20Group%20Rule&search=produced%20an%20unexpected%20new%20value%20for%20was%20present&search=Resource.*aws_s3_bucket.*does%20not%20have%20attribute.*id.*for%20variable&search=error%20deleting%20S3%20Bucket.*BucketNotEmpty&search=Error%20waiting%20for%20instance.*to%20become%20ready:%20Failed%20to%20reach%20target%20state.%20Reason:%20Server.InternalError:%20Internal%20error%20on%20launch
Five more in the past 24 hours [1], but no new AWS events claiming responsibility: $ AWS_PROFILE=ci aws health describe-events | jq -r '.events[] | select(.statusCode == "open" or (.endTime | gmtime | mktime) > now - 12*60*60).arn' arn:aws:health:global::event/BILLING/AWS_BILLING_NOTIFICATION/AWS_BILLING_NOTIFICATION_ded8edff-0517-40ca-b6e5-366a95ab679e Same issue though. Picking terraform-20191018044945211500000001 from [2] and searching with Athena: SELECT eventtime, eventname, useridentity.username, useragent, requestid, requestparameters, responseelements, errorcode, errormessage FROM "default"."cloudtrail_logs_cloud_trail_test_clayton" WHERE eventname LIKE '%Bucket%' AND requestparameters LIKE '%terraform-20191018044945211500000001%' ORDER BY eventtime gives: "eventtime","eventname","username","useragent","requestid","requestparameters","responseelements","errorcode","errormessage" "2019-10-18T04:49:45Z","PutBucketTagging","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","198F4F0D9303D3D8","{""tagging"":[""""],""bucketName"":""terraform-20191018044945211500000001"",""Tagging"":{""xmlns"":""http://s3.amazonaws.com/doc/2006-03-01/"",""TagSet"":{""Tag"":[{""Value"":""2019-10-18T08:49+0000"",""Key"":""expirationDate""},{""Value"":""owned"",""Key"":""kubernetes.io/cluster/ci-op-9g7ssdjx-281c8-bb2tv""},{""Value"":""ci-op-9g7ssdjx-281c8-bb2tv-bootstrap"",""Key"":""Name""}]}},""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""]}","null",, "2019-10-18T04:49:45Z","CreateBucket","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","371265472816F568","{""x-amz-acl"":[""private""],""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001""}","null",, "2019-10-18T04:49:45Z","GetBucketWebsite","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","3F0A01A019B4E5F8","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001"",""website"":[""""]}","null","NoSuchWebsiteConfiguration","The specified bucket does not have a website configuration" "2019-10-18T04:49:45Z","GetBucketCors","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","760059BF39DE77BB","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001"",""cors"":[""""]}","null","NoSuchCORSConfiguration","The CORS configuration does not exist" "2019-10-18T04:49:46Z","GetBucketLocation","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","75FBE0D398055F0C","{""host"":[""s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001"",""location"":[""""]}","null",, "2019-10-18T04:49:46Z","GetBucketRequestPayment","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","1151353FE423EFCD","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001"",""requestPayment"":[""""]}","null",, "2019-10-18T04:49:46Z","GetBucketLifecycle","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","1ADE2241FFCA2A14","{""lifecycle"":[""""],""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001""}","null",, "2019-10-18T04:49:46Z","GetBucketLogging","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","A532B2D915BD3088","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""logging"":[""""],""bucketName"":""terraform-20191018044945211500000001""}","null",, "2019-10-18T04:49:46Z","GetBucketTagging","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","A0A0025ED6B03B65","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""tagging"":[""""],""bucketName"":""terraform-20191018044945211500000001""}","null",, "2019-10-18T04:49:46Z","GetBucketEncryption","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","6BDF478607D29424","{""encryption"":[""""],""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001""}","null","ServerSideEncryptionConfigurationNotFoundError","The server side encryption configuration was not found" "2019-10-18T04:49:46Z","GetBucketObjectLockConfiguration","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","6BA5320ABBB53121","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001"",""object-lock"":[""""]}","null","ObjectLockConfigurationNotFoundError","Object Lock configuration does not exist for this bucket" "2019-10-18T04:49:46Z","GetBucketReplication","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","DB90D38B63C68219","{""replication"":[""""],""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001""}","null","ReplicationConfigurationNotFoundError","The replication configuration was not found" "2019-10-18T04:49:46Z","GetBucketVersioning","origin-ci-robot-provision","[aws-sdk-go/1.17.11 (go1.10.8; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.10]","2A2D7C49593532EB","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001"",""versioning"":[""""]}","null",, "2019-10-18T04:55:17Z","GetBucketTagging","prune_bot","[Boto3/1.9.199 Python/3.7.3 Linux/3.10.0-862.2.3.el7.x86_64 Botocore/1.12.199]","855FAF5188073296","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""tagging"":[""""],""bucketName"":""terraform-20191018044945211500000001""}","null",, "2019-10-18T04:58:24Z","DeleteBucket","origin-ci-robot-provision","[aws-sdk-go/1.16.14 (go1.10.8; linux; amd64) OpenShift/4.x Installer/v4.1.20-201910102034-dirty OpenShift/4.x Destroyer/v4.1.20-201910102034-dirty]","935E0585988358D7","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""bucketName"":""terraform-20191018044945211500000001""}","null",, "2019-10-18T05:01:50Z","PutBucketTagging","prune_bot","AWS Internal","7C027850D3F45F40","{""host"":[""terraform-20191018044945211500000001.s3.amazonaws.com""],""tagging"":[""""],""bucketName"":""terraform-20191018044945211500000001""}","null","NoSuchBucket","The specified bucket does not exist" ... From [3]: time="2019-10-18T04:49:47Z" level=debug msg="2019-10-18T04:49:47.201Z [DEBUG] plugin.terraform-provider-aws: <Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>terraform-20191018044945211500000001</BucketName><RequestId>938E77A11B4E4831</RequestId><HostId>tPx2OtBG0QvfXICF19Ri9QI07Pi9NYG4sJP5ugPqtZgxsdwHy2//kHu18Tcmf9HWoXBgIrOEZx4=</HostId></Error>" I've linked the upstream issue, but until that gets fixed I think this is either a low-probability AWS eventual-consistency flake or an out-and-out AWS backend bug. The lack of a request ID 938E77A11B4E4831 in our CloudTrail logs leads me to suspect an AWS bug. [1]: https://ci-search-ci-search-next.svc.ci.openshift.org/?search=error.*NoSuchBucket&maxAge=24h&context=-1&type=all [2]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.1/631 [3]: https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.1/631/artifacts/e2e-aws-upgrade/installer/.openshift_install.log
Moving to 4.4 unless we receive feedback from either AWS or Upstream Terraform indicating that this should be fixed.
hit this today too in https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-kube-apiserver-operator/656/pull-ci-openshift-cluster-kube-apiserver-operator-master-e2e-aws-operator/1454
apologies, ^ belong to another BZ
Upstream fix in 2.36.0 : https://github.com/terraform-providers/terraform-provider-aws/pull/10863#issuecomment-554091227
Gonna wait for the 4.4 fork: https://github.com/openshift/installer/pull/2676#issuecomment-554361016
Going to be part of https://github.com/openshift/installer/pull/2745
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/17305
In past 24 hours, still found 1 similar failure. https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade/19675
Dropping my assignment, because I'm not going to have time to take another look at this.
In the past 24 hours, still found similar failure: https://search.svc.ci.openshift.org/?search=Error%3A+.*terraform-.*NoSuchBucket&maxAge=24h&context=1&type=build-log https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/kubevirt_hyperconverged-cluster-operator/506/pull-ci-kubevirt-hyperconverged-cluster-operator-master-hco-e2e-aws/1218 https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-csi-snapshot-controller-operator/32/pull-ci-openshift-cluster-csi-snapshot-controller-operator-master-e2e-aws/58
this is still pretty common https://search.apps.build01.ci.devcluster.openshift.com/?search=NoSuchBucket&maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job and we try to fix this.
This seems to be fixed in https://github.com/terraform-providers/terraform-provider-aws/pull/13009. I've grabbed the commit and placed it into a branch of the 2.62.0 release here: https://github.com/openshift/terraform-provider-aws/pull/1 and a pull request against the installer to use our fork here: https://github.com/openshift/installer/pull/3621.
Updated the AWS terraform provider PR here: https://github.com/openshift/terraform-provider-aws/pull/3
Waiting for https://github.com/openshift/installer/pull/3621 to be merged.
Not found “NoSuchBucket” error in the last 3 days, mark this bug as VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409