Bug 2100140

Summary:	[GSS]InvalidBucketState 409 The request is not valid with the current state of the bucket
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	khover
Component:	Multi-Cloud Object Gateway	Assignee:	Nimrod Becker <nbecker>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Ben Eli <belimele>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.8	CC:	assingh, etamir, hnallurv, muagarwa, nbecker, ocs-bugs, odf-bz-bot
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-16 02:48:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description khover 2022-06-22 14:28:28 UTC

Description of problem (please be detailed as possible and provide log
snippests):

InvalidBucketState 409 The request is not valid with the current state of the bucket

A file that is 0 bytes "writes" to the bucket. However, when we check on the AWS S3 console and look at the actual bucket (the one listed in the backing store), we do not see this file there. It appears there is still some connection issue here with the Noobaa layer as we can write directly to the AWS S3 bucket, but not to the Noobaa bucket that's backed by the AWS S3 bucket.

SNIP:

2022-06-16T18:34:32.954487050Z Jun-16 18:34:32.954 [Endpoint/11]    [L0] core.server.object_services.map_server:: enough_room_in_tier: not enough room SENSITIVE-e5f2776edeafbc3f: 0 < 209715200 should move chunks to next tier




Version of all relevant components (if applicable):

ocs-operator.v4.8.6

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.
2.
3.


Actual results:


Expected results:


Additional info:

Comment 2 khover 2022-06-22 14:30:15 UTC

Spec:
  awsS3:
    Region:  us-gov-west-1
    Secret:
      Name:         aws-crunchy-backing-store-creds
      Namespace:    openshift-storage
    Ssl Disabled:   true
    Target Bucket:  aif-crunch-test-dldev
  Type:             aws-s3
Status:
  Conditions:
    Last Heartbeat Time:   2022-05-27T23:06:00Z
    Last Transition Time:  2022-05-27T23:07:24Z
    Message:               noobaa operator completed reconcile - backing store is ready
    Reason:                BackingStorePhaseReady
    Status:                True
    Type:                  Available


- The backingstore is reporting as healthy from noobaa but from Console (Ref. noobaaUIbackingstorecreate.png)

- The bucket list works with AWS CLI. 

-  attempts to hit the bucket and the following :


ERROR: [039]: HTTP request failed with 409 (Conflict):
       *** Path/Query ***:
       /aws-crunch-bucket-supreme-0d1d0462-b369-47f9-955c-ac0230fbaff1/pgbackrest/repo1/archive/db/archive.info
       *** Request Headers ***:
       authorization: <redacted>
       content-length: 253
       content-md5: TYdRkJUQgA53A4ai0ebigg==
       host: s3.openshift-storage.svc
       x-amz-content-sha256: 9b5c9e5253c9c15db1e3ffc4e548fde42cffbb00e600ac68d74a88d6044e84cc
       x-amz-date: <redacted>
       *** Response Headers ***:
       access-control-allow-credentials: true
       access-control-allow-headers: Content-Type,Content-MD5,Authorization,X-Amz-User-Agent,X-Amz-Date,ETag,X-Amz-Content-Sha256
       access-control-allow-methods: GET,POST,PUT,DELETE,OPTIONS
       access-control-allow-origin: *
       access-control-expose-headers: ETag,X-Amz-Version-Id
       connection: keep-alive
       content-length: 333
       content-type: application/xml
       date: Tue, 24 May 2022 15:24:34 GMT
       keep-alive: timeout=5
       x-amz-id-2: l3kb5z6c-5rgmod-12gj
       x-amz-request-id: l3kb5z6c-5rgmod-12gj
       *** Response Content ***:
       <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidBucketState</Code><Message>The request is not valid with the current state of the bucket.</Message><Resource>/aws-crunch-bucket-supreme-0d1d0462-b369-47f9-955c-ac0230fbaff1/pgbackrest/repo1/archive/db/archive.info</Resource><RequestId>l3kb5z6c-5rgmod-12gj</RequestId></Error>

Comment 4 khover 2022-06-23 19:45:36 UTC

Customer restarted the Noobaa stack 


noobaa-db
noobaa-core
noobaa-endpoint 
noobaa-operator

still receiving the same error:


An error occurred (InvalidBucketState) when calling the PutObject operation: The request is not valid with the current state of the bucket


I was under the impression that restarting noobaa-core was the workaround for Dup

https://bugzilla.redhat.com/show_bug.cgi?id=2094020