Bug 2190379

Summary:	Multi-part uploads failing with Bad Gateway using the ocs-storagecluster-cephobjectstore route
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Matthias Muench <mmuench>
Component:	ocs-operator	Assignee:	Jiffin <jthottan>
Status:	CLOSED DUPLICATE	QA Contact:	Elad <ebenahar>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.12	CC:	mmuench, muagarwa, nigoyal, ocs-bugs, odf-bz-bot
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-06-15 12:03:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Matthias Muench 2023-04-28 08:38:14 UTC

Description of problem (please be detailed as possible and provide log
snippests):
Uploads to route ocs-storagecluster-cephobjectstore to RGW are failing - at least for multi-part uploads with "Bad Gateway".
Testing directly against RGW pod using oc port-forward works.


Version of all relevant components (if applicable):
OCP 4.12.13 
ODF 4.12.2


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes


Is there any workaround available to the best of your knowledge?
No viable.


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1


Can this issue reproducible?
- upload a larger file (1GB) to the route using aws CLI
- using OADP Restic based backup to RGW bucket


Can this issue reproduce from the UI?
no


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. create 1-GB-file 
2. aws s3 --endpoint http://ocs-storagecluster-cephobjectstore-openshift-storage.apps.8zc6x.dynamic.opentlc.com cp 1-GB-file s3://<bucketname>
3.


Actual results:
fails after a few parts with Bad Gateway


Expected results:
multi-part uploads should succeed


Additional info:
must-gather available at https://drive.google.com/file/d/1pPfFInP4-PEfaCcpvnZcXW00qDW0ZeZw/view?usp=sharing

Comment 2 Jiffin 2023-04-28 10:08:10 UTC

Since the gateway errors seen only in the route I suspect it is related to port used for the route the https://bugzilla.redhat.com/show_bug.cgi?id=2104148. Can u please try to workaround https://bugzilla.redhat.com/show_bug.cgi?id=2161279#c22 and see if it exists

Comment 4 Matthias Muench 2023-06-15 09:14:57 UTC

Tested with 4.12.4 - the workaround relives the misbehaviour.
Revoking the changes from the workaround reintroduces the misbehaviour.
With this finding, the workaround would fix the reported issue.

Comment 5 Jiffin 2023-06-15 12:03:28 UTC


*** This bug has been marked as a duplicate of bug 2104148 ***