Bug 2005417
| Summary: | [IBM Z/P]: Bad Gateway error with multiple S3 requests while syncing objects to rgw bucket | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Sravika <sbalusu> | ||||||
| Component: | ceph | Assignee: | Scott Ostapovicz <sostapov> | ||||||
| Status: | CLOSED NOTABUG | QA Contact: | Raz Tamir <ratamir> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 4.9 | CC: | akandath, bniver, etamir, madam, muagarwa, nbecker, ocs-bugs, odf-bz-bot, rayalon | ||||||
| Target Milestone: | --- | Keywords: | Automation | ||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2021-11-17 09:22:48 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
Sravika
2021-09-17 15:12:01 UTC
I am able to reproduce it with the ocs-ci tier2 test tests/manage/rgw/test_object_integrity.py::TestObjectIntegrity::test_empty_file_integrity E ocs_ci.ocs.exceptions.CommandFailed: Error during execution of command: oc -n openshift-storage rsh session-awscli-relay-pod-20562f6b72ec44a sh -c "AWS_CA_BUNDLE=/cert/service-ca.crt AWS_ACCESS_KEY_ID=***** AWS_SECRET_ACCESS_KEY=***** AWS_DEFAULT_REGION=us-east-1 aws s3 --endpoint=***** sync test_empty_file_integrity/origin s3://rgw-oc-bucket-1f0ae58edf5b4ae9bc1425f152". E Error is fatal error: Connection was closed before we received a valid response from endpoint URL: "*****/rgw-oc-bucket-1f0ae58edf5b4ae9bc1425f152?list-type=2&prefix=&encoding-type=url". E command terminated with exit code 1 Nimrod, can someone please take a look. This is blocking IBM team Hi, Bad Gateway and ‘Connection was closed before we received a valid response from endpoint URL’ can imply a networking issue. Also from logs, I see that from Sep-16 22:20:42.060 there are many RPC disconnection errors and NO_SUCH_NODE errors inside NooBaa core and NooBaa endpoint logs. Few questions: 1. Do you experience other networking issues on that cluster? 2. Did you reproduce the issue on the same cluster? if not, can you try to reproduce it on another cluster? 3. Can you please provide db-dump from inside the noobaa-db-pg-0 pod run: pg_dump nbcore | gzip > nbcore_postgres.gz Thanks Hi @rayalon , 1. No there is'nt any network issue on the cluster 2. This error has been reproduced on multiple clusters and has occurred each and every time during test case execution 3. db-dump collected and attached to the BZ (nbcore_postgres.gz ) Created attachment 1840949 [details]
nbcore_postgres.gz
Hi Sravika, This is not an MCG issue, these are tests that test RGW OBC and not NooBaa OBC, this bucket is not created in noobaa, but in rook ceph. you can also see that by the test path tests/manage/rgw/test_bucket_deletion.py::TestBucketDeletion::test_bucket_delete_with_objects[RGW-OC] Also, I had a short call with Ben from OCS-CI team, and he is saying that this was an OCS-CI issue that was fixed by this PR: https://github.com/red-hat-storage/ocs-ci/pull/5011/files Please check that and I think you can close the bug afterward. Thanks, Romy Confirmed with Sravika, this issue is not seen now. |