Bug 2096529

Summary: [GSS] s3 calls to noobaa from application are stuck intermittently
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Priya Pandey <prpandey>
Component: Multi-Cloud Object GatewayAssignee: Danny <dzaken>
Status: CLOSED CURRENTRELEASE QA Contact: Ben Eli <belimele>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.8CC: assingh, dzaken, etamir, jalbo, kurathod, nbecker, ocs-bugs, odf-bz-bot, tdesala
Target Milestone: ---Flags: assingh: needinfo? (jalbo)
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2100023 (view as bug list) Environment:
Last Closed: 2022-06-22 08:32:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2100023    

Description Priya Pandey 2022-06-14 07:12:57 UTC
Description of problem (please be detailed as possible and provide log
snippets):

- The s3 call are stuck intermittently from the application.

--------------------------------------

Traceback (most recent call last):
  File "awscli/botocore/utils.py", line 352, in _fetch_metadata_token
  File "awscli/botocore/httpsession.py", line 373, in send
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://169.254.169.254/latest/api/token"
2022-06-08 19:01:44,687 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (2): 169.254.169.254:80
2022-06-08 19:01:44,688 - MainThread - botocore.utils - DEBUG - Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/placement/availability-zone/: Could not connect to the endpoint URL: "http://169.254.169.254/latest/meta-data/placement/availability-zone/"
Traceback (most recent call last):
  File "urllib3/connection.py", line 174, in _new_conn
  File "urllib3/util/connection.py", line 96, in create_connection
  File "urllib3/util/connection.py", line 86, in create_connection
OSError: [Errno 113] No route to host
--------------------------------------

- It's recovered by restarting noobaa-core and noobaa-endpoint pods.


Version of all relevant components (if applicable):

v4.8.12

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


- The application which connects to the S3 service is stuck intermittenty.


Is there any workaround available to the best of your knowledge?

N/A

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
N/A

Can this issue reproducible?
N/A

Can this issue reproduce from the UI?
N/A

If this is a regression, please provide more details to justify this:
N/A

Steps to Reproduce:
N/A

Actual results:

- The application which connects to the S3 service is stuck at times.

Expected results:

- The application which connects to the S3 service should work smoothly.


Additional info:

In the next comments

Comment 3 Eran Tamir 2022-06-14 12:44:04 UTC
Can we please get instructions on how to reproduce? Is there a specific script you are running to make it happen?

Comment 8 Eran Tamir 2022-06-16 10:49:49 UTC
Premkumar, Is there a reproduction script you can provide to help us reproduce it in our lab?