Bug 1977184 - Image registry Degraded caused by requesting to aws sts global endpoint timeout when installing sts cluster in a disconnected network
Summary: Image registry Degraded caused by requesting to aws sts global endpoint timeo...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.z
Assignee: Oleg Bulatov
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On: 1939842
Blocks: 1974499
TreeView+ depends on / blocked
 
Reported: 2021-06-29 06:59 UTC by wang lin
Modified: 2021-07-23 09:40 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1939842
Environment:
Last Closed: 2021-07-20 11:23:42 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Comment 1 wang lin 2021-06-30 06:48:33 UTC
Verified with cluster-bot image with pr#701 merged, cluster image registry can respect serviceEndpoint in infrastructure and work well.

$ oc get co image-registry
NAME             VERSION                                                  AVAILABLE   PROGRESSING   DEGRADED   SINCE
image-registry   4.8.0-0.ci.test-2021-06-30-021220-ci-ln-81p79j2-latest   True        False         False      3h45m

$ oc get infrastructure cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Infrastructure
metadata:
  creationTimestamp: "2021-06-30T02:39:23Z"
  generation: 1
  name: cluster
  resourceVersion: "680"
  uid: d9bd533b-4254-4daa-8921-d74bb153cc0f
spec:
  cloudConfig:
    name: ""
  platformSpec:
    aws:
      serviceEndpoints:
      - name: sts
        url: https://sts.us-east-2.amazonaws.com
    type: AWS
status:
  apiServerInternalURI: https://api-int.lwanipid0630.qe.devcluster.openshift.com:6443
  apiServerURL: https://api.lwanipid0630.qe.devcluster.openshift.com:6443
  controlPlaneTopology: HighlyAvailable
  etcdDiscoveryDomain: ""
  infrastructureName: lwanipid0630-vsbb5
  infrastructureTopology: HighlyAvailable
  platform: AWS
  platformStatus:
    aws:
      region: us-east-2
      serviceEndpoints:
      - name: sts
        url: https://sts.us-east-2.amazonaws.com
    type: AWS

Comment 2 Wenjing Zheng 2021-07-02 02:24:52 UTC
Met below 500 error: err.detail="s3aws: WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post \"https://sts.amazonaws.com/\": dial tcp 52.46.134.192:443: i/o timeout" when trigger build with STS+disconnected cluster


time="2021-07-02T02:14:59.351119646Z" level=info msg="authorized request" go.version=go1.15.7 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=662f0911-5cdc-4b59-b89c-7cca83b3d5ba http.request.method=GET http.request.remoteaddr="10.129.2.24:33082" http.request.uri="/v2/openshift/httpd/manifests/sha256:e48906d6ce958d7b545808fc3b115bb7e60bde9c7a61b4049bf16fa16e480537" http.request.useragent="containers/5.10.6 (github.com/containers/image)" openshift.auth.user="system:serviceaccount:wzheng1:builder" vars.name=openshift/httpd vars.reference="sha256:e48906d6ce958d7b545808fc3b115bb7e60bde9c7a61b4049bf16fa16e480537"
time="2021-07-02T02:19:00.004341792Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post \"https://sts.amazonaws.com/\": dial tcp 52.46.134.192:443: i/o timeout" err.message="unknown error" go.version=go1.15.7 http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=662f0911-5cdc-4b59-b89c-7cca83b3d5ba http.request.method=GET http.request.remoteaddr="10.129.2.24:33082" http.request.uri="/v2/openshift/httpd/manifests/sha256:e48906d6ce958d7b545808fc3b115bb7e60bde9c7a61b4049bf16fa16e480537" http.request.useragent="containers/5.10.6 (github.com/containers/image)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=4m0.669550251s http.response.status=500 http.response.written=104 openshift.auth.user="system:serviceaccount:wzheng1:builder" vars.name=openshift/httpd vars.reference="sha256:e48906d6ce958d7b545808fc3b115bb7e60bde9c7a61b4049bf16fa16e480537"


Note You need to log in before you can comment on or make changes to this bug.