Bug 1878473

Any logs related to this host in the service?

server logs:
ime="2020-09-13T08:05:38Z" level=error msg="Failed to authorize user: can't get access token: can't send request: Post \"https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token\": remote error: tls: internal error" func="github.com/openshift/assisted-service/pkg/auth.(*Au

Related slack discussion: https://coreos.slack.com/archives/CUPJTHQ5P/p1599985225444200

Reproduced to me as well on staging

full log:
{code}
time="2020-09-14T05:52:12Z" level=error msg="Failed to authorize user: can't send request: Post \"https://api.stage.openshift.com/api/authorizations/v1/access_review\": read tcp 10.128.4.222:42782->3.217.136.187:443: read: connection reset by peer" func="github.com/openshift/assisted-service/pkg/auth.(*AuthzHandler).Authorizer" file="/go/src/github.com/openshift/origin/pkg/auth/authz_handler.go:64" pkg=authz
{code}

Another log:
{code}
time="2020-09-13T14:37:24Z" level=error msg="Failed to authorize user: can't get access token: can't send request: Post \"https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token\": remote error: tls: internal error" func="github.com/openshift/assisted-service/pkg/auth.(*AuthzHandler).Authorizer" file="/go/src/github.com/openshift/origin/pkg/auth/authz_handler.go:64" pkg=authz
{code}

I found this https://github.com/openshift-online/ocm-sdk-go/blob/master/send.go#L150
so it probably the source and the error type is fmt.Error so i will search for can't send request in the error and then return 503

It looks like eventually every error will return forbidden https://github.com/go-openapi/runtime/blob/master/middleware/context.go#L436-L440

Created attachment 1714902 [details]
OCM responses

assisted-service https://github.com/openshift/assisted-service/pull/363

Verified
========
on staging Sep 15

Issue still persist in Staging
Sep 17 04:26:27 master-0-0 agent[2002]: time="17-09-2020 04:26:27" level=error msg="User is not authenticated to perform the operation, going to sleep forever" file="step_processor.go:97" error="[GET /clusters/{cluster_id}/hosts/{host_id}/instructions][401] getNextStepsUnauthorized  &{Code:0xc0007e42a0 Message:0xc00080e670}" request_id=85bdfae8-ad9e-4c22-b49f-93fc53a04d51


Cluster id: c097015c-9cd5-4de0-a7be-a7d1e7295983

7h passed (from known --> Disconnected)

9/17/2020, 7:28:09 AM	warning Host master-0-0: updated status from "known" to "disconnected" (Host has stopped communicating with the installation service)
9/17/2020, 3:22:53 AM	Deleted image from backend because it expired. It may be generated again at any time.
9/16/2020, 11:55:53 PM	Updated status of cluster ocp-edge34-cluster-0 to ready
9/16/2020, 11:55:52 PM	Host master-0-0: updated status from "discovering" to "known" (Host is ready to be installed)

host ID: 9ed62e95-27ba-4981-be77-9bbae8777227
Cluster id: c097015c-9cd5-4de0-a7be-a7d1e7295983

OCM debug log:

time="2020-09-17T07:15:10Z" level=debug msg="OpenID token URL wasn't provided, will use 'https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token'" p
kg=ocm
time="2020-09-17T07:15:10Z" level=debug msg="Bearer token isn't available" pkg=ocm
time="2020-09-17T07:15:10Z" level=debug msg="Refresh token isn't available" pkg=ocm
time="2020-09-17T07:15:10Z" level=debug msg="Requesting new token using the client credentials grant" pkg=ocm
time="2020-09-17T07:15:10Z" level=debug msg="Request method is POST" pkg=ocm
time="2020-09-17T07:15:10Z" level=debug msg="Request URL is 'https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token'" pkg=ocm
time="2020-09-17T07:15:10Z" level=debug msg="Request header 'Accept' is 'application/json'" pkg=ocm
time="2020-09-17T07:15:10Z" level=debug msg="Request header 'Content-Type' is 'application/x-www-form-urlencoded'" pkg=ocm
time="2020-09-17T07:15:10Z" level=debug msg="Request header 'User-Agent' is 'OCM/0.1.118'" pkg=ocm
time="2020-09-17T07:15:10Z" level=debug msg="Request body follows" pkg=ocm
time="2020-09-17T07:15:10Z" level=debug msg="client_id=assisted-installer-prod&client_secret=***&grant_type=client_credentials&scope=openid" pkg=ocm
{"file":"/go/src/github.com/openshift/origin/pkg/auth/auth_handler.go:86","func":"github.com/openshift/assisted-service/pkg/auth.(*AuthHandler).AuthAgentAuth","level":"error","msg":"Error Authenticating PullSecret token: can't get access token: can't send request: Post \"https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token\": remote error: tls: internal error","pkg":"auth","time":"2020-09-17T07:15:12Z"}

Handled in https://issues.redhat.com/browse/MGMT-2181 
Added more logs and improve the implementation.

https://github.com/openshift/assisted-service/pull/394
https://github.com/openshift/assisted-service/pull/398
https://github.com/openshift/assisted-service/pull/401

Thanks Fred, we will wait for another release in order to test it.

Verified on Staging v1.0.9.5

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.43 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3197