Bug 2067106 - insights is degraded for failed to pull SCA certs
Summary: insights is degraded for failed to pull SCA certs
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Insights Operator
Version: 4.11
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Tomas Remes
QA Contact: Joao Fula
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-03-23 10:26 UTC by Junqi Zhao
Modified: 2023-06-26 01:34 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-08-29 06:30:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Junqi Zhao 2022-03-23 10:26:56 UTC
Description of problem:
fresh IPI_AWS 4.11.0-0.nightly-2022-03-20-160505 cluster, insights is normal at the beginning, check again later, insights is degraded
# oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-03-20-160505   True        False         9h      Error while reconciling 4.11.0-0.nightly-2022-03-20-160505: the cluster operator insights has not yet successfully rolled out

# oc get co insights
NAME       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
insights   4.11.0-0.nightly-2022-03-20-160505   False       False         True       5h30m   Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/certificates: OCM API https://api.openshift.com/api/accounts_mgmt/v1/certificates returned HTTP 500: {"id":"9","kind":"Error","href":"/api/accounts_mgmt/v1/errors/9","code":"ACCT-MGMT-9","reason":"Unable to get Certificates Serials from RHSM for consumer (UUID=778099db-a840-4bc3-b588-8e91248bf062): 502 Bad Gateway","operation_id":"00b4b7b1-b684-41c8-9668-259a5f537158"}

# oc -n openshift-insights get pod
NAME                                READY   STATUS    RESTARTS     AGE
insights-operator-dd5bf5b57-vq5ws   1/1     Running   1 (9h ago)   9h

# oc -n openshift-insights logs insights-operator-dd5bf5b57-vq5ws | head -n 1
I0323 00:44:33.098156       1 cmd.go:209] Using service-serving-cert provided certificates

almost 5 minutes later, 500 error shows
# oc -n openshift-insights logs insights-operator-dd5bf5b57-vq5ws | grep "OCM API https://api.openshift.com/api/accounts_mgmt/v1/certificates returned HTTP 500" | head -n 2
E0323 00:49:39.743849       1 sca.go:228] OCM API https://api.openshift.com/api/accounts_mgmt/v1/certificates returned HTTP 500: {"id":"9","kind":"Error","href":"/api/accounts_mgmt/v1/errors/9","code":"ACCT-MGMT-9","reason":"Unable to get Certificates Serials from RHSM for consumer (UUID=778099db-a840-4bc3-b588-8e91248bf062): 502 Bad Gateway","operation_id":"f1b83b1a-e004-4182-b8c0-e25d8416b286"}. Trying again in 15m0s
E0323 01:04:39.934512       1 sca.go:228] OCM API https://api.openshift.com/api/accounts_mgmt/v1/certificates returned HTTP 500: {"id":"9","kind":"Error","href":"/api/accounts_mgmt/v1/errors/9","code":"ACCT-MGMT-9","reason":"Unable to get Certificates Serials from RHSM for consumer (UUID=778099db-a840-4bc3-b588-8e91248bf062): 502 Bad Gateway","operation_id":"f3bb4225-d96b-44f8-8872-c00505fdcf06"}. Trying again in 30m0s

Version-Release number of selected component (if applicable):
4.11.0-0.nightly-2022-03-20-160505

How reproducible:
always

Steps to Reproduce:
1. oc get co insights
2.
3.

Actual results:
insights is degraded

Expected results:
should be normal

Additional info:

Comment 2 yasun 2022-03-23 11:13:40 UTC
It is an ocm bug, open a bug https://issues.redhat.com/browse/SDB-2699 to track the issue.

Comment 3 yasun 2022-03-24 07:13:02 UTC
Now we can get the sca successfully on production ocm.

Comment 4 yasun 2022-03-24 07:15:32 UTC
  Please help verify on your side, and close the bug.

Comment 5 Junqi Zhao 2022-03-24 09:18:44 UTC
https://issues.redhat.com/browse/SDB-2699 is fixed, insights is normal
# oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-03-20-160505   True        False         9h      Cluster version is 4.11.0-0.nightly-2022-03-20-160505
# oc get co insights
NAME       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
insights   4.11.0-0.nightly-2022-03-20-160505   True        False         False      9h


Note You need to log in before you can comment on or make changes to this bug.