Bug 1954615 - Undiagnosed panic detected in pod: pods/openshift-cloud-credential-operator_cloud-credential-operator
Summary: Undiagnosed panic detected in pod: pods/openshift-cloud-credential-operator_c...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Credential Operator
Version: 4.8
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.8.0
Assignee: Joel Diaz
QA Contact: wang lin
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-28 13:09 UTC by Petr Muller
Modified: 2021-07-27 23:04 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Incorrect order of operations when applying a resource fails. Consequence: Code errors out because returned data is invalid on error. Fix: Move check for error before processing returned data. Result: Proper error handling on failed application of a resource.
Clone Of:
Environment:
test: Symptom Detection.Undiagnosed panic detected in pod
Last Closed: 2021-07-27 23:04:32 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cloud-credential-operator pull 335 0 None open Bug 1954615: do not assume result is valid after error 2021-05-05 13:57:26 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:04:43 UTC

Description Petr Muller 2021-04-28 13:09:21 UTC
Undiagnosed panic detected in pod expand_less 	0s
pods/openshift-cloud-credential-operator_cloud-credential-operator-955b86d5f-jblj8_cloud-credential-operator_previous.log.gz:E0428 04:40:45.932592       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)

See https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-e2e-aws-serial/1387260819061870592

Comment 3 wang lin 2021-05-08 09:33:31 UTC
Verified on 


Set cpu and memory of container requests to an irrational value(such as 100000000m and 10000000Mi) to let deployment apply fail, need to patch many times to reproduce a failed apply.


Check cco logs, cco shows the error message ,rather than panic.
time="2021-05-08T09:01:52Z" level=error msg="error applying Deployment" controller=awspodidentity error="Operation cannot be fulfilled on deployments.apps \"pod-identity-webhook\": the object has been modified; please apply your changes to the latest version and try again"
time="2021-05-08T09:01:52Z" level=error msg="reconciliation failed, retrying in 10s" controller=awspodidentity

Comment 6 errata-xmlrpc 2021-07-27 23:04:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.