Bug 1939726
| Summary: | clusteroperator/network should not change condition/Degraded during normal serial test execution | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Clayton Coleman <ccoleman> |
| Component: | Networking | Assignee: | Aniket Bhat <anbhat> |
| Networking sub component: | openshift-sdn | QA Contact: | Ying Wang <yingwang> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | aconstan, anbhat, rgudimet, ricarril, wking |
| Version: | 4.8 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.8.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: |
clusteroperator/network should not change condition/Degraded
|
|
| Last Closed: | 2021-07-27 22:53:48 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Clayton Coleman
2021-03-16 21:56:50 UTC
*** Bug 1940992 has been marked as a duplicate of this bug. *** I'm able to repro this by doing:
1. Run bash command like:
for i in {1..100000}; do oc -n openshift-network-diagnostics patch ds network-check-target -p {\"metadata\":{\"annotations\":{\"foo\":\"${i}\"}}}; done
2. Edit network-check-target ds and change the spec (e.g. containerPort)
Then I see this on the CNO logs:
I0412 13:33:46.043376 2624431 log.go:181] Set operator conditions:
- lastTransitionTime: "2021-04-12T10:06:52Z"
status: "False"
type: ManagementStateDegraded
- lastTransitionTime: "2021-04-12T11:32:53Z"
message: 'Error while updating operator configuration: could not apply (apps/v1,
Kind=DaemonSet) openshift-network-diagnostics/network-check-target: could not
update object (apps/v1, Kind=DaemonSet) openshift-network-diagnostics/network-check-target:
Operation cannot be fulfilled on daemonsets.apps "network-check-target": the object
has been modified; please apply your changes to the latest version and try again'
reason: ApplyOperatorConfig
status: "True"
type: Degraded
This is also blocking compact upgrade jobs: https://prow.ci.openshift.org/job-history/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-aws-compact-upgrade Checking CI: $ w3m -dump -cols 200 'https://search.ci.openshift.org/?maxAge=24h&type=junit&search=clusteroperator/network+should+not+change+condition/Degraded' | grep 'failures match' | sort periodic-ci-openshift-release-master-ci-4.8-upgrade-from-stable-4.7-e2e-ovirt-upgrade (all) - 2 runs, 100% failed, 50% of failures match = 50% impact periodic-ci-openshift-release-master-ci-4.9-e2e-gcp-upgrade (all) - 9 runs, 100% failed, 11% of failures match = 11% impact periodic-ci-openshift-release-master-nightly-4.8-e2e-ovirt (all) - 7 runs, 14% failed, 100% of failures match = 14% impact pull-ci-openshift-multus-cni-master-e2e-aws (all) - 12 runs, 100% failed, 8% of failures match = 8% impact Not too many jobs anymore, which is good. 1. deployed openshift cluster using image registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-05-06-190249.
2. run commands as below
for i in {1..100000}; do oc -n openshift-network-diagnostics patch ds network-check-target -p {\"metadata\":{\"annotations\":{\"foo\":\"${i}\"}}}; done
3. Edited network-check-target ds and change the spec containerPort from 8080 to 8081
4. checked CNO logs and didn't find logs like below
I0412 13:33:46.043376 2624431 log.go:181] Set operator conditions:
- lastTransitionTime: "2021-04-12T10:06:52Z"
status: "False"
type: ManagementStateDegraded
- lastTransitionTime: "2021-04-12T11:32:53Z"
message: 'Error while updating operator configuration: could not apply (apps/v1,
Kind=DaemonSet) openshift-network-diagnostics/network-check-target: could not
update object (apps/v1, Kind=DaemonSet) openshift-network-diagnostics/network-check-target:
Operation cannot be fulfilled on daemonsets.apps "network-check-target": the object
has been modified; please apply your changes to the latest version and try again'
reason: ApplyOperatorConfig
status: "True"
type: Degraded
Instead, it showed logs as blow:
I0511 07:23:42.025750 1 log.go:184] Reconciling update to openshift-network-diagnostics/network-check-target
I0511 07:23:42.049289 1 log.go:184] Set operator conditions:
- lastTransitionTime: "2021-05-11T06:41:55Z"
status: "False"
type: ManagementStateDegraded
- lastTransitionTime: "2021-05-11T06:45:01Z"
status: "False"
type: Degraded
- lastTransitionTime: "2021-05-11T06:41:56Z"
status: "True"
type: Upgradeable
- lastTransitionTime: "2021-05-11T07:23:38Z"
message: DaemonSet "openshift-network-diagnostics/network-check-target" update is
rolling out (4 out of 5 updated)
reason: Deploying
status: "True"
type: Progressing
- lastTransitionTime: "2021-05-11T06:42:33Z"
status: "True"
type: Available
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |