Bug 1950112
| Summary: | SNO: machine-config pool is degraded: error running chcon -R -t var_run_t /run/mco-machine-os-content/os-content-321709791 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Alexander Chuzhoy <sasha> | ||||
| Component: | RHCOS | Assignee: | Micah Abbott <miabbott> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Michael Nguyen <mnguyen> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 4.8 | CC: | dornelas, jerzhang, jligon, jnovy, keyoung, mheon, miabbott, mrussell, nstielau, pthomas, sscheink, tsweeney, vrothber | ||||
| Target Milestone: | --- | Flags: | jerzhang:
needinfo-
jerzhang: needinfo- |
||||
| Target Release: | 4.8.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-07-27 23:01:26 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Hi, The specific error happens during os image extraction, specifically if it fails to extract with `oc` and is attempting to fall back to using podman to extract an image. It seems that the podman cp likely failed (unfortunately we don't log that error). Could you provide the whole MCD logs, as well as what the PAO is trying to do in this case? Created attachment 1772248 [details]
mcd log
oc get performanceprofile performance -o yaml
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"performance.openshift.io/v2","kind":"PerformanceProfile","metadata":{"annotations":{},"name":"performance"},"spec":{"cpu":{"isolated":"2-31,34-63","reserved":"0-1,32-33"},"hugepages":{"defaultHugepagesSize":"1G","pages":[{"count":16,"node":0,"size":"1G"},{"count":128,"size":"2M"}]},"nodeSelector":{"node-role.kubernetes.io/master":""},"numa":{"topologyPolicy":"best-effort"},"realTimeKernel":{"enabled":true}}}
creationTimestamp: "2021-04-15T19:57:40Z"
finalizers:
- foreground-deletion
generation: 1
managedFields:
- apiVersion: performance.openshift.io/v2
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
.: {}
f:cpu:
.: {}
f:isolated: {}
f:reserved: {}
f:hugepages:
.: {}
f:defaultHugepagesSize: {}
f:pages: {}
f:nodeSelector:
.: {}
f:node-role.kubernetes.io/master: {}
f:numa:
.: {}
f:topologyPolicy: {}
f:realTimeKernel:
.: {}
f:enabled: {}
manager: kubectl-client-side-apply
operation: Update
time: "2021-04-15T19:57:40Z"
- apiVersion: performance.openshift.io/v2
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.: {}
v:"foreground-deletion": {}
f:status:
.: {}
f:conditions: {}
f:runtimeClass: {}
f:tuned: {}
manager: performance-operator
operation: Update
time: "2021-04-15T19:57:41Z"
name: performance
resourceVersion: "26221"
uid: ecb695f7-0162-4eba-97de-48f71b77e49e
spec:
cpu:
isolated: 2-31,34-63
reserved: 0-1,32-33
hugepages:
defaultHugepagesSize: 1G
pages:
- count: 16
node: 0
size: 1G
- count: 128
size: 2M
nodeSelector:
node-role.kubernetes.io/master: ""
numa:
topologyPolicy: best-effort
realTimeKernel:
enabled: true
status:
conditions:
- lastHeartbeatTime: "2021-04-15T19:59:58Z"
lastTransitionTime: "2021-04-15T19:59:58Z"
status: "False"
type: Available
- lastHeartbeatTime: "2021-04-15T19:59:58Z"
lastTransitionTime: "2021-04-15T19:59:58Z"
status: "False"
type: Upgradeable
- lastHeartbeatTime: "2021-04-15T19:59:58Z"
lastTransitionTime: "2021-04-15T19:59:58Z"
status: "False"
type: Progressing
- lastHeartbeatTime: "2021-04-15T19:59:58Z"
lastTransitionTime: "2021-04-15T19:59:58Z"
message: |
Machine config pool master Degraded Reason: 1 nodes are reporting degraded status on sync.
Machine config pool master Degraded Message: Node openshift-master-0.qe3.kni.lab.eng.bos.redhat.com is reporting: "error running chcon -R -t var_run_t /run/mco-machine-os-content/os-content-219272214: chcon: cannot access '/run/mco-machine-os-content/os-content-219272214': No such file or directory\n: exit status 1".
reason: MCPDegraded
status: "True"
type: Degraded
runtimeClass: performance-performance
tuned: openshift-cluster-node-tuning-operator/openshift-node-performance-performance
[kni@r640-u09 ~]$ oc get po -A|grep performa
openshift-performance-addon-operator performance-operator-66f595c789-qkxpc 1/1 Running 0 29m
[kni@r640-u09 ~]$ oc logs -n openshift-performance-addon-operator performance-operator-66f595c789-qkxpc
I0415 19:57:37.440910 1 main.go:72] Operator Version:
I0415 19:57:37.440968 1 main.go:73] Git Commit:
I0415 19:57:37.440973 1 main.go:74] Build Date: 2021-02-25T12:59:33+0000
I0415 19:57:37.440980 1 main.go:75] Go Version: go1.13.15
I0415 19:57:37.440984 1 main.go:76] Go OS/Arch: linux/amd64
I0415 19:57:38.491963 1 request.go:621] Throttling request took 1.036631243s, request: GET:https://[fd02::1]:443/apis/authorization.k8s.io/v1?timeout=32s
2021-04-15T19:57:40.346Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "0.0.0.0:8383"}
2021-04-15T19:57:40.347Z INFO controller-runtime.builder skip registering a mutating webhook, admission.Defaulter interface is not implemented {"GVK": "performance.openshift.io/v1, Kind=PerformanceProfile"}
2021-04-15T19:57:40.347Z INFO controller-runtime.builder skip registering a validating webhook, admission.Validator interface is not implemented {"GVK": "performance.openshift.io/v1, Kind=PerformanceProfile"}
2021-04-15T19:57:40.347Z INFO controller-runtime.webhook registering webhook {"path": "/convert"}
2021-04-15T19:57:40.347Z INFO controller-runtime.builder conversion webhook enabled {"object": {"metadata":{"creationTimestamp":null},"spec":{},"status":{}}}
I0415 19:57:40.347509 1 main.go:138] Starting the Cmd.
I0415 19:57:40.347599 1 leaderelection.go:242] attempting to acquire leader lease openshift-performance-addon-operator/performance-addon-operators...
2021-04-15T19:57:40.347Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2021-04-15T19:57:40.347Z INFO controller-runtime.webhook.webhooks starting webhook server
I0415 19:57:40.351748 1 leaderelection.go:252] successfully acquired lease openshift-performance-addon-operator/performance-addon-operators
2021-04-15T19:57:40.351Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"openshift-performance-addon-operator","name":"performance-addon-operators","uid":"5be435d7-09db-4119-933e-82ba6a261bdc","apiVersion":"v1","resourceVersion":"25240"}, "reason": "LeaderElection", "message": "performance-operator-66f595c789-qkxpc_33d46b6c-9a90-4d8d-b6d6-6e404145c3cd became leader"}
2021-04-15T19:57:40.351Z INFO controller Starting EventSource {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "source": "kind source: /, Kind="}
2021-04-15T19:57:40.356Z INFO controller-runtime.certwatcher Updated current TLS certificate
2021-04-15T19:57:40.356Z INFO controller-runtime.webhook serving webhook server {"host": "", "port": 4343}
2021-04-15T19:57:40.356Z INFO controller-runtime.certwatcher Starting certificate watcher
2021-04-15T19:57:40.552Z INFO controller Starting EventSource {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "source": "kind source: /, Kind="}
2021-04-15T19:57:40.753Z INFO controller Starting EventSource {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "source": "kind source: /, Kind="}
2021-04-15T19:57:40.953Z INFO controller Starting EventSource {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "source": "kind source: /, Kind="}
2021-04-15T19:57:41.154Z INFO controller Starting EventSource {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "source": "kind source: /, Kind="}
2021-04-15T19:57:41.355Z INFO controller Starting EventSource {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "source": "kind source: /, Kind="}
2021-04-15T19:57:41.555Z INFO controller Starting Controller {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile"}
2021-04-15T19:57:41.556Z INFO controller Starting workers {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "worker count": 1}
I0415 19:57:41.556097 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
2021-04-15T19:57:41.560Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 19:57:41.560662 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:57:41.561667 1 resources.go:69] Create machine-config "performance-performance"
I0415 19:57:41.574141 1 resources.go:219] Create tuned "openshift-node-performance-performance" under the namespace "openshift-cluster-node-tuning-operator"
I0415 19:57:41.581343 1 resources.go:148] Create kubelet-config "performance-performance"
I0415 19:57:41.586199 1 resources.go:307] Create runtime class "performance-performance"
I0415 19:57:41.589923 1 status.go:79] Updating the performance profile "performance" status
2021-04-15T19:57:41.589Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25271"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T19:57:41.593Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 19:57:41.594042 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:57:41.594684 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:57:41.600Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T19:57:41.600Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 19:57:42.263288 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:57:42.263936 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:57:42.269Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T19:57:42.269Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 19:57:42.269358 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:57:42.269970 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:57:42.275Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T19:57:42.275Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 19:57:44.196354 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:57:44.197023 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:57:44.204Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T19:57:44.204Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 19:57:44.204230 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:57:44.204872 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:57:44.211Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T19:57:44.211Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 19:57:46.766420 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:57:46.768589 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:57:46.777Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 19:57:46.778018 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:57:46.782488 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:57:46.783Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T19:57:46.791Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T19:57:46.792Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 19:57:51.727275 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:57:51.728428 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:57:51.734Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T19:57:51.734Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 19:57:51.734327 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:57:51.734943 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:57:51.740Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T19:57:51.740Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 19:57:51.890957 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:57:51.891862 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:57:51.897Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T19:57:51.897Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 19:57:51.897239 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:57:51.897829 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:57:51.903Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T19:57:51.903Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 19:58:02.139172 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:58:02.140390 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:58:02.148Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 19:58:02.148984 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:58:02.149971 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:58:02.151Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T19:58:02.165Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T19:58:02.165Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 19:58:22.628681 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:58:22.629741 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:58:22.640Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T19:58:22.641Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 19:58:22.642116 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:58:22.645434 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:58:22.652Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T19:58:22.652Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 19:59:03.599942 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:59:03.600770 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:59:03.622Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 19:59:03.622681 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:59:03.623377 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:59:03.623Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T19:59:03.633Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T19:59:03.635Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 19:59:58.736642 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:59:58.737907 1 resources.go:80] Update machine-config "performance-performance"
I0415 19:59:58.761013 1 status.go:79] Updating the performance profile "performance" status
2021-04-15T19:59:58.761Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"25280"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T19:59:58.773Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 19:59:58.774605 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 19:59:58.775951 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T19:59:58.808Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T19:59:58.808Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:00:25.527423 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:00:25.528515 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:00:25.535Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:00:25.535908 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:00:25.536835 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:00:25.540Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:00:25.551Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T20:00:25.552Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:03:09.491366 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:03:09.532100 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:03:09.546Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:03:09.547040 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:03:09.563277 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:03:09.563Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:03:09.638Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:03:09.638Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:08:37.093410 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:08:37.095728 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:08:37.107Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:08:37.107881 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:08:37.108837 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:08:37.107Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:08:37.146Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T20:08:37.146Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:19:32.492097 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:19:32.513707 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:19:32.556Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:19:32.560998 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:19:32.564559 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:19:32.565Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:19:32.604Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T20:19:32.604Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:20:32.496479 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:20:32.497975 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:20:32.509Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:20:32.509582 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:20:32.510210 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:20:32.510Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:20:32.519Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T20:20:32.519Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:20:33.200889 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:20:33.202045 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:20:33.209Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:20:33.209571 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:20:33.210367 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:20:33.210Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:20:33.219Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T20:20:33.219Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:20:35.146655 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:20:35.147862 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:20:35.156Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:20:35.157Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:20:35.157564 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:20:35.159585 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:20:35.186Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T20:20:35.186Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:20:37.717457 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:20:37.718886 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:20:37.730Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:20:37.730150 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
2021-04-15T20:20:37.730Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:20:37.731229 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:20:37.744Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T20:20:37.744Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:20:43.073863 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:20:43.134516 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:20:43.147Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:20:43.148334 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:20:43.156010 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:20:43.157Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:20:43.171Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:20:43.171Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:20:53.318190 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:20:53.319773 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:20:53.349Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:20:53.349Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:20:53.349848 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:20:53.350596 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:20:53.362Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:20:53.363Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:21:13.808600 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:21:13.809638 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:21:13.820Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:21:13.820561 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:21:13.821609 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:21:13.822Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:21:13.832Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T20:21:13.832Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:21:54.788107 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:21:54.789406 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:21:54.813Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:21:54.814029 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
2021-04-15T20:21:54.831Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:21:54.845971 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:21:54.856Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T20:21:54.856Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:23:16.717551 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:23:16.718797 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:23:16.730Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:23:16.730256 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
2021-04-15T20:23:16.733Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:23:16.743680 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:23:16.752Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
2021-04-15T20:23:16.752Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
I0415 20:26:00.567246 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:26:00.568126 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:26:00.574Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
I0415 20:26:00.574548 1 performanceprofile_controller.go:197] Reconciling PerformanceProfile
I0415 20:26:00.575148 1 resources.go:80] Update machine-config "performance-performance"
2021-04-15T20:26:00.575Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:26:00.583Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"PerformanceProfile","name":"performance","uid":"ecb695f7-0162-4eba-97de-48f71b77e49e","apiVersion":"performance.openshift.io/v2","resourceVersion":"26221"}, "reason": "Creation succeeded", "message": "Succeeded to create all components"}
2021-04-15T20:26:00.583Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "performance.openshift.io", "reconcilerKind": "PerformanceProfile", "controller": "performanceprofile", "name": "performance", "namespace": ""}
[kni@r640-u09 ~]$
Below is the manifest we create:
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
name: performance
spec:
cpu:
isolated: "2-31,34-63"
reserved: "0-1,32-33"
hugepages:
defaultHugepagesSize: 1G
pages:
- count: 16
node: 0
size: "1G"
- count: 128
size: "2M"
nodeSelector:
node-role.kubernetes.io/master: ""
numa:
topologyPolicy: "best-effort"
realTimeKernel:
enabled: true
Note that I didn't reproduce this on a virtual setup with:
4.8.0-0.nightly-2021-04-15-113950
Note that RealTime kernel is set to false:
[root@sealusa35 terraform]# oc get performanceprofile performance -o json|jq ".spec"
{
"cpu": {
"isolated": "8-31",
"reserved": "0-7"
},
"hugepages": {
"defaultHugepagesSize": "1G",
"pages": [
{
"count": 16,
"node": 0,
"size": "1G"
},
{
"count": 128,
"size": "2M"
}
]
},
"nodeSelector": {
"node-role.kubernetes.io/master": ""
},
"numa": {
"topologyPolicy": "best-effort"
},
"realTimeKernel": {
"enabled": false
}
}
So the actual error is error: unable to connect to image repository quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6aaf5ad4bb837965d0aa94871dcd03c2285ce478f6b8ef1480df435632639a95: Get "https://quay.io/v2/": dial tcp 52.4.104.248:443: connect: network is unreachable It seems that it has networking issues. We can update the error message to properly report the failure (and not the chcon error). Can you check networking for your setup? This is a disconnected setup and a local mirror is used for getting quay.io content: See below how I'm getting the mirrored quay content from the SNO: [kni@r640-u09 ~]$ ssh core@2620:52:0:1386::91 Red Hat Enterprise Linux CoreOS 48.84.202104151145-0 Part of OpenShift 4.8, RHCOS is a Kubernetes native operating system managed by the Machine Config Operator (`clusteroperator/machine-config`). WARNING: Direct SSH access to machines is not recommended; instead, make configuration changes via `machineconfig` objects: https://docs.openshift.com/container-platform/4.8/architecture/architecture-rhcos.html --- Last login: Fri Apr 16 02:14:37 2021 from 2620:52:0:1386::90 [systemd] Failed Units: 1 NetworkManager-wait-online.service [core@openshift-master-0 ~]$ sudo podman pull --authfile /tmp/combined-secret.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6aaf5ad4bb837965d0aa94871dcd03c2285ce478f6b8ef1480df435632639a95 Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6aaf5ad4bb837965d0aa94871dcd03c2285ce478f6b8ef1480df435632639a95... Getting image source signatures Copying blob 55d13a57ff05 skipped: already exists Copying blob cac8a7420844 skipped: already exists Copying blob 217371923d66 [--------------------------------------] 0.0b / 0.0b Copying config 7d1afe89ca done Writing manifest to image destination Storing signatures 7d1afe89ca0a2c2d9938653a9a2f8becbabc10fcee25c16afb16cb4139477a60 [core@openshift-master-0 ~]$ Quick question, is > [systemd] > Failed Units: 1 > NetworkManager-wait-online.service expected? What's the NetworkManager failure, if any? Looking into the context of the bug that added the fallback podman pull, https://bugzilla.redhat.com/show_bug.cgi?id=1862979#c10 It seems that this is expected in a disconnected environment. Reading the code again, I made a mistake in my previous assessment. It seems the command that had an issue was: nice -- ionice -c 3 podman cp 10e547d0bf31469e5779f5802d7678d35cd529d6c9b36a3c187c627ccc2a595b:/ /run/mco-machine-os-content/os-content-070705733 And we don't catch the error from that line, so I cannot tell if that line is failing (might not be since we would have logged a warning), and the following command complains that it can't find the directory. The chain of commands ran is: I0415 20:19:44.801906 36375 update.go:314] Falling back to using podman cp to fetch OS image content I0415 20:19:44.801951 36375 run.go:18] Running: nice -- ionice -c 3 podman pull -q --authfile /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6aaf5ad4bb837965d0aa94871dcd03c2285ce478f6b8ef1480df435632639a95 7d1afe89ca0a2c2d9938653a9a2f8becbabc10fcee25c16afb16cb4139477a60 I0415 20:19:45.184715 36375 rpm-ostree.go:258] Running captured: podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-b2164da0-5729-45d2-ad09-9a3ba6d3ccb3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6aaf5ad4bb837965d0aa94871dcd03c2285ce478f6b8ef1480df435632639a95 I0415 20:19:45.565470 36375 run.go:18] Running: nice -- ionice -c 3 podman cp 10e547d0bf31469e5779f5802d7678d35cd529d6c9b36a3c187c627ccc2a595b:/ /run/mco-machine-os-content/os-content-070705733 I0415 20:19:59.938550 36375 rpm-ostree.go:258] Running captured: chcon -R -t var_run_t /run/mco-machine-os-content/os-content-070705733 I0415 20:19:59.940108 36375 update.go:278] Error changing selinux context on path /run/mco-machine-os-content/os-content-070705733 error running chcon -R -t var_run_t /run/mco-machine-os-content/os-content-070705733: chcon: cannot access '/run/mco-machine-os-content/os-content-070705733': No such file or directory : exit status 1 If you can try running that chain of commands (Ths IDs there are re-generated every time), or give me access to an environment, we can try to see exactly where went wrong. In the meantime I attached a small fix to properly log the error, if it does occur. Discussed with Alexander offline, I was able to get into his environment. Trying to replicate what the MCO would do, I noticed a bunch of weird behaviour. Fundamental to this though I noticed the following difference between 4.7 and 4.8 clusters. In 4.7, when I do the following: sh-4.4# podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-b2164da0-5729-45d2-ad09-9a3ba6d3ccb3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6aaf5ad4bb837965d0aa94871dcd03c2285ce478f6b8ef1480df435632639a95 b17c78172a2e1f8e7f0e8c5587c805f46bdef07677c5a9e28352bba9c8c51086 sh-4.4# podman cp b17c78172a2e1f8e7f0e8c5587c805f46bdef07677c5a9e28352bba9c8c51086:/ /run/mco-machine-os-content/os-content-070705733 sh-4.4# ls /run/mco-machine-os-content/ os-content-070705733 sh-4.4# ls /run/mco-machine-os-content/os-content-070705733/ bin boot dev etc extensions home lib lib64 lost+found media mnt opt pkglist.txt proc root run sbin srv sys tmp usr var You see how it creates it under the full name /run/mco-machine-os-content/os-content-070705733, and if I do the same in 4.8 sh-4.4# podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-b2164da0-5729-45d2-ad09-9a3ba6d3ccb3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6aaf5ad4bb837965d0aa94871dcd03c2285ce478f6b8ef1480df435632639a95 06f9bd5c63e2f161994e58f5f4aab542e77f78dfb3b667e1bdb7266b1654a99f sh-4.4# mkdir /run/mco-machine-os-content sh-4.4# podman cp 06f9bd5c63e2f161994e58f5f4aab542e77f78dfb3b667e1bdb7266b1654a99f:/ /run/mco-machine-os-content/os-content-12345678 sh-4.4# ls /run/mco-machine-os-content/ bin/ dev/ extensions/ lib/ lost+found/ mnt/ pkglist.txt root/ sbin/ sys/ usr/ boot/ etc/ home/ lib64/ media/ opt/ proc/ run/ srv/ tmp/ var/ sh-4.4# ls /run/mco-machine-os-content/os-content-12345678/ ls: cannot access '/run/mco-machine-os-content/os-content-12345678/': No such file or directory It extracts under the /run/mco-machine-os-content directly and not in the subdir! The chcon then fails because /run/mco-machine-os-content/os-content-12345678 doesn't exist. If I specify the full dir as follows: podman cp 06f9bd5c63e2f161994e58f5f4aab542e77f78dfb3b667e1bdb7266b1654a99f:/ /run/mco-machine-os-content/os-content-12345678/ Error: "/run/mco-machine-os-content/os-content-12345678/" could not be found on the host: No such file or directory So I can't just do that either. Maybe this is a podman behaviour change? Will double check with podman team. I do see https://github.com/containers/podman/commit/31b11b5cd62093d86891d32e0912661814d6b78b which is 3.1, current 4.8 uses podman 3.0.1. Maybe that's the fix we need. Moving back to assigned given the PR won't fix the error Version: 4.8.0-0.nightly-2021-04-19-175100
Connected ipv4 deployment.
I actually see that things worked well:
[kni@r640-u09 ~]$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.8.0-0.nightly-2021-04-19-175100 True False 37h Cluster version is 4.8.0-0.nightly-2021-04-19-175100
[kni@r640-u09 ~]$ oc get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
openshift-master-0.qe3.kni.lab.eng.bos.redhat.com Ready master,worker 37h v1.21.0-rc.0+98d91ef 10.19.134.150 <none> Red Hat Enterprise Linux CoreOS 48.84.202104191517-0 (Ootpa) 4.18.0-293.rt7.59.el8.x86_64 cri-o://1.21.0-74.rhaos4.8.gitbc1ef35.el8
[kni@r640-u09 ~]$ oc get performanceprofile performance -o yaml
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"performance.openshift.io/v2","kind":"PerformanceProfile","metadata":{"annotations":{},"name":"performance"},"spec":{"cpu":{"isolated":"2-31,34-63","reserved":"0-1,32-33"},"hugepages":{"defaultHugepagesSize":"1G","pages":[{"count":16,"node":0,"size":"1G"},{"count":128,"size":"2M"}]},"nodeSelector":{"node-role.kubernetes.io/master":""},"numa":{"topologyPolicy":"best-effort"},"realTimeKernel":{"enabled":true}}}
creationTimestamp: "2021-04-20T02:49:28Z"
finalizers:
- foreground-deletion
generation: 1
managedFields:
- apiVersion: performance.openshift.io/v2
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
.: {}
f:cpu:
.: {}
f:isolated: {}
f:reserved: {}
f:hugepages:
.: {}
f:defaultHugepagesSize: {}
f:pages: {}
f:nodeSelector:
.: {}
f:node-role.kubernetes.io/master: {}
f:numa:
.: {}
f:topologyPolicy: {}
f:realTimeKernel:
.: {}
f:enabled: {}
manager: kubectl-client-side-apply
operation: Update
time: "2021-04-20T02:49:28Z"
- apiVersion: performance.openshift.io/v2
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.: {}
v:"foreground-deletion": {}
f:status:
.: {}
f:conditions: {}
f:runtimeClass: {}
f:tuned: {}
manager: performance-operator
operation: Update
time: "2021-04-20T02:49:28Z"
name: performance
resourceVersion: "20059"
uid: 29a21ba9-72a1-462a-ba98-7fdb5ed8d896
spec:
cpu:
isolated: 2-31,34-63
reserved: 0-1,32-33
hugepages:
defaultHugepagesSize: 1G
pages:
- count: 16
node: 0
size: 1G
- count: 128
size: 2M
nodeSelector:
node-role.kubernetes.io/master: ""
numa:
topologyPolicy: best-effort
realTimeKernel:
enabled: true
status:
conditions:
- lastHeartbeatTime: "2021-04-20T02:49:28Z"
lastTransitionTime: "2021-04-20T02:49:28Z"
status: "True"
type: Available
- lastHeartbeatTime: "2021-04-20T02:49:28Z"
lastTransitionTime: "2021-04-20T02:49:28Z"
status: "True"
type: Upgradeable
- lastHeartbeatTime: "2021-04-20T02:49:28Z"
lastTransitionTime: "2021-04-20T02:49:28Z"
status: "False"
type: Progressing
- lastHeartbeatTime: "2021-04-20T02:49:28Z"
lastTransitionTime: "2021-04-20T02:49:28Z"
status: "False"
type: Degraded
runtimeClass: performance-performance
tuned: openshift-cluster-node-tuning-operator/openshift-node-performance-performance
[kni@r640-u09 ~]$
[kni@r640-u09 ~]$ oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-daec39fc795356eca43035b492f43921 True False False 1 1 1 0 37h
worker rendered-worker-6a7b409e479a3da98c3273b78341e5ce True False False 0 0 0 0 37h
[kni@r640-u09 ~]$ oc get pod -A|grep -v Run|grep -v Comple
NAMESPACE NAME READY STATUS RESTARTS AGE
[kni@r640-u09 ~]$
So this bug would only manifest on disconnected environments, because `oc image extract` (the main call we use to extract the os image) does not work with mirrors, and falls back to using podman. See https://github.com/openshift/machine-config-operator/pull/1979/ for some details `podman-3.0.1-6.el8` landed in RHCOS 48.84.202104221621-0 That build (and newer) is available in the OCP 4.8 nightly release payloads. Please let us know if this fixes the original issue. Verified QA: Version: oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-04-23-034722 True False 69m Cluster version is 4.8.0-0.nightly-2021-04-23-034722 [kni@r640-u09 ~]$ oc get performanceprofile performance NAME AGE performance 57m [kni@r640-u09 ~]$ oc get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME openshift-master-0.qe3.kni.lab.eng.bos.redhat.com Ready master,worker 90m v1.21.0-rc.0+0ac2bde 2620:52:0:1386::91 <none> Red Hat Enterprise Linux CoreOS 48.84.202104222319-0 (Ootpa) 4.18.0-293.rt7.59.el8.x86_64 cri-o://1.21.0-76.rhaos4.8.git05bced5.el8 *** Bug 1955194 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |
Version: 4.8.0-0.nightly-2021-04-15-113950: Steps to reproduce: 1. Deploy SNO with ipv6 2. Attempt to apply performance addon operator Result: The mcp switches to updating=True and doesn't return to Updated. [kni@r640-u09 ~]$ oc get mcp NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-7419eb34fa87de7d6a5dd485f58651c6 False True True 1 0 0 1 87m worker rendered-worker-809bda03f02d5e835c45c7bb5eb35885 True False False 0 0 0 0 87m oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.8.0-0.nightly-2021-04-15-113950 True False False 60m baremetal 4.8.0-0.nightly-2021-04-15-113950 True False False 83m cloud-credential 4.8.0-0.nightly-2021-04-15-113950 True False False 87m cluster-autoscaler 4.8.0-0.nightly-2021-04-15-113950 True False False 82m config-operator 4.8.0-0.nightly-2021-04-15-113950 True False False 88m console 4.8.0-0.nightly-2021-04-15-113950 True False False 70m csi-snapshot-controller 4.8.0-0.nightly-2021-04-15-113950 True False False 82m dns 4.8.0-0.nightly-2021-04-15-113950 True False False 82m etcd 4.8.0-0.nightly-2021-04-15-113950 True False False 87m image-registry 4.8.0-0.nightly-2021-04-15-113950 True False False 67m ingress 4.8.0-0.nightly-2021-04-15-113950 True False False 77m insights 4.8.0-0.nightly-2021-04-15-113950 True False False 83m kube-apiserver 4.8.0-0.nightly-2021-04-15-113950 True False False 76m kube-controller-manager 4.8.0-0.nightly-2021-04-15-113950 True False False 81m kube-scheduler 4.8.0-0.nightly-2021-04-15-113950 True False False 76m kube-storage-version-migrator 4.8.0-0.nightly-2021-04-15-113950 True False False 88m machine-api 4.8.0-0.nightly-2021-04-15-113950 True False False 82m machine-approver 4.8.0-0.nightly-2021-04-15-113950 True False False 87m machine-config 4.8.0-0.nightly-2021-04-15-113950 False False True 13m marketplace 4.8.0-0.nightly-2021-04-15-113950 True False False 86m monitoring 4.8.0-0.nightly-2021-04-15-113950 True False False 62m network 4.8.0-0.nightly-2021-04-15-113950 True False False 88m node-tuning 4.8.0-0.nightly-2021-04-15-113950 True False False 23m openshift-apiserver 4.8.0-0.nightly-2021-04-15-113950 True False False 66m openshift-controller-manager 4.8.0-0.nightly-2021-04-15-113950 True False False 66m openshift-samples 4.8.0-0.nightly-2021-04-15-113950 True False False 77m operator-lifecycle-manager 4.8.0-0.nightly-2021-04-15-113950 True False False 87m operator-lifecycle-manager-catalog 4.8.0-0.nightly-2021-04-15-113950 True False False 87m operator-lifecycle-manager-packageserver 4.8.0-0.nightly-2021-04-15-113950 True False False 60m service-ca 4.8.0-0.nightly-2021-04-15-113950 True False False 88m storage 4.8.0-0.nightly-2021-04-15-113950 True False False 83m oc describe co machine-config Name: machine-config Namespace: Labels: <none> Annotations: exclude.release.openshift.io/internal-openshift-hosted: true include.release.openshift.io/self-managed-high-availability: true include.release.openshift.io/single-node-developer: true API Version: config.openshift.io/v1 Kind: ClusterOperator Metadata: Creation Timestamp: 2021-04-15T17:05:44Z Generation: 1 Managed Fields: API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:exclude.release.openshift.io/internal-openshift-hosted: f:include.release.openshift.io/self-managed-high-availability: f:include.release.openshift.io/single-node-developer: f:spec: f:status: .: f:extension: f:worker: f:relatedObjects: f:versions: Manager: cluster-version-operator Operation: Update Time: 2021-04-15T17:05:45Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: f:conditions: f:extension: .: f:master: f:worker: f:relatedObjects: f:versions: Manager: machine-config-operator Operation: Update Time: 2021-04-15T18:50:57Z Resource Version: 42100 UID: 81dc565d-d722-4822-a80d-8e1866f0607b Spec: Status: Conditions: Last Transition Time: 2021-04-15T17:28:11Z Message: Cluster version is 4.8.0-0.nightly-2021-04-15-113950 Status: False Type: Progressing Last Transition Time: 2021-04-15T18:27:30Z Message: One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading Reason: DegradedPool Status: False Type: Upgradeable Last Transition Time: 2021-04-15T18:37:28Z Message: Failed to resync 4.8.0-0.nightly-2021-04-15-113950 because: timed out waiting for the condition during syncRequiredMachineConfigPools: error pool master is not ready, retrying. Status: (pool degraded: true total: 1, ready 0, updated: 0, unavailable: 1) Reason: RequiredPoolsFailed Status: True Type: Degraded Last Transition Time: 2021-04-15T18:37:28Z Message: Cluster not available for 4.8.0-0.nightly-2021-04-15-113950 Status: False Type: Available Extension: Master: pool is degraded because nodes fail with "1 nodes are reporting degraded status on sync": "Node openshift-master-0.qe3.kni.lab.eng.bos.redhat.com is reporting: \"error running chcon -R -t var_run_t /run/mco-machine-os-content/os-content-321709791: chcon: cannot access '/run/mco-machine-os-content/os-content-321709791': No such file or directory\\n: exit status 1\"" Worker: all 0 nodes are at latest configuration rendered-worker-809bda03f02d5e835c45c7bb5eb35885 Related Objects: Group: Name: openshift-machine-config-operator Resource: namespaces Group: machineconfiguration.openshift.io Name: Resource: machineconfigpools Group: machineconfiguration.openshift.io Name: Resource: controllerconfigs Group: machineconfiguration.openshift.io Name: Resource: kubeletconfigs Group: machineconfiguration.openshift.io Name: Resource: containerruntimeconfigs Group: machineconfiguration.openshift.io Name: Resource: machineconfigs Group: Name: Resource: nodes Versions: Name: operator Version: 4.8.0-0.nightly-2021-04-15-113950 Events: <none>