Bug 1977660
| Summary: | the pod events show error codes when crio recreate the missing symlinks | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | MinLi <minmli> |
| Component: | Node | Assignee: | Skyler Clark <skclark> |
| Node sub component: | CRI-O | QA Contact: | MinLi <minmli> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | low | ||
| Priority: | low | CC: | aos-bugs, dwalsh, mburke, nagrawal, pehunt, stevsmit, tsweeney |
| Version: | 4.8 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.12.0 | ||
| Hardware: | x86_64 | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | 4.11 | Doc Type: | Bug Fix |
| Doc Text: |
* Previously, a symlinks error message was printed out as raw data instead of formatted as an error, making it difficult to understand. This fix formats the error message properly, so that it is easily understood. (link:https://bugzilla.redhat.com/show_bug.cgi?id=1977660[*BZ#1977660*])
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-01-17 19:46:24 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
that is an ugly error. I didn't get a chance to take a look, Skyler, can you? oops, it's not that ugly :D fixed in this PR! https://github.com/containers/storage/pull/1048 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> Successfully assigned default/pod-httpd to ci-ln-l21q1yt-72292-tz6bd-worker-b-8wg9r Normal AddedInterface 32s multus Add eth0 [10.128.2.18/23] from openshift-sdn Warning FailedCreatePodSandBox 32s kubelet, ci-ln-l21q1yt-72292-tz6bd-worker-b-8wg9r Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_pod-httpd_default_82ecae98-69aa-45f3-984e-3c792c39d927_0 in pod sandbox k8s_pod-httpd_default_82ecae98-69aa-45f3-984e-3c792c39d927_0(7ac5b9a957d0ff1443e8fd3ff40d73c6ea084df06c44f5266bae2db56be39d5c): Recreating the missing symlinks: 1 error occurred: * reading name of symlink for "0000000000000000000000000000000000000000000000000000000000000000": open /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000/link: no such file or directory reproduced!
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.10.0-0.nightly-2022-01-25-023600 True False 66m Cluster version is 4.10.0-0.nightly-2022-01-25-023600
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> Successfully assigned default/hello-pod-1 to ip-10-0-203-96.us-east-2.compute.internal
Normal AddedInterface 26s multus Add eth0 [10.131.0.24/23] from openshift-sdn
Warning FailedCreatePodSandBox 25s kubelet, ip-10-0-203-96.us-east-2.compute.internal Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_hello-pod-1_default_db81beb4-f9ca-4d3d-8f37-0aa31cadfe55_0 in pod sandbox k8s_hello-pod-1_default_db81beb4-f9ca-4d3d-8f37-0aa31cadfe55_0(ba89443747f620ed2ddd82ad4913c7859934048c037a0ee2ee931ff108303117): Recreating the missing symlinks: 1 error occurred:
* reading name of symlink for &{"0000000000000000000000000000000000000000000000000000000000000000" '\x06' '�' {'�' '�' %!q(*time.Location=&{Local [{UTC 0 false}] [{-576460752303423488 0 false false}] UTC0 9223372036854775807 9223372036854775807 0xc00013f320})} {'𐌄' '�' '\x02' '䇀' '\x00' '\x00' '\x00' '\x00' '\x06' 'က' '\x00' {'�' '�'} {'�' '�'} {'�' '�'} ['\x00' '\x00' '\x00']}}: open /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000/link: no such file or directory
Normal AddedInterface 24s multus Add eth0 [10.131.0.25/23] from openshift-sdn
Normal Pulling 24s kubelet, ip-10-0-203-96.us-east-2.compute.internal Pulling image "httpd:latest"
Normal Pulled 20s kubelet, ip-10-0-203-96.us-east-2.compute.internal Successfully pulled image "httpd:latest" in 3.989657887s
Normal Created 20s kubelet, ip-10-0-203-96.us-east-2.compute.internal Created container hello-pod
Normal Started 20s kubelet, ip-10-0-203-96.us-east-2.compute.internal Started container hello-pod
verified!
% oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.12.0-0.nightly-2022-10-05-053337 True False 5h33m Cluster version is 4.12.0-0.nightly-2022-10-05-053337
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 99s default-scheduler Successfully assigned default/test to minmli-1012-fwqhs-worker-b-ss4ht.c.openshift-qe.internal
Normal AddedInterface 98s multus Add eth0 [10.128.2.15/23] from openshift-sdn
Warning FailedCreatePodSandBox 98s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_test_default_75b869fd-341d-4243-86ac-62da85f54a86_0 in pod sandbox k8s_test_default_75b869fd-341d-4243-86ac-62da85f54a86_0(e67ce787a322ecda070bc0f466894f5724dc9b0faf903d29a14b3d2267226bbf): recreating the missing symlinks: 1 error occurred:
* reading name of symlink for "0000000000000000000000000000000000000000000000000000000000000000": open /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000/link: no such file or directory
Normal AddedInterface 96s multus Add eth0 [10.128.2.16/23] from openshift-sdn
Normal Pulling 96s kubelet Pulling image "docker.io/library/busybox"
Normal Pulled 94s kubelet Successfully pulled image "docker.io/library/busybox" in 1.906702793s
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399 |
Description of problem: the pod events show error codes when crio recreate the missing symlinks: Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-06-23-232238 How reproducible: always Steps to Reproduce: 1.oc debug node/XXX 2.sh-4.4# chroot /host sh-4.4# grep "pause_image" /etc/crio/crio.conf.d/00-default pause_image = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a166dcfd8f85ca9aa9f5c9307a68b4293a882546dad0c4ba631a2a7b50fd19b" sh-4.4# podman inspect quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a166dcfd8f85ca9aa9f5c9307a68b4293a882546dad0c4ba631a2a7b50fd19b --format={{.GrahDriver.Data.UpperDir}} /var/lib/containers/storage/overlay/79ebe873244e43c52545eb880ee0f0c73be5976df75536a7cc076ad1d4855631/diff sh-4.4# cat /var/lib/containers/storage/overlay/79ebe873244e43c52545eb880ee0f0c73be5976df75536a7cc076ad1d4855631/lower l/JSY2TQY3UOJRMRO5PNND63OR6H:l/6JJW4FB2TRS4J6JVEC7CXWHH4G:l/467X23MAQ5657ZKYLLDL4K3J62:l/3SPROF4W57TUWO4GPHN6GW54OJ sh-4.4# mkdir /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000 sh-4.4# chmod 700 /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000 sh-4.4# ls -l /var/lib/containers/storage/overlay/l/3SPROF4W57TUWO4GPHN6GW54OJ lrwxrwxrwx. 1 root root 72 Jun 25 08:03 /var/lib/containers/storage/overlay/l/3SPROF4W57TUWO4GPHN6GW54OJ -> ../98469092e6042f8c9cc81dcb1a710957fb5ef27817c9b178f7b71c4f242cb2ed/diff sh-4.4# rm /var/lib/containers/storage/overlay/l/3SPROF4W57TUWO4GPHN6GW54OJ // delete one symlink sh-4.4# cat /var/lib/containers/storage/overlay/98469092e6042f8c9cc81dcb1a710957fb5ef27817c9b178f7b71c4f242cb2ed/link 3SPROF4W57TUWO4GPHN6GW54OJ sh-4.4# ls /var/lib/containers/storage/overlay/l/3SPROF4W57TUWO4GPHN6GW54OJ ls: cannot access '/var/lib/containers/storage/overlay/l/3SPROF4W57TUWO4GPHN6GW54OJ': No such file or directory sh-4.4# ls -l /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000 total 0 3.create a pod $ oc create -f hello-pod.yaml 4.check the pod description $ oc describe pod hello-pod-1 ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> Successfully assigned default/hello-pod-1 to ci-ln-i38pngk-f76d1-rf7v2-worker-a-44vpq Normal AddedInterface 6m15s multus Add eth0 [10.131.0.33/23] from openshift-sdn Warning FailedCreatePodSandBox 6m14s kubelet, ci-ln-i38pngk-f76d1-rf7v2-worker-a-44vpq Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_hello-pod-1_default_8536b590-d7d5-40a6-8a93-80c9b4f43678_0 in pod sandbox k8s_hello-pod-1_default_8536b590-d7d5-40a6-8a93-80c9b4f43678_0(ca9fa29e6f778c4cdb2e533790aa392b6e4ca181e5a4703063bec827afb11e44): error recreating the missing symlinks: 1 error occurred: * error reading name of symlink for &{"0000000000000000000000000000000000000000000000000000000000000000" '\x06' '�' {'�' '�' %!q(*time.Location=&{Local [{UTC 0 false}] [{-576460752303423488 0 false false}] UTC0 9223372036854775807 9223372036854775807 0xc0002b26e0})} {'ࠄ' '�' '\x02' '䇀' '\x00' '\x00' '\x00' '\x00' '\x06' 'က' '\x00' {'�' '�'} {'�' '�'} {'�' '�'} ['\x00' '\x00' '\x00']}}: open /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000/link: no such file or directory Normal AddedInterface 6m12s multus Add eth0 [10.131.0.34/23] from openshift-sdn Normal Pulling 6m12s kubelet, ci-ln-i38pngk-f76d1-rf7v2-worker-a-44vpq Pulling image "docker.io/ocpqe/hello-pod:latest" Normal Pulled 6m1s kubelet, ci-ln-i38pngk-f76d1-rf7v2-worker-a-44vpq Successfully pulled image "docker.io/ocpqe/hello-pod:latest" in 10.412658807s Normal Created 6m1s kubelet, ci-ln-i38pngk-f76d1-rf7v2-worker-a-44vpq Created container hello-pod Normal Started 6m1s kubelet, ci-ln-i38pngk-f76d1-rf7v2-worker-a-44vpq Started container hello-pod Actual results: 4.crio recreate the missing symlinks,and the pod become running eventually, but the events show error code. Expected results: 4.the events should not show error code, it's not friendly to users. Additional info: