Description of problem: the pod events show error codes when crio recreate the missing symlinks: Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-06-23-232238 How reproducible: always Steps to Reproduce: 1.oc debug node/XXX 2.sh-4.4# chroot /host sh-4.4# grep "pause_image" /etc/crio/crio.conf.d/00-default pause_image = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a166dcfd8f85ca9aa9f5c9307a68b4293a882546dad0c4ba631a2a7b50fd19b" sh-4.4# podman inspect quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a166dcfd8f85ca9aa9f5c9307a68b4293a882546dad0c4ba631a2a7b50fd19b --format={{.GrahDriver.Data.UpperDir}} /var/lib/containers/storage/overlay/79ebe873244e43c52545eb880ee0f0c73be5976df75536a7cc076ad1d4855631/diff sh-4.4# cat /var/lib/containers/storage/overlay/79ebe873244e43c52545eb880ee0f0c73be5976df75536a7cc076ad1d4855631/lower l/JSY2TQY3UOJRMRO5PNND63OR6H:l/6JJW4FB2TRS4J6JVEC7CXWHH4G:l/467X23MAQ5657ZKYLLDL4K3J62:l/3SPROF4W57TUWO4GPHN6GW54OJ sh-4.4# mkdir /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000 sh-4.4# chmod 700 /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000 sh-4.4# ls -l /var/lib/containers/storage/overlay/l/3SPROF4W57TUWO4GPHN6GW54OJ lrwxrwxrwx. 1 root root 72 Jun 25 08:03 /var/lib/containers/storage/overlay/l/3SPROF4W57TUWO4GPHN6GW54OJ -> ../98469092e6042f8c9cc81dcb1a710957fb5ef27817c9b178f7b71c4f242cb2ed/diff sh-4.4# rm /var/lib/containers/storage/overlay/l/3SPROF4W57TUWO4GPHN6GW54OJ // delete one symlink sh-4.4# cat /var/lib/containers/storage/overlay/98469092e6042f8c9cc81dcb1a710957fb5ef27817c9b178f7b71c4f242cb2ed/link 3SPROF4W57TUWO4GPHN6GW54OJ sh-4.4# ls /var/lib/containers/storage/overlay/l/3SPROF4W57TUWO4GPHN6GW54OJ ls: cannot access '/var/lib/containers/storage/overlay/l/3SPROF4W57TUWO4GPHN6GW54OJ': No such file or directory sh-4.4# ls -l /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000 total 0 3.create a pod $ oc create -f hello-pod.yaml 4.check the pod description $ oc describe pod hello-pod-1 ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> Successfully assigned default/hello-pod-1 to ci-ln-i38pngk-f76d1-rf7v2-worker-a-44vpq Normal AddedInterface 6m15s multus Add eth0 [10.131.0.33/23] from openshift-sdn Warning FailedCreatePodSandBox 6m14s kubelet, ci-ln-i38pngk-f76d1-rf7v2-worker-a-44vpq Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_hello-pod-1_default_8536b590-d7d5-40a6-8a93-80c9b4f43678_0 in pod sandbox k8s_hello-pod-1_default_8536b590-d7d5-40a6-8a93-80c9b4f43678_0(ca9fa29e6f778c4cdb2e533790aa392b6e4ca181e5a4703063bec827afb11e44): error recreating the missing symlinks: 1 error occurred: * error reading name of symlink for &{"0000000000000000000000000000000000000000000000000000000000000000" '\x06' '�' {'�' '�' %!q(*time.Location=&{Local [{UTC 0 false}] [{-576460752303423488 0 false false}] UTC0 9223372036854775807 9223372036854775807 0xc0002b26e0})} {'ࠄ' '�' '\x02' '䇀' '\x00' '\x00' '\x00' '\x00' '\x06' 'က' '\x00' {'�' '�'} {'�' '�'} {'�' '�'} ['\x00' '\x00' '\x00']}}: open /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000/link: no such file or directory Normal AddedInterface 6m12s multus Add eth0 [10.131.0.34/23] from openshift-sdn Normal Pulling 6m12s kubelet, ci-ln-i38pngk-f76d1-rf7v2-worker-a-44vpq Pulling image "docker.io/ocpqe/hello-pod:latest" Normal Pulled 6m1s kubelet, ci-ln-i38pngk-f76d1-rf7v2-worker-a-44vpq Successfully pulled image "docker.io/ocpqe/hello-pod:latest" in 10.412658807s Normal Created 6m1s kubelet, ci-ln-i38pngk-f76d1-rf7v2-worker-a-44vpq Created container hello-pod Normal Started 6m1s kubelet, ci-ln-i38pngk-f76d1-rf7v2-worker-a-44vpq Started container hello-pod Actual results: 4.crio recreate the missing symlinks,and the pod become running eventually, but the events show error code. Expected results: 4.the events should not show error code, it's not friendly to users. Additional info:
that is an ugly error. I didn't get a chance to take a look, Skyler, can you?
oops, it's not that ugly :D
fixed in this PR! https://github.com/containers/storage/pull/1048 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> Successfully assigned default/pod-httpd to ci-ln-l21q1yt-72292-tz6bd-worker-b-8wg9r Normal AddedInterface 32s multus Add eth0 [10.128.2.18/23] from openshift-sdn Warning FailedCreatePodSandBox 32s kubelet, ci-ln-l21q1yt-72292-tz6bd-worker-b-8wg9r Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_pod-httpd_default_82ecae98-69aa-45f3-984e-3c792c39d927_0 in pod sandbox k8s_pod-httpd_default_82ecae98-69aa-45f3-984e-3c792c39d927_0(7ac5b9a957d0ff1443e8fd3ff40d73c6ea084df06c44f5266bae2db56be39d5c): Recreating the missing symlinks: 1 error occurred: * reading name of symlink for "0000000000000000000000000000000000000000000000000000000000000000": open /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000/link: no such file or directory
reproduced! $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.nightly-2022-01-25-023600 True False 66m Cluster version is 4.10.0-0.nightly-2022-01-25-023600 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> Successfully assigned default/hello-pod-1 to ip-10-0-203-96.us-east-2.compute.internal Normal AddedInterface 26s multus Add eth0 [10.131.0.24/23] from openshift-sdn Warning FailedCreatePodSandBox 25s kubelet, ip-10-0-203-96.us-east-2.compute.internal Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_hello-pod-1_default_db81beb4-f9ca-4d3d-8f37-0aa31cadfe55_0 in pod sandbox k8s_hello-pod-1_default_db81beb4-f9ca-4d3d-8f37-0aa31cadfe55_0(ba89443747f620ed2ddd82ad4913c7859934048c037a0ee2ee931ff108303117): Recreating the missing symlinks: 1 error occurred: * reading name of symlink for &{"0000000000000000000000000000000000000000000000000000000000000000" '\x06' '�' {'�' '�' %!q(*time.Location=&{Local [{UTC 0 false}] [{-576460752303423488 0 false false}] UTC0 9223372036854775807 9223372036854775807 0xc00013f320})} {'𐌄' '�' '\x02' '䇀' '\x00' '\x00' '\x00' '\x00' '\x06' 'က' '\x00' {'�' '�'} {'�' '�'} {'�' '�'} ['\x00' '\x00' '\x00']}}: open /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000/link: no such file or directory Normal AddedInterface 24s multus Add eth0 [10.131.0.25/23] from openshift-sdn Normal Pulling 24s kubelet, ip-10-0-203-96.us-east-2.compute.internal Pulling image "httpd:latest" Normal Pulled 20s kubelet, ip-10-0-203-96.us-east-2.compute.internal Successfully pulled image "httpd:latest" in 3.989657887s Normal Created 20s kubelet, ip-10-0-203-96.us-east-2.compute.internal Created container hello-pod Normal Started 20s kubelet, ip-10-0-203-96.us-east-2.compute.internal Started container hello-pod
verified! % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.0-0.nightly-2022-10-05-053337 True False 5h33m Cluster version is 4.12.0-0.nightly-2022-10-05-053337 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 99s default-scheduler Successfully assigned default/test to minmli-1012-fwqhs-worker-b-ss4ht.c.openshift-qe.internal Normal AddedInterface 98s multus Add eth0 [10.128.2.15/23] from openshift-sdn Warning FailedCreatePodSandBox 98s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to mount container k8s_POD_test_default_75b869fd-341d-4243-86ac-62da85f54a86_0 in pod sandbox k8s_test_default_75b869fd-341d-4243-86ac-62da85f54a86_0(e67ce787a322ecda070bc0f466894f5724dc9b0faf903d29a14b3d2267226bbf): recreating the missing symlinks: 1 error occurred: * reading name of symlink for "0000000000000000000000000000000000000000000000000000000000000000": open /var/lib/containers/storage/overlay/0000000000000000000000000000000000000000000000000000000000000000/link: no such file or directory Normal AddedInterface 96s multus Add eth0 [10.128.2.16/23] from openshift-sdn Normal Pulling 96s kubelet Pulling image "docker.io/library/busybox" Normal Pulled 94s kubelet Successfully pulled image "docker.io/library/busybox" in 1.906702793s
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7399