Hide Forgot
Description of problem: tested with arm64 cluster, create project and deploy the example deployment, pod is CrashLoopBackOff due to the image is built on linux+amd64 # oc new-project ns3 Now using project "ns3" on server "https://api.**.qe.devcluster.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname # kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname deployment.apps/hello-node created # oc -n ns3 get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES hello-node-6fd44bfdb4-7kmvh 0/1 CrashLoopBackOff 6 (3m53s ago) 9m47s 10.129.2.17 ip-10-0-154-161.us-east-2.compute.internal <none> <none> # oc -n ns3 describe pod hello-node-6fd44bfdb4-7kmvh Name: hello-node-6fd44bfdb4-7kmvh Namespace: ns3 Priority: 0 Node: ip-10-0-154-161.us-east-2.compute.internal/10.0.154.161 Start Time: Mon, 13 Dec 2021 22:35:23 -0500 Labels: app=hello-node pod-template-hash=6fd44bfdb4 Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.129.2.17" ], "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.129.2.17" ], "default": true, "dns": {} }] openshift.io/scc: restricted Status: Running IP: 10.129.2.17 IPs: IP: 10.129.2.17 Controlled By: ReplicaSet/hello-node-6fd44bfdb4 Containers: serve-hostname-c9r5f: Container ID: cri-o://cb6b010ca1ef80dbd8731dee525b6ab7efb269dc023a43254ceb5ddc58223a97 Image: k8s.gcr.io/serve_hostname Image ID: k8s.gcr.io/serve_hostname@sha256:07ab315083303397079209c526f6ed02f731d2fa7ffa0ad5c40b49410b105fee Port: <none> Host Port: <none> State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Mon, 13 Dec 2021 22:38:31 -0500 Finished: Mon, 13 Dec 2021 22:38:31 -0500 Ready: False Restart Count: 5 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gmtzz (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-gmtzz: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 97s default-scheduler Successfully assigned ns3/hello-node-6fd44bfdb4-7kmvh to ip-10-0-154-161.us-east-2.compute.internal Normal AddedInterface 96s multus Add eth0 [10.129.2.17/23] from openshift-sdn Normal Pulled 94s kubelet Successfully pulled image "k8s.gcr.io/serve_hostname" in 2.32054522s Normal Pulled 92s kubelet Successfully pulled image "k8s.gcr.io/serve_hostname" in 450.143679ms Normal Pulled 79s kubelet Successfully pulled image "k8s.gcr.io/serve_hostname" in 445.735955ms Normal Created 52s (x4 over 93s) kubelet Created container serve-hostname-c9r5f Normal Started 52s (x4 over 93s) kubelet Started container serve-hostname-c9r5f Normal Pulled 52s kubelet Successfully pulled image "k8s.gcr.io/serve_hostname" in 434.094282ms Normal Pulling 2s (x5 over 96s) kubelet Pulling image "k8s.gcr.io/serve_hostname" Warning BackOff <invalid> (x25 over 92s) kubelet Back-off restarting failed container found info in journalctl logs, the image is linux+amd64, not fit for linux+arm64 cluster ********************************** Dec 14 03:35:26 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:26.790774222Z" level=info msg="Image operating system mismatch: image uses OS \"linux\"+architecture \"amd64\", expecting one of \"linux+arm64\"Dec 14 03:35:27 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:27.981627728Z" level=info msg="Pulled image: k8s.gcr.io ********************************** # oc debug node/ip-10-0-154-161.us-east-2.compute.internal sh-4.4# chroot /host sh-4.4# journalctl ... Dec 14 03:35:25 ip-10-0-154-161 hyperkube[1387]: I1214 03:35:25.617704 1387 kubelet.go:2084] "SyncLoop UPDATE" source="api" pods=[ns3/hello-node-6fd44bfdb4-7kmvh] aces":[{"name":"eth0","sandbox":"/var/run/netns/c8c4d514-9f1b-4cd8-9c7a-aec68c2c5d54"}],"ips":[{"version":"4","interface":0,"address":"10.129.2.17/23"}],"routes":[{"dst":"0.0.0.0/0","gw":"10.129.2.1"},{"dst":"22.0.0.0/4"},{"dst":"10.128.0.0/14"}],"dns":{}} Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: I1214 03:35:25.595417 102529 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ns3", Name:"hello-node-6fd44bfdb4-7kmvh", UID:"29886699-cdf2-45fb-b3a6-6f36bef2c50", APIVersion:"v1", ResourceVersion:"55318", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [10.129.2.17/23] from openshift-sdn Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.618534981Z" level=info msg="Got pod network &{Name:hello-node-6fd44bfdb4-7kmvh Namespace:ns3 ID:0918a2e604b526f26c20c205b81521e8de79c66d9c9ee8e7971c822fab6d13d UID:29886699-cdf2-45fb-b3a6-6f367bef2c50 NetNS:/var/run/netns/c8c4d514-9f1b-4cd8-9c7a-aec68c2c5d54 Networks:[] RuntimeConfig:map[multus-cni-network:{IP: MAC: PortMappings:[] Bandwidth:<nil> Ipanges:[]}] Aliases:map[]}" Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.618700756Z" level=info msg="Checking pod ns3_hello-node-6fd44bfdb4-7kmvh for CNI network multus-cni-network (type=multus)" Dec 14 03:35:25 ip-10-0-154-161 hyperkube[1387]: W1214 03:35:25.620719 1387 manager.go:1185] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2986699_cdf2_45fb_b3a6_6f367bef2c50.slice/crio-0918a2e604b526f26c20c205b81521e8de79c66d9c9ee08e7971c822fab6d13d.scope WatchSource:0}: Error finding container 0918a2e604b526f26c20c205b81521e8de79c66d9c9ee08e7971c82fab6d13d: Status 404 returned error &{%!s(*http.body=&{0x4002d96ed0 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x7eafa0) %!s(unc() error=0x7eb0c0)} Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.660291204Z" level=info msg="Ran pod sandbox 0918a2e604b526f26c20c205b81521e8de79c66d9c9ee08e7971c822fab6d13d with infra container: ns3/hello-ode-6fd44bfdb4-7kmvh/POD" id=c6c2bce9-818f-47f5-9eab-ac30b1f746f5 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.661168218Z" level=info msg="Checking image status: k8s.gcr.io/serve_hostname:latest" id=078df95c-b124-4aed-a3e6-2a19c1d6a150 name=/runtime.v1lpha2.ImageService/ImageStatus Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.661360594Z" level=info msg="Image k8s.gcr.io/serve_hostname:latest not found" id=078df95c-b124-4aed-a3e6-2a19c1d6a150 name=/runtime.v1alpha2.mageService/ImageStatus Dec 14 03:35:25 ip-10-0-154-161 hyperkube[1387]: I1214 03:35:25.661767 1387 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.663011141Z" level=info msg="Pulling image: k8s.gcr.io/serve_hostname:latest" id=c0948098-6813-4ede-b0d5-e4e48ce317c9 name=/runtime.v1alpha2.IageService/PullImage Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.665236575Z" level=info msg="Trying to access \"k8s.gcr.io/serve_hostname:latest\"" Dec 14 03:35:25 ip-10-0-154-161 hyperkube[1387]: I1214 03:35:25.783307 1387 kubelet.go:2115] "SyncLoop (PLEG): event for pod" pod="ns3/hello-node-6fd44bfdb4-7kmvh" event=&{ID:29886699-cdf2-45fb-b3a6-6f367bef250 Type:ContainerStarted Data:0918a2e604b526f26c20c205b81521e8de79c66d9c9ee08e7971c822fab6d13d} Dec 14 03:35:26 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:26.115642735Z" level=info msg="Trying to access \"k8s.gcr.io/serve_hostname:latest\"" Dec 14 03:35:26 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:26.790774222Z" level=info msg="Image operating system mismatch: image uses OS \"linux\"+architecture \"amd64\", expecting one of \"linux+arm64\"Dec 14 03:35:27 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:27.981627728Z" level=info msg="Pulled image: k8s.gcr.io/serve_hostname@sha256:07ab315083303397079209c526f6ed02f731d2fa7ffa0ad5c40b49410b105fee" d=c0948098-6813-4ede-b0d5-e4e48ce317c9 name=/runtime.v1alpha2.ImageService/PullImage Dec 14 03:35:27 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:27.982512735Z" level=info msg="Checking image status: k8s.gcr.io/serve_hostname" id=1a279481-020d-4257-8340-a60ccdc312fd name=/runtime.v1alpha2.mageService/ImageStatus Dec 14 03:35:27 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:27.983270661Z" level=info msg="Image status: &ImageStatusResponse{Image:&Image{Id:ae63be60a48b6002cabf6e62c5fc3dad170ee8102670406f3a527879a3f0810,RepoTags:[k8s.gcr.io/serve_hostname:latest],RepoDigests:[k8s.gcr.io/serve_hostname@sha256:07ab315083303397079209c526f6ed02f731d2fa7ffa0ad5c40b49410b105fee],Size_:4535115,Uid:nil,Username:,Spec:nil,},Info:map[sring]string{},}" id=1a279481-020d-4257-8340-a60ccdc312fd name=/runtime.v1alpha2.ImageService/ImageStatus Dec 14 03:35:27 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:27.983859694Z" level=info msg="Creating container: ns3/hello-node-6fd44bfdb4-7kmvh/serve-hostname-c9r5f" id=3d7f6164-0a36-48e2-9010-9774d85ce12bname=/runtime.v1alpha2.RuntimeService/CreateContainer # docker inspect k8s.gcr.io/serve_hostname ... "Architecture": "amd64", "Os": "linux", Version-Release number of selected component (if applicable): 4.10.0-0.nightly-arm64-2021-12-13-225101 How reproducible: always Steps to Reproduce: 1. see the description 2. 3. Actual results: pod is CrashLoopBackOff Expected results: pod should be normal Additional info:
serve_hostname is an outdated test image. AFAICS the literal replacement for that would be: kubectl create deployment hello-node --image=k8s.gcr.io/e2e-test-images/agnhost:2.33 -- /agnhost serve-hostname That being said, perhaps there is a better example that could be used not from an e2e-test-image.
checked with 4.10.0-0.nightly-arm64-2021-12-14-101540, pod is Running # kubectl create deployment hello-node --image=k8s.gcr.io/e2e-test-images/agnhost:2.33 -- /agnhost serve-hostname deployment.apps/hello-node created # oc get po NAME READY STATUS RESTARTS AGE hello-node-549d8c98c4-g74sn 1/1 Running 0 18m the image also could be used in amd64 cluster 4.10.0-0.nightly-2021-12-14-083101, pod is also Running
The issue has fixed: [root@localhost Downloads]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.nightly-arm64-2021-12-24-235642 True False 43m Cluster version is 4.10.0-0.nightly-arm64-2021-12-24-235642 [root@localhost ~]# oc new-project zhouyt Now using project "zhouyt" on server "https://api.yinzhou27.qe.devcluster.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/e2e-test-images/agnhost:2.33 -- /agnhost serve-hostname [root@localhost Downloads]# kubectl create deployment hello-node --image=k8s.gcr.io/e2e-test-images/agnhost:2.33 -- /agnhost serve-hostname deployment.apps/hello-node created [root@localhost Downloads]# oc get po NAME READY STATUS RESTARTS AGE hello-node-549d8c98c4-m9bd8 0/1 ContainerCreating 0 7s [root@localhost Downloads]# oc get po NAME READY STATUS RESTARTS AGE hello-node-549d8c98c4-m9bd8 1/1 Running 0 2m12s [root@localhost Downloads]# kubectl version --client Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v0.23.0", GitCommit:"9214112338643662d5922b28720c448b5fbcd8d4", GitTreeState:"clean", BuildDate:"2021-12-20T15:54:53Z", GoVersion:"go1.17.2", Compiler:"gc", Platform:"linux/amd64"}