Bug 2032111 - arm64 cluster, create project and deploy the example deployment, pod is CrashLoopBackOff due to the image is built on linux+amd64
Summary: arm64 cluster, create project and deploy the example deployment, pod is Crash...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.10
Hardware: All
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.10.0
Assignee: Maciej Szulik
QA Contact: zhou ying
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-14 04:10 UTC by Junqi Zhao
Modified: 2022-03-11 18:15 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-11 18:15:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 995 0 None open Bug 2032111: requestproject: use agnhost serve-hostname 2021-12-14 19:47:19 UTC

Description Junqi Zhao 2021-12-14 04:10:06 UTC
Description of problem:
tested with arm64 cluster, create project and deploy the example deployment, pod is CrashLoopBackOff due to the image is built on linux+amd64
# oc new-project ns3
Now using project "ns3" on server "https://api.**.qe.devcluster.openshift.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname

# kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname
deployment.apps/hello-node created

# oc -n ns3 get pod -o wide
NAME                                            READY   STATUS             RESTARTS        AGE     IP             NODE                                         NOMINATED NODE   READINESS GATES
hello-node-6fd44bfdb4-7kmvh                     0/1     CrashLoopBackOff   6 (3m53s ago)   9m47s   10.129.2.17    ip-10-0-154-161.us-east-2.compute.internal   <none>           <none>


# oc -n ns3 describe pod hello-node-6fd44bfdb4-7kmvh
Name:         hello-node-6fd44bfdb4-7kmvh
Namespace:    ns3
Priority:     0
Node:         ip-10-0-154-161.us-east-2.compute.internal/10.0.154.161
Start Time:   Mon, 13 Dec 2021 22:35:23 -0500
Labels:       app=hello-node
              pod-template-hash=6fd44bfdb4
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "openshift-sdn",
                    "interface": "eth0",
                    "ips": [
                        "10.129.2.17"
                    ],
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "openshift-sdn",
                    "interface": "eth0",
                    "ips": [
                        "10.129.2.17"
                    ],
                    "default": true,
                    "dns": {}
                }]
              openshift.io/scc: restricted
Status:       Running
IP:           10.129.2.17
IPs:
  IP:           10.129.2.17
Controlled By:  ReplicaSet/hello-node-6fd44bfdb4
Containers:
  serve-hostname-c9r5f:
    Container ID:   cri-o://cb6b010ca1ef80dbd8731dee525b6ab7efb269dc023a43254ceb5ddc58223a97
    Image:          k8s.gcr.io/serve_hostname
    Image ID:       k8s.gcr.io/serve_hostname@sha256:07ab315083303397079209c526f6ed02f731d2fa7ffa0ad5c40b49410b105fee
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 13 Dec 2021 22:38:31 -0500
      Finished:     Mon, 13 Dec 2021 22:38:31 -0500
    Ready:          False
    Restart Count:  5
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gmtzz (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-gmtzz:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                       From               Message
  ----     ------          ----                      ----               -------
  Normal   Scheduled       97s                       default-scheduler  Successfully assigned ns3/hello-node-6fd44bfdb4-7kmvh to ip-10-0-154-161.us-east-2.compute.internal
  Normal   AddedInterface  96s                       multus             Add eth0 [10.129.2.17/23] from openshift-sdn
  Normal   Pulled          94s                       kubelet            Successfully pulled image "k8s.gcr.io/serve_hostname" in 2.32054522s
  Normal   Pulled          92s                       kubelet            Successfully pulled image "k8s.gcr.io/serve_hostname" in 450.143679ms
  Normal   Pulled          79s                       kubelet            Successfully pulled image "k8s.gcr.io/serve_hostname" in 445.735955ms
  Normal   Created         52s (x4 over 93s)         kubelet            Created container serve-hostname-c9r5f
  Normal   Started         52s (x4 over 93s)         kubelet            Started container serve-hostname-c9r5f
  Normal   Pulled          52s                       kubelet            Successfully pulled image "k8s.gcr.io/serve_hostname" in 434.094282ms
  Normal   Pulling         2s (x5 over 96s)          kubelet            Pulling image "k8s.gcr.io/serve_hostname"
  Warning  BackOff         <invalid> (x25 over 92s)  kubelet            Back-off restarting failed container

found info in journalctl logs, the image is linux+amd64, not fit for linux+arm64 cluster
**********************************
Dec 14 03:35:26 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:26.790774222Z" level=info msg="Image operating system mismatch: image uses OS \"linux\"+architecture \"amd64\", expecting one of \"linux+arm64\"Dec 14 03:35:27 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:27.981627728Z" level=info msg="Pulled image: k8s.gcr.io
**********************************


# oc debug node/ip-10-0-154-161.us-east-2.compute.internal
sh-4.4# chroot /host
sh-4.4# journalctl
...
Dec 14 03:35:25 ip-10-0-154-161 hyperkube[1387]: I1214 03:35:25.617704    1387 kubelet.go:2084] "SyncLoop UPDATE" source="api" pods=[ns3/hello-node-6fd44bfdb4-7kmvh]
aces":[{"name":"eth0","sandbox":"/var/run/netns/c8c4d514-9f1b-4cd8-9c7a-aec68c2c5d54"}],"ips":[{"version":"4","interface":0,"address":"10.129.2.17/23"}],"routes":[{"dst":"0.0.0.0/0","gw":"10.129.2.1"},{"dst":"22.0.0.0/4"},{"dst":"10.128.0.0/14"}],"dns":{}}
Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: I1214 03:35:25.595417  102529 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ns3", Name:"hello-node-6fd44bfdb4-7kmvh", UID:"29886699-cdf2-45fb-b3a6-6f36bef2c50", APIVersion:"v1", ResourceVersion:"55318", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [10.129.2.17/23] from openshift-sdn
Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.618534981Z" level=info msg="Got pod network &{Name:hello-node-6fd44bfdb4-7kmvh Namespace:ns3 ID:0918a2e604b526f26c20c205b81521e8de79c66d9c9ee8e7971c822fab6d13d UID:29886699-cdf2-45fb-b3a6-6f367bef2c50 NetNS:/var/run/netns/c8c4d514-9f1b-4cd8-9c7a-aec68c2c5d54 Networks:[] RuntimeConfig:map[multus-cni-network:{IP: MAC: PortMappings:[] Bandwidth:<nil> Ipanges:[]}] Aliases:map[]}"
Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.618700756Z" level=info msg="Checking pod ns3_hello-node-6fd44bfdb4-7kmvh for CNI network multus-cni-network (type=multus)"
Dec 14 03:35:25 ip-10-0-154-161 hyperkube[1387]: W1214 03:35:25.620719    1387 manager.go:1185] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2986699_cdf2_45fb_b3a6_6f367bef2c50.slice/crio-0918a2e604b526f26c20c205b81521e8de79c66d9c9ee08e7971c822fab6d13d.scope WatchSource:0}: Error finding container 0918a2e604b526f26c20c205b81521e8de79c66d9c9ee08e7971c82fab6d13d: Status 404 returned error &{%!s(*http.body=&{0x4002d96ed0 <nil> <nil> false false {0 0} false false false <nil>}) {%!s(int32=0) %!s(uint32=0)} %!s(bool=false) <nil> %!s(func(error) error=0x7eafa0) %!s(unc() error=0x7eb0c0)}
Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.660291204Z" level=info msg="Ran pod sandbox 0918a2e604b526f26c20c205b81521e8de79c66d9c9ee08e7971c822fab6d13d with infra container: ns3/hello-ode-6fd44bfdb4-7kmvh/POD" id=c6c2bce9-818f-47f5-9eab-ac30b1f746f5 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.661168218Z" level=info msg="Checking image status: k8s.gcr.io/serve_hostname:latest" id=078df95c-b124-4aed-a3e6-2a19c1d6a150 name=/runtime.v1lpha2.ImageService/ImageStatus
Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.661360594Z" level=info msg="Image k8s.gcr.io/serve_hostname:latest not found" id=078df95c-b124-4aed-a3e6-2a19c1d6a150 name=/runtime.v1alpha2.mageService/ImageStatus
Dec 14 03:35:25 ip-10-0-154-161 hyperkube[1387]: I1214 03:35:25.661767    1387 provider.go:102] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.663011141Z" level=info msg="Pulling image: k8s.gcr.io/serve_hostname:latest" id=c0948098-6813-4ede-b0d5-e4e48ce317c9 name=/runtime.v1alpha2.IageService/PullImage
Dec 14 03:35:25 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:25.665236575Z" level=info msg="Trying to access \"k8s.gcr.io/serve_hostname:latest\""
Dec 14 03:35:25 ip-10-0-154-161 hyperkube[1387]: I1214 03:35:25.783307    1387 kubelet.go:2115] "SyncLoop (PLEG): event for pod" pod="ns3/hello-node-6fd44bfdb4-7kmvh" event=&{ID:29886699-cdf2-45fb-b3a6-6f367bef250 Type:ContainerStarted Data:0918a2e604b526f26c20c205b81521e8de79c66d9c9ee08e7971c822fab6d13d}
Dec 14 03:35:26 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:26.115642735Z" level=info msg="Trying to access \"k8s.gcr.io/serve_hostname:latest\""
Dec 14 03:35:26 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:26.790774222Z" level=info msg="Image operating system mismatch: image uses OS \"linux\"+architecture \"amd64\", expecting one of \"linux+arm64\"Dec 14 03:35:27 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:27.981627728Z" level=info msg="Pulled image: k8s.gcr.io/serve_hostname@sha256:07ab315083303397079209c526f6ed02f731d2fa7ffa0ad5c40b49410b105fee" d=c0948098-6813-4ede-b0d5-e4e48ce317c9 name=/runtime.v1alpha2.ImageService/PullImage
Dec 14 03:35:27 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:27.982512735Z" level=info msg="Checking image status: k8s.gcr.io/serve_hostname" id=1a279481-020d-4257-8340-a60ccdc312fd name=/runtime.v1alpha2.mageService/ImageStatus
Dec 14 03:35:27 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:27.983270661Z" level=info msg="Image status: &ImageStatusResponse{Image:&Image{Id:ae63be60a48b6002cabf6e62c5fc3dad170ee8102670406f3a527879a3f0810,RepoTags:[k8s.gcr.io/serve_hostname:latest],RepoDigests:[k8s.gcr.io/serve_hostname@sha256:07ab315083303397079209c526f6ed02f731d2fa7ffa0ad5c40b49410b105fee],Size_:4535115,Uid:nil,Username:,Spec:nil,},Info:map[sring]string{},}" id=1a279481-020d-4257-8340-a60ccdc312fd name=/runtime.v1alpha2.ImageService/ImageStatus
Dec 14 03:35:27 ip-10-0-154-161 crio[1358]: time="2021-12-14 03:35:27.983859694Z" level=info msg="Creating container: ns3/hello-node-6fd44bfdb4-7kmvh/serve-hostname-c9r5f" id=3d7f6164-0a36-48e2-9010-9774d85ce12bname=/runtime.v1alpha2.RuntimeService/CreateContainer

# docker inspect k8s.gcr.io/serve_hostname
...
        "Architecture": "amd64",
        "Os": "linux",

Version-Release number of selected component (if applicable):
4.10.0-0.nightly-arm64-2021-12-13-225101

How reproducible:
always

Steps to Reproduce:
1. see the description
2.
3.

Actual results:
pod is CrashLoopBackOff

Expected results:
pod should be normal


Additional info:

Comment 1 Yaakov Selkowitz 2021-12-14 19:41:25 UTC
serve_hostname is an outdated test image.  AFAICS the literal replacement for that would be:

kubectl create deployment hello-node --image=k8s.gcr.io/e2e-test-images/agnhost:2.33 -- /agnhost serve-hostname

That being said, perhaps there is a better example that could be used not from an e2e-test-image.

Comment 2 Junqi Zhao 2021-12-15 02:50:43 UTC
checked with 4.10.0-0.nightly-arm64-2021-12-14-101540, pod is Running
# kubectl create deployment hello-node --image=k8s.gcr.io/e2e-test-images/agnhost:2.33 -- /agnhost serve-hostname
deployment.apps/hello-node created

# oc get po
NAME                          READY   STATUS    RESTARTS   AGE
hello-node-549d8c98c4-g74sn   1/1     Running   0          18m

the image also could be used in amd64 cluster 4.10.0-0.nightly-2021-12-14-083101, pod is also Running

Comment 5 zhou ying 2021-12-27 02:14:51 UTC
The issue has fixed:

[root@localhost Downloads]# oc get clusterversion 
NAME      VERSION                                    AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-arm64-2021-12-24-235642   True        False         43m     Cluster version is 4.10.0-0.nightly-arm64-2021-12-24-235642


[root@localhost ~]# oc new-project zhouyt
Now using project "zhouyt" on server "https://api.yinzhou27.qe.devcluster.openshift.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=k8s.gcr.io/e2e-test-images/agnhost:2.33 -- /agnhost serve-hostname


[root@localhost Downloads]#  kubectl create deployment hello-node --image=k8s.gcr.io/e2e-test-images/agnhost:2.33 -- /agnhost serve-hostname
deployment.apps/hello-node created
[root@localhost Downloads]# oc get po 
NAME                          READY   STATUS              RESTARTS   AGE
hello-node-549d8c98c4-m9bd8   0/1     ContainerCreating   0          7s
[root@localhost Downloads]# oc get po 
NAME                          READY   STATUS    RESTARTS   AGE
hello-node-549d8c98c4-m9bd8   1/1     Running   0          2m12s

[root@localhost Downloads]# kubectl version --client
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v0.23.0", GitCommit:"9214112338643662d5922b28720c448b5fbcd8d4", GitTreeState:"clean", BuildDate:"2021-12-20T15:54:53Z", GoVersion:"go1.17.2", Compiler:"gc", Platform:"linux/amd64"}


Note You need to log in before you can comment on or make changes to this bug.