Description of problem: Egress router pod not created in SNO ipv6 single stack cluster Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-07-31-065602 How reproducible: Always Steps to Reproduce: #### SNO cluster [root@ocp-edge49 ~]# oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/EgressRouter/ipv6-ovn-egressrouter-redirect-MultipleDestinations.yaml egressrouter.network.operator.openshift.io/egress-router-test created [root@ocp-edge49 ~]# oc get all NAME READY STATUS RESTARTS AGE pod/egress-router-cni-deployment-78f8bc8875-dxlbj 0/1 ContainerCreating 0 13s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/egress-router-cni-deployment 0/1 1 0 13s NAME DESIRED CURRENT READY AGE replicaset.apps/egress-router-cni-deployment-78f8bc8875 1 1 0 13s [root@ocp-edge49 ~]# oc describe pod/egress-router-cni-deployment-78f8bc8875-dxlbj Name: egress-router-cni-deployment-78f8bc8875-dxlbj Namespace: test-ovn-egressrouter-multides Priority: 0 Node: sno-0-0.ocp-edge-cluster-0.ocp-edge49.lab.eng.tlv2.redhat.com/fd2e:6f44:5dd8::65 Start Time: Tue, 03 Aug 2021 18:46:43 +0300 Labels: app=egress-router-cni pod-template-hash=78f8bc8875 Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["fd01:0:0:1::73/64"],"mac_address":"0a:58:85:41:b8:e7","gateway_ips":["fd01:0:0:1::1"],"ip_address":"fd01:0:0:... k8s.v1.cni.cncf.io/networks: egress-router-cni-nad openshift.io/scc: restricted workload.openshift.io/warning: the node "sno-0-0.ocp-edge-cluster-0.ocp-edge49.lab.eng.tlv2.redhat.com" does not have resource "management.workload.openshift.io/cores" Status: Pending IP: IPs: <none> Controlled By: ReplicaSet/egress-router-cni-deployment-78f8bc8875 Containers: egress-router-cni-pod: Container ID: Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9329bdf8d23e5bee621e449b63403d8bc012dde1592454a38f6f6423a755729 Image ID: Port: <none> Host Port: <none> Command: /bin/sh -c sleep infinity State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ls2lq (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-ls2lq: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 81s default-scheduler Successfully assigned test-ovn-egressrouter-multides/egress-router-cni-deployment-78f8bc8875-dxlbj to sno-0-0.ocp-edge-cluster-0.ocp-edge49.lab.eng.tlv2.redhat.com Normal AddedInterface 79s multus Add eth0 [fd01:0:0:1::73/64] from ovn-kubernetes Warning FailedCreatePodSandBox 76s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-78f8bc8875-dxlbj_test-ovn-egressrouter-multides_6aeb8558-3cab-4c1d-ab69-e96477d26b61_0(b93e0e49e2e8e198ef260603b4e6b19860d7b5729e27d4a1c40ab43015ebb366): error adding pod test-ovn-egressrouter-multides_egress-router-cni-deployment-78f8bc8875-dxlbj to CNI network "multus-cni-network": [test-ovn-egressrouter-multides/egress-router-cni-deployment-78f8bc8875-dxlbj:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input Normal AddedInterface 73s multus Add eth0 [fd01:0:0:1::73/64] from ovn-kubernetes Warning FailedCreatePodSandBox 70s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-78f8bc8875-dxlbj_test-ovn-egressrouter-multides_6aeb8558-3cab-4c1d-ab69-e96477d26b61_0(08aa822f968b41c098cb415ee0c3bac1e222828293302378c393dc35f552eafd): error adding pod test-ovn-egressrouter-multides_egress-router-cni-deployment-78f8bc8875-dxlbj to CNI network "multus-cni-network": [test-ovn-egressrouter-multides/egress-router-cni-deployment-78f8bc8875-dxlbj:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input Normal AddedInterface 55s multus Add eth0 [fd01:0:0:1::73/64] from ovn-kubernetes Warning FailedCreatePodSandBox 52s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-78f8bc8875-dxlbj_test-ovn-egressrouter-multides_6aeb8558-3cab-4c1d-ab69-e96477d26b61_0(0716d692b1bdade9c423c00c5f78733c9a845c382ca5131710aa61fb25cb98a4): error adding pod test-ovn-egressrouter-multides_egress-router-cni-deployment-78f8bc8875-dxlbj to CNI network "multus-cni-network": [test-ovn-egressrouter-multides/egress-router-cni-deployment-78f8bc8875-dxlbj:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input Normal AddedInterface 38s multus Add eth0 [fd01:0:0:1::73/64] from ovn-kubernetes Warning FailedCreatePodSandBox 35s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-78f8bc8875-dxlbj_test-ovn-egressrouter-multides_6aeb8558-3cab-4c1d-ab69-e96477d26b61_0(741eeaf7d841975b531f936ab1a85d8b9d1721fc95ef9f15f083c97bcac271e0): error adding pod test-ovn-egressrouter-multides_egress-router-cni-deployment-78f8bc8875-dxlbj to CNI network "multus-cni-network": [test-ovn-egressrouter-multides/egress-router-cni-deployment-78f8bc8875-dxlbj:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input Normal AddedInterface 20s multus Add eth0 [fd01:0:0:1::73/64] from ovn-kubernetes Warning FailedCreatePodSandBox 18s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-78f8bc8875-dxlbj_test-ovn-egressrouter-multides_6aeb8558-3cab-4c1d-ab69-e96477d26b61_0(daea7e84374514d07bfd47ec9f3b5f2ffb286145d84c9fae69d76d275f77d865): error adding pod test-ovn-egressrouter-multides_egress-router-cni-deployment-78f8bc8875-dxlbj to CNI network "multus-cni-network": [test-ovn-egressrouter-multides/egress-router-cni-deployment-78f8bc8875-dxlbj:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input Normal AddedInterface 1s multus Add eth0 [fd01:0:0:1::73/64] from ovn-kubernetes [root@ocp-edge49 ~]# oc get net-attach-def egress-router-cni-nad -o yaml apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: annotations: release.openshift.io/version: 4.8.0-0.nightly-2021-07-31-065602 creationTimestamp: "2021-08-03T17:56:08Z" generation: 1 name: egress-router-cni-nad namespace: test-ovn-egressrouter-multides ownerReferences: - apiVersion: network.operator.openshift.io/v1 controller: true kind: EgressRouter name: egress-router-test uid: 49676e0e-dcd2-4ccc-9c68-30438383ed07 resourceVersion: "116544" uid: 2145bc7e-d5a5-48d1-a787-ef41afafc30e spec: config: |- { "cniVersion": "0.4.0", "type": "egress-router", "name": "egress-router-cni-nad", "ip": { "addresses": [ "fd2e:6f44:5dd8::64/64" ], "destinations": ["80 TCP 2607:f8b0:4004:808::200e","8080 TCP 2600:1408:20:c81::3831 80","8888 TCP 2001:420:1101:1::185 80"], "gateway": "fe80::5054:ff:feb0:6560" }, "log_file": "/tmp/egress-router-log", "log_level": "debug" } [root@ocp-edge49 ~]# #### Regular OCP dual stack cluster [root@ocp-edge50 auth]# oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/EgressRouter/ovn-egressrouter-redirect-MultipleDestinations.yaml egressrouter.network.operator.openshift.io/egress-router-test created [root@ocp-edge50 auth]# oc get all NAME READY STATUS RESTARTS AGE pod/egress-router-cni-deployment-6cd5f9dbc7-mbztd 1/1 Running 0 17s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/egress-router-cni-deployment 1/1 1 1 17s NAME DESIRED CURRENT READY AGE replicaset.apps/egress-router-cni-deployment-6cd5f9dbc7 1 1 1 17s [root@ocp-edge50 auth]# oc describe pod/egress-router-cni-deployment-6cd5f9dbc7-mbztd Name: egress-router-cni-deployment-6cd5f9dbc7-mbztd Namespace: test Priority: 0 Node: worker-0-1.ocp-edge-cluster-0.qe.lab.redhat.com/192.168.123.129 Start Time: Tue, 03 Aug 2021 18:48:52 +0300 Labels: app=egress-router-cni pod-template-hash=6cd5f9dbc7 Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.128.2.94/23","fd01:0:0:5::253/64"],"mac_address":"0a:58:0a:80:02:5e","gateway_ips":["10.128.2.1","fd01:0:0... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.128.2.94", "fd01:0:0:5::253" ], "mac": "0a:58:0a:80:02:5e", "default": true, "dns": {} },{ "name": "test/egress-router-cni-nad", "interface": "net1", "ips": [ "192.168.123.134" ], "mac": "fa:b7:92:0b:2c:ee", "dns": {} }] k8s.v1.cni.cncf.io/networks: egress-router-cni-nad k8s.v1.cni.cncf.io/networks-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.128.2.94", "fd01:0:0:5::253" ], "mac": "0a:58:0a:80:02:5e", "default": true, "dns": {} },{ "name": "test/egress-router-cni-nad", "interface": "net1", "ips": [ "192.168.123.134" ], "mac": "fa:b7:92:0b:2c:ee", "dns": {} }] openshift.io/scc: restricted workload.openshift.io/warning: only single-node clusters support workload partitioning Status: Running IP: 10.128.2.94 IPs: IP: 10.128.2.94 IP: fd01:0:0:5::253 Controlled By: ReplicaSet/egress-router-cni-deployment-6cd5f9dbc7 Containers: egress-router-cni-pod: Container ID: cri-o://305494605b6ec38c76590d76fecdd960d0b633b01850bb4a12271701cfe73c71 Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c507202ceb55d929e2127bd813b08f43527a124dea0bae90102f3c1097680c26 Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c507202ceb55d929e2127bd813b08f43527a124dea0bae90102f3c1097680c26 Port: <none> Host Port: <none> Command: /bin/sh -c sleep infinity State: Running Started: Tue, 03 Aug 2021 18:48:54 +0300 Ready: True Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lt5zk (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: kube-api-access-lt5zk: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 39s default-scheduler Successfully assigned test/egress-router-cni-deployment-6cd5f9dbc7-mbztd to worker-0-1.ocp-edge-cluster-0.qe.lab.redhat.com Normal AddedInterface 38s multus Add eth0 [10.128.2.94/23 fd01:0:0:5::253/64] from ovn-kubernetes Normal AddedInterface 38s multus Add net1 [192.168.123.134/24] from test/egress-router-cni-nad Normal Pulled 38s kubelet Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c507202ceb55d929e2127bd813b08f43527a124dea0bae90102f3c1097680c26" already present on machine Normal Created 38s kubelet Created container egress-router-cni-pod Normal Started 38s kubelet Started container egress-router-cni-pod [root@ocp-edge50 auth]# oc get net-attach-def egress-router-cni-nad -o yaml apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: annotations: release.openshift.io/version: 4.9.0-0.nightly-2021-07-29-004741 creationTimestamp: "2021-08-03T15:48:52Z" generation: 1 name: egress-router-cni-nad namespace: test ownerReferences: - apiVersion: network.operator.openshift.io/v1 controller: true kind: EgressRouter name: egress-router-test uid: 52efad11-b517-4c37-b476-de73c4e09a1f resourceVersion: "2868587" uid: 15eafdde-c398-4611-8412-b0ca12ff2d03 spec: config: |- { "cniVersion": "0.4.0", "type": "egress-router", "name": "egress-router-cni-nad", "ip": { "addresses": [ "192.168.123.134/24" ], "destinations": ["80 TCP 142.250.81.206","8080 TCP 142.250.81.206 80","8888 TCP 142.250.81.206 80"], "gateway": "192.168.123.1" }, "log_file": "/tmp/egress-router-log", "log_level": "debug" } [root@ocp-edge50 auth]# Actual results: pod/egress-router-cni-deployment-78f8bc8875-dxlbj 0/1 ContainerCreating Expected results: pod/egress-router-cni-deployment-78f8bc8875-dxlbj 0/1 Running Additional info: [root@ocp-edge49 ~]# oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/EgressRouter/ipv6-ovn-egressrouter-redirect-MultipleDestinations.yaml egressrouter.network.operator.openshift.io/egress-router-test created [root@ocp-edge49 ~]# oc debug node/sno-0-0.ocp-edge-cluster-0.ocp-edge49.lab.eng.tlv2.redhat.com Starting pod/sno-0-0ocp-edge-cluster-0ocp-edge49labengtlv2redhatcom-debug ... To use host binaries, run `chroot /host` Pod IP: fd2e:6f44:5dd8::65 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host sh-4.4# cat /tmp/egress-router-log 2021-08-03T18:43:10Z [debug] Called CNI ADD 2021-08-03T18:43:10Z [debug] Gateway: fe80::5054:ff:feb0:6560 2021-08-03T18:43:10Z [debug] IP Source Addresses: [fd2e:6f44:5dd8::64/64] 2021-08-03T18:43:10Z [debug] IP Destinations: [80 TCP 2607:f8b0:4004:808::200e 8080 TCP 2600:1408:20:c81::3831 80 8888 TCP 2001:420:1101:1::185 80] 2021-08-03T18:43:10Z [debug] Created macvlan interface 2021-08-03T18:43:10Z [debug] Renamed macvlan to "net1" 2021-08-03T18:43:12Z [debug] Adding route to gateway fe80::5054:ff:feb0:6560 on macvlan interface 2021-08-03T18:43:12Z [error] failed to add new default route : file exists 2021-08-03T18:43:16Z [debug] Called CNI ADD 2021-08-03T18:43:16Z [debug] Gateway: fe80::5054:ff:feb0:6560 2021-08-03T18:43:16Z [debug] IP Source Addresses: [fd2e:6f44:5dd8::64/64] 2021-08-03T18:43:16Z [debug] IP Destinations: [80 TCP 2607:f8b0:4004:808::200e 8080 TCP 2600:1408:20:c81::3831 80 8888 TCP 2001:420:1101:1::185 80] 2021-08-03T18:43:16Z [debug] Created macvlan interface 2021-08-03T18:43:16Z [debug] Renamed macvlan to "net1" 2021-08-03T18:43:18Z [debug] Adding route to gateway fe80::5054:ff:feb0:6560 on macvlan interface 2021-08-03T18:43:18Z [error] failed to add new default route : file exists sh-4.4# cat /tmp/egress-router-log 2021-08-03T18:43:10Z [debug] Called CNI ADD 2021-08-03T18:43:10Z [debug] Gateway: fe80::5054:ff:feb0:6560 2021-08-03T18:43:10Z [debug] IP Source Addresses: [fd2e:6f44:5dd8::64/64] 2021-08-03T18:43:10Z [debug] IP Destinations: [80 TCP 2607:f8b0:4004:808::200e 8080 TCP 2600:1408:20:c81::3831 80 8888 TCP 2001:420:1101:1::185 80] 2021-08-03T18:43:10Z [debug] Created macvlan interface 2021-08-03T18:43:10Z [debug] Renamed macvlan to "net1" 2021-08-03T18:43:12Z [debug] Adding route to gateway fe80::5054:ff:feb0:6560 on macvlan interface 2021-08-03T18:43:12Z [error] failed to add new default route : file exists 2021-08-03T18:43:16Z [debug] Called CNI ADD 2021-08-03T18:43:16Z [debug] Gateway: fe80::5054:ff:feb0:6560 2021-08-03T18:43:16Z [debug] IP Source Addresses: [fd2e:6f44:5dd8::64/64] 2021-08-03T18:43:16Z [debug] IP Destinations: [80 TCP 2607:f8b0:4004:808::200e 8080 TCP 2600:1408:20:c81::3831 80 8888 TCP 2001:420:1101:1::185 80] 2021-08-03T18:43:16Z [debug] Created macvlan interface 2021-08-03T18:43:16Z [debug] Renamed macvlan to "net1" 2021-08-03T18:43:18Z [debug] Adding route to gateway fe80::5054:ff:feb0:6560 on macvlan interface 2021-08-03T18:43:18Z [error] failed to add new default route : file exists 2021-08-03T18:43:35Z [debug] Called CNI ADD 2021-08-03T18:43:35Z [debug] Gateway: fe80::5054:ff:feb0:6560 2021-08-03T18:43:35Z [debug] IP Source Addresses: [fd2e:6f44:5dd8::64/64] 2021-08-03T18:43:35Z [debug] IP Destinations: [80 TCP 2607:f8b0:4004:808::200e 8080 TCP 2600:1408:20:c81::3831 80 8888 TCP 2001:420:1101:1::185 80] 2021-08-03T18:43:35Z [debug] Created macvlan interface 2021-08-03T18:43:35Z [debug] Renamed macvlan to "net1" 2021-08-03T18:43:36Z [debug] Adding route to gateway fe80::5054:ff:feb0:6560 on macvlan interface 2021-08-03T18:43:36Z [error] failed to add new default route : file exists sh-4.4#
raise the severity since this is feature blocker.
Reproduce the same issue in 4.9.0-0.nightly-2021-08-04-131508: [root@ocp-edge50 ~]# oc get nodes NAME STATUS ROLES AGE VERSION sno-0-0.ocp-lab-0.qe.lab.redhat.com Ready master,worker 11h v1.21.1+8268f88 [root@ocp-edge50 ~]# oc new-project test Now using project "test" on server "https://api.ocp-lab-0.qe.lab.redhat.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname [root@ocp-edge50 ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-08-04-131508 True False 11h Cluster version is 4.9.0-0.nightly-2021-08-04-131508 [root@ocp-edge50 ~]# oc get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME sno-0-0.ocp-lab-0.qe.lab.redhat.com Ready master,worker 11h v1.21.1+8268f88 fd2e:6f44:5dd8::75 <none> Red Hat Enterprise Linux CoreOS 49.84.202108021839-0 (Ootpa) 4.18.0-305.10.2.el8_4.x86_64 cri-o://1.22.0-22.rhaos4.9.git79a25c4.el8 [root@ocp-edge50 ~]# oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/EgressRouter/ipv6-ovn-egressrouter-redirect-MultipleDestinations.yaml egressrouter.network.operator.openshift.io/egress-router-test created [root@ocp-edge50 ~]# oc get pods NAME READY STATUS RESTARTS AGE sno-0-0ocp-lab-0qelabredhatcom-debug 1/1 Running 0 2m8s [root@ocp-edge50 ~]# oc describe pod egress-router-cni-deployment-7cd5877f89-2hnv2 Name: egress-router-cni-deployment-7cd5877f89-2hnv2 Namespace: test Priority: 0 Node: sno-0-0.ocp-lab-0.qe.lab.redhat.com/fd2e:6f44:5dd8::75 Start Time: Fri, 06 Aug 2021 16:20:43 +0300 Labels: app=egress-router-cni pod-template-hash=7cd5877f89 Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["fd01:0:0:1::9d/64"],"mac_address":"0a:58:70:b4:56:f3","gateway_ips":["fd01:0:0:1::1"],"ip_address":"fd01:0:0:... k8s.v1.cni.cncf.io/networks: egress-router-cni-nad openshift.io/scc: restricted workload.openshift.io/warning: the node "sno-0-0.ocp-lab-0.qe.lab.redhat.com" does not have resource "management.workload.openshift.io/cores" Status: Pending IP: IPs: <none> Controlled By: ReplicaSet/egress-router-cni-deployment-7cd5877f89 Containers: egress-router-cni-pod: Container ID: Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4744d4dd8b06bc30873d646be02ff8b3bae78f4357fc0badee28514536781471 Image ID: Port: <none> Host Port: <none> Command: /bin/sh -c sleep infinity State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mjlnb (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-mjlnb: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 54s default-scheduler Successfully assigned test/egress-router-cni-deployment-7cd5877f89-2hnv2 to sno-0-0.ocp-lab-0.qe.lab.redhat.com Normal AddedInterface 52s multus Add eth0 [fd01:0:0:1::9d/64] from ovn-kubernetes Warning FailedCreatePodSandBox 50s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-7cd5877f89-2hnv2_test_73ad4fd8-3ab5-4b71-a1bc-101bb639bccd_0(8e1686f88b946bd588bdf675496e7074a9494f6ea54d868ce981068598592d10): error adding pod test_egress-router-cni-deployment-7cd5877f89-2hnv2 to CNI network "multus-cni-network": [test/egress-router-cni-deployment-7cd5877f89-2hnv2:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input Normal AddedInterface 47s multus Add eth0 [fd01:0:0:1::9d/64] from ovn-kubernetes Warning FailedCreatePodSandBox 44s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-7cd5877f89-2hnv2_test_73ad4fd8-3ab5-4b71-a1bc-101bb639bccd_0(fbd2e1364c826bfb0513ad8f92dcb40801a40fce69a8993d2089cfca7a9dba9e): error adding pod test_egress-router-cni-deployment-7cd5877f89-2hnv2 to CNI network "multus-cni-network": [test/egress-router-cni-deployment-7cd5877f89-2hnv2:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input Normal AddedInterface 31s multus Add eth0 [fd01:0:0:1::9d/64] from ovn-kubernetes Warning FailedCreatePodSandBox 28s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-7cd5877f89-2hnv2_test_73ad4fd8-3ab5-4b71-a1bc-101bb639bccd_0(aa78467a4d2eb6bcb2c107523a86fa0a6282fe10d0b3bf92aea130d8e710f313): error adding pod test_egress-router-cni-deployment-7cd5877f89-2hnv2 to CNI network "multus-cni-network": [test/egress-router-cni-deployment-7cd5877f89-2hnv2:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input Normal AddedInterface 12s multus Add eth0 [fd01:0:0:1::9d/64] from ovn-kubernetes Warning FailedCreatePodSandBox 9s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-7cd5877f89-2hnv2_test_73ad4fd8-3ab5-4b71-a1bc-101bb639bccd_0(fd2139b58775d36c009c4509c33c395ac145167a2e814bba77126d0ec8d6d578): error adding pod test_egress-router-cni-deployment-7cd5877f89-2hnv2 to CNI network "multus-cni-network": [test/egress-router-cni-deployment-7cd5877f89-2hnv2:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input [root@ocp-edge50 ~]#
(In reply to Mohamed Mahmoud from comment #7) > I think your configuration is not correct > > gateway: "fe80::5054:ff:feb0:6560" > is IPv6 linklocal subnet and the CNI fail to create default route there > saying the route already exists which is true > fe80::/64 dev eno1 proto kernel metric 100 pref medium > fe80::/64 dev baremetal-0 proto kernel metric 256 pref medium > fe80::/64 dev vnet0 proto kernel metric 256 pref medium > fe80::/64 dev cni-podman0 proto kernel metric 427 linkdown pref medium > > and u see this error in CNI logs > 2021-08-03T18:43:18Z [debug] Adding route to gateway fe80::5054:ff:feb0:6560 > on macvlan interface > 2021-08-03T18:43:18Z [error] failed to add new default route : file exists > > please configure gateway correctly and confirm this issue is no longer seen
Wait for OVN team to guide me which ipv6 address I should use for egressrouter gateway
Tested and verified in 4.9.0-0.nightly-2021-08-14-065522 [root@ocp-edge50 auth]# oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES egress-router-cni-deployment-684c76d444-ztr9b 1/1 Running 0 32s fd01:0:0:1::66 sno-0-0.ocp-lab-0.qe.lab.redhat.com <none> <none> [root@ocp-edge50 auth]# oc get all NAME READY STATUS RESTARTS AGE pod/egress-router-cni-deployment-684c76d444-ztr9b 1/1 Running 0 11m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/egress-router-cni-deployment 1/1 1 1 11m NAME DESIRED CURRENT READY AGE replicaset.apps/egress-router-cni-deployment-684c76d444 1 1 1 11m [root@ocp-edge50 auth]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-08-14-065522 True False 46m Cluster version is 4.9.0-0.nightly-2021-08-14-065522 [root@ocp-edge50 auth]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759