Bug 1989688
Summary: | [SNO] Egress router pod not created in SNO ipv6 single stack cluster | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Weibin Liang <weliang> |
Component: | Networking | Assignee: | Mohamed Mahmoud <mmahmoud> |
Networking sub component: | ovn-kubernetes | QA Contact: | Weibin Liang <weliang> |
Status: | CLOSED ERRATA | Docs Contact: | |
Severity: | urgent | ||
Priority: | unspecified | CC: | astoycos, dosmith, zzhao |
Version: | 4.8 | ||
Target Milestone: | --- | ||
Target Release: | 4.9.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: |
Cause: configure Egress router POD with IPv6 single stack.
Consequence: CNI configuration fails.
Fix: Added IPv6 support.
Result: IPv6 configuration and traffic works.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2021-10-18 17:44:13 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Weibin Liang
2021-08-03 17:57:01 UTC
raise the severity since this is feature blocker. Reproduce the same issue in 4.9.0-0.nightly-2021-08-04-131508: [root@ocp-edge50 ~]# oc get nodes NAME STATUS ROLES AGE VERSION sno-0-0.ocp-lab-0.qe.lab.redhat.com Ready master,worker 11h v1.21.1+8268f88 [root@ocp-edge50 ~]# oc new-project test Now using project "test" on server "https://api.ocp-lab-0.qe.lab.redhat.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname [root@ocp-edge50 ~]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-08-04-131508 True False 11h Cluster version is 4.9.0-0.nightly-2021-08-04-131508 [root@ocp-edge50 ~]# oc get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME sno-0-0.ocp-lab-0.qe.lab.redhat.com Ready master,worker 11h v1.21.1+8268f88 fd2e:6f44:5dd8::75 <none> Red Hat Enterprise Linux CoreOS 49.84.202108021839-0 (Ootpa) 4.18.0-305.10.2.el8_4.x86_64 cri-o://1.22.0-22.rhaos4.9.git79a25c4.el8 [root@ocp-edge50 ~]# oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/EgressRouter/ipv6-ovn-egressrouter-redirect-MultipleDestinations.yaml egressrouter.network.operator.openshift.io/egress-router-test created [root@ocp-edge50 ~]# oc get pods NAME READY STATUS RESTARTS AGE sno-0-0ocp-lab-0qelabredhatcom-debug 1/1 Running 0 2m8s [root@ocp-edge50 ~]# oc describe pod egress-router-cni-deployment-7cd5877f89-2hnv2 Name: egress-router-cni-deployment-7cd5877f89-2hnv2 Namespace: test Priority: 0 Node: sno-0-0.ocp-lab-0.qe.lab.redhat.com/fd2e:6f44:5dd8::75 Start Time: Fri, 06 Aug 2021 16:20:43 +0300 Labels: app=egress-router-cni pod-template-hash=7cd5877f89 Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["fd01:0:0:1::9d/64"],"mac_address":"0a:58:70:b4:56:f3","gateway_ips":["fd01:0:0:1::1"],"ip_address":"fd01:0:0:... k8s.v1.cni.cncf.io/networks: egress-router-cni-nad openshift.io/scc: restricted workload.openshift.io/warning: the node "sno-0-0.ocp-lab-0.qe.lab.redhat.com" does not have resource "management.workload.openshift.io/cores" Status: Pending IP: IPs: <none> Controlled By: ReplicaSet/egress-router-cni-deployment-7cd5877f89 Containers: egress-router-cni-pod: Container ID: Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4744d4dd8b06bc30873d646be02ff8b3bae78f4357fc0badee28514536781471 Image ID: Port: <none> Host Port: <none> Command: /bin/sh -c sleep infinity State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mjlnb (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-mjlnb: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 54s default-scheduler Successfully assigned test/egress-router-cni-deployment-7cd5877f89-2hnv2 to sno-0-0.ocp-lab-0.qe.lab.redhat.com Normal AddedInterface 52s multus Add eth0 [fd01:0:0:1::9d/64] from ovn-kubernetes Warning FailedCreatePodSandBox 50s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-7cd5877f89-2hnv2_test_73ad4fd8-3ab5-4b71-a1bc-101bb639bccd_0(8e1686f88b946bd588bdf675496e7074a9494f6ea54d868ce981068598592d10): error adding pod test_egress-router-cni-deployment-7cd5877f89-2hnv2 to CNI network "multus-cni-network": [test/egress-router-cni-deployment-7cd5877f89-2hnv2:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input Normal AddedInterface 47s multus Add eth0 [fd01:0:0:1::9d/64] from ovn-kubernetes Warning FailedCreatePodSandBox 44s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-7cd5877f89-2hnv2_test_73ad4fd8-3ab5-4b71-a1bc-101bb639bccd_0(fbd2e1364c826bfb0513ad8f92dcb40801a40fce69a8993d2089cfca7a9dba9e): error adding pod test_egress-router-cni-deployment-7cd5877f89-2hnv2 to CNI network "multus-cni-network": [test/egress-router-cni-deployment-7cd5877f89-2hnv2:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input Normal AddedInterface 31s multus Add eth0 [fd01:0:0:1::9d/64] from ovn-kubernetes Warning FailedCreatePodSandBox 28s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-7cd5877f89-2hnv2_test_73ad4fd8-3ab5-4b71-a1bc-101bb639bccd_0(aa78467a4d2eb6bcb2c107523a86fa0a6282fe10d0b3bf92aea130d8e710f313): error adding pod test_egress-router-cni-deployment-7cd5877f89-2hnv2 to CNI network "multus-cni-network": [test/egress-router-cni-deployment-7cd5877f89-2hnv2:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input Normal AddedInterface 12s multus Add eth0 [fd01:0:0:1::9d/64] from ovn-kubernetes Warning FailedCreatePodSandBox 9s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-7cd5877f89-2hnv2_test_73ad4fd8-3ab5-4b71-a1bc-101bb639bccd_0(fd2139b58775d36c009c4509c33c395ac145167a2e814bba77126d0ec8d6d578): error adding pod test_egress-router-cni-deployment-7cd5877f89-2hnv2 to CNI network "multus-cni-network": [test/egress-router-cni-deployment-7cd5877f89-2hnv2:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input [root@ocp-edge50 ~]# (In reply to Mohamed Mahmoud from comment #7) > I think your configuration is not correct > > gateway: "fe80::5054:ff:feb0:6560" > is IPv6 linklocal subnet and the CNI fail to create default route there > saying the route already exists which is true > fe80::/64 dev eno1 proto kernel metric 100 pref medium > fe80::/64 dev baremetal-0 proto kernel metric 256 pref medium > fe80::/64 dev vnet0 proto kernel metric 256 pref medium > fe80::/64 dev cni-podman0 proto kernel metric 427 linkdown pref medium > > and u see this error in CNI logs > 2021-08-03T18:43:18Z [debug] Adding route to gateway fe80::5054:ff:feb0:6560 > on macvlan interface > 2021-08-03T18:43:18Z [error] failed to add new default route : file exists > > please configure gateway correctly and confirm this issue is no longer seen Wait for OVN team to guide me which ipv6 address I should use for egressrouter gateway Tested and verified in 4.9.0-0.nightly-2021-08-14-065522 [root@ocp-edge50 auth]# oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES egress-router-cni-deployment-684c76d444-ztr9b 1/1 Running 0 32s fd01:0:0:1::66 sno-0-0.ocp-lab-0.qe.lab.redhat.com <none> <none> [root@ocp-edge50 auth]# oc get all NAME READY STATUS RESTARTS AGE pod/egress-router-cni-deployment-684c76d444-ztr9b 1/1 Running 0 11m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/egress-router-cni-deployment 1/1 1 1 11m NAME DESIRED CURRENT READY AGE replicaset.apps/egress-router-cni-deployment-684c76d444 1 1 1 11m [root@ocp-edge50 auth]# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-08-14-065522 True False 46m Cluster version is 4.9.0-0.nightly-2021-08-14-065522 [root@ocp-edge50 auth]# Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759 |