Bug 1989688 - [SNO] Egress router pod not created in SNO ipv6 single stack cluster
Summary: [SNO] Egress router pod not created in SNO ipv6 single stack cluster
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.8
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 4.9.0
Assignee: Mohamed Mahmoud
QA Contact: Weibin Liang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-03 17:57 UTC by Weibin Liang
Modified: 2021-10-18 17:44 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: configure Egress router POD with IPv6 single stack. Consequence: CNI configuration fails. Fix: Added IPv6 support. Result: IPv6 configuration and traffic works.
Clone Of:
Environment:
Last Closed: 2021-10-18 17:44:13 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift egress-router-cni pull 52 0 None None None 2021-08-06 16:56:47 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:44:29 UTC

Description Weibin Liang 2021-08-03 17:57:01 UTC
Description of problem:
Egress router pod not created in SNO ipv6 single stack  cluster

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-07-31-065602

How reproducible:
Always

Steps to Reproduce:

#### SNO cluster
[root@ocp-edge49 ~]# oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/EgressRouter/ipv6-ovn-egressrouter-redirect-MultipleDestinations.yaml
egressrouter.network.operator.openshift.io/egress-router-test created
[root@ocp-edge49 ~]# oc get all
NAME                                                READY   STATUS              RESTARTS   AGE
pod/egress-router-cni-deployment-78f8bc8875-dxlbj   0/1     ContainerCreating   0          13s

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/egress-router-cni-deployment   0/1     1            0           13s

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/egress-router-cni-deployment-78f8bc8875   1         1         0       13s
[root@ocp-edge49 ~]# oc describe pod/egress-router-cni-deployment-78f8bc8875-dxlbj
Name:           egress-router-cni-deployment-78f8bc8875-dxlbj
Namespace:      test-ovn-egressrouter-multides
Priority:       0
Node:           sno-0-0.ocp-edge-cluster-0.ocp-edge49.lab.eng.tlv2.redhat.com/fd2e:6f44:5dd8::65
Start Time:     Tue, 03 Aug 2021 18:46:43 +0300
Labels:         app=egress-router-cni
                pod-template-hash=78f8bc8875
Annotations:    k8s.ovn.org/pod-networks:
                  {"default":{"ip_addresses":["fd01:0:0:1::73/64"],"mac_address":"0a:58:85:41:b8:e7","gateway_ips":["fd01:0:0:1::1"],"ip_address":"fd01:0:0:...
                k8s.v1.cni.cncf.io/networks: egress-router-cni-nad
                openshift.io/scc: restricted
                workload.openshift.io/warning:
                  the node "sno-0-0.ocp-edge-cluster-0.ocp-edge49.lab.eng.tlv2.redhat.com" does not have resource "management.workload.openshift.io/cores"
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/egress-router-cni-deployment-78f8bc8875
Containers:
  egress-router-cni-pod:
    Container ID:  
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9329bdf8d23e5bee621e449b63403d8bc012dde1592454a38f6f6423a755729
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      sleep infinity
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ls2lq (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-ls2lq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               81s   default-scheduler  Successfully assigned test-ovn-egressrouter-multides/egress-router-cni-deployment-78f8bc8875-dxlbj to sno-0-0.ocp-edge-cluster-0.ocp-edge49.lab.eng.tlv2.redhat.com
  Normal   AddedInterface          79s   multus             Add eth0 [fd01:0:0:1::73/64] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  76s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-78f8bc8875-dxlbj_test-ovn-egressrouter-multides_6aeb8558-3cab-4c1d-ab69-e96477d26b61_0(b93e0e49e2e8e198ef260603b4e6b19860d7b5729e27d4a1c40ab43015ebb366): error adding pod test-ovn-egressrouter-multides_egress-router-cni-deployment-78f8bc8875-dxlbj to CNI network "multus-cni-network": [test-ovn-egressrouter-multides/egress-router-cni-deployment-78f8bc8875-dxlbj:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input
  Normal   AddedInterface          73s   multus             Add eth0 [fd01:0:0:1::73/64] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  70s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-78f8bc8875-dxlbj_test-ovn-egressrouter-multides_6aeb8558-3cab-4c1d-ab69-e96477d26b61_0(08aa822f968b41c098cb415ee0c3bac1e222828293302378c393dc35f552eafd): error adding pod test-ovn-egressrouter-multides_egress-router-cni-deployment-78f8bc8875-dxlbj to CNI network "multus-cni-network": [test-ovn-egressrouter-multides/egress-router-cni-deployment-78f8bc8875-dxlbj:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input
  Normal   AddedInterface          55s   multus             Add eth0 [fd01:0:0:1::73/64] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  52s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-78f8bc8875-dxlbj_test-ovn-egressrouter-multides_6aeb8558-3cab-4c1d-ab69-e96477d26b61_0(0716d692b1bdade9c423c00c5f78733c9a845c382ca5131710aa61fb25cb98a4): error adding pod test-ovn-egressrouter-multides_egress-router-cni-deployment-78f8bc8875-dxlbj to CNI network "multus-cni-network": [test-ovn-egressrouter-multides/egress-router-cni-deployment-78f8bc8875-dxlbj:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input
  Normal   AddedInterface          38s   multus             Add eth0 [fd01:0:0:1::73/64] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  35s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-78f8bc8875-dxlbj_test-ovn-egressrouter-multides_6aeb8558-3cab-4c1d-ab69-e96477d26b61_0(741eeaf7d841975b531f936ab1a85d8b9d1721fc95ef9f15f083c97bcac271e0): error adding pod test-ovn-egressrouter-multides_egress-router-cni-deployment-78f8bc8875-dxlbj to CNI network "multus-cni-network": [test-ovn-egressrouter-multides/egress-router-cni-deployment-78f8bc8875-dxlbj:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input
  Normal   AddedInterface          20s   multus             Add eth0 [fd01:0:0:1::73/64] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  18s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-78f8bc8875-dxlbj_test-ovn-egressrouter-multides_6aeb8558-3cab-4c1d-ab69-e96477d26b61_0(daea7e84374514d07bfd47ec9f3b5f2ffb286145d84c9fae69d76d275f77d865): error adding pod test-ovn-egressrouter-multides_egress-router-cni-deployment-78f8bc8875-dxlbj to CNI network "multus-cni-network": [test-ovn-egressrouter-multides/egress-router-cni-deployment-78f8bc8875-dxlbj:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input
  Normal   AddedInterface          1s    multus             Add eth0 [fd01:0:0:1::73/64] from ovn-kubernetes
[root@ocp-edge49 ~]# oc get net-attach-def egress-router-cni-nad -o yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    release.openshift.io/version: 4.8.0-0.nightly-2021-07-31-065602
  creationTimestamp: "2021-08-03T17:56:08Z"
  generation: 1
  name: egress-router-cni-nad
  namespace: test-ovn-egressrouter-multides
  ownerReferences:
  - apiVersion: network.operator.openshift.io/v1
    controller: true
    kind: EgressRouter
    name: egress-router-test
    uid: 49676e0e-dcd2-4ccc-9c68-30438383ed07
  resourceVersion: "116544"
  uid: 2145bc7e-d5a5-48d1-a787-ef41afafc30e
spec:
  config: |-
    { "cniVersion": "0.4.0", "type": "egress-router", "name": "egress-router-cni-nad", "ip": { "addresses": [ "fd2e:6f44:5dd8::64/64" ], "destinations": ["80 TCP 2607:f8b0:4004:808::200e","8080 TCP 2600:1408:20:c81::3831 80","8888 TCP 2001:420:1101:1::185 80"],
    "gateway": "fe80::5054:ff:feb0:6560" }, "log_file": "/tmp/egress-router-log", "log_level": "debug" }
[root@ocp-edge49 ~]# 


#### Regular OCP dual stack cluster 
[root@ocp-edge50 auth]# oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/EgressRouter/ovn-egressrouter-redirect-MultipleDestinations.yaml
egressrouter.network.operator.openshift.io/egress-router-test created
[root@ocp-edge50 auth]# oc get all
NAME                                                READY   STATUS    RESTARTS   AGE
pod/egress-router-cni-deployment-6cd5f9dbc7-mbztd   1/1     Running   0          17s

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/egress-router-cni-deployment   1/1     1            1           17s

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/egress-router-cni-deployment-6cd5f9dbc7   1         1         1       17s
[root@ocp-edge50 auth]# oc describe pod/egress-router-cni-deployment-6cd5f9dbc7-mbztd
Name:         egress-router-cni-deployment-6cd5f9dbc7-mbztd
Namespace:    test
Priority:     0
Node:         worker-0-1.ocp-edge-cluster-0.qe.lab.redhat.com/192.168.123.129
Start Time:   Tue, 03 Aug 2021 18:48:52 +0300
Labels:       app=egress-router-cni
              pod-template-hash=6cd5f9dbc7
Annotations:  k8s.ovn.org/pod-networks:
                {"default":{"ip_addresses":["10.128.2.94/23","fd01:0:0:5::253/64"],"mac_address":"0a:58:0a:80:02:5e","gateway_ips":["10.128.2.1","fd01:0:0...
              k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "ovn-kubernetes",
                    "interface": "eth0",
                    "ips": [
                        "10.128.2.94",
                        "fd01:0:0:5::253"
                    ],
                    "mac": "0a:58:0a:80:02:5e",
                    "default": true,
                    "dns": {}
                },{
                    "name": "test/egress-router-cni-nad",
                    "interface": "net1",
                    "ips": [
                        "192.168.123.134"
                    ],
                    "mac": "fa:b7:92:0b:2c:ee",
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks: egress-router-cni-nad
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "ovn-kubernetes",
                    "interface": "eth0",
                    "ips": [
                        "10.128.2.94",
                        "fd01:0:0:5::253"
                    ],
                    "mac": "0a:58:0a:80:02:5e",
                    "default": true,
                    "dns": {}
                },{
                    "name": "test/egress-router-cni-nad",
                    "interface": "net1",
                    "ips": [
                        "192.168.123.134"
                    ],
                    "mac": "fa:b7:92:0b:2c:ee",
                    "dns": {}
                }]
              openshift.io/scc: restricted
              workload.openshift.io/warning: only single-node clusters support workload partitioning
Status:       Running
IP:           10.128.2.94
IPs:
  IP:           10.128.2.94
  IP:           fd01:0:0:5::253
Controlled By:  ReplicaSet/egress-router-cni-deployment-6cd5f9dbc7
Containers:
  egress-router-cni-pod:
    Container ID:  cri-o://305494605b6ec38c76590d76fecdd960d0b633b01850bb4a12271701cfe73c71
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c507202ceb55d929e2127bd813b08f43527a124dea0bae90102f3c1097680c26
    Image ID:      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c507202ceb55d929e2127bd813b08f43527a124dea0bae90102f3c1097680c26
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      sleep infinity
    State:          Running
      Started:      Tue, 03 Aug 2021 18:48:54 +0300
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lt5zk (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-lt5zk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       39s   default-scheduler  Successfully assigned test/egress-router-cni-deployment-6cd5f9dbc7-mbztd to worker-0-1.ocp-edge-cluster-0.qe.lab.redhat.com
  Normal  AddedInterface  38s   multus             Add eth0 [10.128.2.94/23 fd01:0:0:5::253/64] from ovn-kubernetes
  Normal  AddedInterface  38s   multus             Add net1 [192.168.123.134/24] from test/egress-router-cni-nad
  Normal  Pulled          38s   kubelet            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c507202ceb55d929e2127bd813b08f43527a124dea0bae90102f3c1097680c26" already present on machine
  Normal  Created         38s   kubelet            Created container egress-router-cni-pod
  Normal  Started         38s   kubelet            Started container egress-router-cni-pod
[root@ocp-edge50 auth]# oc get net-attach-def egress-router-cni-nad -o yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    release.openshift.io/version: 4.9.0-0.nightly-2021-07-29-004741
  creationTimestamp: "2021-08-03T15:48:52Z"
  generation: 1
  name: egress-router-cni-nad
  namespace: test
  ownerReferences:
  - apiVersion: network.operator.openshift.io/v1
    controller: true
    kind: EgressRouter
    name: egress-router-test
    uid: 52efad11-b517-4c37-b476-de73c4e09a1f
  resourceVersion: "2868587"
  uid: 15eafdde-c398-4611-8412-b0ca12ff2d03
spec:
  config: |-
    { "cniVersion": "0.4.0", "type": "egress-router", "name": "egress-router-cni-nad", "ip": { "addresses": [ "192.168.123.134/24" ], "destinations": ["80 TCP 142.250.81.206","8080 TCP 142.250.81.206 80","8888 TCP 142.250.81.206 80"],
    "gateway": "192.168.123.1" }, "log_file": "/tmp/egress-router-log", "log_level": "debug" }
[root@ocp-edge50 auth]# 


Actual results:
pod/egress-router-cni-deployment-78f8bc8875-dxlbj   0/1     ContainerCreating

Expected results:
pod/egress-router-cni-deployment-78f8bc8875-dxlbj   0/1     Running

Additional info:
[root@ocp-edge49 ~]# oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/EgressRouter/ipv6-ovn-egressrouter-redirect-MultipleDestinations.yaml
egressrouter.network.operator.openshift.io/egress-router-test created
[root@ocp-edge49 ~]# oc debug node/sno-0-0.ocp-edge-cluster-0.ocp-edge49.lab.eng.tlv2.redhat.com
Starting pod/sno-0-0ocp-edge-cluster-0ocp-edge49labengtlv2redhatcom-debug ...
To use host binaries, run `chroot /host`
Pod IP: fd2e:6f44:5dd8::65
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-4.4# cat /tmp/egress-router-log 
2021-08-03T18:43:10Z [debug] Called CNI ADD
2021-08-03T18:43:10Z [debug] Gateway: fe80::5054:ff:feb0:6560
2021-08-03T18:43:10Z [debug] IP Source Addresses: [fd2e:6f44:5dd8::64/64]
2021-08-03T18:43:10Z [debug] IP Destinations: [80 TCP 2607:f8b0:4004:808::200e 8080 TCP 2600:1408:20:c81::3831 80 8888 TCP 2001:420:1101:1::185 80]
2021-08-03T18:43:10Z [debug] Created macvlan interface
2021-08-03T18:43:10Z [debug] Renamed macvlan to "net1"
2021-08-03T18:43:12Z [debug] Adding route to gateway fe80::5054:ff:feb0:6560 on macvlan interface
2021-08-03T18:43:12Z [error] failed to add new default route : file exists
2021-08-03T18:43:16Z [debug] Called CNI ADD
2021-08-03T18:43:16Z [debug] Gateway: fe80::5054:ff:feb0:6560
2021-08-03T18:43:16Z [debug] IP Source Addresses: [fd2e:6f44:5dd8::64/64]
2021-08-03T18:43:16Z [debug] IP Destinations: [80 TCP 2607:f8b0:4004:808::200e 8080 TCP 2600:1408:20:c81::3831 80 8888 TCP 2001:420:1101:1::185 80]
2021-08-03T18:43:16Z [debug] Created macvlan interface
2021-08-03T18:43:16Z [debug] Renamed macvlan to "net1"
2021-08-03T18:43:18Z [debug] Adding route to gateway fe80::5054:ff:feb0:6560 on macvlan interface
2021-08-03T18:43:18Z [error] failed to add new default route : file exists
sh-4.4# cat /tmp/egress-router-log 
2021-08-03T18:43:10Z [debug] Called CNI ADD
2021-08-03T18:43:10Z [debug] Gateway: fe80::5054:ff:feb0:6560
2021-08-03T18:43:10Z [debug] IP Source Addresses: [fd2e:6f44:5dd8::64/64]
2021-08-03T18:43:10Z [debug] IP Destinations: [80 TCP 2607:f8b0:4004:808::200e 8080 TCP 2600:1408:20:c81::3831 80 8888 TCP 2001:420:1101:1::185 80]
2021-08-03T18:43:10Z [debug] Created macvlan interface
2021-08-03T18:43:10Z [debug] Renamed macvlan to "net1"
2021-08-03T18:43:12Z [debug] Adding route to gateway fe80::5054:ff:feb0:6560 on macvlan interface
2021-08-03T18:43:12Z [error] failed to add new default route : file exists
2021-08-03T18:43:16Z [debug] Called CNI ADD
2021-08-03T18:43:16Z [debug] Gateway: fe80::5054:ff:feb0:6560
2021-08-03T18:43:16Z [debug] IP Source Addresses: [fd2e:6f44:5dd8::64/64]
2021-08-03T18:43:16Z [debug] IP Destinations: [80 TCP 2607:f8b0:4004:808::200e 8080 TCP 2600:1408:20:c81::3831 80 8888 TCP 2001:420:1101:1::185 80]
2021-08-03T18:43:16Z [debug] Created macvlan interface
2021-08-03T18:43:16Z [debug] Renamed macvlan to "net1"
2021-08-03T18:43:18Z [debug] Adding route to gateway fe80::5054:ff:feb0:6560 on macvlan interface
2021-08-03T18:43:18Z [error] failed to add new default route : file exists
2021-08-03T18:43:35Z [debug] Called CNI ADD
2021-08-03T18:43:35Z [debug] Gateway: fe80::5054:ff:feb0:6560
2021-08-03T18:43:35Z [debug] IP Source Addresses: [fd2e:6f44:5dd8::64/64]
2021-08-03T18:43:35Z [debug] IP Destinations: [80 TCP 2607:f8b0:4004:808::200e 8080 TCP 2600:1408:20:c81::3831 80 8888 TCP 2001:420:1101:1::185 80]
2021-08-03T18:43:35Z [debug] Created macvlan interface
2021-08-03T18:43:35Z [debug] Renamed macvlan to "net1"
2021-08-03T18:43:36Z [debug] Adding route to gateway fe80::5054:ff:feb0:6560 on macvlan interface
2021-08-03T18:43:36Z [error] failed to add new default route : file exists
sh-4.4#

Comment 1 zhaozhanqi 2021-08-04 03:45:36 UTC
raise the severity since this is feature blocker.

Comment 6 Weibin Liang 2021-08-06 13:26:41 UTC
Reproduce the same issue in 4.9.0-0.nightly-2021-08-04-131508:


[root@ocp-edge50 ~]# oc get nodes
NAME                                  STATUS   ROLES           AGE   VERSION
sno-0-0.ocp-lab-0.qe.lab.redhat.com   Ready    master,worker   11h   v1.21.1+8268f88
[root@ocp-edge50 ~]# oc new-project test
Now using project "test" on server "https://api.ocp-lab-0.qe.lab.redhat.com:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname

[root@ocp-edge50 ~]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-08-04-131508   True        False         11h     Cluster version is 4.9.0-0.nightly-2021-08-04-131508
[root@ocp-edge50 ~]# oc get node -o wide
NAME                                  STATUS   ROLES           AGE   VERSION           INTERNAL-IP          EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
sno-0-0.ocp-lab-0.qe.lab.redhat.com   Ready    master,worker   11h   v1.21.1+8268f88   fd2e:6f44:5dd8::75   <none>        Red Hat Enterprise Linux CoreOS 49.84.202108021839-0 (Ootpa)   4.18.0-305.10.2.el8_4.x86_64   cri-o://1.22.0-22.rhaos4.9.git79a25c4.el8
[root@ocp-edge50 ~]# oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/EgressRouter/ipv6-ovn-egressrouter-redirect-MultipleDestinations.yaml
egressrouter.network.operator.openshift.io/egress-router-test created
[root@ocp-edge50 ~]# oc get pods
NAME                                            READY   STATUS              RESTARTS   AGE
sno-0-0ocp-lab-0qelabredhatcom-debug            1/1     Running             0          2m8s
[root@ocp-edge50 ~]# oc describe pod egress-router-cni-deployment-7cd5877f89-2hnv2
Name:           egress-router-cni-deployment-7cd5877f89-2hnv2
Namespace:      test
Priority:       0
Node:           sno-0-0.ocp-lab-0.qe.lab.redhat.com/fd2e:6f44:5dd8::75
Start Time:     Fri, 06 Aug 2021 16:20:43 +0300
Labels:         app=egress-router-cni
                pod-template-hash=7cd5877f89
Annotations:    k8s.ovn.org/pod-networks:
                  {"default":{"ip_addresses":["fd01:0:0:1::9d/64"],"mac_address":"0a:58:70:b4:56:f3","gateway_ips":["fd01:0:0:1::1"],"ip_address":"fd01:0:0:...
                k8s.v1.cni.cncf.io/networks: egress-router-cni-nad
                openshift.io/scc: restricted
                workload.openshift.io/warning:
                  the node "sno-0-0.ocp-lab-0.qe.lab.redhat.com" does not have resource "management.workload.openshift.io/cores"
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/egress-router-cni-deployment-7cd5877f89
Containers:
  egress-router-cni-pod:
    Container ID:  
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4744d4dd8b06bc30873d646be02ff8b3bae78f4357fc0badee28514536781471
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      sleep infinity
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mjlnb (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-mjlnb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               54s   default-scheduler  Successfully assigned test/egress-router-cni-deployment-7cd5877f89-2hnv2 to sno-0-0.ocp-lab-0.qe.lab.redhat.com
  Normal   AddedInterface          52s   multus             Add eth0 [fd01:0:0:1::9d/64] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  50s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-7cd5877f89-2hnv2_test_73ad4fd8-3ab5-4b71-a1bc-101bb639bccd_0(8e1686f88b946bd588bdf675496e7074a9494f6ea54d868ce981068598592d10): error adding pod test_egress-router-cni-deployment-7cd5877f89-2hnv2 to CNI network "multus-cni-network": [test/egress-router-cni-deployment-7cd5877f89-2hnv2:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input
  Normal   AddedInterface          47s   multus             Add eth0 [fd01:0:0:1::9d/64] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  44s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-7cd5877f89-2hnv2_test_73ad4fd8-3ab5-4b71-a1bc-101bb639bccd_0(fbd2e1364c826bfb0513ad8f92dcb40801a40fce69a8993d2089cfca7a9dba9e): error adding pod test_egress-router-cni-deployment-7cd5877f89-2hnv2 to CNI network "multus-cni-network": [test/egress-router-cni-deployment-7cd5877f89-2hnv2:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input
  Normal   AddedInterface          31s   multus             Add eth0 [fd01:0:0:1::9d/64] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  28s   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-7cd5877f89-2hnv2_test_73ad4fd8-3ab5-4b71-a1bc-101bb639bccd_0(aa78467a4d2eb6bcb2c107523a86fa0a6282fe10d0b3bf92aea130d8e710f313): error adding pod test_egress-router-cni-deployment-7cd5877f89-2hnv2 to CNI network "multus-cni-network": [test/egress-router-cni-deployment-7cd5877f89-2hnv2:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input
  Normal   AddedInterface          12s   multus             Add eth0 [fd01:0:0:1::9d/64] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  9s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_egress-router-cni-deployment-7cd5877f89-2hnv2_test_73ad4fd8-3ab5-4b71-a1bc-101bb639bccd_0(fd2139b58775d36c009c4509c33c395ac145167a2e814bba77126d0ec8d6d578): error adding pod test_egress-router-cni-deployment-7cd5877f89-2hnv2 to CNI network "multus-cni-network": [test/egress-router-cni-deployment-7cd5877f89-2hnv2:egress-router-cni-nad]: error adding container to network "egress-router-cni-nad": unexpected end of JSON input
[root@ocp-edge50 ~]#

Comment 8 Mohamed Mahmoud 2021-08-06 15:11:33 UTC
(In reply to Mohamed Mahmoud from comment #7)
> I think your configuration is not correct
> 
> gateway: "fe80::5054:ff:feb0:6560"
> is IPv6 linklocal subnet and the CNI fail to create default route there
> saying the route already exists which is true
> fe80::/64 dev eno1 proto kernel metric 100 pref medium
> fe80::/64 dev baremetal-0 proto kernel metric 256 pref medium
> fe80::/64 dev vnet0 proto kernel metric 256 pref medium
> fe80::/64 dev cni-podman0 proto kernel metric 427 linkdown pref medium
> 
> and u see this error in CNI logs
> 2021-08-03T18:43:18Z [debug] Adding route to gateway fe80::5054:ff:feb0:6560
> on macvlan interface
> 2021-08-03T18:43:18Z [error] failed to add new default route : file exists
> 
> please configure gateway correctly and confirm this issue is no longer seen

Comment 9 Weibin Liang 2021-08-06 18:20:12 UTC
Wait for OVN team to guide me which ipv6 address I should use for egressrouter gateway

Comment 11 Weibin Liang 2021-08-16 14:54:11 UTC
Tested and verified in 4.9.0-0.nightly-2021-08-14-065522

[root@ocp-edge50 auth]# oc get pod -o wide
NAME                                            READY   STATUS    RESTARTS   AGE   IP               NODE                                  NOMINATED NODE   READINESS GATES
egress-router-cni-deployment-684c76d444-ztr9b   1/1     Running   0          32s   fd01:0:0:1::66   sno-0-0.ocp-lab-0.qe.lab.redhat.com   <none>           <none>
[root@ocp-edge50 auth]# oc get all
NAME                                                READY   STATUS    RESTARTS   AGE
pod/egress-router-cni-deployment-684c76d444-ztr9b   1/1     Running   0          11m

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/egress-router-cni-deployment   1/1     1            1           11m

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/egress-router-cni-deployment-684c76d444   1         1         1       11m
[root@ocp-edge50 auth]# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-08-14-065522   True        False         46m     Cluster version is 4.9.0-0.nightly-2021-08-14-065522
[root@ocp-edge50 auth]#

Comment 14 errata-xmlrpc 2021-10-18 17:44:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.