Description of problem: registry.redhat.io/openshift4/ose-pod image not work registry.redhat.io/openshift3/ose-pod image work Version-Release number of selected component (if applicable): 4.7.0-0.nightly-2020-10-27-051128 How reproducible: Always Steps to Reproduce: [weliang@weliang ~]$ diff egressrouter-1.yaml egressrouter-2.yaml 12c12 < image: registry.redhat.io/openshift3/ose-egress-router --- > image: registry.redhat.io/openshift4/ose-egress-router 27c27 < image: registry.redhat.io/openshift3/ose-pod --- > image: registry.redhat.io/openshift4/ose-pod [weliang@weliang ~]$ oc create -f egressrouter-1.yaml pod/egressrouter-redirect-pod created [weliang@weliang ~]$ oc get pod NAME READY STATUS RESTARTS AGE egressrouter-redirect-pod 1/1 Running 0 60s [weliang@weliang ~]$ oc delete pod egressrouter-redirect-pod pod "egressrouter-redirect-pod" deleted [weliang@weliang ~]$ oc create -f egressrouter-2.yaml pod/egressrouter-redirect-pod created [weliang@weliang ~]$ oc get pod NAME READY STATUS RESTARTS AGE egressrouter-redirect-pod 0/1 Init:CrashLoopBackOff 1 13s [weliang@weliang ~]$ oc logs egressrouter-redirect-pod Error from server (BadRequest): container "egress-pod" in pod "egressrouter-redirect-pod" is waiting to start: PodInitializing [weliang@weliang ~]$ oc describe pod egressrouter-redirect-pod Name: egressrouter-redirect-pod Namespace: test Priority: 0 Node: ip-10-0-147-69.us-east-2.compute.internal/10.0.147.69 Start Time: Mon, 09 Nov 2020 16:00:11 -0500 Labels: name=egressrouter-redirect-pod Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "", "interface": "eth0", "ips": [ "10.128.2.16" ], "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "", "interface": "eth0", "ips": [ "10.128.2.16" ], "default": true, "dns": {} }] openshift.io/scc: node-exporter pod.network.openshift.io/assign-macvlan: true Status: Pending IP: 10.128.2.16 IPs: IP: 10.128.2.16 Init Containers: egress-router: Container ID: cri-o://57ec8e485d567526f54caf09614814f2f1872e342be822cf85b6f1ed3e65542f Image: registry.redhat.io/openshift4/ose-egress-router Image ID: registry.redhat.io/openshift4/ose-egress-router@sha256:079afa6fd2f8e25484b8963ce716942ba9147243168142ee924265b9218dda5b Port: <none> Host Port: <none> State: Terminated Reason: Error Exit Code: 127 Started: Mon, 09 Nov 2020 16:00:30 -0500 Finished: Mon, 09 Nov 2020 16:00:30 -0500 Last State: Terminated Reason: Error Exit Code: 127 Started: Mon, 09 Nov 2020 16:00:14 -0500 Finished: Mon, 09 Nov 2020 16:00:14 -0500 Ready: False Restart Count: 2 Environment: EGRESS_SOURCE: 172.31.249.146 EGRESS_GATEWAY: 172.31.248.1 EGRESS_DESTINATION: 172.217.7.142 EGRESS_ROUTER_MODE: init Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-kpdh8 (ro) Containers: egress-pod: Container ID: Image: registry.redhat.io/openshift4/ose-pod Image ID: Port: <none> Host Port: <none> State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-kpdh8 (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: default-token-kpdh8: Type: Secret (a volume populated by a Secret) SecretName: default-token-kpdh8 Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> Successfully assigned test/egressrouter-redirect-pod to ip-10-0-147-69.us-east-2.compute.internal Normal AddedInterface 29s multus Add eth0 [10.128.2.16/23] Normal Pulled 12s (x3 over 29s) kubelet, ip-10-0-147-69.us-east-2.compute.internal Container image "registry.redhat.io/openshift4/ose-egress-router" already present on machine Normal Created 12s (x3 over 29s) kubelet, ip-10-0-147-69.us-east-2.compute.internal Created container egress-router Normal Started 12s (x3 over 29s) kubelet, ip-10-0-147-69.us-east-2.compute.internal Started container egress-router Warning BackOff 12s (x3 over 27s) kubelet, ip-10-0-147-69.us-east-2.compute.internal Back-off restarting failed container [weliang@weliang ~]$ oc logs -c egress-router egressrouter-redirect-pod /bin/egress-router.sh: line 113: iptables: command not found [weliang@weliang ~]$ Actual results: egressrouter-redirect-pod 0/1 Init:CrashLoopBackOff Expected results: egressrouter-redirect-pod 1/1 Running 0 60s Additional info: registry.redhat.io/openshift4/ose-egress-dns-proxy works registry.redhat.io/openshift4/ose-egress-http-proxy works
It seems the problematic image is registry.redhat.io/openshift4/ose-egress-router. Is it Routing?
(In reply to Oleg Bulatov from comment #1) > It seems the problematic image is > registry.redhat.io/openshift4/ose-egress-router. Is it Routing? From below test results, I do not think it's a routing problem: 1.Test case one pass with below two images in pod yam file: registry.redhat.io/openshift4/ose-egress-router + registry.redhat.io/openshift4/ose-egress-dns-proxy works (https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/EgressRouter/egress-router-dns-pod.yaml) 2.Test case two pass with below two images in pod yam file: registry.redhat.io/openshift4/ose-egress-router + registry.redhat.io/openshift4/ose-egress-http-proxy works 3.Test case three pass with below two images in pod yam file: registry.redhat.io/openshift4/ose-egress-router + registry.redhat.io/openshift3/ose-pod 4.Test case four FAIL with below two images in pod yam file: registry.redhat.io/openshift4/ose-egress-router + registry.redhat.io/openshift4/ose-pod
We need to look into whether (a) registry.redhat.io/openshift4/ose-pod is the correct pullspec in v4 and (b) whether we are failing to ship the image when we should be shipping it.
Got same error as before in the latest v4.7 nightly image, I will try one time tomorrow before I re assign this bug. [weliang@weliang debug-scripts]$ oc logs -c egress-router egressrouter-redirect-pod /bin/egress-router.sh: line 113: iptables: command not found [weliang@weliang debug-scripts]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2020-12-14-165231 True False 5h4m Cluster version is 4.7.0-0.nightly-2020-12-14-165231
Verified failed on latest v4.7 nightly image [weliang@weliang ~]$ oc logs -c egress-router egressrouter-redirect-pod /bin/egress-router.sh: line 113: iptables: command not found [weliang@weliang ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2020-12-18-031435 True False 14m Cluster version is 4.7.0-0.nightly-2020-12-18-031435 [weliang@weliang ~]$
Hi Stephen, Should I try v4.6 nightly image now? Thanks!
Did not see the new PR. Retested and failed in 4.7.0-0.nightly-2021-02-08-052658 [weliang@weliang ~]$ oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/EgressRouter/test.yaml pod/egressrouter-redirect-pod created [weliang@weliang ~]$ oc get pod NAME READY STATUS RESTARTS AGE egressrouter-redirect-pod 0/1 Init:0/1 0 5s [weliang@weliang ~]$ oc get pod NAME READY STATUS RESTARTS AGE egressrouter-redirect-pod 0/1 Init:0/1 0 7s [weliang@weliang ~]$ oc get pod NAME READY STATUS RESTARTS AGE egressrouter-redirect-pod 0/1 Init:0/1 0 9s [weliang@weliang ~]$ oc describe pod egressrouter-redirect-pod Name: egressrouter-redirect-pod Namespace: test Priority: 0 Node: weliang282-r62kn-worker-c-zbwvm.c.openshift-qe.internal/10.0.32.4 Start Time: Mon, 08 Feb 2021 14:30:12 -0500 Labels: name=egressrouter-redirect-pod Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "", "interface": "eth0", "ips": [ "10.129.2.25" ], "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "", "interface": "eth0", "ips": [ "10.129.2.25" ], "default": true, "dns": {} }] openshift.io/scc: node-exporter pod.network.openshift.io/assign-macvlan: true Status: Pending IP: 10.129.2.25 IPs: IP: 10.129.2.25 Init Containers: egress-router: Container ID: cri-o://7299b7c5c507f99048511c07b0fddee5fc1b3300621af00e561b521e4a30a007 Image: registry.redhat.io/openshift4/ose-egress-router Image ID: registry.redhat.io/openshift4/ose-egress-router@sha256:3b1c28d7880f1827d5329ae490021361e0b2a26f1ad6484c22a5729b56b752f4 Port: <none> Host Port: <none> State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 127 Started: Mon, 08 Feb 2021 14:30:23 -0500 Finished: Mon, 08 Feb 2021 14:30:23 -0500 Ready: False Restart Count: 1 Environment: EGRESS_SOURCE: 172.31.249.146 EGRESS_GATEWAY: 172.31.248.1 EGRESS_DESTINATION: 172.217.7.142 EGRESS_ROUTER_MODE: init Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-c4sx2 (ro) Containers: egress-pod: Container ID: Image: registry.redhat.io/openshift4/ose-pod Image ID: Port: <none> Host Port: <none> State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-c4sx2 (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: default-token-c4sx2: Type: Secret (a volume populated by a Secret) SecretName: default-token-c4sx2 Optional: false QoS Class: BestEffort Node-Selectors: app=egressrouter Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 15s default-scheduler Successfully assigned test/egressrouter-redirect-pod to weliang282-r62kn-worker-c-zbwvm.c.openshift-qe.internal Normal AddedInterface 14s multus Add eth0 [10.129.2.25/23] Normal Pulling 13s kubelet Pulling image "registry.redhat.io/openshift4/ose-egress-router" Normal Pulled 5s kubelet Successfully pulled image "registry.redhat.io/openshift4/ose-egress-router" in 8.107812392s Normal Created 4s (x2 over 5s) kubelet Created container egress-router Normal Started 4s (x2 over 5s) kubelet Started container egress-router Normal Pulled 4s kubelet Container image "registry.redhat.io/openshift4/ose-egress-router" already present on machine Warning BackOff 2s (x2 over 3s) kubelet Back-off restarting failed container [weliang@weliang ~]$ oc logs -c egress-router egressrouter-redirect-pod /bin/egress-router.sh: line 113: iptables: command not found [weliang@weliang ~]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2021-02-08-052658 True False 21m Cluster version is 4.7.0-0.nightly-2021-02-08-052658 [weliang@weliang ~]$
Tested and verified in 4.7.0-0.nightly-2021-03-01-085007
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.7.2 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0749