Bug 1896170 - registry.redhat.io/openshift4/ose-pod image dose not work
Summary: registry.redhat.io/openshift4/ose-pod image dose not work
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.7
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.7.0
Assignee: Stephen Greene
QA Contact: Hongan Li
URL:
Whiteboard:
Depends On:
Blocks: 1926267
TreeView+ depends on / blocked
 
Reported: 2020-11-09 21:19 UTC by Weibin Liang
Modified: 2022-08-04 22:30 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-16 08:42:46 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift images pull 55 0 None closed Bug 1896170: Egress router: Add iptables package to Dockerfile 2021-02-19 17:24:43 UTC
Red Hat Product Errata RHBA-2021:0749 0 None None None 2021-03-16 08:43:14 UTC

Description Weibin Liang 2020-11-09 21:19:06 UTC
Description of problem:
registry.redhat.io/openshift4/ose-pod image not work registry.redhat.io/openshift3/ose-pod image work


Version-Release number of selected component (if applicable):
4.7.0-0.nightly-2020-10-27-051128

How reproducible:
Always

Steps to Reproduce:
[weliang@weliang ~]$ diff egressrouter-1.yaml egressrouter-2.yaml
12c12
<     image: registry.redhat.io/openshift3/ose-egress-router
---
>     image: registry.redhat.io/openshift4/ose-egress-router
27c27
<     image: registry.redhat.io/openshift3/ose-pod
---
>     image: registry.redhat.io/openshift4/ose-pod
[weliang@weliang ~]$ oc create -f egressrouter-1.yaml
pod/egressrouter-redirect-pod created
[weliang@weliang ~]$ oc get pod
NAME                        READY   STATUS    RESTARTS   AGE
egressrouter-redirect-pod   1/1     Running   0          60s
[weliang@weliang ~]$ oc delete pod egressrouter-redirect-pod
pod "egressrouter-redirect-pod" deleted
[weliang@weliang ~]$ oc create -f egressrouter-2.yaml
pod/egressrouter-redirect-pod created
[weliang@weliang ~]$ oc get pod
NAME                        READY   STATUS                  RESTARTS   AGE
egressrouter-redirect-pod   0/1     Init:CrashLoopBackOff   1          13s
[weliang@weliang ~]$ oc logs egressrouter-redirect-pod
Error from server (BadRequest): container "egress-pod" in pod "egressrouter-redirect-pod" is waiting to start: PodInitializing
[weliang@weliang ~]$ oc describe pod egressrouter-redirect-pod
Name:         egressrouter-redirect-pod
Namespace:    test
Priority:     0
Node:         ip-10-0-147-69.us-east-2.compute.internal/10.0.147.69
Start Time:   Mon, 09 Nov 2020 16:00:11 -0500
Labels:       name=egressrouter-redirect-pod
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "",
                    "interface": "eth0",
                    "ips": [
                        "10.128.2.16"
                    ],
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "",
                    "interface": "eth0",
                    "ips": [
                        "10.128.2.16"
                    ],
                    "default": true,
                    "dns": {}
                }]
              openshift.io/scc: node-exporter
              pod.network.openshift.io/assign-macvlan: true
Status:       Pending
IP:           10.128.2.16
IPs:
  IP:  10.128.2.16
Init Containers:
  egress-router:
    Container ID:   cri-o://57ec8e485d567526f54caf09614814f2f1872e342be822cf85b6f1ed3e65542f
    Image:          registry.redhat.io/openshift4/ose-egress-router
    Image ID:       registry.redhat.io/openshift4/ose-egress-router@sha256:079afa6fd2f8e25484b8963ce716942ba9147243168142ee924265b9218dda5b
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Error
      Exit Code:    127
      Started:      Mon, 09 Nov 2020 16:00:30 -0500
      Finished:     Mon, 09 Nov 2020 16:00:30 -0500
    Last State:     Terminated
      Reason:       Error
      Exit Code:    127
      Started:      Mon, 09 Nov 2020 16:00:14 -0500
      Finished:     Mon, 09 Nov 2020 16:00:14 -0500
    Ready:          False
    Restart Count:  2
    Environment:
      EGRESS_SOURCE:       172.31.249.146
      EGRESS_GATEWAY:      172.31.248.1
      EGRESS_DESTINATION:  172.217.7.142
      EGRESS_ROUTER_MODE:  init
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kpdh8 (ro)
Containers:
  egress-pod:
    Container ID:   
    Image:          registry.redhat.io/openshift4/ose-pod
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kpdh8 (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-kpdh8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kpdh8
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason          Age                From                                                Message
  ----     ------          ----               ----                                                -------
  Normal   Scheduled       <unknown>                                                              Successfully assigned test/egressrouter-redirect-pod to ip-10-0-147-69.us-east-2.compute.internal
  Normal   AddedInterface  29s                multus                                              Add eth0 [10.128.2.16/23]
  Normal   Pulled          12s (x3 over 29s)  kubelet, ip-10-0-147-69.us-east-2.compute.internal  Container image "registry.redhat.io/openshift4/ose-egress-router" already present on machine
  Normal   Created         12s (x3 over 29s)  kubelet, ip-10-0-147-69.us-east-2.compute.internal  Created container egress-router
  Normal   Started         12s (x3 over 29s)  kubelet, ip-10-0-147-69.us-east-2.compute.internal  Started container egress-router
  Warning  BackOff         12s (x3 over 27s)  kubelet, ip-10-0-147-69.us-east-2.compute.internal  Back-off restarting failed container
[weliang@weliang ~]$ oc logs -c egress-router egressrouter-redirect-pod
/bin/egress-router.sh: line 113: iptables: command not found
[weliang@weliang ~]$ 

Actual results:
egressrouter-redirect-pod   0/1     Init:CrashLoopBackOff

Expected results:
egressrouter-redirect-pod   1/1     Running   0          60s

Additional info:
registry.redhat.io/openshift4/ose-egress-dns-proxy works
registry.redhat.io/openshift4/ose-egress-http-proxy works

Comment 1 Oleg Bulatov 2020-11-10 12:52:00 UTC
It seems the problematic image is registry.redhat.io/openshift4/ose-egress-router. Is it Routing?

Comment 2 Weibin Liang 2020-11-10 14:17:17 UTC
(In reply to Oleg Bulatov from comment #1)
> It seems the problematic image is
> registry.redhat.io/openshift4/ose-egress-router. Is it Routing?

From below test results, I do not think it's a routing problem: 

1.Test case one pass with below two images in pod yam file:
registry.redhat.io/openshift4/ose-egress-router + registry.redhat.io/openshift4/ose-egress-dns-proxy works
(https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/EgressRouter/egress-router-dns-pod.yaml)

2.Test case two pass with below two images in pod yam file:
registry.redhat.io/openshift4/ose-egress-router + registry.redhat.io/openshift4/ose-egress-http-proxy works

3.Test case three pass with below two images in pod yam file:
registry.redhat.io/openshift4/ose-egress-router + registry.redhat.io/openshift3/ose-pod

4.Test case four FAIL with below two images in pod yam file:
registry.redhat.io/openshift4/ose-egress-router + registry.redhat.io/openshift4/ose-pod

Comment 3 Miciah Dashiel Butler Masters 2020-11-12 17:13:10 UTC
We need to look into whether (a) registry.redhat.io/openshift4/ose-pod is the correct pullspec in v4 and (b) whether we are failing to ship the image when we should be shipping it.

Comment 7 Weibin Liang 2020-12-15 21:40:11 UTC
Got same error as before in the latest v4.7 nightly image, I will try one time tomorrow before I re assign this bug.

[weliang@weliang debug-scripts]$ oc logs -c egress-router egressrouter-redirect-pod
/bin/egress-router.sh: line 113: iptables: command not found
[weliang@weliang debug-scripts]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2020-12-14-165231   True        False         5h4m    Cluster version is 4.7.0-0.nightly-2020-12-14-165231

Comment 12 Weibin Liang 2020-12-18 17:03:41 UTC
Verified failed on latest v4.7 nightly image

[weliang@weliang ~]$ oc logs -c egress-router egressrouter-redirect-pod
/bin/egress-router.sh: line 113: iptables: command not found
[weliang@weliang ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2020-12-18-031435   True        False         14m     Cluster version is 4.7.0-0.nightly-2020-12-18-031435
[weliang@weliang ~]$

Comment 14 Weibin Liang 2021-01-04 16:49:02 UTC
Hi Stephen,

Should I try v4.6 nightly image now? Thanks!

Comment 21 Weibin Liang 2021-02-08 19:32:56 UTC
Did not see the new PR.

Retested and failed in 4.7.0-0.nightly-2021-02-08-052658

[weliang@weliang ~]$ oc create -f https://raw.githubusercontent.com/weliang1/Openshift_Networking/master/Features/EgressRouter/test.yaml
pod/egressrouter-redirect-pod created
[weliang@weliang ~]$ oc get pod
NAME                        READY   STATUS     RESTARTS   AGE
egressrouter-redirect-pod   0/1     Init:0/1   0          5s
[weliang@weliang ~]$ oc get pod
NAME                        READY   STATUS     RESTARTS   AGE
egressrouter-redirect-pod   0/1     Init:0/1   0          7s
[weliang@weliang ~]$ oc get pod
NAME                        READY   STATUS     RESTARTS   AGE
egressrouter-redirect-pod   0/1     Init:0/1   0          9s
[weliang@weliang ~]$ oc describe pod egressrouter-redirect-pod
Name:         egressrouter-redirect-pod
Namespace:    test
Priority:     0
Node:         weliang282-r62kn-worker-c-zbwvm.c.openshift-qe.internal/10.0.32.4
Start Time:   Mon, 08 Feb 2021 14:30:12 -0500
Labels:       name=egressrouter-redirect-pod
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "",
                    "interface": "eth0",
                    "ips": [
                        "10.129.2.25"
                    ],
                    "default": true,
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "",
                    "interface": "eth0",
                    "ips": [
                        "10.129.2.25"
                    ],
                    "default": true,
                    "dns": {}
                }]
              openshift.io/scc: node-exporter
              pod.network.openshift.io/assign-macvlan: true
Status:       Pending
IP:           10.129.2.25
IPs:
  IP:  10.129.2.25
Init Containers:
  egress-router:
    Container ID:   cri-o://7299b7c5c507f99048511c07b0fddee5fc1b3300621af00e561b521e4a30a007
    Image:          registry.redhat.io/openshift4/ose-egress-router
    Image ID:       registry.redhat.io/openshift4/ose-egress-router@sha256:3b1c28d7880f1827d5329ae490021361e0b2a26f1ad6484c22a5729b56b752f4
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    127
      Started:      Mon, 08 Feb 2021 14:30:23 -0500
      Finished:     Mon, 08 Feb 2021 14:30:23 -0500
    Ready:          False
    Restart Count:  1
    Environment:
      EGRESS_SOURCE:       172.31.249.146
      EGRESS_GATEWAY:      172.31.248.1
      EGRESS_DESTINATION:  172.217.7.142
      EGRESS_ROUTER_MODE:  init
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-c4sx2 (ro)
Containers:
  egress-pod:
    Container ID:   
    Image:          registry.redhat.io/openshift4/ose-pod
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-c4sx2 (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-c4sx2:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-c4sx2
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  app=egressrouter
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age              From               Message
  ----     ------          ----             ----               -------
  Normal   Scheduled       15s              default-scheduler  Successfully assigned test/egressrouter-redirect-pod to weliang282-r62kn-worker-c-zbwvm.c.openshift-qe.internal
  Normal   AddedInterface  14s              multus             Add eth0 [10.129.2.25/23]
  Normal   Pulling         13s              kubelet            Pulling image "registry.redhat.io/openshift4/ose-egress-router"
  Normal   Pulled          5s               kubelet            Successfully pulled image "registry.redhat.io/openshift4/ose-egress-router" in 8.107812392s
  Normal   Created         4s (x2 over 5s)  kubelet            Created container egress-router
  Normal   Started         4s (x2 over 5s)  kubelet            Started container egress-router
  Normal   Pulled          4s               kubelet            Container image "registry.redhat.io/openshift4/ose-egress-router" already present on machine
  Warning  BackOff         2s (x2 over 3s)  kubelet            Back-off restarting failed container
[weliang@weliang ~]$ oc logs -c egress-router egressrouter-redirect-pod
/bin/egress-router.sh: line 113: iptables: command not found
[weliang@weliang ~]$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2021-02-08-052658   True        False         21m     Cluster version is 4.7.0-0.nightly-2021-02-08-052658
[weliang@weliang ~]$

Comment 24 Weibin Liang 2021-03-01 21:20:58 UTC
Tested and verified in 4.7.0-0.nightly-2021-03-01-085007

Comment 27 errata-xmlrpc 2021-03-16 08:42:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.2 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0749


Note You need to log in before you can comment on or make changes to this bug.