Bug 1565222

Weibin, can you try to reproduce this?  Unfortunately I can't tell what time things started working from the associated information, so I can't correlate things to the logs.

Justin: If you see this happen again, can you grab the output from:
  iptables-save
  ovs-ofctl dump-flows br0 -O OpenFlow13
  oc get ep <service-name>

This did happen again on east-2a's upgrade today. Unfortunately, it occurred before your request. I worked around the issue this time by restarting atomic-openshift-node on all the masters. 
Almost immediately after restarting the node process on one of the masters, the webconsole check by openshift-ansible was satisfied and moved on.

Reproduced above issue in ec2 setup when upgrade v3.7 to v3.9

The reason for curl failed is because web-console pod use wrong repository( registry.access.redhat.com/openshift3/ose-web-console) after update v3.9.

After updating web-console pod to use new repository(registry.reg-aws.openshift.com:443/openshift3/ose-web-console), pod running correctly and curl passed.

#### Test log:

After upgrade to v3.9

[root@ip-172-18-2-112 ~]# oc version
oc v3.9.24
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-2-112.ec2.internal:8443
openshift v3.9.24
kubernetes v1.9.1+a0ce1bc657
[root@ip-172-18-2-112 ~]# 

[root@ip-172-18-2-112 ~]# curl -k https://webconsole.openshift-web-console.svc/healthz
curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused
[root@ip-172-18-2-112 ~]# oc project openshift-web-console
Already on project "openshift-web-console" on server "https://ip-172-18-2-112.ec2.internal:8443".
[root@ip-172-18-2-112 ~]# oc get all
NAME                DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/webconsole   1         1         1            0           8m

NAME                     DESIRED   CURRENT   READY     AGE
rs/webconsole-894b9c44   1         1         0         8m

NAME                           READY     STATUS             RESTARTS   AGE
po/webconsole-894b9c44-qbqm7   0/1       ImagePullBackOff   0          8m

NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
svc/webconsole   ClusterIP   172.30.30.34   <none>        443/TCP   8m
[root@ip-172-18-2-112 ~]# oc describe po/webconsole-894b9c44-qbqm7
Name:           webconsole-894b9c44-qbqm7
Namespace:      openshift-web-console
Node:           ip-172-18-2-112.ec2.internal/172.18.2.112
Start Time:     Fri, 20 Apr 2018 15:27:23 -0400
Labels:         pod-template-hash=45065700
                webconsole=true
Annotations:    openshift.io/scc=restricted
Status:         Pending
IP:             10.129.0.11
Controlled By:  ReplicaSet/webconsole-894b9c44
Containers:
  webconsole:
    Container ID:  
    Image:         registry.access.redhat.com/openshift3/ose-web-console:v3.9.24
    Image ID:      
    Port:          8443/TCP
    Command:
      /usr/bin/origin-web-console
      --audit-log-path=-
      -v=0
      --config=/var/webconsole-config/webconsole-config.yaml
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:     100m
      memory:  100Mi
    Liveness:  exec [/bin/sh -i -c if [[ ! -f /tmp/webconsole-config.hash ]]; then \
  md5sum /var/webconsole-config/webconsole-config.yaml > /tmp/webconsole-config.hash; \
elif [[ $(md5sum /var/webconsole-config/webconsole-config.yaml) != $(cat /tmp/webconsole-config.hash) ]]; then \
  exit 1; \
fi && curl -k -f https://0.0.0.0:8443/console/] delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:8443/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from webconsole-token-6m7gz (ro)
      /var/serving-cert from serving-cert (rw)
      /var/webconsole-config from webconsole-config (rw)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  serving-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  webconsole-serving-cert
    Optional:    false
  webconsole-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      webconsole-config
    Optional:  false
  webconsole-token-6m7gz:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  webconsole-token-6m7gz
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/master=true
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
Events:
  Type     Reason                 Age               From                                   Message
  ----     ------                 ----              ----                                   -------
  Normal   Scheduled              8m                default-scheduler                      Successfully assigned webconsole-894b9c44-qbqm7 to ip-172-18-2-112.ec2.internal
  Normal   SuccessfulMountVolume  8m                kubelet, ip-172-18-2-112.ec2.internal  MountVolume.SetUp succeeded for volume "webconsole-config"
  Normal   SuccessfulMountVolume  8m                kubelet, ip-172-18-2-112.ec2.internal  MountVolume.SetUp succeeded for volume "webconsole-token-6m7gz"
  Warning  FailedMount            8m (x2 over 8m)   kubelet, ip-172-18-2-112.ec2.internal  MountVolume.SetUp failed for volume "serving-cert" : secrets "webconsole-serving-cert" not found
  Normal   SuccessfulMountVolume  8m                kubelet, ip-172-18-2-112.ec2.internal  MountVolume.SetUp succeeded for volume "serving-cert"
  Normal   Pulling                8m (x2 over 8m)   kubelet, ip-172-18-2-112.ec2.internal  pulling image "registry.access.redhat.com/openshift3/ose-web-console:v3.9.24"
  Warning  Failed                 8m (x2 over 8m)   kubelet, ip-172-18-2-112.ec2.internal  Failed to pull image "registry.access.redhat.com/openshift3/ose-web-console:v3.9.24": rpc error: code = Unknown desc = error parsing HTTP 404 response body: invalid character 'F' looking for beginning of value: "File not found.\""
  Warning  Failed                 8m (x2 over 8m)   kubelet, ip-172-18-2-112.ec2.internal  Error: ErrImagePull
  Normal   SandboxChanged         7m (x6 over 8m)   kubelet, ip-172-18-2-112.ec2.internal  Pod sandbox changed, it will be killed and re-created.
  Normal   BackOff                7m (x4 over 8m)   kubelet, ip-172-18-2-112.ec2.internal  Back-off pulling image "registry.access.redhat.com/openshift3/ose-web-console:v3.9.24"
  Warning  Failed                 3m (x22 over 8m)  kubelet, ip-172-18-2-112.ec2.internal  Error: ImagePullBackOff
[root@ip-172-18-2-112 ~]# 


## Workaround:
[root@ip-172-18-2-112 ~]#oc describe pods router-1-w6t24 -n default|grep Image:
Image:          registry.reg-aws.openshift.com:443/openshift3/ose-haproxy-router:v3.7.44
# update registry.reg-aws.openshift.com:443/openshift3/ose-web-console in pod
[root@ip-172-18-2-112 ~]# oc edit po/webconsole-894b9c44-qbqm7
deployment "webconsole" edited

[root@ip-172-18-2-112 ~]# oc get pod
NAME                        READY     STATUS    RESTARTS   AGE
webconsole-894b9c44-28t4m   1/1       Running   0          5m
[root@ip-172-18-2-112 ~]# curl -k https://webconsole.openshift-web-console.svc/healthz
ok
[root@ip-172-18-2-112 ~]#

In the starter environments we've seen the problem resolve itself with no further action on our part. We also see it immediately resolved by restarting the node service. In these environments the images were always set properly so I don't think comment 7 is the same problem. If I had to guess I imagine this is iptables randomly burning CPU while propagating services.

According to Justin this has happened at least twice.

If @sdodson's hunch is right then this is caused by https://bugzilla.redhat.com/show_bug.cgi?id=1569128

I believe that this did not happen with the 3.10 upgrade, and it has not been reported by any customers going to 3.9.