Bug 1453113 - all veth cannot be recovered after restarting openvswitch service
Summary: all veth cannot be recovered after restarting openvswitch service
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.7.0
Assignee: Scott Dodson
QA Contact: Hongan Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-22 08:00 UTC by Hongan Li
Modified: 2017-11-28 21:55 UTC (History)
17 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously the node service was not restarted when openvswitch was restarted which could result in a misconfigured networking environment. The services have been updated to ensure that the node service is restarted whenever openvswitch is restarted.
Clone Of:
Environment:
Last Closed: 2017-11-28 21:55:46 UTC
Target Upstream Version:


Attachments (Terms of Use)
node logs (296.46 KB, application/x-gzip)
2017-07-12 07:39 UTC, Hongan Li
no flags Details
nodelog without restarting ovs (26.83 KB, text/plain)
2017-07-14 07:14 UTC, Meng Bo
no flags Details
nodelog during restart the ovs (28.19 KB, text/plain)
2017-07-14 07:15 UTC, Meng Bo
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description Hongan Li 2017-05-22 08:00:58 UTC
Description of problem:
See error "could not open network device veth46986ce1 (No such device)" after OCP installed. 
And after restart openvswitch, all veth cannot be recovered.

workaround: after restart docker the veth come back.

Version-Release number of selected component (if applicable):
openshift v3.6.78
kubernetes v1.6.1+5115d708d7
etcd 3.1.0
openvswitch-2.6.1-10.git20161206.el7fdp.x86_64

How reproducible:
always

Steps to Reproduce:
1. check ovs-vsctl after fresh install OCP (some pods have been created).
2. systemctl restart openvswitch

Actual results:
step1: 
# ovs-vsctl show
5944be01-d4e7-4677-80d8-ec1df51e47f0
    Bridge "br0"
        fail_mode: secure
        Port "veth46986ce1"
            Interface "veth46986ce1"
                error: "could not open network device veth46986ce1 (No such device)"
        Port "vetha60c6ed6"
            Interface "vetha60c6ed6"
                error: "could not open network device vetha60c6ed6 (No such device)"
        Port "veth0b6b4150"
            Interface "veth0b6b4150"
                error: "could not open network device veth0b6b4150 (No such device)"
        Port "veth2b24d77c"
            Interface "veth2b24d77c"
        Port "vxlan0"
-----------------------<snip>----------------

step2:
# ovs-ofctl show  br0 -O openflow13
OFPT_FEATURES_REPLY (OF1.3) (xid=0x2): dpid:00009e7f42f35a48
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS GROUP_STATS QUEUE_STATS
OFPST_PORT_DESC reply (OF1.3) (xid=0x3):
 1(vxlan0): addr:ba:ef:a1:9c:31:1d
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(tun0): addr:0e:56:5e:a9:8b:1b
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br0): addr:9e:7f:42:f3:5a:48
     config:     PORT_DOWN
     state:      LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (OF1.3) (xid=0x5): frags=nx-match miss_send_len=0


Expected results:
1. should no error like (No such device) after OCP installed.
2. veth should be recovered immediately after restarting openvswitch 

Additional info:

Comment 1 Johnny Liu 2017-05-26 08:25:54 UTC
After restart node service, all the existing pod are not running.

# oc get po
NAME                        READY     STATUS    RESTARTS   AGE
docker-registry-1-4mchj     1/1       Running   0          1h

# service atomic-openshift-node restart

# oc get po
NAME                        READY     STATUS             RESTARTS   AGE
docker-registry-1-4mchj     0/1       CrashLoopBackOff   4          1h

# oc describe po docker-registry-1-4mchj
<--snip-->
Events:
  FirstSeen	LastSeen	Count	From								SubObjectPath			Type		Reason		Message
  ---------	--------	-----	----								-------------			--------	------		-------
  1h		1h		1	default-scheduler										Normal		Scheduled	Successfully assigned docker-registry-1-4mchj to host-8-175-8.host.centralci.eng.rdu2.redhat.com
  1h		1h		1	kubelet, host-8-175-8.host.centralci.eng.rdu2.redhat.com	spec.containers{registry}	Normal		Pulling		pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-docker-registry:v3.6.84"
  1h		1h		1	kubelet, host-8-175-8.host.centralci.eng.rdu2.redhat.com	spec.containers{registry}	Normal		Pulled		Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-docker-registry:v3.6.84"
  1h		1h		1	kubelet, host-8-175-8.host.centralci.eng.rdu2.redhat.com	spec.containers{registry}	Normal		Created		Created container with id 13f2f8c0dda48065e199683800079a3aa5be393c9f41694df252d1203e287354
  1h		1h		1	kubelet, host-8-175-8.host.centralci.eng.rdu2.redhat.com	spec.containers{registry}	Normal		Started		Started container with id 13f2f8c0dda48065e199683800079a3aa5be393c9f41694df252d1203e287354
  16m		16m		1	kubelet, host-8-175-8.host.centralci.eng.rdu2.redhat.com	spec.containers{registry}	Warning		Unhealthy	Liveness probe failed: Get https://10.128.0.4:5000/healthz: http2: no cached connection was available
  23s		23s		1	kubelet, host-8-175-8.host.centralci.eng.rdu2.redhat.com	spec.containers{registry}	Normal		Started		Started container with id 20dcc80dfb76024def7ff88f80159830f26dd7e0a4eadd10e485ab5bf53fac34
  23s		23s		1	kubelet, host-8-175-8.host.centralci.eng.rdu2.redhat.com	spec.containers{registry}	Normal		Created		Created container with id 20dcc80dfb76024def7ff88f80159830f26dd7e0a4eadd10e485ab5bf53fac34
  24s		4s		2	kubelet, host-8-175-8.host.centralci.eng.rdu2.redhat.com	spec.containers{registry}	Normal		Pulled		Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-docker-registry:v3.6.84" already present on machine
  14s		4s		2	kubelet, host-8-175-8.host.centralci.eng.rdu2.redhat.com	spec.containers{registry}	Warning		Unhealthy	Readiness probe failed: Get https://10.128.0.4:5000/healthz: dial tcp 10.128.0.4:5000: getsockopt: no route to host
  4s		4s		1	kubelet, host-8-175-8.host.centralci.eng.rdu2.redhat.com	spec.containers{registry}	Warning		Unhealthy	Liveness probe failed: Get https://10.128.0.4:5000/healthz: dial tcp 10.128.0.4:5000: getsockopt: no route to host
  4s		4s		1	kubelet, host-8-175-8.host.centralci.eng.rdu2.redhat.com	spec.containers{registry}	Normal		Killing		Killing container with id docker://20dcc80dfb76024def7ff88f80159830f26dd7e0a4eadd10e485ab5bf53fac34:pod "docker-registry-1-4mchj_default(6c070063-41e0-11e7-a809-fa163e308807)" container "registry" is unhealthy, it will be killed and re-created.
  3s		3s		1	kubelet, host-8-175-8.host.centralci.eng.rdu2.redhat.com	spec.containers{registry}	Normal		Created		Created container with id 43b9471c49f82c440d93ee9ec38b18454aa1e12f98998a78a320164bc3e9654d
  3s		3s		1	kubelet, host-8-175-8.host.centralci.eng.rdu2.redhat.com	spec.containers{registry}	Normal		Started		Started container with id 43b9471c49f82c440d93ee9ec38b18454aa1e12f98998a78a320164bc3e9654d
<--snip-->

Seem like my issue has the same cause root in this bug's initial report.

Rising its priority.

Comment 2 Dan Williams 2017-05-30 20:14:43 UTC
Note that we do not support restarting openvswitch underneath openshift.

Restarting openshift-node itself, however, should be able to recover.

Comment 3 Ravi Sankar 2017-05-30 20:18:39 UTC
*** Bug 1453190 has been marked as a duplicate of this bug. ***

Comment 4 Hongan Li 2017-05-31 05:43:05 UTC
restarting openvswitch is supported in 3.5 and before, and we had a test case for https://bugzilla.redhat.com/show_bug.cgi?id=1316202.

Comment 5 Ben Bennett 2017-06-02 18:36:06 UTC
Can you run: systemctl cat atomic-openshift-node.service

Mine has:
# /usr/lib/systemd/system/atomic-openshift-node.service.d/openshift-sdn-ovs.conf
[Unit]
Requires=openvswitch.service
After=ovsdb-server.service
After=ovs-vswitchd.service
After=openvswitch.service

But that file comes from atomic-openshift-sdn-ovs-3.6.65-1.git.0.1c96bcd.el7.x86_64, so is done by the productization team.  If that file is missing, then this may be a productization bug.  But if you installed from source then that may not be part of the rpm.

Comment 6 Dan Williams 2017-06-03 03:03:57 UTC
To clarify, we support restarting OVS if-and-only-if OpenShift is restarted immediately after OVS.  We don't support restarting OVS and then restarting OpenShift later.  The file that Ben refers to attempts to ensure that OpenShift is restarted when OVS does.

Comment 7 Meng Bo 2017-06-05 06:35:56 UTC
Yes, the unit files contains the ovs related services as comment#5

# systemctl cat atomic-openshift-node.service
...
...
# /usr/lib/systemd/system/atomic-openshift-node.service.d/openshift-sdn-ovs.conf
[Unit]
Requires=openvswitch.service
After=ovsdb-server.service
After=ovs-vswitchd.service
After=openvswitch.service


And the openshift service will be restarted when the ovs gets restarted.

# ps -ef | grep openshift
root      13444      1  1 Jun01 ?        01:37:35 /usr/bin/openshift start node --config=/etc/origin/node/node-config.yaml --loglevel=0
root     128640   1221  0 14:11 pts/0    00:00:00 grep --color=auto openshift
# systemctl restart openvswitch.service
# ps -ef | grep openshift
root     128922      1 12 14:11 ?        00:00:00 /usr/bin/openshift start node --config=/etc/origin/node/node-config.yaml --loglevel=0
root     129119   1221  0 14:11 pts/0    00:00:00 grep --color=auto openshift


So the problem here should be, all the veth are missing when the openshift-node service restarted.

Comment 8 Dan Winship 2017-06-13 15:45:22 UTC
(In reply to Johnny Liu from comment #1)
> After restart node service, all the existing pod are not running.

> # service atomic-openshift-node restart

> Seem like my issue has the same cause root in this bug's initial report.

No, that's something different (https://github.com/openshift/origin/pull/14446, which doesn't seem to have a bugzilla bug)

Comment 9 Dan Winship 2017-06-13 16:06:09 UTC
OK, so the problem here (currently, which is slightly different from when this was originally filed before PR 14446 merged) is that if openshift does a full network re-setup on restart (which it will have to do if you restarted openvswitch), then the UpdatePod() loop will fail because ovscontroller.UpdatePod() tries to get the containerID out of the OVS flows, but the OVS flows won't be there. (There's also a second problem, which is that ovscontroller.UpdatePod() doesn't call ensureOvsPort() any more, which would require it to know the veth name, which it doesn't know.)

dcbw: thoughts?

Comment 10 Hongan Li 2017-06-14 06:30:00 UTC
tested in latest build atomic-openshift-3.6.106-1.git.0.1072f4f.el7.x86_64 (PR 14446 has been merged) and result as below:
1. restart node service: OK
2. restart openvswitch:  veth cannot be recovered

Comment 11 Dan Williams 2017-06-15 02:35:15 UTC
Upstream fix pull request: https://github.com/openshift/origin/pull/14665

Comment 12 Ben Bennett 2017-06-19 15:34:23 UTC
*** Bug 1461709 has been marked as a duplicate of this bug. ***

Comment 14 Hongan Li 2017-07-05 07:56:20 UTC
verified in atomic-openshift-3.6.133-1.git.0.524e4c8.el7.x86_64 and the issue has been fixed.

Comment 15 Hongan Li 2017-07-11 09:59:29 UTC
Re-open it since the issue still can be reproduced in containerized installation env. Previous verification on RPM installation env is passed.

Comment 16 Dan Williams 2017-07-11 18:08:41 UTC
(In reply to hongli from comment #15)
> Re-open it since the issue still can be reproduced in containerized
> installation env. Previous verification on RPM installation env is passed.

Can you get openshift-node logs with --loglevel=5 when this fails?

Comment 17 Hongan Li 2017-07-12 07:39:06 UTC
Created attachment 1296760 [details]
node logs

Comment 18 Hongan Li 2017-07-12 07:48:30 UTC
reproduced in containerized env build openshift v3.6.140 and attached the node logs.

test steps:
1. restart atomic-openshift-node service, OK
2. restart openvswitch service, failed to connect Pods

Checked OVS then found all openflow rules gone but veth still there, see below:

[root@qe-hongli-bugv-node-registry-router-1 ~]# docker exec openvswitch ovs-ofctl dump-flows br0 -O openflow13
OFPST_FLOW reply (OF1.3) (xid=0x2):

[root@qe-hongli-bugv-node-registry-router-1 ~]# docker exec openvswitch ovs-vsctl show
ba9be912-eaac-4e9d-ae91-8e50b3b0d0bd
    Bridge "br0"
        fail_mode: secure
        Port "tun0"
            Interface "tun0"
                type: internal
        Port "vxlan0"
            Interface "vxlan0"
                type: vxlan
                options: {key=flow, remote_ip=flow}
        Port "veth4c05d454"
            Interface "veth4c05d454"
        Port "br0"
            Interface "br0"
                type: internal
        Port "veth60eb808b"
            Interface "veth60eb808b"
        Port "vethee7b5367"
            Interface "vethee7b5367"
        Port "vethcbe6fe7d"
            Interface "vethcbe6fe7d"
    ovs_version: "2.6.1"

Comment 19 Dan Williams 2017-07-12 18:41:21 UTC
(In reply to hongli from comment #18)
> reproduced in containerized env build openshift v3.6.140 and attached the
> node logs.
> 
> test steps:
> 1. restart atomic-openshift-node service, OK
> 2. restart openvswitch service, failed to connect Pods

Just to be clear; we do not support restarting OVS after restarting OpenShift.  

We do support restarting OVS if OpenShift is restarted right after.

Are you still able to reproduce if you:

test steps:
1. restart atomic-openshift-node service, OK
2. restart openvswitch service, failed to connect Pods
3. restart atomic-openshift-node service

Comment 20 Hongan Li 2017-07-14 01:39:40 UTC
(In reply to Dan Williams from comment #19)
> 
> Just to be clear; we do not support restarting OVS after restarting
> OpenShift.  
> 
> We do support restarting OVS if OpenShift is restarted right after.
> 
> Are you still able to reproduce if you:
> 
> test steps:
> 1. restart atomic-openshift-node service, OK
> 2. restart openvswitch service, failed to connect Pods
> 3. restart atomic-openshift-node service

Thanks Dan, the test results are different in RPM and containerized installation env:

in RPM installation env:

1. systemctl restart openvswitch (it is OK, pod's IP changed and can reach pods after ovs restarted)

in containerized installation env:
1. systemctl restart openvswitch (pod's IP not changed and cannot reach pod)
2. systemctl restart atomic-openshift-node (it is OK, pod's IP changed and can reach pod)

Is this as expected? Why we need restart node service in containerized env but not need it in RPM env?

Comment 21 Dan Williams 2017-07-14 05:01:45 UTC
(In reply to hongli from comment #20)
> (In reply to Dan Williams from comment #19)
> > 
> > Just to be clear; we do not support restarting OVS after restarting
> > OpenShift.  
> > 
> > We do support restarting OVS if OpenShift is restarted right after.
> > 
> > Are you still able to reproduce if you:
> > 
> > test steps:
> > 1. restart atomic-openshift-node service, OK
> > 2. restart openvswitch service, failed to connect Pods
> > 3. restart atomic-openshift-node service
> 
> Thanks Dan, the test results are different in RPM and containerized
> installation env:
> 
> in RPM installation env:
> 
> 1. systemctl restart openvswitch (it is OK, pod's IP changed and can reach
> pods after ovs restarted)
> 
> in containerized installation env:
> 1. systemctl restart openvswitch (pod's IP not changed and cannot reach pod)
> 2. systemctl restart atomic-openshift-node (it is OK, pod's IP changed and
> can reach pod)
> 
> Is this as expected? Why we need restart node service in containerized env
> but not need it in RPM env?

That's quite interesting and unexpected.  Could you grab logs from openshift-node over the time when you do the openvswitch restart?

Comment 22 Meng Bo 2017-07-14 07:14:46 UTC
Created attachment 1298142 [details]
nodelog without restarting ovs

Comment 23 Meng Bo 2017-07-14 07:15:29 UTC
Created attachment 1298143 [details]
nodelog during restart the ovs

Comment 24 Meng Bo 2017-07-14 07:15:49 UTC
The nodelog.1 is node running without restart ovs, the nodelog.2 is the log during the ovs being restarted.

I did not see much difference on them, except one warning about conversion.go

Logs attached.

Comment 25 Dan Williams 2017-07-14 22:39:05 UTC
Thanks, I also don't see much of a difference.  I seem to recall that we have some kind of dependency in the systemd unit files for non-containerized OpenShift.  So I looked around and came across https://bugzilla.redhat.com/show_bug.cgi?id=1316202#c4 which says that due to the Requires+Restart directives in the non-containerized OpenShift node unit file, systemd should restart openshift automatically when openvswitch goes down.

However, I cannot reproduce that behavior.  So I guess we continue to recommend (like in https://docs.openshift.com/container-platform/3.4/install_config/upgrading/manual_upgrades.html.

Looking further, I *can* make openshift restart when OVS restarts by putting "Requires=openvswitch.service" into the system unit file for the node.

Ben/Eric, should we do that for all the unit files in origin tree and ansible?

Comment 26 Weibin Liang 2017-07-17 14:40:40 UTC
Dan, try to understand the order of restarting OVS and openshif-nodes, are my
expectations below correct?

1.Restart OVS only: OK
2.Restart node service only: OK
3.Restart OVS just after node service restarting: OK
4.Restart node service just after OVS restarting: OK
5.Restart OVS, wait for a while, then restart node service: Not OK
6.Restart node service, wait for a while, then restart OVS: OK

Comment 27 Dan Williams 2017-07-17 19:19:07 UTC
(In reply to Weibin Liang from comment #26)
> Dan, try to understand the order of restarting OVS and openshif-nodes, are my
> expectations below correct?

I think it's easier to simply say:

"Whenever you restart OVS, you must restart the node service immediately after."

With that in mind, (1), (5), and (6) should be "not OK".

Comment 28 Weibin Liang 2017-07-17 20:14:15 UTC
(In reply to hongli from comment #20)

> in RPM installation env:
> 
> 1. systemctl restart openvswitch (it is OK, pod's IP changed and can reach
> pods after ovs restarted)
> 

Then above test result is not clear:

hongli, did you restart OVS only? or restart OVS then restart node service immediately after?

Comment 29 Hongan Li 2017-07-18 02:34:39 UTC
(In reply to Weibin Liang from comment #28)
> (In reply to hongli from comment #20)
> 
> > in RPM installation env:
> > 
> > 1. systemctl restart openvswitch (it is OK, pod's IP changed and can reach
> > pods after ovs restarted)
> > 
> 
> Then above test result is not clear:
> 
> hongli, did you restart OVS only? or restart OVS then restart node service
> immediately after?

in RPM installation env, you just need restart OVS only, and you can see the node service is restarted automatically when you restart OVS.

but in containerized env, when you restart OVS then only OVS service is restarted, it won't trigger the node services restarting. This is the problem.

Please help confirm.

Comment 30 Dan Williams 2017-07-20 20:30:08 UTC
So it seems we *used* to have Requires=openvswitch.service in the containerized installs, which would correctly restart openshift-node.  But that was changed to Wants back in May due to:

https://bugzilla.redhat.com/show_bug.cgi?id=1451192
https://github.com/openshift/openshift-ansible/pull/4213

Comment 31 Ben Bennett 2017-09-12 18:22:56 UTC
Scott: Can you help here?  It looks like we have to pick which of two bugs we want to have :-(

Comment 32 Scott Dodson 2017-09-12 19:40:19 UTC
https://github.com/openshift/openshift-ansible/pull/4820 some discussion of other possible solutions here. Asking Giuseppe to look into this

Comment 33 Ben Bennett 2017-10-05 18:17:31 UTC
Scott: Any news?  Thanks!

Comment 34 Scott Dodson 2017-10-06 19:14:19 UTC
https://github.com/openshift/openshift-ansible/pull/4820 merged 9 hours ago

Comment 36 Hongan Li 2017-10-12 05:27:21 UTC
Tested in latest OCP build v3.6.173.0.45 but PR not merged yet.

according to the PR and adding "PartOf=openvswitch.service" in /etc/systemd/system/atomic-openshift-node.service, it can fix the issue in container env.

will re-tested with next build to see if merged and fixed.

Comment 38 Hongan Li 2017-10-13 05:31:07 UTC
verified in atomic-openshift-3.6.173.0.49-1.git.0.0b9377a but sill failed. 

Checked below file but no "PartOf=openvswitch.service" in it.
# cat /etc/systemd/system/atomic-openshift-node.service
[Unit]
After=atomic-openshift-master.service
After=docker.service
After=openvswitch.service
PartOf=docker.service
Requires=docker.service
Wants=openvswitch.service
After=ovsdb-server.service
After=ovs-vswitchd.service
Wants=atomic-openshift-master.service
Requires=atomic-openshift-node-dep.service
After=atomic-openshift-node-dep.service
Requires=dnsmasq.service
After=dnsmasq.service

Comment 41 Ben Bennett 2017-10-17 15:52:23 UTC
Scott: Do we want to backport that fix?

Comment 44 Hongan Li 2017-10-20 07:31:48 UTC
verified in v3.7.0-0.153.0 containerized env and the bug has been fixed.

see also https://bugzilla.redhat.com/show_bug.cgi?id=1453113#c40

Comment 47 errata-xmlrpc 2017-11-28 21:55:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.