Bug 1388856 - SDN will always be fully setup when restarting node service
Summary: SDN will always be fully setup when restarting node service
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Dan Williams
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-26 10:18 UTC by Meng Bo
Modified: 2017-03-08 18:43 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2017-01-18 12:46:08 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Origin (Github) 11613 0 None None None 2016-10-31 15:55:59 UTC
Red Hat Product Errata RHBA-2017:0066 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.4 RPM Release Advisory 2017-01-18 17:23:26 UTC

Description Meng Bo 2016-10-26 10:18:13 UTC
Description of problem:
Setup multinode env, start the node service to initialize the SDN setup. Restart the node service without any operation on the node. The SDN will be fully setup on the 2nd time restart.

Version-Release number of selected component (if applicable):
openshift v3.4.0.15+9c963ec
kubernetes v1.4.0+776c994
etcd 3.1.0-alpha.1
docker version 1.12.1-6

How reproducible:
always

Steps to Reproduce:
1. Setup multi-node env
2. Start the node to get the SDN initialized
3. Restart the node service directly without any other operations
4. Check the node log

Actual results:
The SDN gets fully re-setup on the restart in step3.

Expected results:
Should not re-initialized the SDN if the network config is not changed.

Additional info:
[root@ose-node2 ~]# ip a 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:ec:0e:e9 brd ff:ff:ff:ff:ff:ff
    inet 10.66.140.9/23 brd 10.66.141.255 scope global dynamic eth0
       valid_lft 85315sec preferred_lft 85315sec
    inet6 fe80::5054:ff:feec:ee9/64 scope link 
       valid_lft forever preferred_lft forever
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether a6:81:f4:12:4d:19 brd ff:ff:ff:ff:ff:ff
6: br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether e2:8c:fa:1a:b0:4e brd ff:ff:ff:ff:ff:ff
10: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue master ovs-system state UNKNOWN qlen 1000
    link/ether ca:db:7c:77:36:6f brd ff:ff:ff:ff:ff:ff
11: tun0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN qlen 1000
    link/ether 6e:d1:d3:1f:ef:e0 brd ff:ff:ff:ff:ff:ff
    inet 10.131.0.1/23 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::6cd1:d3ff:fe1f:efe0/64 scope link 
       valid_lft forever preferred_lft forever
[root@ose-node2 ~]# ip r
default via 10.66.141.254 dev eth0  proto static  metric 100 
10.66.140.0/23 dev eth0  proto kernel  scope link  src 10.66.140.9  metric 100 
10.72.17.5 via 10.66.141.254 dev eth0  proto dhcp  metric 100 
10.128.0.0/14 dev tun0  proto kernel  scope link 
172.30.0.0/16 dev tun0  scope link 


Node log:
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.179868   11822 node.go:61] Initializing SDN node of type "redhat/openshift-ovs-subnet" with configured hostname "ose-node2.bmeng.local" (IP ""), iptables sync period "30s"
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.180199   11822 ipcmd.go:44] Executing: /usr/sbin/ip link set lbr0 down
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.184190   11822 ipcmd.go:48] Error executing /usr/sbin/ip: Cannot find device "lbr0"
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.188208   11822 start_node.go:303] Starting node ose-node2.bmeng.local (v3.4.0.15+9c963ec)
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.189267   11822 start_node.go:312] Connecting to API server https://10.66.140.11:8443
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.191899   11822 node.go:141] Connecting to Docker at unix:///var/run/docker.sock
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.237491   11822 controller.go:189] [SDN setup] node pod subnet 10.131.0.0/23 gateway 10.131.0.1
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.237523   11822 ipcmd.go:44] Executing: /usr/sbin/ip addr show dev tun0
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.239038   11822 controller.go:206] [SDN setup] full SDN setup required
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.239270   11822 ovs.go:40] Executing: ovs-vsctl --if-exists del-br br0 -- add-br br0 -- set Bridge br0 fail-mode=secure protocols=OpenFlow13
Oct 26 16:44:18 ose-node2.bmeng.local ovs-vsctl[11854]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-br br0 -- add-br br0 -- set Bridge br0 fail-mode=secure protocols=OpenFlow13
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.260298   11822 ovs.go:40] Executing: ovs-vsctl --if-exists del-port br0 vxlan0
Oct 26 16:44:18 ose-node2.bmeng.local ovs-vsctl[11857]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-port br0 vxlan0
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.269310   11822 ovs.go:40] Executing: ovs-vsctl --may-exist add-port br0 vxlan0 -- set Interface vxlan0 ofport_request=1 type=vxlan options:remote_ip="flow" options:key="flow"
Oct 26 16:44:18 ose-node2.bmeng.local ovs-vsctl[11858]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --may-exist add-port br0 vxlan0 -- set Interface vxlan0 ofport_request=1 type=vxlan "options:remote_ip=\"flow\"" "options:key=\"flow\""
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.279949   11822 ovs.go:40] Executing: ovs-vsctl get Interface vxlan0 ofport
Oct 26 16:44:18 ose-node2.bmeng.local atomic-openshift-node[11822]: I1026 16:44:18.284229   11822 ovs.go:40] Executing: ovs-vsctl --if-exists del-port br0 tun0
Oct 26 16:44:18 ose-node2.bmeng.local ovs-vsctl[11862]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-port br0 tun0

Comment 1 Dan Williams 2016-10-28 22:00:52 UTC
Fix included in https://github.com/openshift/origin/pull/11613

Comment 2 Ben Bennett 2016-11-01 14:07:10 UTC
Not merged yet, so not MODIFIED

Comment 4 zhaozhanqi 2016-11-04 06:22:11 UTC
Verified this bug on 
 openshift version
openshift v3.4.0.21+ca4702d
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0


When restart node service and checking the logs.  The SDN setup logs did not be found.

verified this bug.

Comment 6 errata-xmlrpc 2017-01-18 12:46:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0066


Note You need to log in before you can comment on or make changes to this bug.