This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1275904 - Should clean up the openflow rules for node after delete the node
Should clean up the openflow rules for node after delete the node
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Networking (Show other bugs)
3.x
Unspecified Unspecified
medium Severity low
: ---
: ---
Assigned To: Ravi Sankar
Meng Bo
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-28 01:52 EDT by Yan Du
Modified: 2016-05-12 13:12 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-05-12 13:12:11 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Yan Du 2015-10-28 01:52:27 EDT
Description of problem:
After delete the node, the openflow rules for node is still there.


Version-Release number of selected component (if applicable):
oc v1.0.6-997-gff3b522
kubernetes v1.2.0-alpha.1-1107-g4c8e6f4


How reproducible:
Always


Steps to Reproduce:

1. Set up multi-node env with multi-tenant plugin

 [root@master ~]# oc get node
NAME                LABELS                                     STATUS    AGE
node1.bmeng.local   kubernetes.io/hostname=node1.bmeng.local   Ready     13m
node2.bmeng.local   kubernetes.io/hostname=node2.bmeng.local   Ready     13m
node3.bmeng.local   kubernetes.io/hostname=node3.bmeng.local   Ready     13m

Host/IP map is as below:
master.bmeng.local   10.66.128.62
node1.bmeng.local    10.66.128.60
node2.bmeng.local    10.66.128.61
node3.bmeng.local    10.66.128.57


2. Check the openflow list on node1/node2
# ovs-ofctl dump-flows br0 -O OpenFlow13
 cookie=0xa428039, duration=2713.675s, table=7, n_packets=0, n_bytes=0,  priority=100,ip,nw_dst=10.1.2.0/24  actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.66.128.57->tun_dst,output:1
 cookie=0xa42803d,  duration=2713.695s, table=7, n_packets=0, n_bytes=0,  priority=100,ip,nw_dst=10.1.0.0/24  actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.66.128.61->tun_dst,output:1

 cookie=0xa428039,  duration=2713.670s, table=8, n_packets=0, n_bytes=0,  priority=100,arp,arp_tpa=10.1.2.0/24  actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.66.128.57->tun_dst,output:1
 cookie=0xa42803d,  duration=2713.683s, table=8, n_packets=0, n_bytes=0,  priority=100,arp,arp_tpa=10.1.0.0/24  actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.66.128.61->tun_dst,output:1


3. Delete one node
[root@master ~]# oc delete node node3.bmeng.local
node "node3.bmeng.local" deleted

[root@master ~]# oc get node
NAME                LABELS                                     STATUS    AGE
node1.bmeng.local   kubernetes.io/hostname=node1.bmeng.local   Ready     39m
node2.bmeng.local   kubernetes.io/hostname=node2.bmeng.local   Ready     39m 


4. Check the openflow list again



Actual results:

4. Same as step2, the openflow rules for node3 (10.66.128.57) still exists. 



Expected results:

The openflow rules for node3 should be cleaned up, since node3 is already deleted and not in service any more
Comment 1 Dan Winship 2015-10-28 11:28:31 EDT
Can't reproduce. Can you attach the journal output from node1 or node2?
Comment 2 Yan Du 2015-10-29 07:24:28 EDT
Hi, Dan

Actually I have tried many times, and I found the behavior is a little wired.

If I delete the node at the first time, then the openflow rules is still there even the node have been deleted. 

Then I restart the node to add the node again to master, try to delete the node again then check the openflow rules, the rules disappear.

And here is the log: http://fpaste.org/284758/17727144/
Comment 3 Dan Winship 2015-12-17 08:53:29 EST
so, a bunch more changes have been made since then... is this still reproducible?
Comment 4 Yan Du 2015-12-18 04:38:46 EST
Yes, Test with the latest origin code. It is still could be reproduced when the first time to delete the node.

10.66.128.62    master.bmeng.local
10.66.128.57    node1.bmeng.local
10.66.128.1     node2.bmeng.local

# oc delete node node2.bmeng.local  --config=/tmp/admin.kubeconfig
node "node2.bmeng.local" deleted
# oc get node --config=/tmp/admin.kubeconfig
NAME                LABELS                                     STATUS    AGE
node1.bmeng.local   kubernetes.io/hostname=node1.bmeng.local   Ready     50s

# ovs-ofctl dump-flows br0 -O OpenFlow13 | grep 10.66
 cookie=0xa428001, duration=195.442s, table=0, n_packets=11187, n_bytes=790366, tun_src=10.66.128.1 actions=goto_table:1
 cookie=0xa428001, duration=195.440s, table=8, n_packets=10007, n_bytes=740518, priority=100,ip,nw_dst=10.1.1.0/24 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.66.128.1->tun_dst,output:1
 cookie=0xa428001, duration=195.438s, table=9, n_packets=2, n_bytes=84, priority=100,arp,arp_tpa=10.1.1.0/24 actions=move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31],set_field:10.66.128.1->tun_dst,output:1
Comment 5 Ravi Sankar 2016-01-11 20:45:21 EST
Fixed in https://github.com/openshift/openshift-sdn/pull/241
Comment 6 Ravi Sankar 2016-04-18 15:22:34 EDT
Merged in origin: https://github.com/openshift/origin/pull/8468
Comment 7 Meng Bo 2016-04-19 03:32:37 EDT
Verified on OSE build v3.2.0.17, issue has been fixed.

Note You need to log in before you can comment on or make changes to this bug.