Hide Forgot
Description of problem: Customer is creating egressNetworkPolicy in a project. But system is just removing all rules from OVS for the netnamespace and adding a single "drop all" rule. In node logs we see: atomic-openshift-node[39469]: E0106 17:40:05.187734 39469 controller.go:506] multiple EgressNetworkPolicies in same network namespace (vwc-rec:default, m4d-rec:default) is not allowed; dropping all traffic have checked: - no global projects have egress policy defined. - there are no joined projects. - none of the projects have more than one egress policy defined. Two separate environments (3.3.1.7 and 3.3.1.3) do suffer from the issue. At the beginning only one node was affected by this. But now both nodes have this issue. It looks like more project related. Please advise on what to check/trace. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
This is https://github.com/openshift/origin/pull/12045 and it's fixed in 3.4 (v3.4.0.32). We did not backport the fix to 3.3. The relevant code didn't change much between 3.3 and 3.4 so it would be possible to do, but I don't know what the policy is for 3.3 bugfixes at this point... (There is no way to work around the bug other than backporting the bugfix.)
Tested on OCP 3.3.1.11 After adding multiple egresspolicy to a single namespace, the existing openflow rules will not be affected. And will add a new one to drop the traffic for specific project. From node log: Jan 22 03:03:10 node1 atomic-openshift-node[27026]: E0122 03:03:10.885757 27026 controller.go:506] multiple EgressNetwor kPolicies in same network namespace (bmengp1:default, bmengp1:default2) is not allowed; dropping all traffic Jan 22 03:03:10 node1 atomic-openshift-node[27026]: I0122 03:03:10.885809 27026 ovs.go:37] Executing: /usr/bin/ovs-ofctl -O OpenFlow13 del-flows br0 table=9, reg0=720494 Jan 22 03:03:10 node1 atomic-openshift-node[27026]: I0122 03:03:10.891489 27026 ovs.go:37] Executing: /usr/bin/ovs-ofctl -O OpenFlow13 add-flow br0 table=9, reg0=720494, priority=1, actions=drop Check the openflow rules: # ovs-ofctl dump-flows br0 -O openflow13 OFPST_FLOW reply (OF1.3) (xid=0x2): cookie=0x0, duration=221.113s, table=0, n_packets=0, n_bytes=0, priority=200,arp,in_port=1,arp_spa=10.1.0.0/16,arp_tpa=10.1.1.0/24 actions=move:NXM_NX_TUN_ID[0..31]- >NXM_NX_REG0[],goto_table:1 cookie=0x0, duration=221.110s, table=0, n_packets=0, n_bytes=0, priority=200,ip,in_port=1,nw_src=10.1.0.0/16,nw_dst=10.1.1.0/24 actions=move:NXM_NX_TUN_ID[0..31]->NX M_NX_REG0[],goto_table:1 cookie=0x0, duration=221.105s, table=0, n_packets=45, n_bytes=1890, priority=200,arp,in_port=2,arp_spa=10.1.1.1,arp_tpa=10.1.0.0/16 actions=goto_table:5 cookie=0x0, duration=221.102s, table=0, n_packets=3871, n_bytes=2493051, priority=200,ip,in_port=2 actions=goto_table:5 cookie=0x0, duration=221.095s, table=0, n_packets=2, n_bytes=84, priority=200,arp,in_port=3,arp_spa=10.1.1.0/24 actions=goto_table:5 cookie=0x0, duration=221.085s, table=0, n_packets=0, n_bytes=0, priority=200,ip,in_port=3,nw_src=10.1.1.0/24 actions=goto_table:5 cookie=0x0, duration=221.108s, table=0, n_packets=0, n_bytes=0, priority=150,in_port=1 actions=drop cookie=0x0, duration=221.098s, table=0, n_packets=16, n_bytes=1296, priority=150,in_port=2 actions=drop cookie=0x0, duration=221.058s, table=0, n_packets=38, n_bytes=3132, priority=150,in_port=3 actions=drop cookie=0x0, duration=221.050s, table=0, n_packets=41, n_bytes=1722, priority=100,arp actions=goto_table:2 cookie=0x0, duration=221.044s, table=0, n_packets=2231, n_bytes=239877, priority=100,ip actions=goto_table:2 cookie=0x0, duration=221.004s, table=0, n_packets=45, n_bytes=3558, priority=0 actions=drop cookie=0x0, duration=220.782s, table=1, n_packets=0, n_bytes=0, priority=100,tun_src=10.8.174.9 actions=goto_table:5 cookie=0x0, duration=221.001s, table=1, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x0, duration=220.613s, table=2, n_packets=2, n_bytes=84, priority=100,arp,in_port=11,arp_spa=10.1.1.5,arp_sha=02:42:0a:01:01:05 actions=load:0->NXM_NX_REG0[],goto_table:5 cookie=0x0, duration=220.604s, table=2, n_packets=318, n_bytes=28460, priority=100,ip,in_port=11,nw_src=10.1.1.5 actions=load:0->NXM_NX_REG0[],goto_table:3 cookie=0x0, duration=220.992s, table=2, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x0, duration=220.990s, table=3, n_packets=299, n_bytes=75681, priority=100,ip,nw_dst=172.30.0.0/16 actions=goto_table:4 cookie=0x0, duration=220.981s, table=3, n_packets=1932, n_bytes=164196, priority=0 actions=goto_table:5 cookie=0x0, duration=220.958s, table=4, n_packets=299, n_bytes=75681, priority=200,reg0=0 actions=output:2 ... ... ... cookie=0x0, duration=220.913s, table=8, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x0, duration=22.721s, table=9, n_packets=0, n_bytes=0, priority=1,reg0=0xafe6e actions=drop cookie=0x0, duration=220.911s, table=9, n_packets=496, n_bytes=35890, priority=0 actions=output:2 cookie=0x0, duration=220.822s, table=253, n_packets=0, n_bytes=0, actions=note:01.01.00.00.00.00
Please ignore the comment#11 above. Tested with following steps To reproduce, tested on build 3.3.1.9 1. Create 10 projects 2. Add egress policy to each project 3. Check the openflow 4. Restart openshift node service 5. Check the openflow again Result: In step 3, the openflow rules for the project created in table9 with following contents, cookie=0x0, duration=1.722s, table=9, n_packets=0, n_bytes=0, priority=2,ip,reg0=0x5d687c,nw_dst=172.16.120.0/24 actions=output:2 cookie=0x0, duration=1.715s, table=9, n_packets=0, n_bytes=0, priority=1,ip,reg0=0x5d687c,nw_dst=10.66.140.0/24 actions=drop In step 5, the openflow rules are changed by the restart to cookie=0x0, duration=1.704s, table=9, n_packets=0, n_bytes=0, priority=1,reg0=0x5d687c actions=drop To verify, tested with the same steps above on build 3.3.1.11 The openflow rules for the project with egressnetworkpolicy will not be corrupted by the restart.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0199