Bug 1795790

Summary: ACL with action reject never sends TCP reset
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Tim Rozet <trozet>
Component: OVNAssignee: Numan Siddique <nusiddiq>
Status: CLOSED DUPLICATE QA Contact: Jianlin Shi <jishi>
Severity: high Docs Contact:
Priority: unspecified    
Version: RHEL 8.0CC: ctrautma, nusiddiq
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-06 15:09:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1772655, 1810567, 1810570    
Attachments:
Description Flags
logs
none
fix logs none

Description Tim Rozet 2020-01-28 21:17:02 UTC
Created attachment 1656119 [details]
logs

Description of problem:
Creating an ACL with reject action never actually sends a TCP reset. The packet gets punted to the controller and then eventually dropped in egress pipeline.

Version-Release number of selected component (if applicable):
[root@ovn-control-plane ~]# rpm -qa | egrep 'ovn|openvswitch'
openvswitch-2.12.0-1.fc31.x86_64
ovn-2.12.0-5.fc31.x86_64
ovn-host-2.12.0-5.fc31.x86_64
ovn-central-2.12.0-5.fc31.x86_64


How reproducible:
Always

Steps to Reproduce:
1. Create an ACL in OVN like:
from-lport  1000 (ip4.dst==10.98.35.163 && tcp && tcp.dst==58585) reject
2. curl or create a tcp connection to the address
3. connection will eventually time out

Actual results:
connection timeout

Expected results:
TCP reset should be received.

Additional info:
See attachment for full output. There seems to be several problems with the current behavior:

1. Why are packets punted to the controller for TCP reset? Shouldn't this be handled via OpenFlow and conntrack?
2. If there is some reason for #1, there should be rate limiting on sending these packets to the controller. Otherwise an attacker could exploit this.
3. Why is the egress pipeline dropping the packet?

Comment 1 Tim Rozet 2020-01-28 22:05:29 UTC
The egress pipeline may not be what's dropping the packet. Numan may have some more insights into this.

Comment 2 Tim Rozet 2020-02-04 15:04:11 UTC
Looks like it is not possible to handle the TCP reset via OF:
[root@ovn-control-plane ~]# ovs-ofctl -O openflow14 add-flow br-int 'table=0,tcp,tcp_flags=0,action=load:0x1->NXM_NX_TCP_FLAGS'
2020-02-03T18:30:24Z|00001|meta_flow|WARN|destination field tcp_flags is not writable

There is also no mechanism today for OVN to be able to configure conntrack to send the reset. Therefore the packet is punted to controller, and the controller then sends a packet out.

Numan noticed the packet out is being dropped by ovs-vswitchd:
2020-01-28T17:38:55.498Z|00028|ofproto_dpif_upcall(handler10)|INFO|received packet on unassociated datapath port 4294967295

Comment 3 Numan Siddique 2020-02-04 19:40:04 UTC
@Tim - Can you please try this patch yourself if possible ? https://github.com/numansiddique/ovn/commit/1e83bddfc0e2230830f005cea72beeb53557d00a
https://github.com/numansiddique/ovn/tree/tcp_reset_fix/v1

I am planning to write some system tests. I will submit the patch for review after the system tests.

Thanks
Numan

Comment 4 Tim Rozet 2020-02-05 15:51:02 UTC
I'll give it a try today. Thanks.

Comment 5 Tim Rozet 2020-02-05 22:11:21 UTC
The fix didn't seem to work. I still see the unassociated datapath. I'll ping you tomorrow morning my time and we can take a look at it. Attached the ovn trace output.

Comment 6 Tim Rozet 2020-02-05 22:12:25 UTC
Created attachment 1658027 [details]
fix logs

Comment 7 Tim Rozet 2020-02-06 02:39:00 UTC
Actually I just realized I built off of the wrong ovn branch. Will fix it tmrw.

Comment 8 Numan Siddique 2020-02-06 06:54:03 UTC
Patch submitted for review - https://patchwork.ozlabs.org/patch/1234149/

Comment 9 Tim Rozet 2020-02-06 16:32:01 UTC
Tested and it works for me!

Comment 10 Tim Rozet 2020-02-13 14:51:54 UTC
Fix is now available upstream.

Comment 11 Tim Rozet 2020-03-20 14:10:20 UTC
Is this downstream yet?

Comment 12 Numan Siddique 2020-04-06 15:09:19 UTC
Yes. Its alredy available d/s in ovn2.12 and ovn2.13

And its already addressed in this bz https://bugzilla.redhat.com/show_bug.cgi?id=1805651

I'll close this BZ.

Comment 13 Numan Siddique 2020-04-06 15:09:53 UTC

*** This bug has been marked as a duplicate of bug 1805651 ***