Created attachment 1933931 [details] ovsmirrortest.diff Created attachment 1933931 [details] ovsmirrortest.diff Created attachment 1933931 [details] ovsmirrortest.diff Description of problem: While trying to add a test case related to OVS Mirror in OVN (ovn.at) we are seeing an issue when both 'select-dst-port' and 'select-src-port' are filled. The test we are trying is very simple. Send an ICMP echo request packet and expect an ICMP echo reply packet. An OVS mirror is created and expected to mirror both the request and reply packets. (So both select-dst-port' and 'select-src-port' are filled appropriately) While both ICMP request and reply we see are present (i.e. ping works fine) But both packets are NOT getting mirrored. We see ONLY the ICMP request packet getting mirrored. Alternatively if we select either 'select-dst-port' or 'select-src-port' those packets get mirrored separately well. But when both are selected we see only the request packet getting mirrored. I have attached the test case we used to test it and it has the OVS commands which is used to create mirrors. Please check it. Further we also tried to create a similar environment on the actual HW setup and we do see the same issue there. So, its not just failing in test environment alone. Version-Release number of selected component (if applicable): How reproducible: Please find attached the diff having the test case which uses OVS commands to create mirrors and verify it. Actual results: Only echo request packet is getting mirrored. Expected results: Both echo request and reply packets needs to be mirrored. Additional info:
Hi OVS Team, Could you get chance to look into this issue. More specifically the case which we have shared in the reproducer attached. Currently this blocks port mirroring feature work in OVN for the feature which can mirror packets in both direction. (Note there is not issue while mirroring packets in only one direction). Hence request you to kindly take a look into it. Thanks, Ihar
Hi OVS Team, Could you get chance to look into this issue. More specifically the case which we have shared in the reproducer attached. Currently this blocks port mirroring feature work in OVN for the feature which can mirror packets in both direction. (Note there is not issue while mirroring packets in only one direction). Hence request you to kindly take a look into it. Thanks & Regards, Abhiram R N
Hello Arn, Yes, I've looked into this. It had me confused for a while as the reply packet was seemingly bypassing a lot of code. Finally, I realized that the reply is actually being generated by this flow: > cookie=0xdd2da837, duration=81023.012s, table=11, n_packets=26, n_bytes=2548, idle_age=288, hard_age=65534, priority=90,icmp,metadata=0x1,nw_dst=172.16.1.1,icmp_type=8,icmp_code=0 actions=push:NXM_OF_IP_SRC[],push:NXM_OF_IP_DST[],pop:NXM_OF_IP_SRC[],pop:NXM_OF_IP_DST[],load:0xff->NXM_NX_IP_TTL[],load:0->NXM_OF_ICMP_TYPE[],load:0x1->NXM_NX_REG10[0],resubmit(,12) Currently, the current implementation of mirroring doesn't anticipate one packet becoming its own reply
Hi @mpattric , Thanks for looking into this. So, is it kindof limitation in OVS mirroring? But actually I felt its quite a common scenario. We are just doing a ping from port 1 -> 2 and telling the mirror to any packet coming in and going out of 1 to a destination 3!. Also note if we are mirroring packet coming in to 1 OR going out of 1 it works perfectly well. but packet coming in to 1 AND going out of 1 it doesnt work!. How is it an issue I am unable to understand? Because those 2 packets are totally different ones. Or do you say its a limitation of the Testcase or the way testcase is written? Recently we tried using a playbook and that works perfectly well where we have similar setup ping VM1 --> VM2 and mirror packets coming in and going out same way both directions. There it works well. But the difference I noticed is we dont have an OVN Logical Router there. Reference to the playbook https://github.com/rh-nfv-int/playbooks/blob/main/regression/ovs-hwol/testRemotePortMirroring.yml#L229 Further we tried on same setup to do ping involving a 'logical router'. Again same behavior is happening (i.e only one 1 direction packets is being mirrored) even on actual hardware setup. So, I feel its a genuine bug and NOT limited to 'TEST CASE / ENVIRONMENT'. Next question is, how can the presence or not presence of this 'logical Router' affect 'OVS mirroring' I dont know. Maybe if you know kindly help us understand. OR maybe OVN team can comment. @nusiddiq any thoughts? Thanks & Regards, Abhiram R N
Normally when a packet enters OVS or is created by OVS: OVS makes a flow key for that packet, translates the action for that flow key, and then executes that action. In the case of a mirror, the mirror action is added to the list of actions during translation and before execution. OVS will check to make sure that the packet hasn't already been mirrored, and if not, include an additional output in the list of actions. That duplication check saves us from outputting many duplicates in the case of more complex rule sets. In this case, a single packet is transformed into a response with openflow rules, but that transformation happens after we've already ran the mirror logic. Note that the ICMP echo and response aren't different packets, it's the exact same packet buffer being overwritten. So nothing is even responding to the echo, the openflow rules are faking the response. In cases where a mirror of a ping works, it's because that ping is sent outside of OVS and the response re-enters OVS as a new packet. As you mention "difference I noticed is we dont have an OVN Logical Router there.", this openflow rule comes from: https://github.com/ovn-org/ovn/blob/main/northd/northd.c#L13392 This is not a limitation of the testcase or the environment. But a limitation of how OVS mirrors will only mirror one version of a packet, even if it is changed by openflow rules. Changing this behavior could significantly impact how OVS mirrors work, so I will bring this up at the next OVS meeting to get feedback.
Hi @mpattric , Any update/progress on this? Thanks & Regards, Abhiram R N
Hello Abhiram, I've rebased it to the current state of master and re-sent it in.
Adding link for reference. Mike has upstreamed the fix in https://github.com/openvswitch/ovs/commit/feed7f6775056b3dd55249596a7e587bc9c5fd4a Thanks & Regards, Abhiram R N
This bugfix has now made it to downstream.