Bug 2155579

Hi OVS Team,

Could you get chance to look into this issue. More specifically the case which we have shared in the reproducer attached.

Currently this blocks port mirroring feature work in OVN for the feature which can mirror packets in both direction. (Note there is not issue while mirroring packets in only one direction). Hence request you to kindly take a look into it.

Thanks,
Ihar

Hi OVS Team,

Could you get chance to look into this issue. More specifically the case which we have shared in the reproducer attached.

Currently this blocks port mirroring feature work in OVN for the feature which can mirror packets in both direction. (Note there is not issue while mirroring packets in only one direction). Hence request you to kindly take a look into it.

Thanks & Regards,
Abhiram R N

Hello Arn,

Yes, I've looked into this. It had me confused for a while as the reply packet was seemingly bypassing a lot of code.

Finally, I realized that the reply is actually being generated by this flow:

> cookie=0xdd2da837, duration=81023.012s, table=11, n_packets=26, n_bytes=2548, idle_age=288, hard_age=65534, priority=90,icmp,metadata=0x1,nw_dst=172.16.1.1,icmp_type=8,icmp_code=0 actions=push:NXM_OF_IP_SRC[],push:NXM_OF_IP_DST[],pop:NXM_OF_IP_SRC[],pop:NXM_OF_IP_DST[],load:0xff->NXM_NX_IP_TTL[],load:0->NXM_OF_ICMP_TYPE[],load:0x1->NXM_NX_REG10[0],resubmit(,12)

Currently, the current implementation of mirroring doesn't anticipate one packet becoming its own reply

Hi @mpattric ,

Thanks for looking into this.

So, is it kindof limitation in OVS mirroring? 
    But actually I felt its quite a common scenario. We are just doing a ping from port 1 -> 2 and telling the mirror to any packet coming in and going out of 1 to a destination 3!. Also note if we are mirroring 
    packet coming in to 1 OR going out of 1 it works perfectly well. 
    but
    packet coming in to 1 AND going out of 1 it doesnt work!.

How is it an issue I am unable to understand? Because those 2 packets are totally different ones.

Or do you say its a limitation of the Testcase or the way testcase is written?

Recently we tried using a playbook and that works perfectly well where we have similar setup ping VM1 --> VM2 and mirror packets coming in and going out same way both directions. There it works well. But the difference I noticed is we dont have an OVN Logical Router there.

Reference to the playbook
https://github.com/rh-nfv-int/playbooks/blob/main/regression/ovs-hwol/testRemotePortMirroring.yml#L229

Further we tried on same setup to do ping involving a 'logical router'. Again same behavior is happening (i.e only one 1 direction packets is being mirrored) even on actual hardware setup. So, I feel its a genuine bug and NOT limited to 'TEST CASE / ENVIRONMENT'.
Next question is, how can the presence or not presence of this 'logical Router' affect 'OVS mirroring' I dont know.

Maybe if you know kindly help us understand.
OR
maybe OVN team can comment. @nusiddiq any thoughts?

Thanks & Regards,
Abhiram R N

Normally when a packet enters OVS or is created by OVS: OVS makes a flow key for that packet, translates the action for that flow key, and then executes that action.

In the case of a mirror, the mirror action is added to the list of actions during translation and before execution. OVS will check to make sure that the packet hasn't already been mirrored, and if not, include an additional output in the list of actions. That duplication check saves us from outputting many duplicates in the case of more complex rule sets.

In this case, a single packet is transformed into a response with openflow rules, but that transformation happens after we've already ran the mirror logic. Note that the ICMP echo and response aren't different packets, it's the exact same packet buffer being overwritten. So nothing is even responding to the echo, the openflow rules are faking the response.

In cases where a mirror of a ping works, it's because that ping is sent outside of OVS and the response re-enters OVS as a new packet.

As you mention "difference I noticed is we dont have an OVN Logical Router there.", this openflow rule comes from: https://github.com/ovn-org/ovn/blob/main/northd/northd.c#L13392

This is not a limitation of the testcase or the environment. But a limitation of how OVS mirrors will only mirror one version of a packet, even if it is changed by openflow rules.

Changing this behavior could significantly impact how OVS mirrors work, so I will bring this up at the next OVS meeting to get feedback.

Hi @mpattric ,

Any update/progress on this? 

Thanks & Regards,
Abhiram R N

Hello Abhiram,

I've rebased it to the current state of master and re-sent it in.

Adding link for reference.

Mike has upstreamed the fix in 
https://github.com/openvswitch/ovs/commit/feed7f6775056b3dd55249596a7e587bc9c5fd4a

Thanks & Regards,
Abhiram R N

This bugfix has now made it to downstream.