Bug 1647944

Summary: [clean-traffic-gateway filter] ebtabels is leaking the first packet
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Roni <reliezer>
Component: libvirtAssignee: Virtualization Maintenance <virt-maint>
Status: CLOSED WONTFIX QA Contact: yalzhang <yalzhang>
Severity: medium Docs Contact:
Priority: low    
Version: 8.0CC: dagur, fgarciad, jsuchane, laine, mburman, rbalakri, reliezer, yalzhang
Target Milestone: rcKeywords: Reopened, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-03-15 07:31:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Roni 2018-11-08 15:27:46 UTC
Description of problem:
Libvirt update ebtabels only after a first ping.

Version-Release number of selected component (if applicable):
libvirt-daemon-driver-storage-mpath-4.5.0-10.el7.x86_64
libvirt-daemon-driver-qemu-4.5.0-10.el7.x86_64
libvirt-bash-completion-4.5.0-10.el7.x86_64
libvirt-daemon-4.5.0-10.el7.x86_64
libvirt-daemon-driver-storage-rbd-4.5.0-10.el7.x86_64
libvirt-daemon-config-network-4.5.0-10.el7.x86_64
libvirt-daemon-kvm-4.5.0-10.el7.x86_64
libvirt-libs-4.5.0-10.el7.x86_64
libvirt-daemon-driver-storage-scsi-4.5.0-10.el7.x86_64
libvirt-daemon-driver-storage-logical-4.5.0-10.el7.x86_64
libvirt-daemon-driver-lxc-4.5.0-10.el7.x86_64
libvirt-daemon-driver-interface-4.5.0-10.el7.x86_64
libvirt-4.5.0-10.el7.x86_64
libvirt-python-4.5.0-1.el7.x86_64
libvirt-daemon-driver-nwfilter-4.5.0-10.el7.x86_64
libvirt-daemon-driver-storage-disk-4.5.0-10.el7.x86_64
libvirt-daemon-driver-storage-iscsi-4.5.0-10.el7.x86_64
libvirt-daemon-config-nwfilter-4.5.0-10.el7.x86_64
libvirt-daemon-driver-nodedev-4.5.0-10.el7.x86_64
libvirt-client-4.5.0-10.el7.x86_64
libvirt-daemon-driver-network-4.5.0-10.el7.x86_64
libvirt-daemon-driver-storage-gluster-4.5.0-10.el7.x86_64
libvirt-daemon-driver-secret-4.5.0-10.el7.x86_64
libvirt-daemon-driver-storage-core-4.5.0-10.el7.x86_64
libvirt-daemon-driver-storage-4.5.0-10.el7.x86_64
libvirt-lock-sanlock-4.5.0-10.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
At the oVirt:
1. Use 2 VMs
2. Add one vNIC to each VM
3. Assign the clean-traffic-filter to both vNICs
4. Put fake MAC address in the filter (GATEWAY_MAC=00:00:00:00:12:55)
   parameter of each vNIC
5. Unplug & plug the vNICs
6. Set IP to each vNIC
7. Send 5 ping packets (ping -c 5 [dst_ip])

Actual results:
The first packet pass, 80% fail

Expected results:
No replay 100% fail


More details:
1. Change GATEWAY_MAC "00:00:00:00:12:55" at the VM vNIC filter
2. Unplug & Plug the vNIC
3. run virsh -r dumpxml [VM_ID] |grep "12:55" at the host
    Output:  <parameter name='GATEWAY_MAC' value='00:00:00:00:12:55'/>
    (Assuming parameter successfully sent by ovirt-engine and successfully 
    accepted by libvirt)
4. run: ebtables -t nat -L |grep "12:55" at the host
   Output: empty, does not appear.
   (Assuming libvirt does not update ebtabels)
5. Run ping between VMs
6. run: ebtables -t nat -L |grep "12:55" again
    Output:
    -d 0:0:0:0:12:55 -j ACCEPT 
    -s 0:0:0:0:12:55 -j ACCEPT 
   (Assuming the ping traffic trigger libvirt to update the ebtabels rules)

Additional info:

Comment 2 Laine Stump 2018-11-08 17:24:17 UTC
What do you mean by "fake MAC"?

Can you provide the libvirt XML for the interface? Are you using "<parameter name='CTRL_IP_LEARNING' .... />" ?

There are modes of the clean-traffic filter that setup initial rules when the device is started, and then arae modified as they see traffic (e.g. the setup a pcap socket to monitor traffic and look for a DHCP response to determine the IP address). You can notice this by running "ebtables -t nat -l in a loop, then watching it change twice as you start a guest - once when the guest is initially "turned on", and again as soon as the guest OS comes up and sends out a dhcp request to get its IP address.

When DHCP isn't used, it will look for ARP packets instead (or in some cases will take the first IP packet originating from the guest).

MAC address is of course known at the time the nwfilter rule is instantiated, so there is no "snooping" for that - it comes directly from the <interface> XML, but still if CTRL_IP_LEARNING is turned on, the ebtables entries will still change once the IP address is learned (the initial rules will only be enough to learn the IP address, and then full rules permitting other traffic will be added later)

Does this explain the behavior you're seeing?

Comment 3 Roni 2018-11-11 09:06:22 UTC
Thanks, Laine,
I'm using static IP address and not DHCP
The "fake MAC" means: "not the real Gateway MAC"
Anyway, I add the following keys to the XML as you describe at your comment:
'CTRL_IP_LEARNING'='none'
'IP'=[vNIC IP]
And now the ping is blocked from the first packet as expected.
(Note that the 'IP' key is required if 'CTRL_IP_LEARNING' is set to 'none')
Close as not a bug!

Comment 4 Roni 2018-11-18 08:29:25 UTC
Reopen and change the title from:
"[clean-traffic-gateway filter] ebtabels is updated only after a first ping"
To:
"[clean-traffic-gateway filter] ebtabels is leaking the first packet"

In a second thought, although 'none' eliminating our issue but still the user can't use the keys: CTRL_IP_LEARNING='dhcp'/'any' because they are leaking the first packet.

Comment 5 Laine Stump 2018-11-18 19:43:54 UTC
Can you explain what you mean by "leaking the first packet"?

Comment 6 Roni 2018-11-20 07:46:48 UTC
Hi Laine
Please see the scenario above, from 5 ping packet, the first one was pass successfully and then all other were blocked
Although CTRL_IP_LEARNING=none eliminate this problem, but it still exists with 
the 'any & 'dhcp' parameters

Comment 7 yalzhang@redhat.com 2018-12-29 06:47:54 UTC
Hi Roni,

I want to have a try with your scenario, and there are something unclear:
In comment 0, you said:
3. Assign the clean-traffic-filter to both vNICs

How about the definition of the nwfilter "clean-traffic-filter"?
You can check on the host by:
# virsh dumpxml $vm | grep filter
      <filterref filter='clean-traffic'>  <---- this is the name of the nwfilter
then check the definition:
# virsh nwfilter-dumpxml clean-traffic

Comment 8 Roni 2019-01-17 08:55:34 UTC
[root@vega07 ~]# virsh -r list
 Id    Name                           State
----------------------------------------------------
 3     golden_env_mixed_virtio_1      running

[root@vega07 ~]# 
[root@vega07 ~]# virsh -r dumpxml 3 | grep filter
      <filterref filter='vdsm-no-mac-spoofing'/>
      <filterref filter='clean-traffic-gateway'>
      </filterref>
[root@vega07 ~]# 
[root@vega07 ~]# 
[root@vega07 ~]# virsh -r nwfilter-dumpxml clean-traffic-gateway
<filter name='clean-traffic-gateway' chain='root'>
  <uuid>0f4f1db2-59fb-40b4-a250-4500ac9da460</uuid>
  <filterref filter='no-mac-spoofing'/>
  <filterref filter='no-ip-spoofing'/>
  <filterref filter='no-arp-spoofing'/>
  <rule action='accept' direction='inout' priority='-500'>
    <mac protocolid='arp'/>
  </rule>
  <rule action='accept' direction='in' priority='500'>
    <mac srcmacaddr='$GATEWAY_MAC'/>
  </rule>
  <rule action='accept' direction='out' priority='500'>
    <mac dstmacaddr='$GATEWAY_MAC'/>
  </rule>
  <filterref filter='no-other-l2-traffic'/>
  <filterref filter='qemu-announce-self'/>
</filter>

Comment 9 yalzhang@redhat.com 2019-03-07 02:51:29 UTC
I think I have reproduced this issue with below packages:
# rpm -q libvirt kernel iptables-ebtables
libvirt-5.0.0-5.module+el8+2850+33063f9c.x86_64
kernel-4.18.0-73.el8.x86_64
iptables-ebtables-1.8.2-9.el8.x86_64

This is my steps:
1. Prepare 2 vms as below setting:
# virsh net-dumpxml default   | grep mac
  <mac address='52:54:00:54:9f:87'/>

# virsh dumpxml rhel | grep /interface -B9
...
      <mac address='52:54:00:0d:42:2f'/>
      <source network='default'/>
      <model type='virtio'/>
      <filterref filter='clean-traffic-gateway'>
        <parameter name='GATEWAY_MAC' value='00:00:00:00:12:55'/>
      </filterref>
...

# virsh dumpxml avocado-vt-vm1 | grep /interface -B9
...
  <source network='default'/>
      <model type='virtio'/>
      <filterref filter='clean-traffic-gateway'>
        <parameter name='GATEWAY_MAC' value='00:00:00:00:12:55'/>
      </filterref>
...
(the definition of clean-traffic-gateway is described in above comment 8)

2. Start the 2 guests, and loggin to config the static ip for them:
on rhel: # nmcli con add type ethernet con-name eth0 ifname eth0 ipv4.method manual ipv4.addresses 10.10.10.10/24
on avocado-vt-vm1: # nmcli con add type ethernet  con-name enp1s0 ifname enp1s0 ipv4.method manual ipv4.addresses 10.10.10.20/24

3. on avocado-vt-vm1, try to ping rhe other vm rhel:
# ping 10.10.10.10 -c 10
PING 10.10.10.10 (10.10.10.10) 56(84) bytes of data.
64 bytes from 10.10.10.10: icmp_seq=1 ttl=64 time=1.38 ms
64 bytes from 10.10.10.10: icmp_seq=2 ttl=64 time=0.336 ms

--- 10.10.10.10 ping statistics ---
10 packets transmitted, 2 received, 80% packet loss, time 169ms
rtt min/avg/max/mdev = 0.336/0.858/1.380/0.522 ms

The first 2 packages can received successfully, which is unexpected, as the GATEWAY_MAC('00:00:00:00:12:55') in both interface are not the correct gateway mac (default virbr0 has '52:54:00:54:9f:87').

Comment 12 RHEL Program Management 2021-03-15 07:31:19 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.