Bug 1746123

Erratum: "unroutable" is generated by the guest itself. In fact, no routable ipv6 address on host acts the same as disabled ipv6 (as far as guests are concerned).

Also, for completeness, the behavior for ipv4 is that, with no ipv4 address on host and br_netfilter being on, ipv4 works in guests. Rules in iptables are hit and packets are properly bridged.

More info: link-local addresses behave the same, i.e. it has to be assigned (so at least disable_ipv6=0) on the host to work on guests.

More info: openvswitch is affected too (not only linux bridges).

Preciser statement: by br_netfilter being on I mean that the module is loaded and relevant variables activated, i.e.:

# sysctl net.bridge
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
...

More info: this may affect OpenStack (neutron) data plane because it requires this exact scenario, see https://opendev.org/openstack/neutron/src/commit/555535977662e505dfe219d6918ac0fb41624eaa/doc/source/install/compute-install-option2-rdo.rst

This behaviour of the br_netfilter module is indeed uphelpful. When net.bridge.bridge-nf-call-XXXtables = 1 all guest traffic is processed by the host firewall, and depending on what rules you have in that you may well see traffic dropped. 

With the virtual networks, libvirt tries to add rules that should workaround this problem by explicitly allowing traffic for guest tap devices on the bridge libvirt creates. If this isn't working for IPv6 isolated network though, then we have a bug in libvirt rules in this respect.

To me it looks like ip6tables does something awry. Libvirt simply mimicks the behavior of iptables there.

The ACCEPT rules, as mentioned, are added by libvirt, but they do not help as: a) they are completely ignored in the disable_ipv6 case or b) they accept traffic but it still never reaches other guests as host finds them unroutable.

Could you share the output of 'virsh net-dumpxml $NETWORK' for the network you created, so I can be sure I'm testing/debugging the exact same setup as you.

Ignoring VMs for a minute, does your host itself actually have IPv6 connectivity / addresses configured for its own LAN ?

Sure, here:
<network connections='7' ipv6='yes'>
  <name>kolla-internal</name>
  <uuid>399ddf2e-686f-4438-9627-ac392937ecef</uuid>
  <bridge name='virbr2' stp='on' delay='0'/>
  <mac address='52:54:00:2d:e9:3e'/>
</network>

As mentioned, ipv6 toggle is useless in either br_netfilter state (either unused or not effective).

Yes, the host deals with IPv6 connectivity just fine. The point is isolated VM networks do not want to work with IPv6.
(So the host does not have an IPv6 on this virbr2 on purpose.)

Thanks for the config & hints.

I've confirmed I see the same behaviour on a rhel 7.6 host with the config you have here.

For sake of testing I created two Cirros guests http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img with their NIC connected to the 'kolla-internal' network.

After waiting for them to boot & fail DHCP (expected), they have a link local IPv6 address each.

Trying to ping6 from one guest to the other guest fails.

Running tcpdump on the vnet0 TAP device on the rhel7 host I can see repeated neighbor solicitation packets for the other guests' ipv6 link local address, eg

12:55:01.518691 IP6 fe80::5054:ff:fe50:2e4e > ff02::1:ffb7:64d7: ICMP6, neighbor solicitation, who has fe80::5054:ff:feb7:64d7, length 32
 
No reply is ever seen


Now, testing the identical configuration on a rhel8 host, everything works as intended - ping6 works, ssh works.

Running tcpdump on the vnet0 TAP device on the rhel8 host I can see the expected replies

12:52:36.153381 IP6 fe80::5054:ff:fea6:497b > ff02::1:ff47:8904: ICMP6, neighbor solicitation, who has fe80::5054:ff:fe47:8904, length 32
12:52:36.155194 IP6 fe80::5054:ff:fe47:8904 > fe80::5054:ff:fea6:497b: ICMP6, neighbor advertisement, tgt is fe80::5054:ff:fe47:8904, length 32
12:52:36.156380 IP6 fe80::5054:ff:fea6:497b > fe80::5054:ff:fe47:8904: ICMP6, echo request, seq 0, length 64
12:52:36.158254 IP6 fe80::5054:ff:fe47:8904 > fe80::5054:ff:fea6:497b: ICMP6, echo reply, seq 0, length 64

Some we have some problem with ICMP6 neighbor advertisement on rhel7.

Slight correction - my RHEL-8 host wasn't configured quite the same way - firewalld was disabled.  When I start firewalld on RHEL-8, IPv6 breaks too, though the tcpdump traffic is slightly different.

Back on RHEL-7 I see the problem is that firewalld is adding rules to the 'raw' table that block traffic


Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     ipv6-icmp    any    any     anywhere             anywhere             ipv6-icmp router-advertisement
  685 50696 DROP       all	any    any     anywhere             anywhere             rpfilter invert


Can you confirm that you have 'firewalld' running on your host too ?

# firewall-cmd --list-all
FirewallD is not running

Do you have any other kind of firewall setup on the host ?  Can you attach output of 'ip6tables-save' to show the full ruleset ?

It probably doesn't matter, but for the record, what exact kernel version + libvirt version do you have.

kernel: 3.10.0-957.27.2.el7.x86_64
libvirt: 4.5.0-10.el7_6.12

Thanks for thorough investigation.
I completely forgot about raw tables (checked mangle though :-) ).
I found out that on one tested host firewalld was indeed up and running.
The culprit was:
-A PREROUTING -m rpfilter --invert -j DROP
So in this case firewalld was dropping traffic and stopping it would fix it.

The ipv4 rp_filter is exempt from bridge traffic but the one in ip6tables seems not.
This is the bug.
Can this rule be made exempt from the bridge traffic?

I have to redo the test on the other machine where there was no firewalld running and get back to you.
It probably had some other filtering rule that caught the traffic. It could be unrelated.

Hey Eric - is there something that firewalld could/should do to solve this? Or maybe something extra that we should be doing to counter-act firewalld's rules?

A short synopsis is that we're setting up a bridge that has no IP connectivity on the host itself, but want the guests connected to that bridge to be able to send/receive IPv6 traffic to each other, but when br_netfilter is loaded and firewalld is active, that doesn't work.

(In reply to Laine Stump from comment #16)
> Hey Eric - is there something that firewalld could/should do to solve this?
> Or maybe something extra that we should be doing to counter-act firewalld's
> rules?
> 
> A short synopsis is that we're setting up a bridge that has no IP
> connectivity on the host itself, but want the guests connected to that
> bridge to be able to send/receive IPv6 traffic to each other, but when
> br_netfilter is loaded and firewalld is active, that doesn't work.

Seems like it's not passing the IPv6 rpfilter.

Workaround: In /etc/firewalld/firewalld.conf set IPv6_rpfilter=no.

Long-term: This comes down to ip6tables rpfilter in the kernel. The host has no route to the source, so it makes sense the check fails. I'm not sure what, if anything, we can do in the case of br_netfilter.

What about extending that rule with:
-m physdev ! --physdev-is-in
Though I believe this switches off valid security for bridges in general...
If only one could postpone rpfilter till bridge/route decision. :-)
Maybe let libvirt set bridges it controls exempt from this check?
But then again docker/podman bridges would be affected anyway.

(In reply to Radosław Piliszek from comment #18)
> What about extending that rule with:
> -m physdev ! --physdev-is-in
> Though I believe this switches off valid security for bridges in general...

You are correct. Unfortunately there in no nftables equivalent, which is what firewalld uses in RHEL-8.

For completeness, I confirm my other issue was unrelated. This bug is relevant to libvirt+firewalld.

Folks, I gave it some thought and I think it would be best, in the RHEL7 case, to convert the rpfilter DROP to MARK and then in filter FORWARDING do DROP if marked and routed (as opposed to bridged). This would ensure firewalld provides equal security while bridging still works.

(In reply to Radosław Piliszek from comment #24)
> Folks, I gave it some thought and I think it would be best, in the RHEL7
> case, to convert the rpfilter DROP to MARK and then in filter FORWARDING do
> DROP if marked and routed (as opposed to bridged). This would ensure
> firewalld provides equal security while bridging still works.

We'd effectively be marking all IPv6 packets. This means user won't be able to set their own marks. firewalld would either overwrite their mark or they'd overwrite the rpfilter mark.

> We'd effectively be marking all IPv6 packets. This means user won't be able
> to set their own marks. firewalld would either overwrite their mark or
> they'd overwrite the rpfilter mark.

I disagree with the first sentence. We would only be marking packets failing rpfilter.
Though I agree that introducing a magic number is a breaking change.

Then it falls to kernel to let rpfilter DROP only routed packets (as it does for IPv4 as controlled by sysctl vars)...

(In reply to Radosław Piliszek from comment #26)
> > We'd effectively be marking all IPv6 packets. This means user won't be able
> > to set their own marks. firewalld would either overwrite their mark or
> > they'd overwrite the rpfilter mark.
> 
> I disagree with the first sentence. We would only be marking packets failing
> rpfilter.

Right. I misspoke.

> Though I agree that introducing a magic number is a breaking change.
> 
> Then it falls to kernel to let rpfilter DROP only routed packets (as it does
> for IPv4 as controlled by sysctl vars)...

I'm currently discussing the options with my colleagues.

I'd like to see the following:

  - Use "-m physdev ! --physdev-is-in" for the firewalld iptables backend
  - Change upstream nftables (kernel) to completely ignore packets coming from bridge devices
    - depends on upstream acceptance
    - br_netfilter doesn't make sense with nftables as nftables as full bridge filtering capabilities

Thank you for reporting this issue to the libvirt project. Unfortunately we have been unable to resolve this issue due to insufficient maintainer capacity and it will now be closed. This is not a reflection on the possible validity of the issue, merely the lack of resources to investigate and address it, for which we apologise. If you none the less feel the issue is still important, you may choose to report it again at the new project issue tracker https://gitlab.com/libvirt/libvirt/-/issues The project also welcomes contribution from anyone who believes they can provide a solution.