Description of problem: Isolated IPv6 bridge network does not work with br_netfilter being on. Version-Release number of selected component (if applicable): How reproducible: Always. Steps to Reproduce: 1. Create isolated (no host IP address) network in libvirt and enable ipv6 for guests on it. (i.e. set ipv6="yes" on network XML element). 2. Load br_netfilter module (e.g. using: modprobe br_netfilter). 3. Boot off some guests on this network (their OS does not matter but assume it's another RHEL for consistency). 4. Configure them for IPv6 addressing. Actual results: No IPv6 connectivity between guests. Expected results: Working IPv6 connectivity between guests. Additional info: Enabling ipv6 on network keeps bridge with ipv6 disabled (disable_ipv6=1). Enabling br_netfilter causes ip6tables to apply to bridges. Disabled ipv6 causes all ipv6 traffic to be dropped before it hits ip6tables at all (counters never increase). Enabling ipv6 on the bridge (disable_ipv6=0) makes packets hit the libvirt-created FORWARD ACCEPT rules in ip6tables (that's what ipv6="yes" does) but, since the host has no routable address on the interface, guests still cannot work with ipv6 because they get "unroutable" back from the host. Only configuring a correct routable ipv6 address on the bridge (libvirt network) will allow ipv6 to work. Hence, there is no way to create a truly isolated ipv6 network with br_netfilter being on (and calls to ip6tables being enabled for bridges). Note there is some logic flaw in this in general because without br_netfilter the ipv6 being yes/no does not matter at all - it would always work because ip6tables is not consulted. On the other hand, having br_netfilter makes ipv6 yes/no do not matter as well, only that it now never works unless you enable ipv6 and assign a proper address. I believe this issue is likely to need handling at the kernel level. I chose libvirt because it is affected.
Erratum: "unroutable" is generated by the guest itself. In fact, no routable ipv6 address on host acts the same as disabled ipv6 (as far as guests are concerned). Also, for completeness, the behavior for ipv4 is that, with no ipv4 address on host and br_netfilter being on, ipv4 works in guests. Rules in iptables are hit and packets are properly bridged.
More info: link-local addresses behave the same, i.e. it has to be assigned (so at least disable_ipv6=0) on the host to work on guests. More info: openvswitch is affected too (not only linux bridges).
Preciser statement: by br_netfilter being on I mean that the module is loaded and relevant variables activated, i.e.: # sysctl net.bridge net.bridge.bridge-nf-call-arptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 ... More info: this may affect OpenStack (neutron) data plane because it requires this exact scenario, see https://opendev.org/openstack/neutron/src/commit/555535977662e505dfe219d6918ac0fb41624eaa/doc/source/install/compute-install-option2-rdo.rst
This behaviour of the br_netfilter module is indeed uphelpful. When net.bridge.bridge-nf-call-XXXtables = 1 all guest traffic is processed by the host firewall, and depending on what rules you have in that you may well see traffic dropped. With the virtual networks, libvirt tries to add rules that should workaround this problem by explicitly allowing traffic for guest tap devices on the bridge libvirt creates. If this isn't working for IPv6 isolated network though, then we have a bug in libvirt rules in this respect.
To me it looks like ip6tables does something awry. Libvirt simply mimicks the behavior of iptables there.
The ACCEPT rules, as mentioned, are added by libvirt, but they do not help as: a) they are completely ignored in the disable_ipv6 case or b) they accept traffic but it still never reaches other guests as host finds them unroutable.
Could you share the output of 'virsh net-dumpxml $NETWORK' for the network you created, so I can be sure I'm testing/debugging the exact same setup as you. Ignoring VMs for a minute, does your host itself actually have IPv6 connectivity / addresses configured for its own LAN ?
Sure, here: <network connections='7' ipv6='yes'> <name>kolla-internal</name> <uuid>399ddf2e-686f-4438-9627-ac392937ecef</uuid> <bridge name='virbr2' stp='on' delay='0'/> <mac address='52:54:00:2d:e9:3e'/> </network> As mentioned, ipv6 toggle is useless in either br_netfilter state (either unused or not effective). Yes, the host deals with IPv6 connectivity just fine. The point is isolated VM networks do not want to work with IPv6. (So the host does not have an IPv6 on this virbr2 on purpose.)
Thanks for the config & hints. I've confirmed I see the same behaviour on a rhel 7.6 host with the config you have here. For sake of testing I created two Cirros guests http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img with their NIC connected to the 'kolla-internal' network. After waiting for them to boot & fail DHCP (expected), they have a link local IPv6 address each. Trying to ping6 from one guest to the other guest fails. Running tcpdump on the vnet0 TAP device on the rhel7 host I can see repeated neighbor solicitation packets for the other guests' ipv6 link local address, eg 12:55:01.518691 IP6 fe80::5054:ff:fe50:2e4e > ff02::1:ffb7:64d7: ICMP6, neighbor solicitation, who has fe80::5054:ff:feb7:64d7, length 32 No reply is ever seen Now, testing the identical configuration on a rhel8 host, everything works as intended - ping6 works, ssh works. Running tcpdump on the vnet0 TAP device on the rhel8 host I can see the expected replies 12:52:36.153381 IP6 fe80::5054:ff:fea6:497b > ff02::1:ff47:8904: ICMP6, neighbor solicitation, who has fe80::5054:ff:fe47:8904, length 32 12:52:36.155194 IP6 fe80::5054:ff:fe47:8904 > fe80::5054:ff:fea6:497b: ICMP6, neighbor advertisement, tgt is fe80::5054:ff:fe47:8904, length 32 12:52:36.156380 IP6 fe80::5054:ff:fea6:497b > fe80::5054:ff:fe47:8904: ICMP6, echo request, seq 0, length 64 12:52:36.158254 IP6 fe80::5054:ff:fe47:8904 > fe80::5054:ff:fea6:497b: ICMP6, echo reply, seq 0, length 64 Some we have some problem with ICMP6 neighbor advertisement on rhel7.
Slight correction - my RHEL-8 host wasn't configured quite the same way - firewalld was disabled. When I start firewalld on RHEL-8, IPv6 breaks too, though the tcpdump traffic is slightly different. Back on RHEL-7 I see the problem is that firewalld is adding rules to the 'raw' table that block traffic Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT ipv6-icmp any any anywhere anywhere ipv6-icmp router-advertisement 685 50696 DROP all any any anywhere anywhere rpfilter invert Can you confirm that you have 'firewalld' running on your host too ?
# firewall-cmd --list-all FirewallD is not running
Do you have any other kind of firewall setup on the host ? Can you attach output of 'ip6tables-save' to show the full ruleset ? It probably doesn't matter, but for the record, what exact kernel version + libvirt version do you have.
kernel: 3.10.0-957.27.2.el7.x86_64 libvirt: 4.5.0-10.el7_6.12
Thanks for thorough investigation. I completely forgot about raw tables (checked mangle though :-) ). I found out that on one tested host firewalld was indeed up and running. The culprit was: -A PREROUTING -m rpfilter --invert -j DROP So in this case firewalld was dropping traffic and stopping it would fix it. The ipv4 rp_filter is exempt from bridge traffic but the one in ip6tables seems not. This is the bug. Can this rule be made exempt from the bridge traffic? I have to redo the test on the other machine where there was no firewalld running and get back to you. It probably had some other filtering rule that caught the traffic. It could be unrelated.
Hey Eric - is there something that firewalld could/should do to solve this? Or maybe something extra that we should be doing to counter-act firewalld's rules? A short synopsis is that we're setting up a bridge that has no IP connectivity on the host itself, but want the guests connected to that bridge to be able to send/receive IPv6 traffic to each other, but when br_netfilter is loaded and firewalld is active, that doesn't work.
(In reply to Laine Stump from comment #16) > Hey Eric - is there something that firewalld could/should do to solve this? > Or maybe something extra that we should be doing to counter-act firewalld's > rules? > > A short synopsis is that we're setting up a bridge that has no IP > connectivity on the host itself, but want the guests connected to that > bridge to be able to send/receive IPv6 traffic to each other, but when > br_netfilter is loaded and firewalld is active, that doesn't work. Seems like it's not passing the IPv6 rpfilter. Workaround: In /etc/firewalld/firewalld.conf set IPv6_rpfilter=no. Long-term: This comes down to ip6tables rpfilter in the kernel. The host has no route to the source, so it makes sense the check fails. I'm not sure what, if anything, we can do in the case of br_netfilter.
What about extending that rule with: -m physdev ! --physdev-is-in Though I believe this switches off valid security for bridges in general... If only one could postpone rpfilter till bridge/route decision. :-) Maybe let libvirt set bridges it controls exempt from this check? But then again docker/podman bridges would be affected anyway.
(In reply to Radosław Piliszek from comment #18) > What about extending that rule with: > -m physdev ! --physdev-is-in > Though I believe this switches off valid security for bridges in general... You are correct. Unfortunately there in no nftables equivalent, which is what firewalld uses in RHEL-8.
For completeness, I confirm my other issue was unrelated. This bug is relevant to libvirt+firewalld.
Folks, I gave it some thought and I think it would be best, in the RHEL7 case, to convert the rpfilter DROP to MARK and then in filter FORWARDING do DROP if marked and routed (as opposed to bridged). This would ensure firewalld provides equal security while bridging still works.
(In reply to Radosław Piliszek from comment #24) > Folks, I gave it some thought and I think it would be best, in the RHEL7 > case, to convert the rpfilter DROP to MARK and then in filter FORWARDING do > DROP if marked and routed (as opposed to bridged). This would ensure > firewalld provides equal security while bridging still works. We'd effectively be marking all IPv6 packets. This means user won't be able to set their own marks. firewalld would either overwrite their mark or they'd overwrite the rpfilter mark.
> We'd effectively be marking all IPv6 packets. This means user won't be able > to set their own marks. firewalld would either overwrite their mark or > they'd overwrite the rpfilter mark. I disagree with the first sentence. We would only be marking packets failing rpfilter. Though I agree that introducing a magic number is a breaking change. Then it falls to kernel to let rpfilter DROP only routed packets (as it does for IPv4 as controlled by sysctl vars)...
(In reply to Radosław Piliszek from comment #26) > > We'd effectively be marking all IPv6 packets. This means user won't be able > > to set their own marks. firewalld would either overwrite their mark or > > they'd overwrite the rpfilter mark. > > I disagree with the first sentence. We would only be marking packets failing > rpfilter. Right. I misspoke. > Though I agree that introducing a magic number is a breaking change. > > Then it falls to kernel to let rpfilter DROP only routed packets (as it does > for IPv4 as controlled by sysctl vars)... I'm currently discussing the options with my colleagues. I'd like to see the following: - Use "-m physdev ! --physdev-is-in" for the firewalld iptables backend - Change upstream nftables (kernel) to completely ignore packets coming from bridge devices - depends on upstream acceptance - br_netfilter doesn't make sense with nftables as nftables as full bridge filtering capabilities
Thank you for reporting this issue to the libvirt project. Unfortunately we have been unable to resolve this issue due to insufficient maintainer capacity and it will now be closed. This is not a reflection on the possible validity of the issue, merely the lack of resources to investigate and address it, for which we apologise. If you none the less feel the issue is still important, you may choose to report it again at the new project issue tracker https://gitlab.com/libvirt/libvirt/-/issues The project also welcomes contribution from anyone who believes they can provide a solution.