Description of problem: Sometimes, libvirt (presumably) adds additional entries to iptables, causing iptables to grow without end. Version-Release number of selected component (if applicable): libvirt-0.9.4-12.el6.x86_64 iptables-1.4.7-4.el6.x86_64 kernel-2.6.32-202.el6.x86_64 How reproducible: Likely synchronous, but conditions are unknown. Steps to Reproduce: *unknown* Actual results: The iptables tables grow without end. Expected results: Additional entries add and disappear. Additional info: The iptables configuration has handcrafted entries, but they are rather limited in scope. I tried to catch the exact condition when this happens, but I was unable. No matter what I do with the box, libvirt carefuly adds and removes the entries. But then I look away for a month and voila: a leak.
Created attachment 551308 [details] iptables-save output
This could be a tough one to figure out. One clue is that some of the rules are duplicated and some aren't. By feeding the iptables rules through this filter: grep virbr0 | sort | uniq -c | sort we can see which rules were duplicated and which weren't: 1 -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 1 -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 1 -A INPUT -i virbr0 -j ACCEPT <-- **THIS RULE ISN'T ADDED BY LIBVIRT** 2 -A FORWARD -d 192.168.129.16/28 -o virbr0 -j ACCEPT 2 -A FORWARD -i virbr0 -o virbr0 -j ACCEPT 2 -A FORWARD -s 192.168.129.16/28 -i virbr0 -j ACCEPT 2 -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 2 -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 2 -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 2 -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 2 -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill I can't think of a way that libvirt would be actually adding two identical rules at the same time, so possibly there's some condition where the rules are being removed, and only some of them get removed. The rules are removed in 3 conditions I can think of: 1) when a network is destroyed (i.e. "virsh net-destroy default") 2) when there is an error while a network is being started. 3) when libvirtd is restarted (it removes and re-adds all the rules for all active networks). (1) and (3) both call the same function (networkRemoveIptablesRules). It would make sense that the non-duplicated rules would be the ones being removed first, and the duplicated ones would be removed later (with some sort of problem causing the removal to abort midstream). However, when I look at networkRemoveIptablesRules, this is the order I see: 2 -A FORWARD -d 192.168.129.16/28 -o virbr0 -j ACCEPT 2 -A FORWARD -s 192.168.129.16/28 -i virbr0 -j ACCEPT 2 -A FORWARD -i virbr0 -o virbr0 -j ACCEPT 1 -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 1 -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 2 -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 2 -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 2 -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill 2 -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 2 -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT If there was a failure during rule removal, the first three rules would have already been removed, so they wouldn't be duplicated. On the other hand, taking the order they're being added: 2 -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 2 -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 2 -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill 2 -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 2 -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 1 -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 1 -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 2 -A FORWARD -i virbr0 -o virbr0 -j ACCEPT 2 -A FORWARD -s 192.168.129.16/28 -i virbr0 -j ACCEPT 2 -A FORWARD -d 192.168.129.16/28 -o virbr0 -j ACCEPT If there was an error after adding the "--dport 53" rules that resulted in a failure to cleanup, there wouldn't be any extras of the final three rules. So neither of those scenarios makes sense. Basically, if this is happening during the adding or removing of rules by libvirt, there needs to be a situation where the REJECT rules are added last, or removed first. A manual search of bridge_driver.c didn't show either of those cases. At this point I think the best that can be done is to try checking the iptables rules after every system restart, libvirtd restart, ????, (maybe a short script that does grep virbr0 | sort | uniq -c | grep " 2" and raises some kind of alarm when the output isn't empty; it could for example grab the last bit of log from /var/log/libvirt/libvirtd.log (and /var/log/messages) and email that off to or something). Hopefully another clue will pop up.
I am having the same issue on a fully up-2-date Fedora 16 system as well. After changing 'IPTABLES_SAVE_ON_STOP' to no (/etc/sysconfig/iptables-config) the accumulation stops. But this can only be used as a workaround, it is not a solution.
I would be surprised if the root cause were the same, as RHEL 6 roughly corresponds to F13, and F16 is way newer. But thanks for the hint, I'll give it a try.
I used Freddy's workaround from comment #4, but forgot if I trimmed the duplicates. Trimming now, zero day for my testing. Also, updated to RHEL 6.3 Beta.
If this helps, the problem apparently occurs on boot, not on shutdown. If you look at /etc/sysconfig/iptables on a running system, the number of duplicates in the iptables-save output is 1 more than in the state file.
Pete - do you still see the extra entries when using 6.3 beta?
Yes. Still happens in the 6.3 Beta. However, I am using Freddy's suggestion to keep them under control, so it's not as urgent.
Given the age of this bug, and all the changes in this area, this bug may very well be gone. Closing as DEFERRED; if anyone can still reproduce with modern libvirt, please reopen
I have the same issue on Ubuntu 17.10 with libvirt 3.10.0. For instance on virbr0 (I use 10 virbrn Linux bridges): Before "iptables-save > /etc/iptables/rules.v4" ----------------------------------------------- # cat iptables/rules.v4 | grep virbr0 | sort | uniq -c | sort 14 -A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 14 -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 14 -A FORWARD -i virbr0 -o virbr0 -j ACCEPT 14 -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 14 -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT 14 -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 14 -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 14 -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 14 -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 14 -A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT 1 -A ufw-user-input -i virbr0 -j ACCEPT 1 -A ufw-user-output -o virbr0 -j ACCEPT 33 -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill After "iptables-save > /etc/iptables/rules.v4" ---------------------------------------------- # cat iptables/rules.v4 | grep virbr0 | sort | uniq -c | sort 15 -A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 15 -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 15 -A FORWARD -i virbr0 -o virbr0 -j ACCEPT 15 -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 15 -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT 15 -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 15 -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 15 -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 15 -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 15 -A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT 1 -A ufw-user-input -i virbr0 -j ACCEPT 1 -A ufw-user-output -o virbr0 -j ACCEPT 34 -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill It looks like iptables-save is confused by virbrn entries and duplicates them each time it is run.
Actually, this issue is even broader as it relates to all types of Linux bridges, so libvirt does not appear as the root cause: # cat iptables/rules.v4 | grep lxcbr0 | sort | uniq -c 7 -A FORWARD -i lxcbr0 -j ACCEPT 7 -A FORWARD -o lxcbr0 -j ACCEPT 7 -A INPUT -i lxcbr0 -p tcp -m tcp --dport 53 -j ACCEPT 7 -A INPUT -i lxcbr0 -p tcp -m tcp --dport 67 -j ACCEPT 7 -A INPUT -i lxcbr0 -p udp -m udp --dport 53 -j ACCEPT 7 -A INPUT -i lxcbr0 -p udp -m udp --dport 67 -j ACCEPT 13 -A POSTROUTING -o lxcbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill