Bug 772368
Summary: | Accumulating entries in iptables | ||||||
---|---|---|---|---|---|---|---|
Product: | [Community] Virtualization Tools | Reporter: | Pete Zaitcev <zaitcev> | ||||
Component: | libvirt | Assignee: | Libvirt Maintainers <libvirt-maint> | ||||
Status: | CLOSED DEFERRED | QA Contact: | |||||
Severity: | unspecified | Docs Contact: | |||||
Priority: | unspecified | ||||||
Version: | unspecified | CC: | actionmystique, crobinso, cwei, dyuan, freddy, mshao, mzhan, rbalakri | ||||
Target Milestone: | --- | ||||||
Target Release: | --- | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2016-03-23 21:29:04 UTC | Type: | --- | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Pete Zaitcev
2012-01-07 00:54:32 UTC
Created attachment 551308 [details]
iptables-save output
This could be a tough one to figure out. One clue is that some of the rules are duplicated and some aren't. By feeding the iptables rules through this filter: grep virbr0 | sort | uniq -c | sort we can see which rules were duplicated and which weren't: 1 -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 1 -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 1 -A INPUT -i virbr0 -j ACCEPT <-- **THIS RULE ISN'T ADDED BY LIBVIRT** 2 -A FORWARD -d 192.168.129.16/28 -o virbr0 -j ACCEPT 2 -A FORWARD -i virbr0 -o virbr0 -j ACCEPT 2 -A FORWARD -s 192.168.129.16/28 -i virbr0 -j ACCEPT 2 -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 2 -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 2 -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 2 -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 2 -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill I can't think of a way that libvirt would be actually adding two identical rules at the same time, so possibly there's some condition where the rules are being removed, and only some of them get removed. The rules are removed in 3 conditions I can think of: 1) when a network is destroyed (i.e. "virsh net-destroy default") 2) when there is an error while a network is being started. 3) when libvirtd is restarted (it removes and re-adds all the rules for all active networks). (1) and (3) both call the same function (networkRemoveIptablesRules). It would make sense that the non-duplicated rules would be the ones being removed first, and the duplicated ones would be removed later (with some sort of problem causing the removal to abort midstream). However, when I look at networkRemoveIptablesRules, this is the order I see: 2 -A FORWARD -d 192.168.129.16/28 -o virbr0 -j ACCEPT 2 -A FORWARD -s 192.168.129.16/28 -i virbr0 -j ACCEPT 2 -A FORWARD -i virbr0 -o virbr0 -j ACCEPT 1 -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 1 -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 2 -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 2 -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 2 -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill 2 -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 2 -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT If there was a failure during rule removal, the first three rules would have already been removed, so they wouldn't be duplicated. On the other hand, taking the order they're being added: 2 -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 2 -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 2 -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill 2 -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 2 -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 1 -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 1 -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 2 -A FORWARD -i virbr0 -o virbr0 -j ACCEPT 2 -A FORWARD -s 192.168.129.16/28 -i virbr0 -j ACCEPT 2 -A FORWARD -d 192.168.129.16/28 -o virbr0 -j ACCEPT If there was an error after adding the "--dport 53" rules that resulted in a failure to cleanup, there wouldn't be any extras of the final three rules. So neither of those scenarios makes sense. Basically, if this is happening during the adding or removing of rules by libvirt, there needs to be a situation where the REJECT rules are added last, or removed first. A manual search of bridge_driver.c didn't show either of those cases. At this point I think the best that can be done is to try checking the iptables rules after every system restart, libvirtd restart, ????, (maybe a short script that does grep virbr0 | sort | uniq -c | grep " 2" and raises some kind of alarm when the output isn't empty; it could for example grab the last bit of log from /var/log/libvirt/libvirtd.log (and /var/log/messages) and email that off to or something). Hopefully another clue will pop up. I am having the same issue on a fully up-2-date Fedora 16 system as well. After changing 'IPTABLES_SAVE_ON_STOP' to no (/etc/sysconfig/iptables-config) the accumulation stops. But this can only be used as a workaround, it is not a solution. I would be surprised if the root cause were the same, as RHEL 6 roughly corresponds to F13, and F16 is way newer. But thanks for the hint, I'll give it a try. I used Freddy's workaround from comment #4, but forgot if I trimmed the duplicates. Trimming now, zero day for my testing. Also, updated to RHEL 6.3 Beta. If this helps, the problem apparently occurs on boot, not on shutdown. If you look at /etc/sysconfig/iptables on a running system, the number of duplicates in the iptables-save output is 1 more than in the state file. Pete - do you still see the extra entries when using 6.3 beta? Yes. Still happens in the 6.3 Beta. However, I am using Freddy's suggestion to keep them under control, so it's not as urgent. Given the age of this bug, and all the changes in this area, this bug may very well be gone. Closing as DEFERRED; if anyone can still reproduce with modern libvirt, please reopen I have the same issue on Ubuntu 17.10 with libvirt 3.10.0. For instance on virbr0 (I use 10 virbrn Linux bridges): Before "iptables-save > /etc/iptables/rules.v4" ----------------------------------------------- # cat iptables/rules.v4 | grep virbr0 | sort | uniq -c | sort 14 -A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 14 -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 14 -A FORWARD -i virbr0 -o virbr0 -j ACCEPT 14 -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 14 -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT 14 -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 14 -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 14 -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 14 -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 14 -A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT 1 -A ufw-user-input -i virbr0 -j ACCEPT 1 -A ufw-user-output -o virbr0 -j ACCEPT 33 -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill After "iptables-save > /etc/iptables/rules.v4" ---------------------------------------------- # cat iptables/rules.v4 | grep virbr0 | sort | uniq -c | sort 15 -A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 15 -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable 15 -A FORWARD -i virbr0 -o virbr0 -j ACCEPT 15 -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable 15 -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT 15 -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT 15 -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT 15 -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT 15 -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT 15 -A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT 1 -A ufw-user-input -i virbr0 -j ACCEPT 1 -A ufw-user-output -o virbr0 -j ACCEPT 34 -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill It looks like iptables-save is confused by virbrn entries and duplicates them each time it is run. Actually, this issue is even broader as it relates to all types of Linux bridges, so libvirt does not appear as the root cause: # cat iptables/rules.v4 | grep lxcbr0 | sort | uniq -c 7 -A FORWARD -i lxcbr0 -j ACCEPT 7 -A FORWARD -o lxcbr0 -j ACCEPT 7 -A INPUT -i lxcbr0 -p tcp -m tcp --dport 53 -j ACCEPT 7 -A INPUT -i lxcbr0 -p tcp -m tcp --dport 67 -j ACCEPT 7 -A INPUT -i lxcbr0 -p udp -m udp --dport 53 -j ACCEPT 7 -A INPUT -i lxcbr0 -p udp -m udp --dport 67 -j ACCEPT 13 -A POSTROUTING -o lxcbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill Actually, this issue is even broader as it relates to all types of Linux bridges, so libvirt does not appear as the root cause: # cat iptables/rules.v4 | grep lxcbr0 | sort | uniq -c 7 -A FORWARD -i lxcbr0 -j ACCEPT 7 -A FORWARD -o lxcbr0 -j ACCEPT 7 -A INPUT -i lxcbr0 -p tcp -m tcp --dport 53 -j ACCEPT 7 -A INPUT -i lxcbr0 -p tcp -m tcp --dport 67 -j ACCEPT 7 -A INPUT -i lxcbr0 -p udp -m udp --dport 53 -j ACCEPT 7 -A INPUT -i lxcbr0 -p udp -m udp --dport 67 -j ACCEPT 13 -A POSTROUTING -o lxcbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill |