Red Hat Bugzilla – Bug 676622
libvirtd errors after upgrading libvirtd
Last modified: 2012-02-21 01:17:54 EST
Created attachment 478053 [details] /var/log/messages on affected server Description of problem: After upgrading various virtualisation packages included in the RHEL5.6 release I got the following errors during the patching process: Feb 10 08:00:34 drake libvirtd: 08:00:34.475: error : virRunWithHook:856 : internal error '/sbin/iptables --table nat --delete POSTROUTING --source 192.168.122.0/255.255.255.0 -p tcp ! --destination 192.168.122.0/255.255.255.0 --jump MASQUERADE --to-ports 1024-65535' exited with non-zero status 1 and signal 0: iptables: No chain/target/match by that name (this second one multiple times) and Feb 10 08:00:35 drake libvirtd: 08:00:35.728: warning : qemudStartup:1662 : Unable to create cgroup for driver: No such device or address This has happened on two servers. Both are running RHEL5 64-bit, with five guests. These messages also appeared again on reboot into the new kernel. I have a third server which I have not yet updated. Version-Release number of selected component (if applicable): Updated and erroring servers: # uname -a Linux drake 2.6.18-238.1.1.el5xen #1 SMP Tue Jan 4 13:52:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux # rpm -qa | grep virt | sort libvirt-0.8.2-15.el5_6.1 libvirt-0.8.2-15.el5_6.1 libvirt-python-0.8.2-15.el5_6.1 python-virtinst-0.400.3-11.el5 rhn-virtualization-common-5.4.14-1.el5sat rhn-virtualization-host-5.4.14-1.el5sat virt-manager-0.6.1-13.el5 virt-top-1.0.4-3.1.el5 virt-viewer-0.0.2-3.el5 # rpm -qa | grep xen | sort kernel-xen-2.6.18-194.32.1.el5 kernel-xen-2.6.18-238.1.1.el5 kernel-xen-devel-2.6.18-194.32.1.el5 kernel-xen-devel-2.6.18-238.1.1.el5 xen-3.0.3-120.el5 xen-libs-3.0.3-120.el5 xen-libs-3.0.3-120.el5 Not updated and not erroring server: # uname -a Linux ayres 2.6.18-194.32.1.el5xen #1 SMP Mon Dec 20 11:01:33 EST 2010 x86_64 x86_64 x86_64 GNU/Linux # rpm -qa | grep virt | sort libvirt-0.6.3-33.el5_5.3 libvirt-0.6.3-33.el5_5.3 libvirt-python-0.6.3-33.el5_5.3 python-virtinst-0.400.3-9.el5_5.1 rhn-virtualization-common-5.4.14-1.el5sat rhn-virtualization-host-5.4.14-1.el5sat virt-manager-0.6.1-12.el5 virt-top-1.0.4-3.1.el5 virt-viewer-0.0.2-3.el5 # rpm -qa | grep xen | sort kernel-xen-2.6.18-194.32.1.el5 kernel-xen-devel-2.6.18-194.32.1.el5 xen-3.0.3-105.el5_5.5 xen-libs-3.0.3-105.el5_5.5 xen-libs-3.0.3-105.el5_5.5 How reproducible: Every time. Restarting libvirtd while the machine is running also produces the following: Feb 10 13:07:05 drake libvirtd: 13:07:05.067: warning : qemudDispatchSignalEvent:402 : Shutting down on signal 15 Feb 10 13:07:05 drake libvirtd: 13:07:05.251: error : virRunWithHook:856 : internal error '/sbin/iptables --table filter --delete INPUT --in-interface virbr0 --protocol udp --destination-port 69 --jump ACCEPT' exited with non-zero status 1 and signal 0: iptables: Bad rule (does a matching rule exist in that chain?) Feb 10 13:07:05 drake libvirtd: 13:07:05.506: warning : qemudStartup:1662 : Unable to create cgroup for driver: No such device or address Steps to Reproduce: 1. Run yum update on a RHEL5.5 box 2. Watch /var/log/messages during patching, shutdown and reboot Actual results: There doesn't _seem_ to be an impact on guests starting. Although two guests on one Dom0 didn't restart on reboot and had to be started manually using "xm create -c <guest>" after the Dom0 had booted. The other three came up properly. Expected results: No errors or warnings. Additional info: See attached /var/log/messages and /var/log/xen/xend.log from one of the affected servers. This is the server that (as mentioned above) skipped starting two of its guests (it now have vif1.0 to vif7.0 rather than just 1.0-5.0 as two got disabled). This too is a bit worrying.
Created attachment 478054 [details] xend.log on affected server
Any word on this? Has anyone else experienced it, or knows how to make it go away?
I'm also confirming this one after upgrading to 5.6 I've even undefined all filters with virsh nwfilter-undefine but I still get this. According to https://bugzilla.redhat.com/show_bug.cgi?id=433484 this has to do with 'nat' mode. This causes many problems, and firewall should be manually restarted after all VMs come up, in order for their network to work. There should be a way to disable these automatic iptables rules. We don't want another program to mess with our firewall rules, unless we choose to do so. Giannis
In advance this happens on both KVM and XEN. It has to do libvirtd
I would like to confirm that the networks and guests all operate normally after this error is logged, and that the problem is just the log messages themselves. Is this the case?
For me, yes, the Dom0 and DomUs all appear to be unaffected network connectivity-wise. The problem for me is that if something's being logged _something_ is being affected. It'd be nice to know what, and why.
(In reply to comment #0) > After upgrading various virtualisation packages included in the RHEL5.6 release > I got the following errors during the patching process: > > > Feb 10 08:00:34 drake libvirtd: 08:00:34.475: error : virRunWithHook:856 : > internal error '/sbin/iptables --table nat --delete POSTROUTING --source > 192.168.122.0/255.255.255.0 -p tcp ! --destination 192.168.122.0/255.255.255.0 > --jump MASQUERADE --to-ports 1024-65535' exited with non-zero status 1 and > signal 0: iptables: No chain/target/match by that name Examining the source code of the previous release (0.6.3) shows that this rule in fact would *NOT* be preset during an upgrade, but should be present during any subsequent reloading of the rules, so this message is a benign, one time artifact of the upgrade process. (I also looked at the code that adds and deletes the iptables rules again, and was reminded that, due to the extreme amount of code sharing and levels of nesting, modifying the code to not log a message when deletion of a rule fails would be very invasive, so it would not be advisable to patch the RHEL5 version of the software in such a general way.) > > Feb 10 08:00:35 drake libvirtd: 08:00:35.728: warning : qemudStartup:1662 : > Unable to create cgroup for driver: No such device or address This message is also benign. > Every time. Restarting libvirtd while the machine is running also produces the > following: > > Feb 10 13:07:05 drake libvirtd: 13:07:05.067: warning : > qemudDispatchSignalEvent:402 : Shutting down on signal 15 > Feb 10 13:07:05 drake libvirtd: 13:07:05.251: error : virRunWithHook:856 : > internal error '/sbin/iptables --table filter --delete INPUT --in-interface > virbr0 --protocol udp --destination-port 69 --jump ACCEPT' exited with non-zero > status 1 and signal 0: iptables: Bad rule (does a matching rule exist in that > chain?) The cause of this message (also benign, by the way) is best described by the commit comment of the upstream patch that fixed it: commit 0111cebb5a430b67e6579efe0a0bc0b39d6002c3 Author: Laine Stump <laine@laine.org> Date: Wed Oct 27 22:45:43 2010 -0400 Only attempt removal of the rule allowing tftp if it was added During virtual network startup, the iptables rule that allows tftp traffic is only added if network->def->tftproot is non-empty, but when the virtual network is destroyed, we had been unconditionally trying to delete the rule. This was harmless, except that it created a bogus error message. This patch conditionalizes the delete command in the same manner that the insert command is already conditionalized.
Verify this bug with libvirt-0.8.2-23.el5 kernel-xen-2.6.18-238.21.1.el5 After upgrade libvirt the /var/log/messages no "error" iptables messages ,but "warning" is still there : tail -f /var/log/messages Oct 26 10:44:18 intel-5130-16-2 libvirtd: 10:44:18.602: warning : qemudDispatchSignalEvent:402 : Shutting down on signal 15 Oct 26 10:44:19 intel-5130-16-2 libvirtd: 10:44:19.162: warning : qemudStartup:1667 : Unable to create cgroup for driver: No such device or address
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2012-0248.html