Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 676622 - libvirtd errors after upgrading libvirtd
libvirtd errors after upgrading libvirtd
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: libvirt (Show other bugs)
5.6
x86_64 Linux
medium Severity medium
: rc
: ---
Assigned To: Laine Stump
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-02-10 08:40 EST by Ben
Modified: 2012-02-21 01:17 EST (History)
12 users (show)

See Also:
Fixed In Version: libvirt-0.8.2-23.el5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-02-21 01:17:54 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/var/log/messages on affected server (56.46 KB, text/plain)
2011-02-10 08:40 EST, Ben
no flags Details
xend.log on affected server (636.04 KB, text/x-log)
2011-02-10 08:41 EST, Ben
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Legacy) 56601 None None None Never
Red Hat Product Errata RHBA-2012:0248 normal SHIPPED_LIVE libvirt bug fix update 2012-02-20 10:07:14 EST

  None (edit)
Description Ben 2011-02-10 08:40:00 EST
Created attachment 478053 [details]
/var/log/messages on affected server

Description of problem:

After upgrading various virtualisation packages included in the RHEL5.6 release I got the following errors during the patching process:


Feb 10 08:00:34 drake libvirtd: 08:00:34.475: error : virRunWithHook:856 : internal error '/sbin/iptables --table nat --delete POSTROUTING --source 192.168.122.0/255.255.255.0 -p tcp ! --destination 192.168.122.0/255.255.255.0 --jump MASQUERADE --to-ports 1024-65535' exited with non-zero status 1 and signal 0: iptables: No chain/target/match by that name
(this second one multiple times)

and 

Feb 10 08:00:35 drake libvirtd: 08:00:35.728: warning : qemudStartup:1662 : Unable to create cgroup for driver: No such device or address


This has happened on two servers.  Both are running RHEL5 64-bit, with five guests.  These messages also appeared again on reboot into the new kernel.  I have a third server which I have not yet updated.


Version-Release number of selected component (if applicable):

Updated and erroring servers:

# uname -a
Linux drake 2.6.18-238.1.1.el5xen #1 SMP Tue Jan 4 13:52:39 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa | grep virt | sort
libvirt-0.8.2-15.el5_6.1
libvirt-0.8.2-15.el5_6.1
libvirt-python-0.8.2-15.el5_6.1
python-virtinst-0.400.3-11.el5
rhn-virtualization-common-5.4.14-1.el5sat
rhn-virtualization-host-5.4.14-1.el5sat
virt-manager-0.6.1-13.el5
virt-top-1.0.4-3.1.el5
virt-viewer-0.0.2-3.el5

# rpm -qa | grep xen | sort
kernel-xen-2.6.18-194.32.1.el5
kernel-xen-2.6.18-238.1.1.el5
kernel-xen-devel-2.6.18-194.32.1.el5
kernel-xen-devel-2.6.18-238.1.1.el5
xen-3.0.3-120.el5
xen-libs-3.0.3-120.el5
xen-libs-3.0.3-120.el5


Not updated and not erroring server:

# uname -a
Linux ayres 2.6.18-194.32.1.el5xen #1 SMP Mon Dec 20 11:01:33 EST 2010 x86_64 x86_64 x86_64 GNU/Linux

# rpm -qa | grep virt | sort
libvirt-0.6.3-33.el5_5.3
libvirt-0.6.3-33.el5_5.3
libvirt-python-0.6.3-33.el5_5.3
python-virtinst-0.400.3-9.el5_5.1
rhn-virtualization-common-5.4.14-1.el5sat
rhn-virtualization-host-5.4.14-1.el5sat
virt-manager-0.6.1-12.el5
virt-top-1.0.4-3.1.el5
virt-viewer-0.0.2-3.el5

# rpm -qa | grep xen | sort
kernel-xen-2.6.18-194.32.1.el5
kernel-xen-devel-2.6.18-194.32.1.el5
xen-3.0.3-105.el5_5.5
xen-libs-3.0.3-105.el5_5.5
xen-libs-3.0.3-105.el5_5.5


How reproducible:

Every time.  Restarting libvirtd while the machine is running also produces the following:

Feb 10 13:07:05 drake libvirtd: 13:07:05.067: warning : qemudDispatchSignalEvent:402 : Shutting down on signal 15
Feb 10 13:07:05 drake libvirtd: 13:07:05.251: error : virRunWithHook:856 : internal error '/sbin/iptables --table filter --delete INPUT --in-interface virbr0 --protocol udp --destination-port 69 --jump ACCEPT' exited with non-zero status 1 and signal 0: iptables: Bad rule (does a matching rule exist in that chain?)
Feb 10 13:07:05 drake libvirtd: 13:07:05.506: warning : qemudStartup:1662 : Unable to create cgroup for driver: No such device or address


Steps to Reproduce:
1. Run yum update on a RHEL5.5 box
2. Watch /var/log/messages during patching, shutdown and reboot

  
Actual results:

There doesn't _seem_ to be an impact on guests starting.  Although two guests on one Dom0 didn't restart on reboot and had to be started manually using "xm create -c <guest>" after the Dom0 had booted.  The other three came up properly.


Expected results:

No errors or warnings.


Additional info:

See attached /var/log/messages and /var/log/xen/xend.log from one of the affected servers.  This is the server that (as mentioned above) skipped starting two of its guests (it now have vif1.0 to vif7.0 rather than just 1.0-5.0 as two got disabled).  This too is a bit worrying.
Comment 1 Ben 2011-02-10 08:41:02 EST
Created attachment 478054 [details]
xend.log on affected server
Comment 2 Ben 2011-04-01 08:31:53 EDT
Any word on this?  Has anyone else experienced it, or knows how to make it go away?
Comment 3 Kapetanakis Giannis 2011-04-16 06:10:31 EDT
I'm also confirming this one after upgrading to 5.6

I've even undefined all filters 
with virsh nwfilter-undefine but I still get this.

According to https://bugzilla.redhat.com/show_bug.cgi?id=433484 this has
to do with 'nat' mode.

This causes many problems, and firewall should be manually restarted after all VMs come up, in order for their network to work.

There should be a way to disable these automatic iptables rules. We don't want another program to mess with our firewall rules, unless we choose to do so.

Giannis
Comment 4 Kapetanakis Giannis 2011-04-16 06:12:29 EDT
In advance this happens on both KVM and XEN. It has to do libvirtd
Comment 9 Laine Stump 2011-09-02 14:02:13 EDT
I would like to confirm that the networks and guests all operate normally after this error is logged, and that the problem is just the log messages themselves. Is this the case?
Comment 10 Ben 2011-09-02 14:26:58 EDT
For me, yes, the Dom0 and DomUs all appear to be unaffected network connectivity-wise.

The problem for me is that if something's being logged _something_ is being affected.  It'd be nice to know what, and why.
Comment 11 Laine Stump 2011-09-05 00:52:44 EDT
(In reply to comment #0)

> After upgrading various virtualisation packages included in the RHEL5.6 release
> I got the following errors during the patching process:
> 
> 
> Feb 10 08:00:34 drake libvirtd: 08:00:34.475: error : virRunWithHook:856 :
> internal error '/sbin/iptables --table nat --delete POSTROUTING --source
> 192.168.122.0/255.255.255.0 -p tcp ! --destination 192.168.122.0/255.255.255.0
> --jump MASQUERADE --to-ports 1024-65535' exited with non-zero status 1 and
> signal 0: iptables: No chain/target/match by that name


Examining the source code of the previous release (0.6.3) shows that this rule in fact would *NOT* be preset during an upgrade, but should be present during any subsequent reloading of the rules, so this message is a benign, one time artifact of the upgrade process.

(I also looked at the code that adds and deletes the iptables rules again, and was reminded that, due to the extreme amount of code sharing and levels of nesting, modifying the code to not log a message when deletion of a rule fails would be very invasive, so it would not be advisable to patch the RHEL5 version of the software in such a general way.)


> 
> Feb 10 08:00:35 drake libvirtd: 08:00:35.728: warning : qemudStartup:1662 :
> Unable to create cgroup for driver: No such device or address

This message is also benign.

> Every time.  Restarting libvirtd while the machine is running also produces the
> following:
> 
> Feb 10 13:07:05 drake libvirtd: 13:07:05.067: warning :
> qemudDispatchSignalEvent:402 : Shutting down on signal 15
> Feb 10 13:07:05 drake libvirtd: 13:07:05.251: error : virRunWithHook:856 :
> internal error '/sbin/iptables --table filter --delete INPUT --in-interface
> virbr0 --protocol udp --destination-port 69 --jump ACCEPT' exited with non-zero
> status 1 and signal 0: iptables: Bad rule (does a matching rule exist in that
> chain?)

The cause of this message (also benign, by the way) is best described by the commit comment of the upstream patch that fixed it:

commit 0111cebb5a430b67e6579efe0a0bc0b39d6002c3
Author: Laine Stump <laine@laine.org>
Date:   Wed Oct 27 22:45:43 2010 -0400

    Only attempt removal of the rule allowing tftp if it was added
    
    During virtual network startup, the iptables rule that allows tftp
    traffic is only added if network->def->tftproot is non-empty, but when
    the virtual network is destroyed, we had been unconditionally trying
    to delete the rule. This was harmless, except that it created a bogus
    error message.
    
    This patch conditionalizes the delete command in the same manner that
    the insert command is already conditionalized.
Comment 14 Huang Wenlong 2011-10-25 22:50:01 EDT
Verify this bug with libvirt-0.8.2-23.el5 kernel-xen-2.6.18-238.21.1.el5 

After upgrade libvirt the /var/log/messages  no "error" iptables messages  ,but "warning" is still there :

tail -f /var/log/messages

Oct 26 10:44:18 intel-5130-16-2 libvirtd: 10:44:18.602: warning : qemudDispatchSignalEvent:402 : Shutting down on signal 15 
Oct 26 10:44:19 intel-5130-16-2 libvirtd: 10:44:19.162: warning : qemudStartup:1667 : Unable to create cgroup for driver: No such device or address
Comment 15 errata-xmlrpc 2012-02-21 01:17:54 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0248.html

Note You need to log in before you can comment on or make changes to this bug.