Bug 633346 - virInterface* functions don't work after reboot until libvirtd is restarted
virInterface* functions don't work after reboot until libvirtd is restarted
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: netcf (Show other bugs)
6.0
All Linux
high Severity medium
: rc
: ---
Assigned To: Laine Stump
qe-baseos-daemons
:
Depends On: 651032
Blocks:
  Show dependency treegraph
 
Reported: 2010-09-13 11:49 EDT by Alex Jia
Modified: 2011-05-19 09:43 EDT (History)
8 users (show)

See Also:
Fixed In Version: netcf-0.1.7-1.el6
Doc Type: Bug Fix
Doc Text:
Previously, after a reboot, virInterface functions did not work until libvirt was restarted and instead failed with a "error: Failed to list active interfaces. error: this function is not supported by the connection driver: virConnectNumOfInterfaces" error message. This is now fixed so that the code that caused the problem is removed and virInterface functions operate as expected after a reboot.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-05-19 09:43:36 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Alex Jia 2010-09-13 11:49:08 EDT
Description of problem:
virInterface* functions don't work after reboot until libvirtd is restarted on RHEL6.0 RC3.

Version-Release number of selected component (if applicable):
# uname -r
2.6.32-71.el6.x86_64

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.0 (Santiago)

# rpm -q libvirt
libvirt-0.8.1-27.el6.x86_64

# rpm -q netcf-libs
netcf-libs-0.1.6-4.el6.x86_64


How reproducible:
always

Steps to Reproduce:
1.install RHEL6.0 RC3
2.run virsh iface-list or use python API to list number of interface  
  
Actual results:
# virsh iface-list
error: Failed to list active interfaces
error: this function is not supported by the connection driver: virConnectNumOfInterfaces

Expected results:
fix it

Additional info:
# rpm -q NetworkManager
NetworkManager-0.8.1-5.el6.x86_64

# service NetworkManager status
NetworkManager is stopped

or 

# service NetworkManager status
NetworkManager is running

The result is the same for installing/uninstalling netcf package.
Comment 1 Laine Stump 2010-09-13 11:55:44 EDT
Some questions:

1) does it fail after *every* system reboot

2) does it succeed after *every* libvirtd restart (including multiple times between reboots)?

3) Does the behavior change if you change:

   net.bridge.bridge-nf-call-iptables = 1

in /etc/sysctl.conf to:

   net.bridge.bridge-nf-call-iptables = 0

(1 & 2 are general curiousity. If the answer to 3 is "yes", then the netcf code that modifies the iptables configuration will need to be removed sooner rather than later)
Comment 3 Alex Jia 2010-09-14 08:35:40 EDT
(In reply to comment #1)
> Some questions:
> 
> 1) does it fail after *every* system reboot

Yep, it will be successful if I reboot system every time or multiple times
> 
> 2) does it succeed after *every* libvirtd restart (including multiple times
> between reboots)?

Yep, it will be be successful if I reboot libvirtd service every time or multiple times

> 
> 3) Does the behavior change if you change:
> 
>    net.bridge.bridge-nf-call-iptables = 1
> 
> in /etc/sysctl.conf to:
> 
>    net.bridge.bridge-nf-call-iptables = 0

My configuration is "net.bridge.bridge-nf-call-iptables = 0" in /etc/sysctl.conf
> 
> (1 & 2 are general curiousity. If the answer to 3 is "yes", then the netcf code
> that modifies the iptables configuration will need to be removed sooner rather
> than later)
Comment 4 Laine Stump 2010-09-14 11:13:21 EDT
> My configuration is "net.bridge.bridge-nf-call-iptables = 0" in
> /etc/sysctl.conf

I'm fairly certain that yesterday I ran "sysctl -a | grep bridge" on your test machine, and saw that net.bridge.bridge-nf-call-iptables was set to 1. I'm unable to login to the machine now (did its IP address change?) to verify, though.

Can you check the setting of that variable with the sysctl command? It would be useful to know what process is changing it from the setting in /etc/sysctl.conf (libvirt has nothing to do with it, and netcf only examines it).
Comment 5 Alex Jia 2010-09-15 02:23:59 EDT
(In reply to comment #4)
> > My configuration is "net.bridge.bridge-nf-call-iptables = 0" in
> > /etc/sysctl.conf
> 
> I'm fairly certain that yesterday I ran "sysctl -a | grep bridge" on your test
> machine, and saw that net.bridge.bridge-nf-call-iptables was set to 1. I'm
> unable to login to the machine now (did its IP address change?) to verify,
> though.

IP has no change, and you can access that machine using previous user/password now.
> 
> Can you check the setting of that variable with the sysctl command? It would be
> useful to know what process is changing it from the setting in /etc/sysctl.conf
> (libvirt has nothing to do with it, and netcf only examines it).

Their value are inconsistent between sysctl.conf and current environment variable, it seems that the changing is only temporary and value is written into
sysctl.conf.

# cat /etc/sysctl.conf | grep bridge
# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

# sysctl -a | grep bridge
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0
Comment 6 Alex Jia 2010-09-15 02:49:44 EDT
(In reply to comment #5)
> (In reply to comment #4)
> > > My configuration is "net.bridge.bridge-nf-call-iptables = 0" in
> > > /etc/sysctl.conf
> > 
> > I'm fairly certain that yesterday I ran "sysctl -a | grep bridge" on your test
> > machine, and saw that net.bridge.bridge-nf-call-iptables was set to 1. I'm
> > unable to login to the machine now (did its IP address change?) to verify,
> > though.
> 
> IP has no change, and you can access that machine using previous user/password
> now.
> > 
> > Can you check the setting of that variable with the sysctl command? It would be
> > useful to know what process is changing it from the setting in /etc/sysctl.conf
> > (libvirt has nothing to do with it, and netcf only examines it).
> 
> Their value are inconsistent between sysctl.conf and current environment
> variable, it seems that the changing is only temporary and value is written
> into sysctl.conf.
In fact, after setting net.bridge.bridge-nf-call-iptables to 1 by you, I reboot host machine many times, but net.bridge.bridge-nf-call-iptables always is 1 using sysctl -a to see, so I think the changing is not temporary, it only hasn't been written sysctl.conf instead of writing to /proc/sys/net/bridge/bridge-nf-call-iptables and sysctl will read the file to get value of bridge-nf-call-iptables every time:

# cat /proc/sys/net/bridge/bridge-nf-call-iptables
1

> 
> # cat /etc/sysctl.conf | grep bridge
> # Disable netfilter on bridges.
> net.bridge.bridge-nf-call-ip6tables = 0
> net.bridge.bridge-nf-call-iptables = 0
> net.bridge.bridge-nf-call-arptables = 0
> 
> # sysctl -a | grep bridge
> net.bridge.bridge-nf-call-arptables = 1
> net.bridge.bridge-nf-call-iptables = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-filter-vlan-tagged = 0
> net.bridge.bridge-nf-filter-pppoe-tagged = 0
Comment 7 Laine Stump 2010-09-15 10:18:28 EDT
Yep. I see that behavior on both of my RHEL6 machines now - some process is setting all three net.bridge.bridge-nf-call-*tables variables to 1 unconditionally (which is bad enough by itself), and then also not updating sysctl.conf.

We really need to find out who is doing this and make them stop!

Note that if I restart either NetworkManager or the network service, the variables will be reset to their proper value (as given in sysctl.conf)
Comment 8 Laine Stump 2010-09-15 11:46:24 EDT
Okay, it's not *too* bad. Here's the situation:

1) the default settings of net.bridge.bridge-nf-call-*tables in the bridge module is 1.

2) the default in /etc/sysctl.conf is 0.

3) Since net.bridge.bridge-nf-call-iptables doesn't exist in the system until the bridge module is loaded, processing sysctl.conf prior to loading the bridge module doesn't change the module default.

4) if there is a bridge defined in the ifcfg scripts (eg /etc/sysconfig/network-scripts/ifcfg-br0), the "network" service will load the bridge module prior to (re)processing /etc/sysctl.conf, and the variable will be properly set to the value in /etc/sysctl.conf prior to libvirtd being loaded.

So once you've defined at least one bridge interface on the host, everything will work properly.

Removing the code in netcf that checks/sets the iptables config will completely solve the problem.
Comment 9 Laine Stump 2010-11-04 11:16:10 EDT
This problem is completely remedied by rebasing to netcf-0.1.7, which has removed all the code that manipulates iptables rules (and it's that code that's causing trouble).

There isn't much new functionality in 0.1.7, so a rebase carries very low risk of regression.
Comment 11 Laine Stump 2011-01-13 15:37:36 EST
The fix for this problem is in netcf-0.17-1, which has been built:

https://brewweb.devel.redhat.com/buildinfo?buildID=154128
Comment 14 Laine Stump 2011-01-17 06:49:37 EST
Your test is invalid in determining whether or not the bug is fixed.

The change to netcf *does not* modify the setting of net.bridge.bridge-nf-call-iptables, nor does it remove any existing iptables rules. It only removes all of the code in netcf that *examines or changes these two items*.

Thus, looking at the setting of net.bridge.bridge-nf-call-iptables will not tell you anything about whether or not the bug has been fixed. The only external changes that you will be able to notice are:

1) netcf will no longer *add* the following rule to iptables (it also will not *remove* the rule if it was already added by someone else, including an earlier version of netcf):

   -A FORWARD -m physdev --physdev-is-bridged -j ACCEPT 

2) In certain cases where netcf previously failed to initialize due to inability to parse the iptables configuration, it will now succeed. Past experience has shown that this has always been the cause of failures to initialize.

The test you should run is to make sure that ncftool is able to start (and whether the iface-* commands are available in virsh) after rebooting the host (without needing an extra restart of libirtd), whether the setting of net.bridge.bridge-nf-call-iptables is 0 or 1.
Comment 17 Laine Stump 2011-01-18 02:10:37 EST
Yes, this verifies that it passes.

Thanks for going through all that!
Comment 18 Misha H. Ali 2011-05-09 01:40:15 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, after a reboot, virInterface functions did not work until libvirt was restarted and instead failed with a "error: Failed to list active interfaces. error: this function is not supported by the connection driver: virConnectNumOfInterfaces" error message. This is now fixed so that the code that caused the problem is removed and virInterface functions operate as expected after a reboot.
Comment 19 errata-xmlrpc 2011-05-19 09:43:36 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0620.html

Note You need to log in before you can comment on or make changes to this bug.