Bug 633346

Some questions:

1) does it fail after *every* system reboot

2) does it succeed after *every* libvirtd restart (including multiple times between reboots)?

3) Does the behavior change if you change:

   net.bridge.bridge-nf-call-iptables = 1

in /etc/sysctl.conf to:

   net.bridge.bridge-nf-call-iptables = 0

(1 & 2 are general curiousity. If the answer to 3 is "yes", then the netcf code that modifies the iptables configuration will need to be removed sooner rather than later)

(In reply to comment #1)
> Some questions:
> 
> 1) does it fail after *every* system reboot

Yep, it will be successful if I reboot system every time or multiple times
> 
> 2) does it succeed after *every* libvirtd restart (including multiple times
> between reboots)?

Yep, it will be be successful if I reboot libvirtd service every time or multiple times

> 
> 3) Does the behavior change if you change:
> 
>    net.bridge.bridge-nf-call-iptables = 1
> 
> in /etc/sysctl.conf to:
> 
>    net.bridge.bridge-nf-call-iptables = 0

My configuration is "net.bridge.bridge-nf-call-iptables = 0" in /etc/sysctl.conf
> 
> (1 & 2 are general curiousity. If the answer to 3 is "yes", then the netcf code
> that modifies the iptables configuration will need to be removed sooner rather
> than later)

> My configuration is "net.bridge.bridge-nf-call-iptables = 0" in
> /etc/sysctl.conf

I'm fairly certain that yesterday I ran "sysctl -a | grep bridge" on your test machine, and saw that net.bridge.bridge-nf-call-iptables was set to 1. I'm unable to login to the machine now (did its IP address change?) to verify, though.

Can you check the setting of that variable with the sysctl command? It would be useful to know what process is changing it from the setting in /etc/sysctl.conf (libvirt has nothing to do with it, and netcf only examines it).

(In reply to comment #4)
> > My configuration is "net.bridge.bridge-nf-call-iptables = 0" in
> > /etc/sysctl.conf
> 
> I'm fairly certain that yesterday I ran "sysctl -a | grep bridge" on your test
> machine, and saw that net.bridge.bridge-nf-call-iptables was set to 1. I'm
> unable to login to the machine now (did its IP address change?) to verify,
> though.

IP has no change, and you can access that machine using previous user/password now.
> 
> Can you check the setting of that variable with the sysctl command? It would be
> useful to know what process is changing it from the setting in /etc/sysctl.conf
> (libvirt has nothing to do with it, and netcf only examines it).

Their value are inconsistent between sysctl.conf and current environment variable, it seems that the changing is only temporary and value is written into
sysctl.conf.

# cat /etc/sysctl.conf | grep bridge
# Disable netfilter on bridges.
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

# sysctl -a | grep bridge
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-filter-pppoe-tagged = 0

(In reply to comment #5)
> (In reply to comment #4)
> > > My configuration is "net.bridge.bridge-nf-call-iptables = 0" in
> > > /etc/sysctl.conf
> > 
> > I'm fairly certain that yesterday I ran "sysctl -a | grep bridge" on your test
> > machine, and saw that net.bridge.bridge-nf-call-iptables was set to 1. I'm
> > unable to login to the machine now (did its IP address change?) to verify,
> > though.
> 
> IP has no change, and you can access that machine using previous user/password
> now.
> > 
> > Can you check the setting of that variable with the sysctl command? It would be
> > useful to know what process is changing it from the setting in /etc/sysctl.conf
> > (libvirt has nothing to do with it, and netcf only examines it).
> 
> Their value are inconsistent between sysctl.conf and current environment
> variable, it seems that the changing is only temporary and value is written
> into sysctl.conf.
In fact, after setting net.bridge.bridge-nf-call-iptables to 1 by you, I reboot host machine many times, but net.bridge.bridge-nf-call-iptables always is 1 using sysctl -a to see, so I think the changing is not temporary, it only hasn't been written sysctl.conf instead of writing to /proc/sys/net/bridge/bridge-nf-call-iptables and sysctl will read the file to get value of bridge-nf-call-iptables every time:

# cat /proc/sys/net/bridge/bridge-nf-call-iptables
1

> 
> # cat /etc/sysctl.conf | grep bridge
> # Disable netfilter on bridges.
> net.bridge.bridge-nf-call-ip6tables = 0
> net.bridge.bridge-nf-call-iptables = 0
> net.bridge.bridge-nf-call-arptables = 0
> 
> # sysctl -a | grep bridge
> net.bridge.bridge-nf-call-arptables = 1
> net.bridge.bridge-nf-call-iptables = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-filter-vlan-tagged = 0
> net.bridge.bridge-nf-filter-pppoe-tagged = 0

Yep. I see that behavior on both of my RHEL6 machines now - some process is setting all three net.bridge.bridge-nf-call-*tables variables to 1 unconditionally (which is bad enough by itself), and then also not updating sysctl.conf.

We really need to find out who is doing this and make them stop!

Note that if I restart either NetworkManager or the network service, the variables will be reset to their proper value (as given in sysctl.conf)

Okay, it's not *too* bad. Here's the situation:

1) the default settings of net.bridge.bridge-nf-call-*tables in the bridge module is 1.

2) the default in /etc/sysctl.conf is 0.

3) Since net.bridge.bridge-nf-call-iptables doesn't exist in the system until the bridge module is loaded, processing sysctl.conf prior to loading the bridge module doesn't change the module default.

4) if there is a bridge defined in the ifcfg scripts (eg /etc/sysconfig/network-scripts/ifcfg-br0), the "network" service will load the bridge module prior to (re)processing /etc/sysctl.conf, and the variable will be properly set to the value in /etc/sysctl.conf prior to libvirtd being loaded.

So once you've defined at least one bridge interface on the host, everything will work properly.

Removing the code in netcf that checks/sets the iptables config will completely solve the problem.

This problem is completely remedied by rebasing to netcf-0.1.7, which has removed all the code that manipulates iptables rules (and it's that code that's causing trouble).

There isn't much new functionality in 0.1.7, so a rebase carries very low risk of regression.

The fix for this problem is in netcf-0.17-1, which has been built:

https://brewweb.devel.redhat.com/buildinfo?buildID=154128

Your test is invalid in determining whether or not the bug is fixed.

The change to netcf *does not* modify the setting of net.bridge.bridge-nf-call-iptables, nor does it remove any existing iptables rules. It only removes all of the code in netcf that *examines or changes these two items*.

Thus, looking at the setting of net.bridge.bridge-nf-call-iptables will not tell you anything about whether or not the bug has been fixed. The only external changes that you will be able to notice are:

1) netcf will no longer *add* the following rule to iptables (it also will not *remove* the rule if it was already added by someone else, including an earlier version of netcf):

   -A FORWARD -m physdev --physdev-is-bridged -j ACCEPT 

2) In certain cases where netcf previously failed to initialize due to inability to parse the iptables configuration, it will now succeed. Past experience has shown that this has always been the cause of failures to initialize.

The test you should run is to make sure that ncftool is able to start (and whether the iface-* commands are available in virsh) after rebooting the host (without needing an extra restart of libirtd), whether the setting of net.bridge.bridge-nf-call-iptables is 0 or 1.

Yes, this verifies that it passes.

Thanks for going through all that!

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Previously, after a reboot, virInterface functions did not work until libvirt was restarted and instead failed with a "error: Failed to list active interfaces. error: this function is not supported by the connection driver: virConnectNumOfInterfaces" error message. This is now fixed so that the code that caused the problem is removed and virInterface functions operate as expected after a reboot.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0620.html