Bug 1122700

Summary:	sysctl.conf / sysctl.conf.d settings not read when modules are loaded (one result being that libvirt bridged networking with NetworkManager does not work correctly)
Product:	Red Hat Enterprise Linux 7	Reporter:	Adam Williamson <awilliam>
Component:	kmod	Assignee:	David Shea <dshea>
Status:	CLOSED DUPLICATE	QA Contact:	Network QE <network-qe>
Severity:	high	Docs Contact:
Priority:	low
Version:	7.0	CC:	abisogia, awilliam, bgoncalv, harald, jpsandys, laine, msekleta, systemd-maint-list, tis
Target Milestone:	rc
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	634736
Clones:	1182105 (view as bug list)		Environment:
Last Closed:	2015-04-14 17:45:58 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	634736
Bug Blocks:	1205790

Description Adam Williamson 2014-07-23 19:46:30 UTC

+++ This bug was initially created as a clone of Bug #634736 +++

Description of problem:

On a system that uses NetworkManager and has bridges, when system boot is complete the settings of net.bridge.bridge-nf-call-iptables , net.bridge.bridge-nf-call-ip6tables and net.bridge.bridge-nf-call-arptables are always "1", despite /usr/lib/sysctl.d/00-system.conf attempting to set them to "0". This prevents bridged connections to virtual machines working almost at all (if you set a static IP address a VM can ping numeric IPs, but that's about all).

The problem, which is rather well known by now, is that the settings only actually exist once the 'bridge' module is loaded, but the systemd service which applies these settings, systemd-sysctl.service , runs before the module is loaded. There is no mechanism in the kernel, kmod, or systemd which would cause sysctl settings to be re-read and re-applied when a kernel module is loaded.

The old 'network' service has a hack for this. It runs the function apply_sysctl() from /etc/init.d/functions a couple of times when bringing up interfaces. Hence folks using bridging with the old network service didn't see this problem. NetworkManager does not have an equivalent hack, however.

The 'bridge' incarnation of this problem is the one people are always running into, but in theory at least it is a generic problem, it could affect any other module with tunable settings.

There are various proposals for fixing this in the Fedora bug, notably from Edgar Hoch:

EH#1: Implement a hack in NetworkManager similar to the one in the old network service: "I have created a dispatcher file for NetworkManager that reloads the kernel parameter after change of a network interface." https://bugzilla.redhat.com/show_bug.cgi?id=634736#c5

EH#2: "A real solution - the best? - should be that every kernel module will check for parameters in sysctl config files immediate after loading the kernel module. This would avoid the con listed above." https://bugzilla.redhat.com/show_bug.cgi?id=634736#c7

EH#3: "For solving the general problem, I have the following idea:

- The kernel should remember parameters set by sysctl even if this parameter does not exist in the kernel at the time when the parameter was set.
  When the parameter is created later (for example by loading a module), the kernel should check the list of previous loaded parameters and if the parameter is found, this value should be used as default instead of other defaults.

This solution needs changes in the kernel. But it sets the values at the right time (only the kernel knows when a parameter is created) and it has less overhead than other solutions (for example like checking sysctl config files after every module load)." https://bugzilla.redhat.com/show_bug.cgi?id=634736#c9

The sysctl.d manpage explicitly notes the 'bridge' incarnation of this bug, and suggests two other approaches:

SYSCTL#1: create a udev rule ACTION=="add", SUBSYSTEM=="module", KERNEL=="bridge", RUN+="/usr/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/bridge"

SYSCTL#2: cause the 'bridge' module to be loaded statically very early in boot by listing it in a file in /usr/lib/modules-load.d/

Both of these approaches are specific to the 'bridge' incarnation of the problem.

However we decide to do it, at *least* the bridge incarnation of this issue really needs to be solved; if you're trying to set up libvirt bridged networking with NetworkManager you *will* trip over this bug, and it's quite frustrating to diagnose until you stumble across an old Fedora guide that mentions the sysctl parameters or something.

RHEL 7 and all now-supported Fedora versions are basically identical so far as this bug goes, so the same fix can and probably should be used for both. There's a whole other mess related to these parameters for RHEL 6 - see https://bugzilla.redhat.com/show_bug.cgi?id=919472 , but don't dive in too deep - but thankfully that should not be at all relevant to RHEL 7, as our attempt to override the defaults is no longer in /etc/sysctl.conf .

Comment 2 David Shea 2014-08-12 20:26:24 UTC

Since we agreed in bug 919472 and bug 1101045 to just move these defaults from one static file to another static file, I'm inclined to stick with that solution here, too. And as with that pair of bugs, this would require a new RHEL-7 bug to move the settings out of /usr/lib/sysctl.d/00-system.conf, which is still owned by initscripts. Sound good?

Comment 3 Adam Williamson 2014-08-13 00:12:23 UTC

eeeegad, that's the ugliest solution I've seen so far, but I guess it'd *work*. from that perspective it seems fine, but really, stuffing sysctl calls into a modprobe.d file? well, if you're okay with it, i guess i am...:P it seems icky to me, but hey.

Comment 4 Harald Hoyer 2014-08-20 15:03:45 UTC

I would go with:

SYSCTL#1: create a udev rule ACTION=="add", SUBSYSTEM=="module", KERNEL=="bridge", RUN+="/usr/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/bridge"

as a short term solution.

Long term, we should probably add a modinfo alias like we did for devname.

# modinfo loop
filename:       /lib/modules/3.16.0-1.fc22.x86_64/kernel/drivers/block/loop.ko.xz
alias:          devname:loop-control
alias:          char-major-10-237
alias:          block-major-7-*
....


So, I could imagine:

alias: sysctl:/proc/sys/net/bridge

Comment 5 Laine Stump 2014-10-28 18:25:01 UTC

Note that the Fedora version of this bug (Bug 634736) has been resolved with a downstream-only patch to the systemd package. It would be *very* useful for that patch to also be in RHEL7.1, as it prevents surprises with bridge-connected virtual guests.

I *think* this BZ should be moved to systemd and the flags set to rhel-7.1.0, but am not comfortable doing that myself on a bug where I'm not the reporter or assignee for change between two packages I don't work on :-)

Adam, am I asking for the right thing?

Comment 6 Adam Williamson 2014-10-28 20:33:22 UTC

The re-assignment seems reasonable to me. I'm not gonna touch any flags as I'm way too rusty on RHEL workflows to touch things. Note this bug is phrased to cover the general case as it may theoretically apply to *any* module; I haven't looked at the systemd fix yet, if it only covers the bridge module case specifically, arguably the general case still remains unaddressed, and we should handle that somehow (by splitting off a new bug report which covers only the bridge case and marking that as fixed, or closing this but filing a new one, or something).

Comment 7 Michal Sekletar 2015-01-14 13:08:39 UTC

Short term solution will be fixed as #1182105. As per general case, I don't think systemd is a component where fix for this should be introduced.

Comment 8 Laine Stump 2015-02-23 19:49:28 UTC

This was posted to the original of this BZ (filed against Fedora) last week:

(Scott Shambarger from comment #43 of Bug 634736)
> Just noticed the boot message:
> 
> bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update
> your scripts to load br_netfilter if you need this.
> 
> The new module appears to export the bridge sysctl values, but the udev rule
> in /lib/udev/rules.d/99-bridge.rules still refers to the bridge module (not
> the br_netfilter module), and so the bridge netfilter values after loading
> br_netfilter are:
> 
> net.bridge.bridge-nf-call-arptables = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-call-iptables = 1
> 
> Granted, if you load br_netfilter explicitly, you probably want to use it,
> but then /usr/sysctl.d/50-bridge.conf should probably be removed as it is no
> longer needed to disable bridge netfilter (by default) anymore.

Here is my comment on that, lifted from the same BZ:

(Laine Stump from comment #44 of Bug 634736)
> After all these years of asking for the default of bridge filtering to be
> "disabled" and the change being refused by the kernel maintainers, now the
> filtering has been moved into a separate module that isn't loaded (by
> default) when the bridge module is loaded, effectively making the default
> "disabled". Yay!
> 
> I *think* this is in the kernel as of 3.17 (It definitely is in kernel
> 3.18.7-200.fc21, and appears to be in git prior to the tag "v3.17-rc4")
> 
> So again the problem has disappeared for our case, so we're happy, but the
> general problem (as in the latest version of this bug's summary) is still
> there. What is the proper action for this BZ then? Do we leave it around
> (knowing that, based on the experience up to now, the general problem will
> likely never be fixed), close it as WONTFIX, or close it with one or another
> of the "resolved" reasons, since our particular problem has been fixed by
> the addition of the new module?

There is a similar question for RHEL - when RHEL7 moves up to a new enough kernel (or backports the br_netfilter module) should this BZ be closed as a result, or do we leave it open in the likely fruitless hope that a general purpose solution for setting non-compiled-in defaults of kernel module settings at module load time will be implemented?

Comment 9 David Shea 2015-04-14 17:45:58 UTC

Marking as a duplicate per comment 7, and since the proposed long-term solutions don't involve kmod changes.

*** This bug has been marked as a duplicate of bug 1182105 ***

Comment 10 Jeff Sandys 2015-12-10 07:24:21 UTC

Help, after upgrading to fedora 21 boot stops after the message "bridge: automatic filtering via arp/ip/ip6table has been deprecated."  After reading these bugs and the sysctl.d manpage I believe I can fix this by using a rescue disk to implement the changes that Adam lists as SYSCTL#1 or SYSCTL#2 above.

Adam or other experts, can you clarify the edit I should make to which file with a rescue disk?

Comment 11 Laine Stump 2015-12-10 19:47:23 UTC

No, the problem detailed in this bug has nothing to do with a hanging boot. It only affects whether or not packets will be forwarded across a bridge device without iptables processing; that would not cause the boot to hang. It's just a coincidence that the above message just happened to be the last message logged before it hung (as a matter of fact, that message is indicating that your host would *not* have the problem that all the discussion in this bug has been trying to eliminate).

You'll need to look elsewhere for the source of your hang.