Bug 1022977

Summary: DOC: document that sysctl variables for loadable modules must be set using udev rules
Product: [Fedora] Fedora Reporter: Paul Knox-Kennedy <paul.knox-kennedy>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: agk, awilliam, cra, dcbw, deanhunter, harald, herrold, johannbg, kay, lnykryn, lpoetter, martin.wilck, mmarzantowicz, mschmidt, msekleta, plautrba, s, systemd-maint, vpavlin, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: systemd-208-20.fc20 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-07-22 03:30:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Paul Knox-Kennedy 2013-10-24 11:51:26 UTC
Description of problem:

Using nm controlled bridge, with sysctl.conf containing:
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
After a reboot, all these have been set to 1. This then stops, for example, 
DHCP from working on virtual machines using the "shared physical device"
setup

Version-Release number of selected component (if applicable):
NetworkManager-0.9.9.0-14.git20131003.fc20.x86_64

How reproducible:
Every time

Steps to Reproduce:

1. Configure a bridge following http://wiki.libvirt.org/page/Networking,
section on Bridged networking, particularly the part which uses sysctl
to disable netfilter on the bridge  
2. Reboot and verify that cat /proc/sys/net/bridge/bridge-nf-call-* returns zeros
3. Disable network service and enable NetworkManager
4. Reboot
5. Verify that NetworkManager has set up the bridge correctly (network settings, 
brctl show)
3. cat /proc/sys/net/bridge/bridge-nf-call-*

Actual results:

All return 1 - netfilter is enabled on the bridge

Expected results:

All return 0 - netfilter should be disabled on the bridge

Additional info:
Workaround is to create /etc/NetworkManager/dispatcher.d/00-sysctl with the following:
#!/bin/sh

/usr/lib/systemd/systemd-sysctl

exit 0

Comment 1 Paul Knox-Kennedy 2013-10-25 07:59:20 UTC
Raised upstream bug: https://bugzilla.gnome.org/show_bug.cgi?id=710858

Comment 2 Martin Wilck 2013-11-15 17:19:08 UTC
You found an interesting workaround, but I think this is rather a systemd problem than a NetworkManager bug.

The problem is caused simply by the fact that systemd-sysctl service starts before "bridge.ko" is loaded in the kernel. NetworkManager is not necessary to trigger this bug. Simply running "modprobe bridge" in runlevel S will have the same effect. This effect has been known at least since 2010 (see https://bugzilla.redhat.com/show_bug.cgi?id=512206#c13).

I came up with a different workaround:
 1. echo bridge >/etc/modules-load.d/bridge.conf
 2. add "systemd-modules-load.service" in the "After" clause in /usr/lib/systemd/system/systemd-sysctl.service.

This is slightly more complex as your workaround, but it works also in the non-NM case.

I'm not sure how the systemd folks could fix this cleanly. It's a generic problem - some sysctl settings depend on modules being loaded.

Comment 3 Zbigniew Jędrzejewski-Szmek 2014-05-27 02:28:24 UTC
*** Bug 1093421 has been marked as a duplicate of this bug. ***

Comment 4 Zbigniew Jędrzejewski-Szmek 2014-05-27 02:29:36 UTC
I'm reassigning this to systemd, since it does not seem NM related. Not really a bug, but rather a missing feature.

Comment 5 Lennart Poettering 2014-06-18 09:27:27 UTC
This is always, and necessarily racy. If you want to set sysctls for loadable modules I fear there's no way around doing so from udev rules, that are run when the kernel module is loaded. 

This pops up regularly, for example here:

http://lists.freedesktop.org/archives/systemd-devel/2013-March/009851.html

or here:

http://lists.freedesktop.org/archives/systemd-devel/2014-June/020135.html

But this is nothing we can fix really. I will now rename the bug and turn this into a documentation thing: we should document this better so that we at least can point people to docs about this the next time.

Comment 6 Zbigniew Jędrzejewski-Szmek 2014-06-18 13:16:10 UTC
Looking at http://lists.freedesktop.org/archives/systemd-devel/2013-March/009851.html again, I think we are making it unnecessarily hard to apply sysctl settings. It *is* true that modules can be loaded at any time asynchronously, but

- adding a module name to /etc/modules-load.d/...
- making systemd-sysctl.service After=systemd-modules-load.service
- adding stuff /etc/sysctl.d/ for settings which are created when the module is loaded

is a reliable and *easy*. It might not be the perfect way, but it is hard to screw up, especially in comparison to udev rules. And adding this additional dependency is not going to make things measurably slower..., systemd-sysctl.service is nearly instantenous, and we have the rich dependency language to make such things easy. I'm inclined to apply the patch.

Comment 7 Lennart Poettering 2014-06-20 00:04:28 UTC
Zbigniew, if you apply it, i wouldn't revert it...

I am still favouring to add some easy example to do the same properly with udev rules though somewhere, and then guiding people to that... But maybe that solution isn't exclusive to yours...

Comment 9 Zbigniew Jędrzejewski-Szmek 2014-06-20 01:16:02 UTC
http://cgit.freedesktop.org/systemd/systemd/commit/?id=7141829

Comment 10 Fedora Update System 2014-07-20 00:39:53 UTC
systemd-208-20.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/systemd-208-20.fc20

Comment 11 Dean Hunter 2014-07-20 18:35:04 UTC
I am sorry, but I am having difficulty following the patches and change this but dont change that.  Please, could you post here the recommended bridge configuration changes so that my virtual machines can participate on my network as peers of my physical machines?  I have one of the alternatives discussed here or another place implemented, but I sense now that it is not the "preferred / recommended / least likely to cause problems in the future" implementation.

Comment 12 Fedora Update System 2014-07-21 05:21:56 UTC
Package systemd-208-20.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-208-20.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-8572/systemd-208-20.fc20
then log in and leave karma (feedback).

Comment 13 Michal Schmidt 2014-07-21 09:47:51 UTC
(In reply to Dean Hunter from comment #11)
Dean, you do not need to read the patches. Just update to systemd-208-20.fc20, and then read "man sysctl.d". The man page has been updated with helpful hints, including the examples. If something is still unclear, let us know.

Comment 14 Fedora Update System 2014-07-22 03:30:28 UTC
systemd-208-20.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 15 Adam Williamson 2014-10-03 17:25:27 UTC
https://bugzilla.redhat.com/show_bug.cgi?id=634736 is the sort of parent bug for this issue.