Description of problem: If you add any kernel arguments that are interpreted by systemd-network-generator then the systemd-network-generator.service will fail to start because of SELinux denials. ``` Jan 4 16:58:05.149000 audit[888]: AVC avc: denied { add_name } for pid=888 comm="systemd-network" name="90-ens5.network" scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=dir permissive=0 Jan 4 16:58:05.149000 audit[888]: SYSCALL arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=7ffc53f5b7e0 a2=802c1 a3=1b6 items=0 ppid=1 pid=888 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="s Jan 4 16:58:05.149000 audit: PROCTITLE proctitle="/usr/lib/systemd/systemd-network-generator" Jan 4 16:58:05.165000 audit: CONFIG_CHANGE op=set audit_enabled=1 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:syslogd_t:s0 res=1 Jan 4 16:58:05.165000 audit[886]: SYSCALL arch=c000003e syscall=46 success=yes exit=60 a0=6 a1=7ffc68b1def0 a2=4000 a3=7ffc68b1df7c items=0 ppid=1 pid=886 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="s Jan 4 16:58:05.165000 audit: PROCTITLE proctitle="/usr/lib/systemd/systemd-journald" Jan 4 16:58:05.216000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kmod-static-nodes comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jan 4 16:58:05.225000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@configfs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jan 4 16:58:05.225000 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@configfs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jan 4 16:58:05.233000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@drm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jan 4 16:58:05.233000 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@drm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jan 4 16:58:05.236000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jan 4 16:58:05.239000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@fuse comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jan 4 16:58:05.239000 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@fuse comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jan 4 16:58:05.243000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-modules-load comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jan 4 16:58:04.922425 systemd[1]: Queued start job for default target multi-user.target. Jan 4 16:58:04.930405 systemd[1]: systemd-journald.service: Deactivated successfully. Jan 4 16:58:05.250000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-network-generator comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' Jan 4 16:58:05.253000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Jan 4 16:58:05.150573 systemd-modules-load[887]: Module 'msr' is built in Jan 4 16:58:05.189028 systemd-modules-load[887]: Failed to insert module 'ipmi_si': No such device Jan 4 16:58:05.237505 systemd[1]: modprobe: Deactivated successfully. Jan 4 16:58:05.240207 systemd[1]: Finished modprobe - Load Kernel Module fuse. Jan 4 16:58:05.243910 systemd[1]: Finished systemd-modules-load.service - Load Kernel Modules. Jan 4 16:58:05.244729 systemd[1]: systemd-network-generator.service: Main process exited, code=exited, status=1/FAILURE Jan 4 16:58:05.247246 systemd[1]: systemd-network-generator.service: Failed with result 'exit-code'. Jan 4 16:58:05.250339 systemd[1]: Failed to start systemd-network-generator.service - Generate network units from Kernel command line. Jan 4 16:58:05.254181 systemd[1]: Finished systemd-remount-fs.service - Remount Root and Kernel File Systems. Jan 4 16:58:05.255029 systemd[1]: Reached target network-pre.target - Preparation for Network. Jan 4 16:58:05.278242 systemd-network-generator[888]: Failed to create unit file /run/systemd/network/90-ens5.network: Permission denied ``` This particular run had: `ip=10.10.10.10::10.10.10.1:255.255.255.0:myhostname:ens5:none:8.8.8.8`, but this is easily reproduced with just `nameserver=8.8.8.8`. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Start latest rawhide (from 20220103.n.0 compose) 2. Add `nameserver=8.8.8.8` to kernel command line arguments 3. See systemd-network-generator fail. Actual results: systemd-network-generator fails because it can't write to /run/systemd/network/ Expected results: No failure Additional info: This was originally discovered in Fedora CoreOS CI but was also reproduced on the Fedora Cloud Base image: https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20220103.n.0/compose/Cloud/x86_64/images/Fedora-Cloud-Base-Rawhide-20220103.n.0.x86_64.qcow2
For completeness: ``` [root@fedora ~]# rpm -q systemd selinux-policy systemd-250-3.fc36.x86_64 selinux-policy-35.8-1.fc36.noarch ```
Jan 4 16:58:05.278242 systemd-network-generator[888]: Failed to create unit file /run/systemd/network/90-ens5.network: Permission denied Yeah, systemd-network-generator will create files under /run/systemd/network/.
Note a real simple reproducer would be: ``` sudo su - systemd-run /usr/lib/systemd/systemd-network-generator -- nameserver=8.8.8.8 journalctl -u $unit ausearch -m AVC ``` Here's an example run: ``` [dustymabe@media fR]$ vagrant ssh [vagrant@vanilla-rawhide ~]$ sudo su - [root@vanilla-rawhide ~]# id uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [root@vanilla-rawhide ~]# systemd-run /usr/lib/systemd/systemd-network-generator -- nameserver=8.8.8.8 Running as unit: run-raf26d43a650c400ca241189406d58075.service [root@vanilla-rawhide ~]# journalctl -u run-raf26d43a650c400ca241189406d58075.service | cat Jan 11 15:08:00 vanilla-rawhide systemd[1]: Started run-raf26d43a650c400ca241189406d58075.service - /usr/lib/systemd/systemd-network-generator -- nameserver=8.8.8.8. Jan 11 15:08:00 vanilla-rawhide systemd-network-generator[4541]: Failed to create unit file /run/systemd/network/91-default.network: Permission denied Jan 11 15:08:00 vanilla-rawhide systemd[1]: run-raf26d43a650c400ca241189406d58075.service: Main process exited, code=exited, status=1/FAILURE Jan 11 15:08:00 vanilla-rawhide systemd[1]: run-raf26d43a650c400ca241189406d58075.service: Failed with result 'exit-code'. [root@vanilla-rawhide ~]# ausearch -m avc ---- time->Tue Jan 11 15:08:00 2022 type=AVC msg=audit(1641913680.338:1145): avc: denied { add_name } for pid=4541 comm="systemd-network" name="91-default.network" scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=dir permissive=0 ```
I need this fixed in f35 too as we are starting to see this there now with the new systemd: https://github.com/coreos/fedora-coreos-tracker/issues/1059#issuecomment-1013766710
Caught in enforcing mode: ---- type=PROCTITLE msg=audit(01/18/2022 02:26:36.120:550) : proctitle=/usr/lib/systemd/systemd-network-generator -- nameserver=8.8.8.8 type=PATH msg=audit(01/18/2022 02:26:36.120:550) : item=0 name=/run/systemd/network/ inode=1301 dev=00:1a mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:net_conf_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=CWD msg=audit(01/18/2022 02:26:36.120:550) : cwd=/ type=SYSCALL msg=audit(01/18/2022 02:26:36.120:550) : arch=x86_64 syscall=openat success=no exit=EACCES(Permission denied) a0=0xffffff9c a1=0x7ffdd043c760 a2=O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_CLOEXEC a3=0x1b6 items=1 ppid=1 pid=4272 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=systemd-network exe=/usr/lib/systemd/systemd-network-generator subj=system_u:system_r:init_t:s0 key=(null) type=AVC msg=audit(01/18/2022 02:26:36.120:550) : avc: denied { add_name } for pid=4272 comm=systemd-network name=91-default.network scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=dir permissive=0 ---- Caught in permissive mode: ---- type=PROCTITLE msg=audit(01/18/2022 02:27:49.905:559) : proctitle=/usr/lib/systemd/systemd-network-generator -- nameserver=8.8.8.8 type=PATH msg=audit(01/18/2022 02:27:49.905:559) : item=1 name=/run/systemd/network/91-default.network inode=1312 dev=00:1a mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:net_conf_t:s0 nametype=CREATE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=PATH msg=audit(01/18/2022 02:27:49.905:559) : item=0 name=/run/systemd/network/ inode=1301 dev=00:1a mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:net_conf_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 type=CWD msg=audit(01/18/2022 02:27:49.905:559) : cwd=/ type=SYSCALL msg=audit(01/18/2022 02:27:49.905:559) : arch=x86_64 syscall=openat success=yes exit=3 a0=0xffffff9c a1=0x7ffcf845ed90 a2=O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_CLOEXEC a3=0x1b6 items=2 ppid=1 pid=4292 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=systemd-network exe=/usr/lib/systemd/systemd-network-generator subj=system_u:system_r:init_t:s0 key=(null) type=AVC msg=audit(01/18/2022 02:27:49.905:559) : avc: denied { write } for pid=4292 comm=systemd-network path=/run/systemd/network/91-default.network dev="tmpfs" ino=1312 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=file permissive=1 type=AVC msg=audit(01/18/2022 02:27:49.905:559) : avc: denied { create } for pid=4292 comm=systemd-network name=91-default.network scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=file permissive=1 type=AVC msg=audit(01/18/2022 02:27:49.905:559) : avc: denied { add_name } for pid=4292 comm=systemd-network name=91-default.network scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=dir permissive=1 ---- # rpm -qa selinux\* systemd\* | sort selinux-policy-35.8-1.fc35.noarch selinux-policy-targeted-35.8-1.fc35.noarch systemd-249.7-2.fc35.x86_64 systemd-libs-249.7-2.fc35.x86_64 systemd-networkd-249.7-2.fc35.x86_64 systemd-oomd-defaults-249.7-2.fc35.noarch systemd-pam-249.7-2.fc35.x86_64 systemd-resolved-249.7-2.fc35.x86_64 systemd-udev-249.7-2.fc35.x86_64 #
Hey Team. Any updates here?
(In reply to Dusty Mabe from comment #6) > Hey Team. Any updates here? Requires a new policy which takes time.
Hey Zdenek, Thanks for the info. I understand it takes time. Do you mind giving a rough estimate (a week, 2 weeks, a month, multiple months)? Currently we've pinned systemd in Fedora Coreos (based on F35) on an older version. The rough estimate will help us plan accordingly.
This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle. Changing version to 36.
Moving to Fedora 35 since that's where we're feeling the most pain right now because of the pinned systemd.
In yesterday's meeting, we agreed to accept this as a Prioritized Bug because it affects a common use case for a flagship offering https://meetbot.fedoraproject.org/teams/fedora_prioritized_bugs_and_issues/fedora_prioritized_bugs_and_issues.2022-02-09-15.00.log.html#l-65
(In reply to Dusty Mabe from comment #8) > Hey Zdenek, > > Thanks for the info. I understand it takes time. Do you mind giving a rough > estimate (a week, 2 weeks, a month, multiple months)? Currently we've pinned > systemd in Fedora Coreos (based on F35) on an older version. The rough > estimate will help us plan accordingly. Rough estimate atm (no promises) is 3 weeks to get to F35. Any suggestions or help with testing can effect this in a positive manner.
Does the reproducer in https://bugzilla.redhat.com/show_bug.cgi?id=2037047#c3 help? We can certainly test this if you come up with a potential fix and provide us a scratch build. Thank you!
I can reproduce from comment 3 on a clean installed system using Fedora-Server-netinst-x86_64-36-20220308.n.0.iso (today). type=AVC msg=audit(1646781760.098:273): avc: denied { add_name } for pid=1023 comm="systemd-network" name="91-default.network" scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=dir permissive=0 Also fails if I use `nameserver=8.8.8.8` as a kernel boot parameter. ● systemd-network-generator.service loaded failed failed Generate network units from Kernel command line
Proposed as a Blocker for 36-beta by Fedora user chrismurphy using the blocker tracking app because: https://fedoraproject.org/wiki/Basic_Release_Criteria#Cockpit_management_interface The default system init daemon (e.g. systemd) must be capable of starting, stopping, enabling and disabling correctly-defined services. a. Performing this kind of network service manipulation is pretty basic functionality for any server/VM/cloud, i.e. it is a correctly defined service. b. systemd isn't capable of enabling it due to the selinux issue
Pretty sure the blocker bug tracker needs to see the bug set to 36 to show it as blocking 36.
>b. systemd isn't capable of enabling it due to the selinux issue Correction b. systemd isn't capable of starting it due to the selinux issue
OK per the note in the release criterion about "Correctly-defined services" - I'll argue that systemd is inhibited by SELinux from starting the service, i.e. systemd is functionally prevented from working correctly by this issue.
No. The criterion is about Cockpit functionality. It is definitely *not* meant to be judo'ed into a criterion that makes it a blocker bug if absolutely any systemd service shipped in the distro doesn't work as intended.
Oh I copied the wrong URL it's: https://fedoraproject.org/wiki/Basic_Release_Criteria#System_service_manipulation Nothing to do with cockpit.
Same deal. That means that systemd must be able to start, stop, enable and disable services. It's about systemd's operation, it is not supposed to be judo'd into this kind of situation.
Ok well the plain language doesn't explain it that way, nor does the example under it. There is nothing wrong with the service unit. Systemd is partially made defective in a narrow scope due to this SELinux problem. But let's flip this around: Is it OK to release Server/Cloud edition in a state in which select basic services like this can't be started? Release blocking desktops have "applications must start successfully and withstand a basic functionality test" and the cited criterion is the closest equivalent to that for Server and Cloud edition. This bug directly inhibits the four basic objectives too. https://fedoraproject.org/wiki/Basic_Release_Criteria#Basic_Objectives There is a basic release criterion that requires SELinux enabled, so we can't test if there are subsequent or underlying bugs in this same area while SELinux is enabled due to the bug. It's a catch-22.
Let's move that discussion to the ticket, I guess. https://pagure.io/fedora-qa/blocker-review/issue/651
-5 in https://pagure.io/fedora-qa/blocker-review/issue/651 , marking rejected. Someone voted +1 Beta FE, so proposing as Beta FE for others to vote on.
btw, *is* this a "basic service"? Is this a common or supported way to configure networking on Server or Cloud? I don't recall hearing of it before.
I think systemd-network-generator was enabled by default fairly recently in an update to systemd. Obviously the logical conclusion that people could come to is "We don't use systemd-networkd, so we don't need this generator", but that's not totally true because there are other things (like udev rules) that this generator creates that are needed. Example: https://github.com/systemd/systemd/pull/21766#issuecomment-994411870
Discussed during the 2022-03-14 blocker review meeting: [0] The decision to classify this bug as an "AcceptedFreezeException (Beta)" was made as it is a noticeable issue that cannot be fixed with an update. [0] https://meetbot.fedoraproject.org/fedora-blocker-review/2022-03-14/f36-blocker-review.2022-03-14-16.01.txt
hey @zpytela - can we get an updated estimate? Thanks!
(In reply to Dusty Mabe from comment #28) > hey @zpytela - can we get an updated estimate? Thanks! Next week if I can reproduce and test it.
There is a PR adding the new domain: https://github.com/fedora-selinux/selinux-policy/pull/1126 The reproducer from #c3 does not produce any denials, so the fix is expected to be in the next build, but I'd like anybody using this feature in a different way to check the build, rpms for testing can be downloaded from the PR: Checks -> Details -> Artifacts -> rpms
The CI RPM from the PR (`selinux-policy-36.5-1.20220404_083611.bff1cac.fc37.noarch.rpm`) corrects this issue for me. Thanks!
Thanks for checking.
Sadly, rawhide composes still fail. ;( https://pagure.io/releng/failed-composes/issue/3311 I'm not sure if this is the same issue or not however. It looks like anaconda cannot connect to nm-client...
Kevin, I doubt it's the same issue. This issue has been around since the start of the year. Zdenek, Testing of the issue in rawhide for us seems to look good with selinux-policy-36.6-1.fc37. Can we get an update for Fedora 35/36?
Well, untagging systemd-251~rc1-2.fc37 got us a rawhide compose. I guess we can put it back and see if it fails again?
(In reply to Dusty Mabe from comment #34) > Zdenek, > > Testing of the issue in rawhide for us seems to look good with > selinux-policy-36.6-1.fc37. Can we get an update for Fedora 35/36? It's on the way since yesterday, it just takes a long time.
FEDORA-2022-690770081a has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-690770081a
(In reply to Kevin Fenzi from comment #35) > Well, untagging systemd-251~rc1-2.fc37 got us a rawhide compose. > This bug is against selinux-policy and the fix is in the selinux-policy rpm. I don't think untagging the newest systemd would affect this issue.
FEDORA-2022-690770081a has been pushed to the Fedora 36 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-690770081a` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-690770081a See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
(In reply to Dusty Mabe from comment #38) > (In reply to Kevin Fenzi from comment #35) > > Well, untagging systemd-251~rc1-2.fc37 got us a rawhide compose. > > > > This bug is against selinux-policy and the fix is in the selinux-policy rpm. > I don't think untagging the newest systemd would affect this issue. Oops. Wrong bug. ;( I was meanting to post that in https://bugzilla.redhat.com/show_bug.cgi?id=2071069 Sorry for the noise.
This was an Accepted Beta FreezeException, re-proposing as Final FE.
Discussed in ticket: https://pagure.io/fedora-qa/blocker-review/issue/698 The decision to classify this bug as an AcceptedFreezeException was made: "It is a noticeable issue that cannot be fixed with an update."
FEDORA-2022-690770081a has been pushed to the Fedora 36 stable repository. If problem still persists, please make note of it in this bug report.
Hi all. Not sure if this BZ is a proper one to update, but looks like the issue could be reproduced on RHCOS 4.12: ``` [core@localhost ~]$ cat /etc/os-release NAME="Red Hat Enterprise Linux CoreOS" VERSION_ID="4.12" RHEL_VERSION="9.0" [core@localhost ~]$ ip a 2: enc2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute enc2 [core@localhost ~]$ sudo rpm-ostree kargs --append="ip=10.0.2.15::10.0.2.2:255.255.255.0:rhcos:enc2:none" && sudo reboot ---- reboot ---- [core@localhost ~]$ systemctl status systemd-network-generator.service Jan 17 09:58:34 localhost systemd-network-generator[805]: Failed to create unit file /run/systemd/network/90-enc2.network: Permission denied [core@localhost ~]$ rpm -q selinux-policy selinux-policy-34.1.29-1.el9_0.2.noarch [core@localhost ~]$ rpm -q systemd systemd-250-6.el9_0.1.s390x [core@localhost ~]$ ls -Z /usr/lib/systemd/systemd-network-generator system_u:object_r:init_exec_t:s0 /usr/lib/systemd/systemd-network-generator [core@localhost ~]$ ls -dZ /run/systemd/network/ system_u:object_r:net_conf_t:s0 /run/systemd/network/ ```
@ndubrovs I think it'd be better to file a new issue against RHEL 9 instead. You can link to this RHBZ there for more details.