Bug 2111069

+++ This bug was initially created as a clone of Bug #2037047 +++

Description of problem:

If you add any kernel arguments that are interpreted by systemd-network-generator then the systemd-network-generator.service will fail to start because of SELinux denials. 



```
Jan  4 16:58:05.149000 audit[888]: AVC avc:  denied  { add_name } for  pid=888 comm="systemd-network" name="90-ens5.network" scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=dir permissive=0                                   
Jan  4 16:58:05.149000 audit[888]: SYSCALL arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=7ffc53f5b7e0 a2=802c1 a3=1b6 items=0 ppid=1 pid=888 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="s
Jan  4 16:58:05.149000 audit: PROCTITLE proctitle="/usr/lib/systemd/systemd-network-generator"                                                                                                                                                                       
Jan  4 16:58:05.165000 audit: CONFIG_CHANGE op=set audit_enabled=1 old=1 auid=4294967295 ses=4294967295 subj=system_u:system_r:syslogd_t:s0 res=1                                                                                                                    
Jan  4 16:58:05.165000 audit[886]: SYSCALL arch=c000003e syscall=46 success=yes exit=60 a0=6 a1=7ffc68b1def0 a2=4000 a3=7ffc68b1df7c items=0 ppid=1 pid=886 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="s
Jan  4 16:58:05.165000 audit: PROCTITLE proctitle="/usr/lib/systemd/systemd-journald"                                                                                                                                                                                
Jan  4 16:58:05.216000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=kmod-static-nodes comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'                       
Jan  4 16:58:05.225000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@configfs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'                       
Jan  4 16:58:05.225000 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@configfs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'                        
Jan  4 16:58:05.233000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@drm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'                            
Jan  4 16:58:05.233000 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@drm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'                             
Jan  4 16:58:05.236000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'                        
Jan  4 16:58:05.239000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@fuse comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'                           
Jan  4 16:58:05.239000 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=modprobe@fuse comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'                            
Jan  4 16:58:05.243000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-modules-load comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'                    
Jan  4 16:58:04.922425 systemd[1]: Queued start job for default target multi-user.target.                                                                                                                                                                            
Jan  4 16:58:04.930405 systemd[1]: systemd-journald.service: Deactivated successfully.                                                                                                                                                                               
Jan  4 16:58:05.250000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-network-generator comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'                
Jan  4 16:58:05.253000 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'                      
Jan  4 16:58:05.150573 systemd-modules-load[887]: Module 'msr' is built in                                                                                                                                                                                           
Jan  4 16:58:05.189028 systemd-modules-load[887]: Failed to insert module 'ipmi_si': No such device                                                                                                                                                                  
Jan  4 16:58:05.237505 systemd[1]: modprobe: Deactivated successfully.                                                                                                                                                                                  
Jan  4 16:58:05.240207 systemd[1]: Finished modprobe - Load Kernel Module fuse.                                                                                                                                                                         
Jan  4 16:58:05.243910 systemd[1]: Finished systemd-modules-load.service - Load Kernel Modules.                                                                                                                                                                      
Jan  4 16:58:05.244729 systemd[1]: systemd-network-generator.service: Main process exited, code=exited, status=1/FAILURE                                                                                                                                             
Jan  4 16:58:05.247246 systemd[1]: systemd-network-generator.service: Failed with result 'exit-code'.                                                                                                                                                                
Jan  4 16:58:05.250339 systemd[1]: Failed to start systemd-network-generator.service - Generate network units from Kernel command line.                                                                                                                              
Jan  4 16:58:05.254181 systemd[1]: Finished systemd-remount-fs.service - Remount Root and Kernel File Systems.                                                                                                                                                       
Jan  4 16:58:05.255029 systemd[1]: Reached target network-pre.target - Preparation for Network.                                                                                                                                                                      
Jan  4 16:58:05.278242 systemd-network-generator[888]: Failed to create unit file /run/systemd/network/90-ens5.network: Permission denied
```

This particular run had: `ip=10.10.10.10::10.10.10.1:255.255.255.0:myhostname:ens5:none:8.8.8.8`, but this is easily reproduced with just `nameserver=8.8.8.8`.


Version-Release number of selected component (if applicable):


How reproducible:

Always

Steps to Reproduce:
1. Start latest rawhide (from 20220103.n.0 compose)
2. Add `nameserver=8.8.8.8` to kernel command line arguments
3. See systemd-network-generator fail. 

Actual results:

systemd-network-generator fails because it can't write to /run/systemd/network/


Expected results:

No failure

Additional info:

This was originally discovered in Fedora CoreOS CI but was also reproduced on the Fedora Cloud Base image:
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20220103.n.0/compose/Cloud/x86_64/images/Fedora-Cloud-Base-Rawhide-20220103.n.0.x86_64.qcow2

--- Additional comment from Dusty Mabe on 2022-01-04 17:53:11 UTC ---

For completeness:

```
[root@fedora ~]# rpm -q systemd selinux-policy
systemd-250-3.fc36.x86_64
selinux-policy-35.8-1.fc36.noarch
```

--- Additional comment from Zbigniew Jędrzejewski-Szmek on 2022-01-05 20:23:46 UTC ---

Jan  4 16:58:05.278242 systemd-network-generator[888]: Failed to create unit file /run/systemd/network/90-ens5.network: Permission denied

Yeah, systemd-network-generator will create files under /run/systemd/network/.

--- Additional comment from Dusty Mabe on 2022-01-11 15:09:28 UTC ---

Note a real simple reproducer would be:


```
sudo su -
systemd-run /usr/lib/systemd/systemd-network-generator -- nameserver=8.8.8.8
journalctl -u $unit
ausearch -m AVC
```

Here's an example run:

```
[dustymabe@media fR]$ vagrant ssh 
[vagrant@vanilla-rawhide ~]$ sudo su -

[root@vanilla-rawhide ~]# id
uid=0(root) gid=0(root) groups=0(root) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

[root@vanilla-rawhide ~]# systemd-run /usr/lib/systemd/systemd-network-generator -- nameserver=8.8.8.8
Running as unit: run-raf26d43a650c400ca241189406d58075.service

[root@vanilla-rawhide ~]# journalctl -u run-raf26d43a650c400ca241189406d58075.service | cat
Jan 11 15:08:00 vanilla-rawhide systemd[1]: Started run-raf26d43a650c400ca241189406d58075.service - /usr/lib/systemd/systemd-network-generator -- nameserver=8.8.8.8.
Jan 11 15:08:00 vanilla-rawhide systemd-network-generator[4541]: Failed to create unit file /run/systemd/network/91-default.network: Permission denied
Jan 11 15:08:00 vanilla-rawhide systemd[1]: run-raf26d43a650c400ca241189406d58075.service: Main process exited, code=exited, status=1/FAILURE
Jan 11 15:08:00 vanilla-rawhide systemd[1]: run-raf26d43a650c400ca241189406d58075.service: Failed with result 'exit-code'.

[root@vanilla-rawhide ~]# ausearch -m avc
----
time->Tue Jan 11 15:08:00 2022
type=AVC msg=audit(1641913680.338:1145): avc:  denied  { add_name } for  pid=4541 comm="systemd-network" name="91-default.network" scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=dir permissive=0
```

--- Additional comment from Dusty Mabe on 2022-01-16 04:03:46 UTC ---

I need this fixed in f35 too as we are starting to see this there now with the new systemd:
https://github.com/coreos/fedora-coreos-tracker/issues/1059#issuecomment-1013766710

--- Additional comment from Milos Malik on 2022-01-18 07:30:03 UTC ---

Caught in enforcing mode:
----
type=PROCTITLE msg=audit(01/18/2022 02:26:36.120:550) : proctitle=/usr/lib/systemd/systemd-network-generator -- nameserver=8.8.8.8 
type=PATH msg=audit(01/18/2022 02:26:36.120:550) : item=0 name=/run/systemd/network/ inode=1301 dev=00:1a mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:net_conf_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(01/18/2022 02:26:36.120:550) : cwd=/ 
type=SYSCALL msg=audit(01/18/2022 02:26:36.120:550) : arch=x86_64 syscall=openat success=no exit=EACCES(Permission denied) a0=0xffffff9c a1=0x7ffdd043c760 a2=O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_CLOEXEC a3=0x1b6 items=1 ppid=1 pid=4272 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=systemd-network exe=/usr/lib/systemd/systemd-network-generator subj=system_u:system_r:init_t:s0 key=(null) 
type=AVC msg=audit(01/18/2022 02:26:36.120:550) : avc:  denied  { add_name } for  pid=4272 comm=systemd-network name=91-default.network scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=dir permissive=0
----

Caught in permissive mode:
----
type=PROCTITLE msg=audit(01/18/2022 02:27:49.905:559) : proctitle=/usr/lib/systemd/systemd-network-generator -- nameserver=8.8.8.8 
type=PATH msg=audit(01/18/2022 02:27:49.905:559) : item=1 name=/run/systemd/network/91-default.network inode=1312 dev=00:1a mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:net_conf_t:s0 nametype=CREATE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=PATH msg=audit(01/18/2022 02:27:49.905:559) : item=0 name=/run/systemd/network/ inode=1301 dev=00:1a mode=dir,755 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:net_conf_t:s0 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0 
type=CWD msg=audit(01/18/2022 02:27:49.905:559) : cwd=/ 
type=SYSCALL msg=audit(01/18/2022 02:27:49.905:559) : arch=x86_64 syscall=openat success=yes exit=3 a0=0xffffff9c a1=0x7ffcf845ed90 a2=O_WRONLY|O_CREAT|O_EXCL|O_TRUNC|O_CLOEXEC a3=0x1b6 items=2 ppid=1 pid=4292 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=systemd-network exe=/usr/lib/systemd/systemd-network-generator subj=system_u:system_r:init_t:s0 key=(null) 
type=AVC msg=audit(01/18/2022 02:27:49.905:559) : avc:  denied  { write } for  pid=4292 comm=systemd-network path=/run/systemd/network/91-default.network dev="tmpfs" ino=1312 scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=file permissive=1 
type=AVC msg=audit(01/18/2022 02:27:49.905:559) : avc:  denied  { create } for  pid=4292 comm=systemd-network name=91-default.network scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=file permissive=1 
type=AVC msg=audit(01/18/2022 02:27:49.905:559) : avc:  denied  { add_name } for  pid=4292 comm=systemd-network name=91-default.network scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=dir permissive=1
----

# rpm -qa selinux\* systemd\* | sort
selinux-policy-35.8-1.fc35.noarch
selinux-policy-targeted-35.8-1.fc35.noarch
systemd-249.7-2.fc35.x86_64
systemd-libs-249.7-2.fc35.x86_64
systemd-networkd-249.7-2.fc35.x86_64
systemd-oomd-defaults-249.7-2.fc35.noarch
systemd-pam-249.7-2.fc35.x86_64
systemd-resolved-249.7-2.fc35.x86_64
systemd-udev-249.7-2.fc35.x86_64
#

--- Additional comment from Dusty Mabe on 2022-01-20 20:39:59 UTC ---

Hey Team. Any updates here?

--- Additional comment from Zdenek Pytela on 2022-01-26 18:33:47 UTC ---

(In reply to Dusty Mabe from comment #6)
> Hey Team. Any updates here?

Requires a new policy which takes time.

--- Additional comment from Dusty Mabe on 2022-01-27 04:07:12 UTC ---

Hey Zdenek,

Thanks for the info. I understand it takes time. Do you mind giving a rough estimate (a week, 2 weeks, a month, multiple months)? Currently we've pinned systemd in Fedora Coreos (based on F35) on an older version. The rough estimate will help us plan accordingly.

--- Additional comment from Ben Cotton on 2022-02-08 20:42:01 UTC ---

This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle.
Changing version to 36.

--- Additional comment from Dusty Mabe on 2022-02-09 15:24:06 UTC ---

Moving to Fedora 35 since that's where we're feeling the most pain right now because of the pinned systemd.

--- Additional comment from Ben Cotton on 2022-02-10 15:44:05 UTC ---

In yesterday's meeting, we agreed to accept this as a Prioritized Bug because it affects a common use case for a flagship offering

https://meetbot.fedoraproject.org/teams/fedora_prioritized_bugs_and_issues/fedora_prioritized_bugs_and_issues.2022-02-09-15.00.log.html#l-65

--- Additional comment from Zdenek Pytela on 2022-02-22 14:00:51 UTC ---

(In reply to Dusty Mabe from comment #8)
> Hey Zdenek,
> 
> Thanks for the info. I understand it takes time. Do you mind giving a rough
> estimate (a week, 2 weeks, a month, multiple months)? Currently we've pinned
> systemd in Fedora Coreos (based on F35) on an older version. The rough
> estimate will help us plan accordingly.

Rough estimate atm (no promises) is 3 weeks to get to F35. Any suggestions or help with testing can effect this in a positive manner.

--- Additional comment from Dusty Mabe on 2022-02-23 01:42:46 UTC ---

Does the reproducer in https://bugzilla.redhat.com/show_bug.cgi?id=2037047#c3 help?

We can certainly test this if you come up with a potential fix and provide us a scratch build.

Thank you!

--- Additional comment from Chris Murphy on 2022-03-08 23:37:56 UTC ---

I can reproduce from comment 3 on a clean installed system using Fedora-Server-netinst-x86_64-36-20220308.n.0.iso (today).

type=AVC msg=audit(1646781760.098:273): avc:  denied  { add_name } for  pid=1023 comm="systemd-network" name="91-default.network" scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=dir permissive=0

Also fails if I use `nameserver=8.8.8.8` as a kernel boot parameter.
● systemd-network-generator.service                                                        loaded failed failed    Generate network units from Kernel command line

--- Additional comment from Fedora Blocker Bugs Application on 2022-03-08 23:40:59 UTC ---

Proposed as a Blocker for 36-beta by Fedora user chrismurphy using the blocker tracking app because:

 https://fedoraproject.org/wiki/Basic_Release_Criteria#Cockpit_management_interface
The default system init daemon (e.g. systemd) must be capable of starting, stopping, enabling and disabling correctly-defined services. 

a. Performing this kind of network service manipulation is pretty basic functionality for any server/VM/cloud, i.e. it is a correctly defined service.
b. systemd isn't capable of enabling it due to the selinux issue

--- Additional comment from Chris Murphy on 2022-03-08 23:42:51 UTC ---

Pretty sure the blocker bug tracker needs to see the bug set to 36 to show it as blocking 36.

--- Additional comment from Chris Murphy on 2022-03-08 23:44:07 UTC ---

>b. systemd isn't capable of enabling it due to the selinux issue

Correction
b. systemd isn't capable of starting it due to the selinux issue

--- Additional comment from Chris Murphy on 2022-03-08 23:49:43 UTC ---

OK per the note in the release criterion about "Correctly-defined services" - I'll argue that systemd is inhibited by SELinux from starting the service, i.e. systemd is functionally prevented from working correctly by this issue.

--- Additional comment from Adam Williamson on 2022-03-08 23:54:54 UTC ---

No. The criterion is about Cockpit functionality. It is definitely *not* meant to be judo'ed into a criterion that makes it a blocker bug if absolutely any systemd service shipped in the distro doesn't work as intended.

--- Additional comment from Chris Murphy on 2022-03-09 00:00:50 UTC ---

Oh I copied the wrong URL it's: https://fedoraproject.org/wiki/Basic_Release_Criteria#System_service_manipulation
Nothing to do with cockpit.

--- Additional comment from Adam Williamson on 2022-03-09 00:53:43 UTC ---

Same deal. That means that systemd must be able to start, stop, enable and disable services. It's about systemd's operation, it is not supposed to be judo'd into this kind of situation.

--- Additional comment from Chris Murphy on 2022-03-09 01:16:43 UTC ---

Ok well the plain language doesn't explain it that way, nor does the example under it. There is nothing wrong with the service unit. Systemd is partially made defective in a narrow scope due to this SELinux problem.

But let's flip this around:
Is it OK to release Server/Cloud edition in a state in which select basic services like this can't be started?

Release blocking desktops have "applications must start successfully and withstand a basic functionality test" and the cited criterion is the closest equivalent to that for Server and Cloud edition. This bug directly inhibits the four basic objectives too.  https://fedoraproject.org/wiki/Basic_Release_Criteria#Basic_Objectives  There is a basic release criterion that requires SELinux enabled, so we can't test if there are subsequent or underlying bugs in this same area while SELinux is enabled due to the bug. It's a catch-22.

--- Additional comment from Adam Williamson on 2022-03-09 01:23:43 UTC ---

Let's move that discussion to the ticket, I guess. https://pagure.io/fedora-qa/blocker-review/issue/651

--- Additional comment from Adam Williamson on 2022-03-14 16:08:34 UTC ---

-5 in https://pagure.io/fedora-qa/blocker-review/issue/651 , marking rejected. Someone voted +1 Beta FE, so proposing as Beta FE for others to vote on.

--- Additional comment from Adam Williamson on 2022-03-14 16:09:55 UTC ---

btw, *is* this a "basic service"? Is this a common or supported way to configure networking on Server or Cloud? I don't recall hearing of it before.

--- Additional comment from Dusty Mabe on 2022-03-14 16:18:11 UTC ---

I think systemd-network-generator was enabled by default fairly recently in an update to systemd. Obviously the logical conclusion that people could come to is "We don't use systemd-networkd, so we don't need this generator", but that's not totally true because there are other things (like udev rules) that this generator creates that are needed. Example: https://github.com/systemd/systemd/pull/21766#issuecomment-994411870

--- Additional comment from Geoffrey Marr on 2022-03-14 18:50:28 UTC ---

Discussed during the 2022-03-14 blocker review meeting: [0]

The decision to classify this bug as an "AcceptedFreezeException (Beta)" was made as it is a noticeable issue that cannot be fixed with an update.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2022-03-14/f36-blocker-review.2022-03-14-16.01.txt

--- Additional comment from Dusty Mabe on 2022-03-24 14:07:14 UTC ---

hey @zpytela - can we get an updated estimate? Thanks!

--- Additional comment from Zdenek Pytela on 2022-03-30 15:52:13 UTC ---

(In reply to Dusty Mabe from comment #28)
> hey @zpytela - can we get an updated estimate? Thanks!

Next week if I can reproduce and test it.

--- Additional comment from Zdenek Pytela on 2022-04-04 08:57:15 UTC ---

There is a PR adding the new domain:
https://github.com/fedora-selinux/selinux-policy/pull/1126

The reproducer from #c3 does not produce any denials, so the fix is expected to be in the next build, but I'd like anybody using this feature in a different way to check the build, rpms for testing can be downloaded from the PR:

Checks -> Details -> Artifacts -> rpms

--- Additional comment from Dusty Mabe on 2022-04-04 14:06:49 UTC ---

The CI RPM from the PR (`selinux-policy-36.5-1.20220404_083611.bff1cac.fc37.noarch.rpm`) corrects this issue for me. Thanks!

--- Additional comment from Zdenek Pytela on 2022-04-04 14:12:48 UTC ---

Thanks for checking.

--- Additional comment from Kevin Fenzi on 2022-04-05 20:12:05 UTC ---

Sadly, rawhide composes still fail. ;( 

https://pagure.io/releng/failed-composes/issue/3311

I'm not sure if this is the same issue or not however. It looks like anaconda cannot connect to nm-client...

--- Additional comment from Dusty Mabe on 2022-04-06 02:56:23 UTC ---

Kevin,

I doubt it's the same issue. This issue has been around since the start of the year.

Zdenek,

Testing of the issue in rawhide for us seems to look good with selinux-policy-36.6-1.fc37. Can we get an update for Fedora 35/36?