systemd-resolved 253.7-1.fc38 no longer populates /run/systemd/resolve/ on startup, although the service is enabled. Manually calling resolvectl status resolves the issue immediately. Reproducible: Always Steps to Reproduce: 1. Boot up system and log in 2. ls -l /etc/resolv.conf /run/systemd/resolve/ to show dangling link and empty dir 2. Open a terminal window and pinging any site by DNS name (e.g. kernel.org) fails 3. call resolvectl. This gives good output and immediately solves the issue Actual Results: Manual call to resolvectl needed to get functional network name resolution. Expected Results: Fully functional network name resolution automatically enabled at boot.
Prior to calling resolvectl: systemctl status systemd-resolved.service × systemd-resolved.service - Network Name Resolution Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; preset: enabled) Drop-In: /usr/lib/systemd/system/service.d └─10-timeout-abort.conf Active: failed (Result: exit-code) since Wed 2023-08-02 20:44:53 CEST; 1min 13s ago Docs: man:systemd-resolved.service(8) man:org.freedesktop.resolve1(5) https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients Process: 779 ExecStart=/usr/lib/systemd/systemd-resolved (code=exited, status=1/FAILURE) Main PID: 779 (code=exited, status=1/FAILURE) Error: 13 (Keine Berechtigung) CPU: 132ms Aug 02 20:44:53 phoenix.fritz.box systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5. Aug 02 20:44:53 phoenix.fritz.box systemd[1]: Stopped systemd-resolved.service - Network Name Resolution. Aug 02 20:44:53 phoenix.fritz.box systemd[1]: systemd-resolved.service: Start request repeated too quickly. Aug 02 20:44:53 phoenix.fritz.box systemd[1]: systemd-resolved.service: Failed with result 'exit-code'. Aug 02 20:44:53 phoenix.fritz.box systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution. Aug 02 20:44:57 phoenix.fritz.box systemd[1]: systemd-resolved.service: Start request repeated too quickly. Aug 02 20:44:57 phoenix.fritz.box systemd[1]: systemd-resolved.service: Failed with result 'exit-code'. Aug 02 20:44:57 phoenix.fritz.box systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution. After calling resolvectl: systemctl status systemd-resolved.service ● systemd-resolved.service - Network Name Resolution Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; preset: enabled) Drop-In: /usr/lib/systemd/system/service.d └─10-timeout-abort.conf Active: active (running) since Wed 2023-08-02 20:46:39 CEST; 3min 15s ago Docs: man:systemd-resolved.service(8) man:org.freedesktop.resolve1(5) https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients Main PID: 3330 (systemd-resolve) Status: "Processing requests..." Tasks: 1 (limit: 38289) Memory: 5.0M CPU: 122ms CGroup: /system.slice/systemd-resolved.service └─3330 /usr/lib/systemd/systemd-resolved Aug 02 20:46:39 phoenix.fritz.box systemd[1]: Starting systemd-resolved.service - Network Name Resolution... Aug 02 20:46:39 phoenix.fritz.box systemd-resolved[3330]: Positive Trust Anchors: Aug 02 20:46:39 phoenix.fritz.box systemd-resolved[3330]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d Aug 02 20:46:39 phoenix.fritz.box systemd-resolved[3330]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in> Aug 02 20:46:39 phoenix.fritz.box systemd-resolved[3330]: Using system hostname 'phoenix.fritz.box'. Aug 02 20:46:39 phoenix.fritz.box systemd[1]: Started systemd-resolved.service - Network Name Resolution. Aug 02 20:46:39 phoenix.fritz.box systemd-resolved[3330]: enp0s31f6: Bus client set search domain list to: fritz.box Aug 02 20:46:39 phoenix.fritz.box systemd-resolved[3330]: enp0s31f6: Bus client set default route setting: yes Aug 02 20:46:39 phoenix.fritz.box systemd-resolved[3330]: enp0s31f6: Bus client set DNS server list to: 192.168.48.1, fd00::3681:c4ff:feb0:e17a, 2003:cb:5717> Aug 02 20:47:04 phoenix.fritz.box systemd-resolved[3330]: Using degraded feature set UDP instead of UDP+EDNS0 for DNS server 192.168.48.1.
I'm not able to reproduce it. I'm booting fresh f38 with systemd-253.7-1. # ls -l /etc/resolv.conf lrwxrwxrwx. 1 root root 39 Aug 1 03:28 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf # ls -a /run/systemd/resolve/ . .. io.systemd.Resolve io.systemd.Resolve.Monitor netif resolv.conf stub-resolv.conf
@Arne, Please provide more detailed logs of systemd-resolved. 1. Enable debug log level for systemd-resolved service: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [Service] Environment=SYSTEMD_LOG_LEVEL=debug ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2. Get the logs: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ journalctl -u systemd-resolved ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Thanks!
Created attachment 1982211 [details] journalctl -u systemd-resolved.service output with systemd.log_level=debug
Lines 1-856 in the log are before manually typing "resolvectl", lines 857-end after calling resolvectl.
@Arne The issue is with inotify_add_watch returning EACCES (Keine Berechtigung). Did you try to run it without SELinux enabled? Could you share your `mount -v` output?
@Jacek, many thanks for following up! In permissive SELinux mode DNS resolution seems to work right away. These are the last three (out of many, and only the final two with permissive=1) "denied" messages from /var/log/audit/audit.log: type=AVC msg=audit(1693219864.883:82): avc: denied { watch } for pid=766 comm="systemd-resolve" path="/" dev="nvme0n1p3" ino=2 scontext=system_u:system_r:systemd_resolved_t:s0 tcontext=system_u:object_r:man_t:s0 tclass=dir permissive=0 type=AVC msg=audit(1693220633.706:57): avc: denied { watch } for pid=749 comm="systemd-resolve" path="/" dev="nvme0n1p3" ino=2 scontext=system_u:system_r:systemd_resolved_t:s0 tcontext=system_u:object_r:man_t:s0 tclass=dir permissive=1 type=AVC msg=audit(1693221287.162:265): avc: denied { mounton } for pid=4191 comm="(sd-parse-elf)" path="/" dev="nvme0n1p3" ino=2 scontext=system_u:system_r:systemd_coredump_t:s0 tcontext=system_u:object_r:man_t:s0 tclass=dir permissive=1 And this is 'mount -v' output: proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel) devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=4096k,nr_inodes=4084255,mode=755,inode64) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel,inode64) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=6543696k,nr_inodes=819200,mode=755,inode64) cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel) efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) ramfs on /run/credentials/systemd-vconsole-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,seclabel,mode=700) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) /dev/nvme0n1p3 on / type ext4 (rw,noatime,seclabel) selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=33,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=21172) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime,seclabel) tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime,seclabel) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,seclabel,mode=700) ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,seclabel,mode=700) tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel,nr_inodes=1048576,inode64) /dev/nvme0n1p2 on /boot type ext4 (rw,noatime,seclabel) /dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro) /dev/sda2 on /Foto type ext4 (rw,relatime,seclabel) /dev/sda1 on /old type ext4 (rw,relatime,seclabel) /dev/sda3 on /virtualization type ext4 (rw,relatime,seclabel) /dev/sda4 on /home type ext4 (rw,relatime,seclabel) ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,seclabel,mode=700) ramfs on /run/credentials/systemd-resolved.service type ramfs (ro,nosuid,nodev,noexec,relatime,seclabel,mode=700) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=3271844k,nr_inodes=817961,mode=700,uid=1000,gid=1000,inode64) gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000) portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
I saw that on my system too but only after I upgraded F38 to F39. I ran "fixfiles onboot" to run an SELinux relabel on reboot. After that the problem was gone.
Many thanks, Simon, relabeling seems to have solved it. Once every few dozen boots the resolver would be configured properly and I was beginning to fear hardware related timing issues as early signs of imminent hardware failure. But no, it was SELinux related after all, and no indication of that at all in the logs...