After upgrading from 40 to 41, I had lost all internet connection. Looking for a solution I came across https://discussion.fedoraproject.org/t/no-internet-on-silverblue-41/132113 Same sympthoms as in the above link, also the indicated workaround works. journalctl around the time of the upgrade tells me ``` Okt 29 21:49:29 fedora dnf-3[728]: systemd-resolved x86_64 256.7-1.fc41 fedora 310 k Okt 29 21:56:21 fedora dnf-3[728]: Aktualisieren : systemd-resolved-256.7-1.fc41.x86_64 853/3698 Okt 29 21:56:22 fedora dnf-3[728]: Ausgeführtes Scriptlet: systemd-resolved-256.7-1.fc41.x86_64 853/3698 Okt 29 22:15:21 fedora dnf-3[728]: Ausgeführtes Scriptlet: systemd-resolved-255.13-1.fc40.x86_64 3501/3698 Okt 29 22:15:21 fedora dnf-3[728]: Aufräumen : systemd-resolved-255.13-1.fc40.x86_64 3501/3698 Okt 29 22:15:22 fedora dnf-3[728]: Ausgeführtes Scriptlet: systemd-resolved-255.13-1.fc40.x86_64 3501/3698 Okt 29 22:15:53 fedora dnf-3[728]: Ausgeführtes Scriptlet: systemd-resolved-256.7-1.fc41.x86_64 3698/3698 Okt 29 22:21:03 fedora systemd[1]: Stopping systemd-resolved.service - Network Name Resolution... Okt 29 22:21:03 fedora systemd[1]: systemd-resolved.service: Deactivated successfully. Okt 29 22:21:03 fedora systemd[1]: Stopped systemd-resolved.service - Network Name Resolution. Okt 29 22:21:03 fedora systemd[1]: Starting systemd-resolved.service - Network Name Resolution... Okt 29 22:21:03 fedora audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-resolved comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Okt 29 22:21:03 fedora systemd-resolved[29311]: Positive Trust Anchors: Okt 29 22:21:03 fedora systemd-resolved[29311]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d Okt 29 22:21:03 fedora systemd-resolved[29311]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 170.0.0.192.in-addr.arpa 171.0.0.192.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa ipv4only.arpa resolver.arpa corp home internal intranet lan local private test Okt 29 22:21:03 fedora systemd-resolved[29311]: Could not create manager: Cannot allocate memory Okt 29 22:21:03 fedora systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=1/FAILURE Okt 29 22:21:03 fedora systemd[1]: systemd-resolved.service: Failed with result 'exit-code'. Okt 29 22:21:03 fedora systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution. Okt 29 22:21:03 fedora audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-resolved comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' Okt 29 22:21:03 fedora systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 1. Okt 29 22:21:04 fedora systemd[1]: Starting systemd-resolved.service - Network Name Resolution... Okt 29 22:21:04 fedora dnf-3[728]: Job for systemd-resolved.service failed because the control process exited with error code. Okt 29 22:21:04 fedora dnf-3[728]: See "systemctl status systemd-resolved.service" and "journalctl -xeu systemd-resolved.service" for details. ``` Seemingly a memory issue - yet the upgrade worked fine. Reproducible: Didn't try Actual Results: After a smooth update from 40 -> 41 I lost all internet connections Expected Results: After a smooth update from 40 -> 41 I can access the internet * `cat /proc/sys/vm/overcommit_memory` gives `0` * during the upgrade I had also lost the gnome-wayland-session - still investigating.
It's coming from here: int manager_new(Manager **ret) { _cleanup_(manager_freep) Manager *m = NULL; int r; assert(ret); m = new(Manager, 1); if (!m) return -ENOMEM; So it really looks like you were legitimately out of memory at the time, in which case the service is expected to fail. Does this happen every time you attempt to restart systemd-resolved ('systemctl restart systemd-resolved.service) or reboot Fedora? If it's happening consistently, that's weird.
Let me put it this way ``` maymage@Host-015:~$ systemctl restart systemd-resolved Job for systemd-resolved.service failed because the control process exited with error code. See "systemctl status systemd-resolved.service" and "journalctl -xeu systemd-resolved.service" for details. maymage@Host-015:~$ systemctl status systemd-resolved.service × systemd-resolved.service - Network Name Resolution Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; disabled; preset: enabled) Drop-In: /usr/lib/systemd/system/service.d └─10-timeout-abort.conf Active: failed (Result: exit-code) since Thu 2024-10-31 15:00:54 CET; 16s ago Invocation: 71a89f7e294f454591c8f4230611d3f4 Docs: man:systemd-resolved.service(8) man:org.freedesktop.resolve1(5) https://systemd.io/WRITING_NETWORK_CONFIGURATION_MANAGERS https://systemd.io/WRITING_RESOLVER_CLIENTS Process: 68267 ExecStart=/usr/lib/systemd/systemd-resolved (code=exited, status=1/FAILURE) Main PID: 68267 (code=exited, status=1/FAILURE) Error: 12 (Nicht genügend Hauptspeicher verfügbar) Okt 31 15:00:54 Host-015 systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5. Okt 31 15:00:54 Host-015 systemd[1]: systemd-resolved.service: Start request repeated too quickly. Okt 31 15:00:54 Host-015 systemd[1]: systemd-resolved.service: Failed with result 'exit-code'. Okt 31 15:00:54 Host-015 systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution. ```
And ``` maymage@Host-015:~$ journalctl -xeu systemd-resolved.service ░░ ░░ An ExecStart= process belonging to unit systemd-resolved.service has exited. ░░ ░░ The process' exit code is 'exited' and its exit status is 1. Okt 31 15:00:53 Host-015 systemd[1]: systemd-resolved.service: Failed with result 'exit-code'. ░░ Subject: Unit failed ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit systemd-resolved.service has entered the 'failed' state with result 'exit-code'. Okt 31 15:00:53 Host-015 systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution. ░░ Subject: A start job for unit systemd-resolved.service has failed ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ A start job for unit systemd-resolved.service has finished with a failure. ░░ ░░ The job identifier is 20125 and the job result is failed. Okt 31 15:00:54 Host-015 systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5. ░░ Subject: Automatic restarting of a unit has been scheduled ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ Automatic restarting of the unit systemd-resolved.service has been scheduled, as the result for ░░ the configured Restart= setting for the unit. Okt 31 15:00:54 Host-015 systemd[1]: systemd-resolved.service: Start request repeated too quickly. Okt 31 15:00:54 Host-015 systemd[1]: systemd-resolved.service: Failed with result 'exit-code'. ░░ Subject: Unit failed ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit systemd-resolved.service has entered the 'failed' state with result 'exit-code'. Okt 31 15:00:54 Host-015 systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution. ░░ Subject: A start job for unit systemd-resolved.service has failed ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ A start job for unit systemd-resolved.service has finished with a failure. ░░ ░░ The job identifier is 20156 and the job result is failed. lines 181-220/220 (END) ```
I think this is just bad logging, not a real memory failure. I filed https://github.com/systemd/systemd/pull/35006 to improve logging in the spot that seems most likely. Please post the output of 'systemd-analyze cat-config --tldr systemd/resolved.conf' and 'update-crypto-policies --show'.
``` systemd-analyze cat-config --tldr systemd/resolved.conf # /etc/systemd/resolved.conf ``` --- ``` update-crypto-policies --show LEGACY ```
I'm the person who opened the discussion on Fedora forum (link in first comment). Here's what I see on Silverblue 41: $ systemd-analyze cat-config --tldr systemd/resolved.conf # /usr/lib/systemd/resolved.conf $ update-crypto-policies --show DEFAULT
I forgot to update the status here. systemd-256.8-1.fc41 from 2024-11-14 includes a patch the does more reliable logging. Is anyone still seeing this failure? If yes, please attach logs from a newer version.
Yes, I can still reproduce it. Here the logs: $ journalctl -u systemd-resolved.service -b Nov 30 21:12:15 fedora systemd[1]: Starting systemd-resolved.service - Network Name Resolution... Nov 30 21:12:16 fedora systemd-resolved[848]: Positive Trust Anchors: Nov 30 21:12:16 fedora systemd-resolved[848]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d Nov 30 21:12:16 fedora systemd-resolved[848]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa > Nov 30 21:12:16 fedora systemd-resolved[848]: Failed to create SSL context: error:07800069:common libcrypto routines::provider section error Nov 30 21:12:16 fedora systemd-resolved[848]: Could not create manager: State not recoverable Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=1/FAILURE Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Failed with result 'exit-code'. Nov 30 21:12:16 fedora systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution. Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 1. Nov 30 21:12:16 fedora systemd[1]: Starting systemd-resolved.service - Network Name Resolution... Nov 30 21:12:16 fedora systemd-resolved[883]: Positive Trust Anchors: Nov 30 21:12:16 fedora systemd-resolved[883]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d Nov 30 21:12:16 fedora systemd-resolved[883]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa > Nov 30 21:12:16 fedora systemd-resolved[883]: Failed to create SSL context: error:07800069:common libcrypto routines::provider section error Nov 30 21:12:16 fedora systemd-resolved[883]: Could not create manager: State not recoverable Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=1/FAILURE Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Failed with result 'exit-code'. Nov 30 21:12:16 fedora systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution. Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 2. Nov 30 21:12:16 fedora systemd[1]: Starting systemd-resolved.service - Network Name Resolution... Nov 30 21:12:16 fedora systemd-resolved[888]: Positive Trust Anchors: Nov 30 21:12:16 fedora systemd-resolved[888]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d Nov 30 21:12:16 fedora systemd-resolved[888]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa > Nov 30 21:12:16 fedora systemd-resolved[888]: Failed to create SSL context: error:07800069:common libcrypto routines::provider section error Nov 30 21:12:16 fedora systemd-resolved[888]: Could not create manager: State not recoverable Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=1/FAILURE Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Failed with result 'exit-code'. Nov 30 21:12:16 fedora systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution. Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 3. Nov 30 21:12:16 fedora systemd[1]: Starting systemd-resolved.service - Network Name Resolution... Nov 30 21:12:16 fedora systemd-resolved[893]: Positive Trust Anchors: Nov 30 21:12:16 fedora systemd-resolved[893]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d Nov 30 21:12:16 fedora systemd-resolved[893]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa > Nov 30 21:12:16 fedora systemd-resolved[893]: Failed to create SSL context: error:07800069:common libcrypto routines::provider section error Nov 30 21:12:16 fedora systemd-resolved[893]: Could not create manager: State not recoverable Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=1/FAILURE Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Failed with result 'exit-code'. Nov 30 21:12:16 fedora systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution. Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 4. Nov 30 21:12:16 fedora systemd[1]: Starting systemd-resolved.service - Network Name Resolution... Nov 30 21:12:16 fedora systemd-resolved[898]: Positive Trust Anchors: Nov 30 21:12:16 fedora systemd-resolved[898]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d Nov 30 21:12:16 fedora systemd-resolved[898]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa > Nov 30 21:12:16 fedora systemd-resolved[898]: Failed to create SSL context: error:07800069:common libcrypto routines::provider section error Nov 30 21:12:16 fedora systemd-resolved[898]: Could not create manager: State not recoverable Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=1/FAILURE Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Failed with result 'exit-code'. Nov 30 21:12:16 fedora systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution. Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 5. Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Start request repeated too quickly. Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Failed with result 'exit-code'. Nov 30 21:12:16 fedora systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution. Nov 30 21:12:23 fedora systemd[1]: systemd-resolved.service: Start request repeated too quickly. Nov 30 21:12:23 fedora systemd[1]: systemd-resolved.service: Failed with result 'exit-code'. Nov 30 21:12:23 fedora systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution.
Have you enabled FIPS mode?
No, I haven't. $ sudo fips-mode-setup --check FIPS mode is disabled. Initramfs fips module is disabled. The current crypto policy (DEFAULT) neither is the FIPS policy nor is based on the FIPS policy.
I don't know if it matters... today I happened to run a dig command and I got a coredump related to OpenSSL: $ dig @1.1.1.1 google.com ANY ../../../lib/isc/tls.c:88:tls_initialize(): fatal error: RUNTIME_CHECK(OPENSSL_init_ssl(OPENSSL_INIT_ENGINE_ALL_BUILTIN | OPENSSL_INIT_LOAD_CONFIG, NULL) == 1) failed Aborted (core dumped) This happens only on my laptop where I have Silverblue 41 installed. No problem on my desktop, where I run the Workstation 41.
Yeah, it matters, as resolved is likely hitting the same issue: Nov 30 21:12:16 fedora systemd-resolved[848]: Failed to create SSL context: error:07800069:common libcrypto routines::provider section error Nov 30 21:12:16 fedora systemd-resolved[848]: Could not create manager: State not recoverable Nov 30 21:12:16 fedora systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=1/FAILURE
> Nov 30 21:12:16 fedora systemd-resolved[848]: Failed to create SSL context: error:07800069:common libcrypto routines::provider section error This error indicates wrong system-wide openssl configuration, I presume introduced manually. Could you please provide your openssl.cnf file?
It's passed almost a year. I'm no longer using Silverblue on my laptop; I switched to Workstation for other reasons. I can't remember if I did any manual change to openssl.cnf, but it's possible. I cannot check it anymore, sorry.
Thanks, closing then.