Originally problem was reported on Unbound github: https://github.com/NLnetLabs/unbound/issues/509 note: Start delay can be elliminated by: echo DISABLE_UNBOUND_ANCHOR="yes" >> /etc/sysconfig/unbound Need to investigate the way unbound-anchor is started.
Unbound developers have offered: > @birdie-github, I would assume that problem is here: > ExecStartPre=-/usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem -f /etc/resolv.conf -R > could you try to delete -f /etc/resolv.conf and start and stop the unbound to see if it'll change the boot time. which has indeed fixed the issue for me. Since I didn't know which unit was responsible for the delay I've removed it from both unbound.service and unbound-anchor.service, now unbound starts in less than 3 seconds vs over 30 seconds earlier. Fedora uses 127.0.0.53 by default, so it looks like removing `-f /etc/resolv.conf` is a must.
Another way is to ensure your resolver passes and understands DNSSEC, which I think systemd-resolved in rawhide should already do. If you want to use unbound in default configuration, your resolvers need to support DNSSEC. In most cases disabling systemd-resolved and using unbound instead of it should help. Consider using dnssec-trigger to autoconfigure it on moving laptops. Both commands 'unbound-host -vrD -t ns .' and 'unbound-host -vD -t ns .' should return the same results, marked with (secure). -R flag of unbound-anchor should ensure resolv.conf is not used if it does not provide working reply. It works to me ok when local resolver does not reply with DNSSEC bit. But it seems systemd is unable to switch to dnssec working resolver if broken one is chosen by default. # tcpdump -ni any port domain & # echo nameserver 127.0.0.53 > systemd.conf # time /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem -f ./systemd.conf -Rvvv /var/lib/unbound/root.key has content 08:07:48.470526 IP localhost.37608 > localhost.domain: 40589+ [1au] DNSKEY? . (28) 08:07:48.473421 IP localhost.domain > localhost.37608: 40589 ServFail* 0/0/1 (28) 08:07:48.473629 IP localhost.41312 > localhost.domain: 42536+% [1au] DNSKEY? . (28) 08:07:48.762289 IP localhost.50084 > localhost.domain: 8911+% [1au] DNSKEY? . (28) 08:07:49.050951 IP localhost.43291 > localhost.domain: 51452+% [1au] DNSKEY? . (28) 08:07:49.627865 IP localhost.51937 > localhost.domain: 36987+% [1au] DNSKEY? . (28) 08:07:50.204795 IP localhost.35445 > localhost.domain: 13448+% [1au] DNSKEY? . (28) 08:07:51.358301 IP localhost.40215 > localhost.domain: 56871+% [1au] DNSKEY? . (28) 08:07:52.511805 IP localhost.33094 > localhost.domain: 19977+% [1au] DNSKEY? . (28) 08:07:54.818488 IP localhost.42630 > localhost.domain: 56792+% [1au] DNSKEY? . (28) ./systemd.conf failed, retrying direct 08:07:58.682350 IP localhost.domain > localhost.43291: 51452 3/0/1 DNSKEY, DNSKEY, RRSIG (864) 08:07:58.682516 IP localhost.domain > localhost.41312: 42536 3/0/1 DNSKEY, DNSKEY, RRSIG (864) 08:07:58.682615 IP localhost.domain > localhost.42630: 56792 3/0/1 DNSKEY, DNSKEY, RRSIG (864) 08:07:58.682706 IP localhost.domain > localhost.40215: 56871 3/0/1 DNSKEY, DNSKEY, RRSIG (864) 08:07:58.682799 IP localhost.domain > localhost.33094: 19977 3/0/1 DNSKEY, DNSKEY, RRSIG (864) 08:07:58.682891 IP localhost.domain > localhost.50084: 8911 3/0/1 DNSKEY, DNSKEY, RRSIG (864) 08:07:58.682985 IP localhost.domain > localhost.35445: 13448 3/0/1 DNSKEY, DNSKEY, RRSIG (864) 08:07:58.683084 IP localhost.domain > localhost.51937: 36987 3/0/1 DNSKEY, DNSKEY, RRSIG (864) success: the anchor is ok real 0m15.549s user 0m0.013s sys 0m0.023s When broken server is used directly in resolv.conf, it skips it well. Also if first nameserver does not deliver DNSSEC enabled reply, it uses next immediately. # time /usr/sbin/unbound-anchor -a /var/lib/unbound/root.key -c /etc/unbound/icannbundle.pem -f ./resolv.conf -Rvvv /var/lib/unbound/root.key has content ./resolv.conf failed, retrying direct success: the anchor is ok real 0m0.115s user 0m0.008s sys 0m0.019s Which leads me to conclusion the actual bug is inside systemd-resolved. Because it does not deliver reply to dig @127.0.0.53 -t DNSKEY . for 10 seconds. Or it does, ServFail on first time, then nothing more. It seems it takes long to get response from systemd for some reason, delaying this step significantly.
This bug appears to have been reported against 'rawhide' during the Fedora 36 development cycle. Changing version to 36.
Hmm, I am not able to reproduce it on current rawhide, with systemd-resolved-251.1-2.fc37.x86_64. The thing is it should not fail with servfail, but deliver just reply without signatures. That should be notified by unbound-anchor immediately and switch to direct root query. time unbound-anchor -a /tmp/root.key -c /etc/unbound/icannbundle.pem -f /run/systemd/resolve/stub-resolv.conf -Rvvv /tmp/root.key has content success: the anchor is ok real 0m0.017s user 0m0.010s sys 0m0.007s 23:15:02.987838 lo In IP 127.0.0.1.40163 > 127.0.0.53.domain: 51634+ [1au] DNSKEY? . (28) 23:15:02.988119 lo In IP 127.0.0.53.domain > 127.0.0.1.40163: 51634 2/0/1 DNSKEY, DNSKEY (578) 23:15:02.988494 lo In IP 127.0.0.1.58321 > 127.0.0.53.domain: 24998+ [1au] NULL? _ta-4f66. (37) 23:15:02.988570 lo In IP 127.0.0.1.34530 > 127.0.0.53.domain: 58581+% [1au] DNSKEY? . (28) 23:15:02.988624 lo In IP 127.0.0.53.domain > 127.0.0.1.58321: 24998 Refused 0/0/1 (37) 23:15:02.988771 lo In IP 127.0.0.1.49671 > 127.0.0.53.domain: 7299+% [1au] NULL? _ta-4f66. (37) 23:15:02.988807 lo In IP 127.0.0.53.domain > 127.0.0.1.34530: 58581 3/0/1 DNSKEY, DNSKEY, RRSIG (864) 23:15:02.988864 lo In IP 127.0.0.53.domain > 127.0.0.1.49671: 7299 Refused 0/0/1 (37) 23:15:02.991985 lo In IP 127.0.0.1.42236 > 127.0.0.53.domain: 32150+% [1au] DNSKEY? . (28) 23:15:02.992072 lo In IP 127.0.0.1.34396 > 127.0.0.53.domain: 32820+% [1au] NULL? _ta-4f66. (37) 23:15:02.992307 lo In IP 127.0.0.53.domain > 127.0.0.1.42236: 32150 3/0/1 DNSKEY, DNSKEY, RRSIG (864) 23:15:02.992391 lo In IP 127.0.0.53.domain > 127.0.0.1.34396: 32820 Refused 0/0/1 (37)
I have very similar results on f36 live with systemd 250.3-8. Not sure how did RRSIG appeared there in systemd response, but it seems present. So it passed fast.
To debug this issue, I think I would require port domain pcap file recording communication on localhost and to root servers. Another fix might be changing /etc/resolv.conf in unbound-anchor to /run/systemd/resolve/resolv.conf, which does not include resolved in query. Common servers should be able to provide replies with signatures.
Oh, I think issue #597 describes what is required to long timeouts. If outgoing plain dns queries are all blocked, then unbound-anchor will not work. It cannot use TLS without extra configuration. But it can be fixed by passing -C /etc/unbound/unbound.conf explicitly to it: sudo -u unbound unbound-anchor -C /etc/unbound/unbound.conf This way it can prepare root.key even using TLS!
Or it seems more appropriate would be storing just forwarder configuration in separate file and feeding unbound-anchor such file: cat >tls.conf << CONF server: tls-cert-bundle: "/etc/pki/tls/cert.pem" forward-zone: name: "." forward-tls-upstream: yes forward-addr: 1.1.1.1@853#cloudflare-dns.com forward-addr: 9.9.9.9@853#dns.quad9.net CONF sudo -u unbound unbound-anchor -C tls.conf
FEDORA-2022-afc2d2e53c has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-afc2d2e53c
Okay, I made it simpler to disable unbound-anchor altogether. Just systemctl disable --now unbound-anchor.timer Also, I moved /etc/sysconfig/unbound to new unbound-anchor package. It has own UNBOUND_ANCHOR_OPTIONS environment, which can be used to pass extra parameters to unbound-anchor. Such as custom root hints or TLS configuration snippet. Which would solve your delay when blocking outgoing DNS queries on port 53. It makes it possible to also not use resolv.conf and contact directly root servers, which might help in some cases too.
FEDORA-2022-afc2d2e53c has been pushed to the Fedora 36 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-afc2d2e53c` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-afc2d2e53c See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2022-afc2d2e53c has been pushed to the Fedora 36 stable repository. If problem still persists, please make note of it in this bug report.