Description of problem: The DNS server '172.30.0.10' provided by the cluster cannot resolve 'mirrors.fedoraproject.org'. Version-Release number of selected component (if applicable): OCP 4.8 How reproducible: 100% Steps to Reproduce: 1. Check default dns server $ oc describe dns.operator/default Name: default Namespace: Labels: <none> Annotations: <none> API Version: operator.openshift.io/v1 Kind: DNS ... Status: Cluster Domain: cluster.local Cluster IP: 172.30.0.10 ... 2. Create a pod or VM in the cluster and try to resolve host 'mirrors.fedoraproject.org'. Actual results: $ curl mirrors.fedoraproject.org curl: (6) Couldn't resolve host 'mirrors.fedoraproject.org' $ curl google.com <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>302 Moved</TITLE></HEAD><BODY> <H1>302 Moved</H1> The document has moved <A HREF="http://www.google.com/sorry/index?continue=http://google.com/&q=EgRCu-h_GOe5noUGIhBHxrQe2Ie5MyWFRDNW3rk8MgFy">here</A>. </BODY></HTML> Expected results: It could resolve host 'mirrors.fedoraproject.org'. Additional info: It could resolve other host like google/twitter, but not `mirrors.fedoraproject.org`.
In your pod can you: $ cat /etc/resolv.conf $ dig +trace www.google.com $ dig +trace mirrors.fedoraproject.org and attach the output.
$ cat /etc/resolv.conf # Generated by NetworkManager search default.svc.cluster.local svc.cluster.local cluster.local uit-03.cnv-qe.rhcloud.com nameserver 172.30.0.10 $ dig +trace www.google.com ; <<>> DiG 9.11.28-RedHat-9.11.28-1.fc32 <<>> +trace www.google.com ;; global options: +cmd . 30 IN NS m.root-servers.net. . 30 IN NS b.root-servers.net. . 30 IN NS j.root-servers.net. . 30 IN NS i.root-servers.net. . 30 IN NS l.root-servers.net. . 30 IN NS k.root-servers.net. . 30 IN NS f.root-servers.net. . 30 IN NS e.root-servers.net. . 30 IN NS a.root-servers.net. . 30 IN NS c.root-servers.net. . 30 IN NS h.root-servers.net. . 30 IN NS d.root-servers.net. . 30 IN NS g.root-servers.net. . 30 IN RRSIG NS 8 0 518400 20210602210000 20210520200000 14631 . dZX8GzySKRQi36EaA85gE2qPxKt039w0OUg i9Cu8+ZmpP/HJz43mX6qq aR+w+j5XjxMqEoDmgeWEex6b8wUzO+eqoqFPhW2r+1yFycQN8OirDdnn zLT51sG6V3YgCockoU+MzoxMMTLq2ndbIVXHMt48/IXwW0b/KaGAmm9E GuF6g0mMCMNh4P8i0U6ik/dVfK+dCnTrDiTw9MxfkCT2wU7tP6l5qIHd QPIVrg+BX5gZ0aq/hKtJW6WTu+qS70z+qRqaFGzuzRTGK7+Lt6dzbFQ9 yLlYBMNIJDLrDUjrCsBV s14U6cpIDpntARRdclD3byLJdliafPNF6XFn 68QNqQ== ;; Received 1109 bytes from 172.30.0.10#53(172.30.0.10) in 5 ms com. 172800 IN NS a.gtld-servers.net. com. 172800 IN NS b.gtld-servers.net. com. 172800 IN NS c.gtld-servers.net. com. 172800 IN NS d.gtld-servers.net. com. 172800 IN NS e.gtld-servers.net. com. 172800 IN NS f.gtld-servers.net. com. 172800 IN NS g.gtld-servers.net. com. 172800 IN NS h.gtld-servers.net. com. 172800 IN NS i.gtld-servers.net. com. 172800 IN NS j.gtld-servers.net. com. 172800 IN NS k.gtld-servers.net. com. 172800 IN NS l.gtld-servers.net. com. 172800 IN NS m.gtld-servers.net. com. 86400 IN DS 30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766 com. 86400 IN RRSIG DS 8 1 86400 20210603050000 20210521040000 14631 . Uh6uqTNgc1YwKCOvRb8DWjAWEfs2bYaLHV6f DgfuFDc7FkYYnK3s9SQ6 xuzF2C+DEmDehCUIU1MakZpim161N1jcjfPcecKOILng7vt04ziKsd79 /Qiu0A6TnpG0xK03wXbkFYIm8nspLj9sIV2Vzwmh9M5X93QHgbhmKYxi pvbAwxhfFyZD3b05QPbYrS+cOsEsk8FRpHmkmgOT8GpzaS2AZogo122O qjm1087Xtv46wASExRRtNMCazkc/G7pm08xfhAwgLcx9z+VLuZiRm9mL e61eGRoL80PNpj0+WW3JQ vxLKucaxEJhy9GhMhCVMQkyN6EjV/WLHJ+i 4e2XRA== ;; Received 1174 bytes from 193.0.14.129#53(k.root-servers.net) in 47 ms google.com. 172800 IN NS ns2.google.com. google.com. 172800 IN NS ns1.google.com. google.com. 172800 IN NS ns3.google.com. google.com. 172800 IN NS ns4.google.com. CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN NSEC3 1 1 0 - CK0Q1GIN43N1ARRC9OSM6QPQR81H5M9A NS SOA RRSIG DNSKEY NSEC3PARAM CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86400 IN RRSIG NSEC3 8 2 86400 20210528042349 20210521031349 54714 com. b19R2S3segOcVOlRCfZ/PUDGf v6zzUL+2vPmXnBkv7SRo6n527FLObof 9Bfom9SCd9+19H8HlVFlZnXW9oQryoXq7cFy1W/R6LQoWVLDIM/nuI3Q Qkci5vTpctbICUwkAWhrxg9bsM0DgXJj2FRI8eLcs8j4pU 5ktIlYj2g1 0ahIm/WhEXsopfZhFNhijEdscL2y9VrjcDVBiG/IdlIgjw== S84BDVKNH5AGDSI7F5J0O3NPRHU0G7JQ.com. 86400 IN NSEC3 1 1 0 - S84BUO64GQCVN69RJFUO6LVC7FSLUNJ5 NS DS RRSIG S84BDVKNH5AGDSI7F5J0O3NPRHU0G7JQ.com. 86400 IN RRSIG NSEC3 8 2 86400 20210525042148 20210518031148 54714 com. QZCpMUBFhy1MRjB99oEA+1fBL zWmpXl0Fwl6TcRCfuBU3ye1YcuwL9Q/ RgoQmiUYq+51/XuLJserReWDpclwoH+roMiLWuzcgzrIguBmlkYvWJvS Fi9gObcyMZKxa92Kf+29bhG0N+WG/9COt/1PVKObe9Me1g guwWq2f8AA SKl4EkWmbC23WYE21m9IVFyOyyvgeQjasPzdltzhmRS0fA== ;; Received 840 bytes from 192.12.94.30#53(e.gtld-servers.net) in 12 ms www.google.com. 300 IN A 142.250.73.196 ;; Received 59 bytes from 216.239.36.10#53(ns3.google.com) in 18 ms $ dig +trace mirrors.fedoraproject.org ; <<>> DiG 9.11.28-RedHat-9.11.28-1.fc32 <<>> +trace mirrors.fedoraproject.org ;; global options: +cmd . 30 IN NS a.root-servers.net. . 30 IN NS m.root-servers.net. . 30 IN NS j.root-servers.net. . 30 IN NS e.root-servers.net. . 30 IN NS d.root-servers.net. . 30 IN NS f.root-servers.net. . 30 IN NS l.root-servers.net. . 30 IN NS h.root-servers.net. . 30 IN NS i.root-servers.net. . 30 IN NS b.root-servers.net. . 30 IN NS k.root-servers.net. . 30 IN NS c.root-servers.net. . 30 IN NS g.root-servers.net. . 30 IN RRSIG NS 8 0 518400 20210602210000 20210520200000 14631 . dZX8GzySKRQi36EaA85gE2qPxKt039w0OUg i9Cu8+ZmpP/HJz43mX6qq aR+w+j5XjxMqEoDmgeWEex6b8wUzO+eqoqFPhW2r+1yFycQN8OirDdnn zLT51sG6V3YgCockoU+MzoxMMTLq2ndbIVXHMt48/IXwW0b/KaGAmm9E GuF6g0mMCMNh4P8i0U6ik/dVfK+dCnTrDiTw9MxfkCT2wU7tP6l5qIHd QPIVrg+BX5gZ0aq/hKtJW6WTu+qS70z+qRqaFGzuzRTGK7+Lt6dzbFQ9 yLlYBMNIJDLrDUjrCsBV s14U6cpIDpntARRdclD3byLJdliafPNF6XFn 68QNqQ== ;; Received 1109 bytes from 172.30.0.10#53(172.30.0.10) in 4 ms org. 172800 IN NS a0.org.afilias-nst.info. org. 172800 IN NS a2.org.afilias-nst.info. org. 172800 IN NS b0.org.afilias-nst.org. org. 172800 IN NS b2.org.afilias-nst.org. org. 172800 IN NS c0.org.afilias-nst.info. org. 172800 IN NS d0.org.afilias-nst.org. org. 86400 IN DS 26974 8 2 4FEDE294C53F438A158C41D39489CD78A86BEB0D8A0AEAFF14745C0D 16E1DE32 org. 86400 IN RRSIG DS 8 1 86400 20210603050000 20210521040000 14631 . g52DI+uhjL/iPq3qtzO54fQ2feWx3PbajO7v mUV7EZGKUqkQ98F6v+EG eoDkBEge4e0DzbgBLrST5vlofs7ftrnobYfqrj+1vAqConEZ9ew8kI0m ZmJJ+06ctJ1IGiIq1ijB0gfDJgsK6X1mDmBoaqH/r0WCcdACSMe7mmRF 4MviNcTeNuJgOWwXbDr/8HL6li6GwCq5xawal37LbTAB02YIilskC5IV qYmX27pulDawIvGzbNGHA5Djq8dRteqzWxTevFkDPwp/srO8/CcwyeDF VcUj0pojRdOnLzLJO0oO2 udRG8JlkiggSCtQ1m13v+ZH2Gk+vP+pWrRb LMhgQw== ;; Received 791 bytes from 199.7.83.42#53(l.root-servers.net) in 28 ms fedoraproject.org. 86400 IN NS ns-iad02.fedoraproject.org. fedoraproject.org. 86400 IN NS ns-iad01.fedoraproject.org. fedoraproject.org. 86400 IN NS ns05.fedoraproject.org. fedoraproject.org. 86400 IN NS ns02.fedoraproject.org. fedoraproject.org. 86400 IN DS 16207 5 1 8DD099791A2A110851FDE5D14F6C62ADC3DD7C18 fedoraproject.org. 86400 IN DS 16207 5 2 A7C9BF5AFE374C9650ED678F3D36931A7DE9256B86A7BC34D6DEED7D 4E492E5E fedoraproject.org. 86400 IN RRSIG DS 8 2 86400 20210610015245 20210520005245 30453 org. PfNOXbqEajGW4mH+0CRyRGqzTeZUNPT3D EDuxC2ff7AFvnkUulrz2lYt pcgK5OOg+TfS/oMCdjryOtzLp+ldQ8dsk1YDjYCdPOOJv8uxV5kv+jfJ urqPeJKtxoLohG5sJfwMU0sFn2WSv+PgI3yq7FxO94t3S+94pGqG36 hU 7IU= couldn't get address for 'ns-iad02.fedoraproject.org': not found couldn't get address for 'ns-iad01.fedoraproject.org': not found couldn't get address for 'ns05.fedoraproject.org': not found couldn't get address for 'ns02.fedoraproject.org': not found dig: couldn't get address for 'ns-iad02.fedoraproject.org': no more
And if you explicitly dig against: $ dig @1.1.1.1 mirrors.fedoraproject.org $ dig @8.8.8.8 mirrors.fedoraproject.org
$ dig @1.1.1.1 mirrors.fedoraproject.org ; <<>> DiG 9.11.28-RedHat-9.11.28-1.fc32 <<>> @1.1.1.1 mirrors.fedoraproject.org ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37059 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 11, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;mirrors.fedoraproject.org. IN A ;; ANSWER SECTION: mirrors.fedoraproject.org. 274 IN CNAME wildcard.fedoraproject.org. wildcard.fedoraproject.org. 34 IN A 8.43.85.67 wildcard.fedoraproject.org. 34 IN A 8.43.85.73 wildcard.fedoraproject.org. 34 IN A 38.145.60.20 wildcard.fedoraproject.org. 34 IN A 38.145.60.21 wildcard.fedoraproject.org. 34 IN A 67.219.144.68 wildcard.fedoraproject.org. 34 IN A 140.211.169.196 wildcard.fedoraproject.org. 34 IN A 140.211.169.206 wildcard.fedoraproject.org. 34 IN A 152.19.134.198 wildcard.fedoraproject.org. 34 IN A 209.132.190.2 wildcard.fedoraproject.org. 34 IN A 152.19.134.142 ;; Query time: 12 msec ;; SERVER: 1.1.1.1#53(1.1.1.1) ;; WHEN: Fri May 21 13:05:28 UTC 2021 ;; MSG SIZE rcvd: 237 $ dig @8.8.8.8 mirrors.fedoraproject.org ; <<>> DiG 9.11.28-RedHat-9.11.28-1.fc32 <<>> @8.8.8.8 mirrors.fedoraproject.org ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53806 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 11, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 512 ;; QUESTION SECTION: ;mirrors.fedoraproject.org. IN A ;; ANSWER SECTION: mirrors.fedoraproject.org. 273 IN CNAME wildcard.fedoraproject.org. wildcard.fedoraproject.org. 33 IN A 209.132.190.2 wildcard.fedoraproject.org. 33 IN A 8.43.85.67 wildcard.fedoraproject.org. 33 IN A 38.145.60.20 wildcard.fedoraproject.org. 33 IN A 8.43.85.73 wildcard.fedoraproject.org. 33 IN A 140.211.169.196 wildcard.fedoraproject.org. 33 IN A 67.219.144.68 wildcard.fedoraproject.org. 33 IN A 140.211.169.206 wildcard.fedoraproject.org. 33 IN A 152.19.134.198 wildcard.fedoraproject.org. 33 IN A 38.145.60.21 wildcard.fedoraproject.org. 33 IN A 152.19.134.142 ;; Query time: 12 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Fri May 21 13:06:10 UTC 2021 ;; MSG SIZE rcvd: 237
What platform is this? What are the upstream resolvers? Is there any firewall active? At face value there doesn't appear to be any issue with cluster DNS here.
Also could you please share the coredns corefile? Should be available as a config map in the `openshift-dns` namespace. `oc get configmap -n openshift-dns dns-default -o yaml`
(In reply to Stephen Greene from comment #6) > Also could you please share the coredns corefile? > > Should be available as a config map in the `openshift-dns` namespace. > > `oc get configmap -n openshift-dns dns-default -o yaml` oc get configmap -n openshift-dns dns-default -o yaml vm_creation apiVersion: v1 data: Corefile: | .:5353 { bufsize 1232 errors health { lameduck 20s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus 127.0.0.1:9153 forward . /etc/resolv.conf { policy sequential } cache 900 { denial 9984 30 } reload } kind: ConfigMap metadata: creationTimestamp: "2021-05-19T08:14:45Z" labels: dns.operator.openshift.io/owning-dns: default name: dns-default namespace: openshift-dns ownerReferences: - apiVersion: operator.openshift.io/v1 controller: true kind: DNS name: default uid: d60d3e5b-56bf-45d3-af5f-1637bc61eb3e resourceVersion: "8215" uid: 265cf386-2059-4cfe-9063-35c5bc35b205
(In reply to Andrew McDermott from comment #5) > What platform is this? it's OCP 4.8 + CNV which is deployed on PSI. > What are the upstream resolvers? Not sure what's it? > Is there any firewall active? the firewall is not active in the VM. $ systemctl status firewalld Unit firewalld.service could not be found. > At face value there doesn't appear to be any issue with cluster DNS here.
(In reply to Guohua Ouyang from comment #8) > (In reply to Andrew McDermott from comment #5) > > What platform is this? > it's OCP 4.8 + CNV which is deployed on PSI. > > > What are the upstream resolvers? > Not sure what's it? The upstream resolver would be the DNS server _outside_ of the OpenShift cluster. When OpenShift's DNS sees a request for a name external to the cluster, CoreDNS forwards the request to the upstream resolver. Can you check if the upstream resolver is able to resolve mirrors.fedoraproject.org using a machine outside of the OpenShift cluster? The following errors from Comment 2 would suggest that the issue is with some sort of firewall preventing the upstream resolver from completely resolving mirrors.fedoraproject.org. Can you please try to investigate these specific errors that are happening outside of OpenShift more thoroughly? This does not look like an issue with OpenShift DNS. ``` couldn't get address for 'ns-iad02.fedoraproject.org': not found couldn't get address for 'ns-iad01.fedoraproject.org': not found couldn't get address for 'ns05.fedoraproject.org': not found couldn't get address for 'ns02.fedoraproject.org': not found dig: couldn't get address for 'ns-iad02.fedoraproject.org': no more ```
(In reply to Stephen Greene from comment #9) > (In reply to Guohua Ouyang from comment #8) > > (In reply to Andrew McDermott from comment #5) > > > What platform is this? > > it's OCP 4.8 + CNV which is deployed on PSI. > > > > > What are the upstream resolvers? > > Not sure what's it? > > > The upstream resolver would be the DNS server _outside_ of the OpenShift > cluster. When OpenShift's DNS sees a request for a name external to the > cluster, CoreDNS forwards the request to the upstream resolver. Can you > check if the upstream resolver is able to resolve mirrors.fedoraproject.org > using a machine outside of the OpenShift cluster? > > > The following errors from Comment 2 would suggest that the issue is with > some sort of firewall preventing the upstream resolver from completely > resolving mirrors.fedoraproject.org. Can you please try to investigate these > specific errors that are happening outside of OpenShift more thoroughly? > This does not look like an issue with OpenShift DNS. > > ``` > couldn't get address for 'ns-iad02.fedoraproject.org': not found > couldn't get address for 'ns-iad01.fedoraproject.org': not found > couldn't get address for 'ns05.fedoraproject.org': not found > couldn't get address for 'ns02.fedoraproject.org': not found > dig: couldn't get address for 'ns-iad02.fedoraproject.org': no more > ``` The problem does not happen outside the cluster, it only happens within the cluster. I suspect it relates to cluster firewall, but not sure of it. Currently, my co-worker from the infra team is investigating it. @Motty, If you find something, please update it here.
All evidence is that this issue is environmental. This is best investigated by support before coming to engineering.
Seems awfully similar to https://bugzilla.redhat.com/show_bug.cgi?id=1991067, tentatively marking as a duplicate. *** This bug has been marked as a duplicate of bug 1991067 ***