This bug also affects Red Hat Enterprise Linux 7. It reproduces outside of OCP,  too.

+++ This bug was initially created as a clone of Bug #1868106 +++

Description of problem:

When making a curl request for an external site within a pod, (e.g. www.redhat.com) the DNS lookup takes around 5s (default timeout)

Version-Release number of selected component (if applicable):

All versions currently.

How reproducible:
Consistently

Steps to Reproduce:
1. On an s390x OCP cluster spin up a test pod with curl (e.g. nginx pod)
2. # oc exec $pod -- bash -c 'cat /etc/resolv.conf; while true; do echo -n "$(date)  "; curl -s -o /dev/null -w "%{time_namelookup} %{time_total} %{http_code}\n" https://www.redhat.com -k; done'

Actual results:

The time_namelookup value is just over 5s reasonably consistently.

Expected results:

The time_namelookup value is consistently well below 1s.

Additional info:

I was unable to recreate this on a non-s390x cluster.

tcpdump from the container's interface shows the requests going out and being responded to, but the 5s gap occurs and then the request happens again, which succeeds (I've attached the tcpdump).

The issue looks to be related to the optimizations that mean A & AAAA records are requested in parallel, since you can adjust the pod to use "options single-request" within the pod's /etc/resolv.conf and the requests will go resolve quickly, as they should.
# oc exec $pod -- bash -c 'echo "options single-request" >> /etc/resolv.conf'

This is related to the timeout value, since adjusting the DNS timeout to 2 seconds would adjust the time_namelookup to be around 2 seconds instead:
# oc exec $pod -- bash -c 'echo "options timeout:2" >> /etc/resolv.conf'