Bug 2065058

Summary: glibc: Transaction ID collisions cause slow DNS lookups in getaddrinfo [rhel-7.9.z]
Product: Red Hat Enterprise Linux 7 Reporter: Florian Weimer <fweimer>
Component: glibcAssignee: DJ Delorie <dj>
Status: CLOSED ERRATA QA Contact: Sergey Kolosov <skolosov>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.9CC: ashankar, codonell, dj, fweimer, kpfleming, mnewsome, pfrankli, sipoyare, skolosov, tstaudt
Target Milestone: rcKeywords: Bugfix, Patch, Triaged, ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: glibc-2.17-326.el7_9 Doc Type: Bug Fix
Doc Text:
.The `glibc` DNS stub resolver correctly processes parallel queries with identical transaction IDs Prior to this update, the DNS stub resolver in the GNU C library `glibc` did not process responses to parallel queries with identical transaction IDs correctly. Consequently, when the transaction IDs were equal, the second parallel response was never matched to a query, resulting in a timeout and retry. With this update, the second parallel response is now recognized as valid. As a result, the `glibc` DNS stub resolver avoids excessive timeouts due to unrecognized responses.
 Story Points: ---
Clone Of: 1868106 Environment:
Last Closed: 2022-05-18 16:15:28 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1868106    
Bug Blocks:    

Description Florian Weimer 2022-03-17 09:12:50 UTC 
This bug also affects Red Hat Enterprise Linux 7. It reproduces outside of OCP,  too.

+++ This bug was initially created as a clone of Bug #1868106 +++

Description of problem:

When making a curl request for an external site within a pod, (e.g. www.redhat.com) the DNS lookup takes around 5s (default timeout)

Version-Release number of selected component (if applicable):

All versions currently.

How reproducible:
Consistently

Steps to Reproduce:
1. On an s390x OCP cluster spin up a test pod with curl (e.g. nginx pod)
2. # oc exec $pod -- bash -c 'cat /etc/resolv.conf; while true; do echo -n "$(date)  "; curl -s -o /dev/null -w "%{time_namelookup} %{time_total} %{http_code}\n" https://www.redhat.com -k; done'

Actual results:

The time_namelookup value is just over 5s reasonably consistently.

Expected results:

The time_namelookup value is consistently well below 1s.

Additional info:

I was unable to recreate this on a non-s390x cluster.

tcpdump from the container's interface shows the requests going out and being responded to, but the 5s gap occurs and then the request happens again, which succeeds (I've attached the tcpdump).

The issue looks to be related to the optimizations that mean A & AAAA records are requested in parallel, since you can adjust the pod to use "options single-request" within the pod's /etc/resolv.conf and the requests will go resolve quickly, as they should.
# oc exec $pod -- bash -c 'echo "options single-request" >> /etc/resolv.conf'

This is related to the timeout value, since adjusting the DNS timeout to 2 seconds would adjust the time_namelookup to be around 2 seconds instead:
# oc exec $pod -- bash -c 'echo "options timeout:2" >> /etc/resolv.conf'
Comment 11 errata-xmlrpc 2022-05-18 16:15:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (glibc bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:4641