Bug 1027452

Thanks for the report. Which application is making the request, and how did you determine that it was the glibc resolver that made the request? Can you provide a step-by-step set of actions to reproduce the issue including any tcpdump logs?

The original problem was that the webpage "www.shellcardonline.shell.com" was not reachable via Squid proxy server - the loading stucked on "https://www.shellcardonline.shell.com/authenticateusertoken.aspx". When the computer reach the site directly without proxy. So I started to investigate why this happen and I found, that AAAA name resolution is timing out for "www.shellcardonline.extha.shell.com" - which is CNAME of "www.shellcardonline.shell.com" and for "www-cardauth-services-prd.extha.shell.com" which is CNAME of "www.shellcardonline.extha.shell.com" (example: http://www.dnswatch.info/dns/dnslookup?la=en&host=www-cardauth-services-prd.extha.shell.com&type=AAAA&submit=Resolve).

So I have disabled IPv6 completely as this Proxy server does not have IPv6 connectivity and I hoped that AAAA requets will stop raising at all. But AAAA resolving remains same even if I disabled IPv6 on the host. 

> Which application is making the request?
It is squid (as proxy server) and telnet (wanted to replicate connection establishing). For both cases both records where requested A and AAAA. 

> how did you determine that it was the glibc resolver that made the request?
Attaching strace of telnet output. As I can see the "/lib64/libresolv.so.2" is loaded to handle DNS reqests. I think this is same for Squid process as well.

Created attachment 822445 [details]
Strace for telnet command

Created attachment 822446 [details]
Packet capture from telnet command

it was provided from different computer, please ignore IP address differences from previous strace output. 
As you can see here the delay between executing "telnet www.shellcardonline.shell.com 443" and getting "Connected..." response is 15 seconds.

(In reply to Michal Bruncko from comment #3)
> So I have disabled IPv6 completely as this Proxy server does not have IPv6
> connectivity and I hoped that AAAA requets will stop raising at all. But
> AAAA resolving remains same even if I disabled IPv6 on the host. 

I don't think we have a mechanism in place to disable AAAA lookups in glibc via a configuration - a program that makes an AF_UNSPEC or AF_INET6 request will get IPv6 results if the nameserver supports it.  Disabling IPv6 networking is something very different - it simply disables IPv6 support in the kernel and prevents the relevant network interfaces from being created.  It does not result in disabling IPv6 name lookups.  Maybe there should be a feature request for this.  This is probably another good use case for tunables.

Hi Siddesh,
yes exactly, such option ("inet4only") is missing in this situation. in current case resolver is asking for both records (==two queries) even if there are scenarios where the AAAA records are not necessary and this doubles every name resolution which also can increase delay for waiting for responses from both requests (like in reported example). 
and this is what I have tried to discuss here. this option will not be "feature", but desired option for "legacy" servers on IPv4 networks only.

Unfortunately the solution we were expecting to use to solve this issue has been shown to violate the POSIX standard wording for getaddrinfo. Therefore we have had to change the implementation plan. That places this solution outside the scope of rhel-6.6. I have moved this bug to rhel-6.7. In the meantime we will be working on an upstream solution to attempt to provide a glibc tunnable to completely disable the ipv6 queries (orthogonal to the usage of AI_ADDRCONFIG). Such a tunable could be used to prevent AAAA queries from being issued by the glibc stub resolver when AF_UNSPEC queries are made, regardless of the state of the interfaces.

Was this bug ever fixed? I am still seeing the unwanted AAAA queries in Centos 6.7 and 7.2 even after disabling ipv6.

(In reply to ozzzo from comment #13)
> Was this bug ever fixed? I am still seeing the unwanted AAAA queries in
> Centos 6.7 and 7.2 even after disabling ipv6.

Thank you for your inquiry. This bug is not fixed in upstream, and is not fixed in RHEL6 or RHEL7 yet.

When Red Hat shipped 6.8 on May 10, 2016 RHEL 6 entered Production Phase 2. 
https://access.redhat.com/support/policy/updates/errata#Production_2_Phase
That means only "Critical and Important Security errata advisories (RHSAs) and Urgent Priority Bug Fix errata advisories (RHBAs) may be released"
That also means no new RFEs so this BZ is being moved to RHEL 7 which is still in Production Phase 1.

1. Proposed title of this feature request
  * [RFE] glibc: implement GAI modifier for AAAA? DNS queries

2. Who is the customer behind the request?
Account name: Confidential
SRM customer: Confidential
TAM customer: Confidential
Strategic Customer: Confidential
 
3. What is the nature and description of the request?
  * Currently, in order to be fully adherent with RFC 2553, getaddrinfo() performs both a AAAA (IPv6 address query) query and a A (IPv4 address query) query to its DNS server.
  * getaddrinfo() is full-blocking, meaning: the getaddrinfo() function will not return before it gets either a reply or a timeout for each of the A and AAAA queries.
  * This request aims to add a new glibc functionality, mitigating potential problem scenarios where the RFC 2553 adherence cause problems.
 
4. Why does the customer need this? (List the business requirements here)
  * There are scenarios where in a AAAA lookup is not wanted / desired and may cause problems. For example:
    - IPv6 stack for local-only traffic
      In this scenario, the client would be connecting to sites that resolves AAAA addresses, but that is not desired
    - DNS servers that does not replies AAAA queries
      In this scenario, we have long getaddrinfo() calls, that returns only after the resolver timeout.  
  * There might exist other non-envisioned scenarios here, benefitting them all.

5. How would the customer like to achieve this? (List the functional requirements here)
  * We would suggest adding a configuration clause to /etc/gai.conf. This flag would change getaddrinfo() behavior and if present, would NOT perform the AAAA queries.

6. For each functional requirement listed in question 5, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.
  * getaddrinfo() resolves BY DEFAULT both A and AAAA for a host name
  * IF present /etc/gai.conf and getaddrinfo() AAAA modifier clause ; THEN
    * DO NOT try AAAA name resolution
  * FI

7. Is there already an existing RFE upstream or in Red Hat bugzilla?
  * None known.

8. Does the customer have any specific timeline dependencies?
  * Desirable: RHEL 7.6
 
9. Is the sales team involved in this request and do they have any additional input?
  * No Sales knowledge.

10. List any affected packages or components.
  * glibc

11. Would the customer be able to assist in testing this functionality if implemented?
  * Yes.

Wow, 5 years later this bug is finally being fixed. I would be ecstatic if I still used RHEL.

Hi, I'm much interested in this bug and implementing the solution to it, on behalf of my organization. Not sure if I can somehow vote for this using my Redhat support contract.

I tested this on Centos 7.6 and the vanilla build still has this issue. Of course I don't know what modifications have to be done to /etc/gai.conf to test disabling IPv6 queries, if it was implemented in 7.6 as targeted.

Also, I have few questions:
Why it cannot be done in /etc/resolv.conf? All other options are going there, so it is the "standard breaking of RFC 2553" that you mentioned in comment above that prevents this option to go in to /etc/resolv.conf? And is /etc/gai.conf not covered under this restriction?

We actually found this to be a problem in our environment, because by default IPv6 and IPv4 queries are done from the same source port. Microsoft DNS server on Windows 2012 and Windows 2016 is not sending the response to the first query that arrives (something to do with Windows OS waiting for the ARP response, we don't know yet), which results in the DNS request to time out. After the retry, it works fine until the ARP entry expires.
Testing on SLES (SUSE Linux) show that they do both DNS request (IPv4 and IPv6) using different source port and avoid the above problem. This is probably equivalent to the "single-request-reopen" that you can set in /etc/resolv.conf. And is probably some default option that SLES compiles into their glibc.
So setting that option could work around it (we tested and it helps in above scenario), but disabling the IPv6 query altogether would be much better solution.

Similar request to this - https://sourceware.org/bugzilla/show_bug.cgi?id=14799

(In reply to Piotr Kierklo from comment #28)
> Hi, I'm much interested in this bug and implementing the solution to it, on
> behalf of my organization. Not sure if I can somehow vote for this using my
> Redhat support contract.
> 
> I tested this on Centos 7.6 and the vanilla build still has this issue. Of
> course I don't know what modifications have to be done to /etc/gai.conf to
> test disabling IPv6 queries, if it was implemented in 7.6 as targeted.

This feature has not been implemented.  It is not currently targeted for any particular release of Red Hat Enterprise Linux.

> Also, I have few questions:
> Why it cannot be done in /etc/resolv.conf? All other options are going
> there, so it is the "standard breaking of RFC 2553" that you mentioned in
> comment above that prevents this option to go in to /etc/resolv.conf? And is
> /etc/gai.conf not covered under this restriction?

We could offer a configuration knob to get different behavior for AF_UNSPEC, as long as we keep the default behavior as it exists today.

There are actually multiple related RFEs here.  Some people want to suppress all AAAA DNS queries, some want to change AF_UNSPEC to send only A queries.  Others want to filter out all AF_INET6 results from the name lookup results, not just DNS, and not just for getaddrinfo.

It's probably best to continue the discussion on the upstream libc-alpha list.  You can subscribe here:

  <https://sourceware.org/lists.html#ml-requestor>

We will not implement a downstream-only change for this.

> We actually found this to be a problem in our environment, because by
> default IPv6 and IPv4 queries are done from the same source port. Microsoft
> DNS server on Windows 2012 and Windows 2016 is not sending the response to
> the first query that arrives (something to do with Windows OS waiting for
> the ARP response, we don't know yet), which results in the DNS request to
> time out. After the retry, it works fine until the ARP entry expires.
> Testing on SLES (SUSE Linux) show that they do both DNS request (IPv4 and
> IPv6) using different source port and avoid the above problem. This is
> probably equivalent to the "single-request-reopen" that you can set in
> /etc/resolv.conf. And is probably some default option that SLES compiles
> into their glibc.
> So setting that option could work around it (we tested and it helps in above
> scenario), but disabling the IPv6 query altogether would be much better
> solution.

Interesting.  Yes, single-request-reopen should work around this, but this looks to me like a bug in the server.  We do not want to enable single-request-reopen by default because it increases latency.  Changing the code to open two sockets and send those queries in parallel would help, but the code has been proven to be rather resistant to change.

We already have an upstream enhancement request to make AF_UNSPEC behavior configurable through /etc/gai.conf: https://sourceware.org/bugzilla/show_bug.cgi?id=19697

Once the upstream enhancement has landed, we can consider backporting it.

The issue is not yet resolved. The Curl/libcurl still queries for both AAAA and A records, which becomes overhead for the DNS resolvers.

Can someone reopen this bug.

Thanks for updating this bug.

I'm going to review this bug again as part of our release process for the next release of Red Hat Enterprise Linux 8.

Any hope?

*** Bug 2060556 has been marked as a duplicate of this bug. ***

Closed Upstream..... and the backport?

(In reply to yuk from comment #47)
> Closed Upstream..... and the backport?

There is no upstream fix to backport.

I posted an upstream patch:

resolv: Implement no-aaaa stub resolver option
https://sourceware.org/pipermail/libc-alpha/2022-June/139341.html

This is purely DNS-based, so if getaddrinfo gets IPv6 addresses from any other source (such as /etc/hosts), it will still be returned to applications.

(In reply to Florian Weimer from comment #49)
> I posted an upstream patch:

Has the patch been accepted?
Thank you

(In reply to yuk from comment #50)
> (In reply to Florian Weimer from comment #49)
> > I posted an upstream patch:
> 
> Has the patch been accepted?

No, I posted it a few hours ago.

I have filed bug 2096189 for the DNS AAAA query submission specifically, to clarify the narrow scope of that change.

We are not going to target AF_UNSPEC directly, so adjusting the resolution. See bug 2096189 for the actual approach (which also affects “getent hosts“; an AF_UNSPEC-specific approach would not).

Upstream has added "no-aaaa" to the resolver with glibc 2.36:

https://lists.gnu.org/archive/html/info-gnu/2022-08/msg00000.html

> The “no-aaaa” DNS stub resolver option has been added.  System
> administrators can use it to suppress AAAA queries made by the stub
> resolver, including AAAA lookups triggered by NSS-based interfaces
> such as getaddrinfo.  Only DNS lookups are affected: IPv6 data in
> /etc/hosts is still used, getaddrinfo with AI_PASSIVE will still
> produce IPv6 addresses, and configured IPv6 name servers are still
> used.  To produce correct Name Error (NXDOMAIN) results, AAAA queries
> are translated to A queries.  The new resolver option is intended
> primarily for diagnostic purposes, to rule out that AAAA DNS queries
> have adverse impact.  It is incompatible with EDNS0 usage and DNSSEC
> validation by applications.

iiuc that would be used by adding the following to resolv.conf:

> options no-aaaa

Reopening this request to backport this new option to RHEL 9's glibc.

As explained in comment 60, this bug requested a change in behavior for AF_UNSPEC, and we are not going to implement that particular approach.

The backport for Red Hat Enterprise Linux 9 is tracked in bug 2096191.

`options no-aaaa` works fine in EL9.3, thank you very much! no longer observing AAAA queries with this enabled in resolv.conf

as per https://access.redhat.com/solutions/15863, 

sudo nmcli connection modify public +ipv4.dns-options no-aaaa
sudo nmcli device reapply ens192
grep aaaa /etc/resolv.conf
options edns0 no-aaaa