Bug 771204

(In reply to comment #0)
> curl has name resolving problems when only a IPV6 resolver is available
> (or it hits a IPV6 resolver through a resolver rotate)

What kind of problems?  I do not think that RHEL-6 curl does anything special in the terms of name resolving.  It uses the standard NSS service from glibc.  If it fails, it is most likely problem in your network configuration and I bet that other clients on your machine suffer from the same problems.

> The problem was already reported and solved for Fedora 13 in Bug #548269 
> 
> Please fix this problem also for RHEL 6

Bug #548269 was caused by using c-ares for name resolving.  This problem was specific to Fedora 13 and it has never reached any RHEL.

Hi,

i did some tests now, and can prove the problem.

It happens only when "option rotate" is set inside /etc/resolv.conf

For example:

nameserver 2001:4ca0:4103:3::2
nameserver 2001:4ca0:4103:1::10
search physik.uni-muenchen.de 
options rotate


without "options rotate" it works very well, but with "options rotate" set and ipv6 DNS servers yum fails:


[root@z-sv-rheltst etc]# yum update
Loaded plugins: product-id, rhnplugin, security, subscription-manager
Updating certificate-based repositories.
Bei der Kommunikation mit RHN trat ein Fehler auf.
RHN Satellite oder RHN Classic Unterstützung wird deaktiviert.
Fehler beim Kommunizieren mit dem Server. Die Meldung war:
Unable to connect to the host and port specified
Setting up Update Process
No Packages marked for Update
[root@z-sv-rheltst etc]# 


The bug is probably not in yum, but somewhere in the deeper layers, as the very same problem happens with IBM Tivoli client, but ping/ping6 works as expected.

This does not show me any problem in libcurl.  What does 'getent ahosts' say about the server you are trying to connect?

getent ahosts:
[root@z-sv-rheltst ~]# getent ahosts
127.0.0.1       localhost localhost.localdomain localhost4 localhost4.localdomain4
127.0.0.1       localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@z-sv-rheltst ~]#

Id did some tracing with strace. I did the following:

yum clean all
strace yum update 2> tracefile

in the working case (no "options rotate"):

munmap(0x7faee7497000, 4096)            = 0
socket(PF_INET6, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 10
connect(10, {sa_family=AF_INET6, sin6_port=htons(53), inet_pton(AF_INET6, "2001:4ca0:4103:3::2", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
gettimeofday({1326117741, 811677}, NULL) = 0
poll([{fd=10, events=POLLOUT}], 1, 0)   = 1 ([{fd=10, revents=POLLOUT}])
sendto(10, "\316f\1\0\0\1\0\0\0\0\0\0\6xmlrpc\3rhn\6redhat\3c"..., 39, MSG_NOSIGNAL, NULL, 0) = 39
poll([{fd=10, events=POLLIN}], 1, 5000) = 1 ([{fd=10, revents=POLLIN}])
ioctl(10, FIONREAD, [279])              = 0
recvfrom(10, "\316f\201\200\0\1\0\1\0\r\0\0\6xmlrpc\3rhn\6redhat\3c"..., 1024, 0, {sa_family=AF_INET6, sin6_port=htons(53), inet_pton(AF_INET6, "2001:4ca0:4103:3::2", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 279
close(10)                               = 0


In the non working case (with "option rotate"):


munmap(0x7f5917a6b000, 4096)            = 0
socket(PF_INET6, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 10
connect(10, {sa_family=AF_INET6, sin6_port=htons(53), inet_pton(AF_INET6, "2001:4ca0:4103:3::2", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
gettimeofday({1326117821, 673407}, NULL) = 0
poll([{fd=10, events=POLLOUT}], 1, 0)   = 1 ([{fd=10, revents=POLLOUT}])
sendto(10, "v\220\1\0\0\1\0\0\0\0\0\0\6xmlrpc\3rhn\6redhat\3c"..., 62, MSG_NOSIGNAL, NULL, 0) = 62
poll([{fd=10, events=POLLIN}], 1, 5000) = 1 ([{fd=10, revents=POLLIN}])
ioctl(10, FIONREAD, [117])              = 0
recvfrom(10, "v\220\205\203\0\1\0\0\0\1\0\0\6xmlrpc\3rhn\6redhat\3c"..., 1024, 0, {sa_family=AF_INET6, sin6_port=htons(53), inet_pton(AF_INET6, "2001:4ca0:4103:3::2", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 117
close(10)                               = 0
close(9)                                = 0
gettimeofday({1326117821, 674909}, NULL) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0
open("/var/log/up2date", O_WRONLY|O_CREAT|O_APPEND, 0666) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=5033, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5917a6b000
fstat(9, {st_mode=S_IFREG|0644, st_size=5033, ...}) = 0
lseek(9, 5033, SEEK_SET)                = 5033
fstat(9, {st_mode=S_IFREG|0644, st_size=5033, ...}) = 0
write(9, "[Mon Jan  9 15:03:41 2012] up2da"..., 121) = 121
close(9)                                = 0
munmap(0x7f5917a6b000, 4096)            = 0
gettimeofday({1326117821, 675925}, NULL) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0
open("/var/log/up2date", O_WRONLY|O_CREAT|O_APPEND, 0666) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=5154, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5917a6b000
fstat(9, {st_mode=S_IFREG|0644, st_size=5154, ...}) = 0
lseek(9, 5154, SEEK_SET)                = 5154
fstat(9, {st_mode=S_IFREG|0644, st_size=5154, ...}) = 0
write(9, "[Mon Jan  9 15:03:41 2012] up2da"..., 143) = 143
close(9)                                = 0
munmap(0x7f5917a6b000, 4096)            = 0
gettimeofday({1326117821, 676649}, NULL) = 0
write(2, "Bei der Kommunikation mit RHN tr"..., 223Bei der Kommunikation mit RHN trat ein Fehler auf.


Looks like the resolver library didn't give back an IP Address if "options  rotate" was set ....

(In reply to comment #5)
> [root@z-sv-rheltst ~]# getent ahosts

I meant the output of 'getent ahost YOUR_MIRROR'.  Please attach also your /etc/nsswitch.conf and /etc/gai.conf.

[root@z-sv-rheltst ~]# getent ahosts xmlrpc.rhn.redhat.com
209.132.183.44  STREAM xmlrpc.rhn.redhat.com
209.132.183.44  DGRAM  
209.132.183.44  RAW    
[root@z-sv-rheltst ~]# 

/etc/gai.conf is empty (I didn't change anything)

I will append /etc/nsswitch.conf

Created attachment 551784 [details]
nsswitch.conf

Please export URLGRABBER_DEBUG=DEBUG and attach the output of both working and non-working cases as text files.

Created attachment 551815 [details]
Not working case (options rotate in resolv.conf and only IPV6 DNS)

Not working case (options rotate in resolv.conf and only IPV6 DNS)

Created attachment 551816 [details]
Working case (only IPV6 DNS, but no options line in resolv.conf)

Working case (only IPV6 DNS, but no options line in resolv.conf)

(In reply to comment #10)
> Created attachment 551815 [details]
> Not working case (options rotate in resolv.conf and only IPV6 DNS)

I cannot see any failure of libcurl in the non-working case.  Could this be narrowed down to a curl-based test-case?

Its getting wierd more and more:

Some more testing on a Scientific linux host (as I have more control over yum):

2 or less IPV6 nameservers with options rotate fails, but curl works. 3 IPV6 nameservers are failing.

Here a failing yum session:

[root@z-sv-dns1 ~]# yum update
Repository sl-fastbugs is listed more than once in the configuration
internal                                                                                                                                                                                       | 1.1 kB     00:00     
internal/primary                                                                                                                                                                               |  12 kB     00:00     
internal                                                                                                                                                                                                        28/28
http://sl.physik.lmu.de/mirror/scientific/6.1/x86_64/os/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - "Couldn't resolve host 'sl.physik.lmu.de'"
Trying other mirror.
Error: Cannot retrieve repository metadata (repomd.xml) for repository: sl. Please verify its path and try again


Wget works:


[root@z-sv-dns1 ~]# wget http://sl.physik.lmu.de/mirror/scientific/6.1/x86_64/os/repodata/repomd.xml
--2012-01-10 14:09:26--  http://sl.physik.lmu.de/mirror/scientific/6.1/x86_64/os/repodata/repomd.xml
Auflösen des Hostnamen »sl.physik.lmu.de«.... 2001:4ca0:4102::1:34, 192.54.42.34
Verbindungsaufbau zu sl.physik.lmu.de|2001:4ca0:4102::1:34|:80... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 3307 (3,2K) [text/xml]
In »»repomd.xml.15«« speichern.

100%[============================================================================================================================================================================>] 3.307       --.-K/s   in 0,001s  

2012-01-10 14:09:26 (3,96 MB/s) - »»repomd.xml.15«« gespeichert [3307/3307]


Curl works too:


[root@z-sv-dns1 ~]# curl http://sl.physik.lmu.de/mirror/scientific/6.1/x86_64/os/repodata/repomd.xml
<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
  <revision>1311779765</revision>
  <data type="other_db">
    <location href="repodata/other.sqlite.bz2"/>
    <checksum type="sha256">45ff434da6c30d2f46500217e298a768b911227ecb2869d7db7862e7445e8fc9</checksum>
    <timestamp>1311779867.07</timestamp>
    <size>1892419</size>
    <open-size>11474944</open-size>
    <open-checksum type="sha256">a666818788ea9301c533d9adc882ea4fc693f2f8e60a9020579d026797b52492</open-checksum>
    <database_version>10</database_version>
  </data>
  <data type="other">
    <checksum type="sha256">a51438d73d4fc3c03c3cd8427240c4a1702b3ca835fbb5a7ca9a36ff27725864</checksum>
    <timestamp>1311779863</timestamp>
    <size>1227256</size>
    <open-size>12197914</open-size>
    <open-checksum type="sha256">a96571baeb21fd2c6735f2f2cde8deafafd807dd7e6a82961df2af97c24f3fd4</open-checksum>
    <location href="repodata/other.xml.gz"/>
  </data>
  <data type="filelists_db">
    <location href="repodata/filelists.sqlite.bz2"/>
    <checksum type="sha256">b2fad8e4d87c5430532c33ceed3be73f343b23886431df2db84c104123fc84db</checksum>
    <timestamp>1311779880.91</timestamp>
    <size>5201988</size>
    <open-size>32673792</open-size>
    <open-checksum type="sha256">1e9b00c12bfd18d4eccfe745a736c25a0ea5b8d83d3253e21096231f0ba6dcee</open-checksum>
    <database_version>10</database_version>
  </data>
  <data type="filelists">
    <checksum type="sha256">77e5d95c3a6f30f8c078598b36a301c0dbf61be7c7646bc660d238f8d10739f8</checksum>
    <timestamp>1311779863</timestamp>
    <size>4937567</size>
    <open-size>69945751</open-size>
    <open-checksum type="sha256">9e050ae0ae9f961bf6b45ef84df90dc48f43d30b9e79e14aa5ebad8397ba6713</open-checksum>
    <location href="repodata/filelists.xml.gz"/>
  </data>
  <data type="primary_db">
    <location href="repodata/primary.sqlite.bz2"/>
    <checksum type="sha256">65bd042497bbace157417c781d735a92d717aa91fba1f0f4fba006e7e661ca61</checksum>
    <timestamp>1311779889.48</timestamp>
    <size>4146868</size>
    <open-size>19913728</open-size>
    <open-checksum type="sha256">e3a4cd13113a7c6c8142768a59ef1a3a60668b298e0570678d6c7b3ab520d5bc</open-checksum>
    <database_version>10</database_version>
  </data>
  <data type="primary">
    <checksum type="sha256">4ef02f15480e16b9f40bdc385dcb8357efca586766a88d6f9f53b07ed09b19f1</checksum>
    <timestamp>1311779863</timestamp>
    <size>1938149</size>
    <open-size>16663128</open-size>
    <open-checksum type="sha256">3e60ecc392154ae88a0018ad091f60279d440ed9fadba2a8494ddcc202605d29</open-checksum>
    <location href="repodata/primary.xml.gz"/>
  </data>
  <data type="group_gz">
    <location href="repodata/comps-sl6-x86_64.xml.gz"/>
    <checksum type="sha256">f80a106c4baa2952da9453e20e185c097983d772ed26aa39e4dc07f8e558aeeb</checksum>
    <open-checksum type="sha256">1dd8f07850968f91b23df430db695fafccd3074042d0a6dd945b0e2f80adc99c</open-checksum>
    <timestamp>1311779889</timestamp>
  </data>
  <data type="group">
    <location href="repodata/comps-sl6-x86_64.xml"/>
    <checksum type="sha256">1dd8f07850968f91b23df430db695fafccd3074042d0a6dd945b0e2f80adc99c</checksum>
    <timestamp>1311779889</timestamp>
  </data>
</repomd>
[root@z-sv-dns1 ~]# 


I suspect now some very curious side effect inside the resolver libraries!

Is the failure reproducible independently on yum then?

Yes, I can reproduce it with IBM Tivoli Storage Client:

[root@z-sv-rheltst adsm]# dsmc q filespace
ANS0102W Unable to open the message repository /opt/tivoli/tsm/client/ba/bin/DE_DE/dsmclientV3.cat. The American English repository will be used instead.
IBM Tivoli Storage Manager
Command Line Backup-Archive Client Interface
  Client Version 6, Release 3, Level 0.0  
  Client date/time: 11.01.2012 15:06:36
(c) Copyright by IBM Corporation and other(s) 1990, 2011. All Rights Reserved.

Node Name: Z-SV-RHELTST.PH
ANS1033E An invalid TCP/IP address was specified.

[root@z-sv-rheltst adsm]# 

In this case with only one IPV6 Nameserver and options rotate

and I was able to reproduce it with telnet client:

[root@z-sv-rheltst adsm]# telnet www.physik.uni-muenchen.de 80
telnet: www.physik.uni-muenchen.de: Name or service not known
www.physik.uni-muenchen.de: Host name lookup failure
[root@z-sv-rheltst adsm]# host www.physik.uni-muenchen.de
www.physik.uni-muenchen.de has address 192.54.42.2
www.physik.uni-muenchen.de has IPv6 address 2001:4ca0:4102::1:2
[root@z-sv-rheltst adsm]#

It is not a yum problem then.  I am switching the component to glibc although it can be a configuration/network problem.

Klaus,

The bad news, telnet explicitly only asks for IPV4 names because it simply doesn't support IPV6, so it's clearly not a good test.

The good news, I have a patch which *might* fix this problem, but I don't have a good test environment handy.  If I was to build you a version of glibc with the potential fix, could you test it and report the results here?

Yes of course. Testing should be no problem

Hi Jeff,

I tested the patch on a 64bit system, looks like it works.

Sincerly,
Klaus

Thanks for testing the patch.  I'm going to ack this from the dev side for 6.3; I'm not sure if the QA team has any bandwidth left though.

QA: If it helps, I think I can probably get a simpler testcase now that I know the root cause of this problem.

Same underlying issue as 841787, which is scheduled to be fixed in RHEL 6.4.

*** This bug has been marked as a duplicate of bug 841787 ***